Examples of experiment design for analysis, simulation, and measurement

[Japanese]

This page contains several experimet design examples for analysis, simulation, and measurement according to the guideline given in [Jain91]. For all performance analysit, I strongly recommend to carefully read [Jain91]. The following guidelines are excerpt from [Jain91].

Common Mistakes in Performance Evaluation

  1. No Goals
  2. Biased Goals
  3. Unsystematic Approach
  4. Analysis without Understanding the Problem
  5. Incorrect Performance Metrics
  6. Unrepresentative Workload
  7. Wrong Evaluation Technique
  8. Overlooking Important Parameters
  9. Ignoring Significant Factors
  10. Inappropriate Experimental Design
  11. Inappropriate Level of Detail
  12. No analysis
  13. Errorneous Analysis
  14. No Sensitivity Analysis
  15. Ignoring Errors
  16. Improper Treatment of Outliers
  17. Assuming No Change in the Future
  18. Ignoring Variability
  19. Too Complex Analysis
  20. Improper Presentation of Results
  21. Ignoring Social Aspects
  22. Omitting Assumptions

A Systematic Approach to Performance Evaluation

  1. State Goals and Define the System
  2. List Services and Outcomes
  3. Select Metrics
  4. List Parameters
  5. Select Factors to Study
  6. Select Evaluation Technique
  7. Select Workload
  8. Design Experiments
  9. Analyze and Interpret Data
  10. Present Results

Experiment Design Examples

According to the above guideline, "A Systematic Approach to Performance Evaluation", I have designed several REAL experiments in my research work. Performace analysis results based on these experiment designs will be soon available at my publication page.

References

Hiroyuki Ohsaki (oosaki[atmark]ist.osaka-u.ac.jp)