Performance evaluation is a critical stage of software- and hardware-system development that every computer engineer and scientist should master. Although complex – requiring skills in mathematics, measurement techniques and simulation – performance evaluation is primarily an art; indeed, the most difficult stage in a performance analysis is defining the approach: once you know what to do, it is less difficult to define a plan of attack with your familiar software tools. We present a set of topics, which we believe should be part of every engineer's intellectual toolkit. This includes the statistical exploitation of numerical results in an efficient and ethical way, for example: how to summarize variability or fairness; what transient removal in a simulation is; and how to make predictions from a time series. We also present well-known performance patterns, which helps to quickly bring the engineer to the main issues. For queuing theory, we focus on a subset of very useful results, such as operational laws. A highlight of the book is the development of Palm calculus, also called ìthe importance of the viewpoint,î which is central to queuing theory. Indeed, this topic has so many applications to simulation and to system analysis in general that it is a very good time investment. This book began as a set of lecture notes for a course given at EPFL.
Table of Contents:
Methodology
What is Performance Evaluation ?
Factors
Evaluation Methods
The Scientific Method
Performance Patterns
Summarizing Performance Data, Confidence Intervals
Summarized Performance Data
Confidence Intervals
The Independence Assumption
Prediction Interval
Which Summarization To Use?
Other Aspects of Confidence/Prediction Intervals
Proofs
Model Fitting
Model Fitting Criteria
Linear Regression
Linear Regression with Norm Minimization
Choosing a Distribution
Heavy Tail
Proofs
Tests
The Neyman Pearson Framework
Likelihood Ratio Tests
ANOVA
Asymptotic Results
Other Tests
Proofs
Forecasting
What is Forecasting ?
Linear Regression
The Overfitting Problem
Differencing the Data
Fitting Differenced Data to an ARMA Model
Sparse ARMA and ARIMA Models
Proofs
Discrete Event Simulation
What is a Simulation?
Simulation Tehniques
Computing the Accuracy of Stochastic Simulations
Monte Carlo Simulation
Random Number Generators
How to Sample from a Distribution
Importance Sampling
Proofs
Palm Calculus, or the Importance of the Viewpoint
An Informal Introduction
Palm Calculus
Other Useful Palm Calculus Results
Simulation Defined as Stochastic Recurrence
Application to Markov Chain Models and the PASTA Property
Appendix: Quick Review of Markov Chains
Proofs
Review Questions
Queuing Theory for Those Who Cannot Wait
Deterministic Analysis
Operational Laws For Queuing Systems
Classical Results for a Single Queue
Definitions for Queuing Networks
The Product-Form Theorem
Computational Aspects
What This Tells Us
Mathematical Details About Product-Form Queuing Networks
Case Study
Proofs
Each chapter concludes with a Review that includes review questions.
