Calibration Statistics
Selecting a Statistic and Setting a Standard
1996 System Dynamics Conference
Kimberly Sklar Reichelt,
Dr. James M. Lyneis & Carl G. Bespolka
PUGH-ROBERTS ASSOCIATES
Why We Care about Statistics
« To enhance the confidence of others
« For comparison against other calibrations
To assure that minimum standards are met
For use in “automatic” calibration and
sensitivity analysis software
The Ideal Stat
ic
+ Provides meamungful results in a variety of
circumstances
+ Easily interpreted
«Easy to explain (“intuitve")
« Provides a consistent measure
« Is sensitive to important differences
Is relatively insensitive to small differences when
either the simulation or the data is close to zero
Calibration Statistics
« Why We Care About Statistics
« Alternative Statistics to Consider
Selecting a Best Statistic
Setting Standards
ro.
Simulation
Noisy daca davies
vas sh variazion
Alternative Statistics
oR
Mean Absolute Percent Error (MAPE)
Root Mean Square Percent Error (RMSPE)
Modified Mean Absolute Percent Error (MMAPE)
‘© Modified Root Mean Square Percent Error
(MRMSPE)
‘Theil Statistics (U)
Average Absolute Error (AAE)
.
42s
rar etek % Bron)
Handling Noisy Data
Response of AAE, R? and MAPE to Variations in “Noise”
Jeo
Measuring Sensitivity to Tail Length
MAPE ana
Ure nfinite errar
between the magnitude
of the error at amy one
point nowever, They ae
return greater errors
igh, Lior tae
ror Satine (Error)
UR
‘Response of RMSPE, MRMSPE,MMAPE and AAE
to Variations in "Noie”
woe
om
soe
om
gS SS RR FRESE
Measuring Sensitivity to Tail Height
mt: a
ey I oH
IMAPE and PMGPE
- rewen nfinite error,
MMAPE and MRMSPE
ml van cannot disenguits
between the magnitude
lof the error
* te
R? Ignores Magnitude of the Data
dentate a heat
vos meanest aa
site
00
oe TTT RT
TM
Most Statistics Do Not Produce
Symmetrical Results for Multiples
ne
os
Pn eats
£ [o-ncranasnare
3 wanes)
a
H
How Good a Fit is Necessary?
Standard will depend on ...
« The purpose for which the model is
constructed
¢ The amount of noise in the data
Time and budget available
Noise Levels Vary Within a Project
‘Simuiate vata for Design Labor ‘Strate ¥. Dota for erementat Revisions
bet >
o
AAE Provides the Best Measure for
Life Cycle Models
« Retums symmetric errors for bias
+ Gives reasonable errors for increasing magnitude and
duration of any “tails”
« Provides predictable response to increasing levels of
noise
«Yields similar values to other statistics “the rest of the
time”
« Is intuitive and easy to explain
All other measures suffer some fatal flaw
Noise Levels Vary Between Projects
‘Stated Act for Danlgn Labor ‘Shot v Data or Sofie Design Labor
cement tacomry = —Oaetieent __—Smaaadinamn = = Ostet
‘ l\
\
[7
Examples of Good Fit
Design Labor, Skmuletad va, Acti
—amuananeccum —_ = ~Conmécun
Examples of Good Fit
osign Labor, Simulsad vs. Actual
—SmuaudHendcoun = =OaiaHewiene
AAE= 15%
U0
Uu=98
Summary
Statistics cannot substitute for a careful visual
inspection and analysis of the fit to historical data
We should continue to compute and report all
statistics to allow comparison to other results
The “Average Absolute Error” statistic should be
used as the standard
What fit is “good enough” is inherently subjective.
‘The Theil components should be computed and
monitored to ensure that “bias” is small (<=10%)
and most error is due to covariance (>=70%).
For a copy of the complete paper. contact
the authors at:
Pugh-Roberts Associates
41 William Linskey Way
Cambridge, MA 02142
(617) 864-8880
(617) 864-8884
telephone:
facsimile:
Smoothed Data May Provide a More
Fair Basis for Comparison
mee
References
‘sls, Yanan. "Muhipl Tess fer Validation of Sytem Dyna
Models." European Joural of Op ational Research 42 (1989).
Fonsi Jay W. Insial Dynanis, The MIT Press, Cambridge, Mastachusts,
196.
Mathews, Brian P. set Adumartios Damanopatos: Towards a Taxonomy of Forecast
ror Meares: A Petr omaparuivelnvextgation of Force! Eor Dmensios
Sournal of Forecasting, Vol 13, 409-416, 1994,
Susan, Son, Nelson Repeoing and Feed Kolaan. Unanicipaed Side Elec of
‘Sccrestol Quality Reprams: Expaing «Paco of Organizational lmprovement
Monagenant Scie (orécomig)
[ype James Man Aletander L. Pgh I. “Hard vs, Anemated Taarg: An
Expentenal Analy,” Precedings ofthe 1996 lnsratonal Sytem Dyas
Confrence, Cambridge, Masctsets, Jl 1996,
‘Stee an Jahn D. Apgropsiae Surtnay Stas fot Evahising the Hato Fi of
Syten Dynamics Models, Dynan. 102) pp. 51-56, 1984
Sema, lin, GouseP Retain aoa Deion, Meer fe Eman
othe Une Sites Technolog Fotecasing and Sia
Chang 3-219 28,8
jes Type of Simltion
4r%
