8/29/2018
Wirele (=e eB L=tleMme Long Live Modeling!
onstruction
International Conference of the System
Reykjavik, Iceland 2018
The Setting
* The System Dynamics Society provides access to models submitted as
Supporting Material to their annual conference.
* These models are free to anyone*
* These models may represent the Society and the System Dynamics
field.
* Model quality (as defined by units, layout, naming, logic, interface, etc.) varies
* Conclusions (about the methodology, field, and Society) may be drawn from
model quality
* Learning may be enabled or hampered by theses models
*In fact, system dynamics models are available on many public websites.
Terminology:
Module, model, interface, application
This study
is only
interested
in models.
+ =
Ground
Water
Typical system dynamics application
variable classes
control objects,
graphs, tables, ...
Data from
external sources
Aggregations,
disaggregations,
All model
variables
Internal calculations
(the system dynamics)
Exposed to user
via interface
8/29/2018
Distribution of Supporting Documents
International Conference of the System Dynamics
Society
2009 - 2013
Model Vensim 19%
Model stella 2%
Model Think 2%
The Method
* The list of all papers from the International
Conference of the System Dynamics Society years
2009-2017 were loaded into a file and parsed for the
word ‘Supporting’.
* The supporting files were downloaded and
individually examined.
* Models using Vensim, and Studio were selected and
examined using objective and subjective criteria.
8/29/2018
8/29/2018
The Evidence
* 329 models were downloaded (Vensim and Studio)
¢ 35 had a sufficient number of problems that they could not be
simulated
¢ 294 were of sufficient quality to run.
* The number of models by year is 33, 13, 19, 40, 40, 40, 39, 29, 35 for
2009-2017 respectively.
Variables captured
*Variables / Module
*Variables / Unit error
*Unit errors / Variable
«(Levels + Constants) / Total variables
*Levels / Total variables
Results
* Acursory view of some of these models shows considerable poor quality in
almost all (but not all) of them.
* A pessimistic conclusion might be that no model, other than very small
models, models much smaller than the average number of variables in the
models sampled (71.8 levels, 313.9 auxiliaries, 47.9 constants) can be
examined for model construction quality in a tractable manner and period
of time.
* As a model passes each stage of construction quality the time required to
test it increases (Wakeland and Hoarfrost 2005). In the extreme, no model
can be verified but confidence in a model can be raised.
* There is little, if any, indication in the model files of how the model was
built, how long it took, who were the authors, how many persons
participated, what was its purpose, etc.
Note: Wakeland and Hoarfrost performed no model construction quality tests.
8/29/2018
Data
Table 3 Descriptive statistics of all models 2009-2017
DN eae onstants unit Errors
5005 30006 972 201 54
0 3) 3 0 al
71.8 313.9 47.9 ipa 3.3
Large numbers are a result of Vensim’s way of counting variables.
Studio for example, considers a ‘subscripted’ variable to be one variable.
Data
bed oe 6 oe ee
20-00 lth oon 000 oe "8 5B E ees? Sele 0 a “soe sae” oo
Shall we go on?
De gustibus non disputandem est —
About matters of taste there is no point
in arguing.
De veritate disputandum est — About
matters of truth, dispute is fruitful.
Are we discussing taste or truth?
8/29/2018
What can we do?
* Adopt a personal modeling process
* Develop worksheets
* Develop checklists
* Develop habits based on routine use of checklists
* Model, model, model — practice makes perfect
* Convert models from accessible languages (DYNAMO,
Vensim) to your preferred language
* Join a user group
* Develop and use standards
Objective criteria
Variable count ‘The total number of variables include inputs, endogenous model calculati ables for
the interface.
Element count This represents the number of model values. An arrayed-variable counts as one variable and as many
elements as its dimension.
Element/Variable ratio This
details.
Relative model size ‘Model size in variables divided by the average model size in variables. This is useful for groups that have an
archive of models.
(EXPE 4 range is a variable’s dimension, e.g. a range called ‘States’ would have 50 elements, one for each state.
Hae atomic Bile or Slunits..
in some sense a measure of model leverage. Variables represent dynamics, elements represent
‘able Units or not all variables have units.
eka mode is get Brae into relatively self-contained sections if necessa
ree pI tabs. The number of model tabs signals
the degree eee decomposition and re-usability.
interface ‘The number of interface elements or views.
lOther Other model decompositions
8/29/2018
Objective criteria
Stocks or levels are the model variables that accumulate material, information, persons, etc.
/Auxiliaries/Flows ‘Auxiliaries are composed of rates (flow into stocks per unit of time) and variables used for any purpose other
than stock or rate.
[Constants Constants signal the degi ich the is controlled by external not by causal
Geer tris ns iS Are all constants documented?
Modeling conventions
by input from many sources. If followed, they improve the
understandability and reusability of the model.
1s a well-defined naming convention
used?
|Embedded constants Are there auxiliaries with embedded and undefined constants?
Variable names well defined Are the variables named using the naming convention?
Subjective criteria
Sufficient documentation exists: to undertake improvement by original author(s)
Sufficient documentation exists: to reproduce results by non-authors
Percent of constants documented and documented sufficiently
ibjective criteria ied as experience in modeling is gained
8/29/2018
Problems with this research
* Samples versus population: is this the right ‘geography’
* Is the sample/population representative?
* What about other software?
* Objective versus subjective measures of quality
* Ease of measurement
* Binary or scaled performance?
* No causality or correlation examined
* Does experience matter?
* Does institution matter?
* Does the software matter?
8/29/2018
Finally — close your eyes, please
¢Think of a realistic goal for your models.
° Pleture a future where you have achieved that
goal.
° Piet irs the obstacles between you and that
goal.
*Overcome them and don’t stop until your are
proud of your work!
