Go Back
Table of Contents
Exploring rationality with system dynamics based
simulators: A literature review
Andrees Grofle?, Ftiénne A. J. A. Rouwette* and Jac A. M. Vennix?
1 Tndustieseminar, Mannheim Univesity, Gamany
? Methodology Department, University of Nijmegen, The Nethedands
* Corresponding author:
Methodology Department, University of Nijmegen 6500 HK Nijmegen, The Netherlands
phone: 431 24 36 72 55 68, fax: +31 24 36 72 357
e nail: erouwetp@nsmkinnl
Exploring rationality with system dynamics based
simulators: A literature review
Extended Abstract
Simulators, interactive leaming environments and microwodds have attracted attention for various
reasons, both within and outside the fied of system dynamics? For more than 15 years many studies
have either used simulators as research tools to explore human and organizational characteristics, or
have presented simulators as instruments for teaching. For both purposes, simulators promise to be
velid means to achieve the goals intended as exemplified by the following two quotes considering the
potential value of simulators:
“... simulation-based leaming is usually expected to motivate, to invite active and deep
processing of subject matter, to allow for systematic exploration, for fruitful failure, and for
unlimited practice, all of which should contribute to better leaming outcomes, reduced leaming
time, orboth.” (Goodyear et al. 1991, 274)
“(With simulators we will] be able to do fiddwork in the laboratory, albeit under conditions
where the characteristics of the fidd are known in detail. Thus, we will have escaped the
narrow straits of the laboratory, as well as the deep blue sea of the field study, without losing too
much of the advantages of either approach.” (Brehmer/Démer 1993, 183)
Simulators are computer based simulation games of real world soenarios. Regularly, simulators of
social systems like, for instance, organizations are operating with a reduced level of detail compared
to reality. Users of simulators take on the role of decision makers within the systems. In many cases
they posses practically unlimited power to decide on available decision variables, similar to a
manager: owner of a firm.
Together, (1) a preconfigured formal simulation modd, (2) a human-computer interaction
component and (3) gaming functionality build the three basic aspects of a simulator (Maier and
Gdler 2000). The formal model, which undedlies the simulator, determines how user decisions are
processed and what outcome they produce. The humancomputer interaction component is
responsible for presenting the current state of the model and allowing the user to input decisions. The
ganing functionality lays down, for example, for what time interval decisions have to be made,
whether and how different agents compde in the simulation, or the contextual story in which the
simulation game is embedded. Simulators can be used for a variety of different purposes: research,
teaching and training, entertainment, personnel selection, motivation, etc. In the context of this paper
‘we concentrate on the first area of application. It is investigated, to which extent simulator based
studies allow the examination of the kind and the degree of human rationality.
From the further discussion in this paper we exclude the following tools as not being a simulator:
1. modding enironments (eg. Vensim, Powersim or Stella/iThink) because they focus on the users
building modds on their own, experimenting with them and changing them, not on pre- configured
modds;
2. board games (eg. the “beer game”, Senge 1990) because they are not computerized and,
therefore, miss a lot of the features and peculiatities of simulators;
3. stand-alone simulation models (eg. the “market growth modal”, Forrester 1968) because they
usually do not comprise an daborated usercomputer interaction component or gaming
functionality;
1. Wewill use the tam,,simulaior” throughout this paper. Our arguments apply to simulators of
2
4. role playing games because they do not use a formal modal to calculate outoomes of decisions;
5. group model building interventions (Richardson and Andersen, 1995; Vennix 1996) because
they concentrate on the creation of a mode (not using an existing one) and draw their power
mainly from the interaction between humans.
The objectives of this paper are twofold. First, we want to demonstrate the state of simulators as
‘well accepted instruments in the system dynamics area. Despite of some open questions concaming
their validity they are used in a variety of ways to explore human and organizational decision making.
Second, we aim at summarizing empirical findings derived from a literature analysis. This endeavor is
connected to providing starting: points for identifying a number of remaining issues of simulator usage
in research.
Our review wes ddiberately constrained to literature from within the system dynamics literature.
With this restriction we do not deny the extensive and daborated work that has been done in other
fidds, for example, in psychology or in the teaching sciences. However, constraints on breadth and
focus of the papas to be included seemed unavoidable in order to achieve interpretable results from
the review. Literature reviews stemming from other branches of science can, for example, be found
in Wolfe (1985), Klanmumiz (1987), Funke (1991), Brehmer (1992), Funke (1995), Buchner
(1995).
Besides that, the concentration on system dynamics based literature has two reasons:
1. We want to contribute to the ongoing discussion between system dynamicists whether simulation
experiments without modeling can yield substantial gains for the user (Forrester 1961, Machuca
1992, Davidsen 1996).
social, not merely technical systems.
2. System dynamics appears to be an appropiate and effective tool to create the undedying fonnal
model of simulators (Gro@ler 2001).
Thus, we used two major sourves of references: the proceedings of the annual intemational system
dynamics conferences from 1985 to 2002 and the back issues of the System Dynamics Review
(Volumes 1 to 18). In addition, we included relevant papers from Morecroft and Starman (1994)
and two special issues of Simulation and Gaming (Symposium Issue: System Dynamics and
Interactive leaming Environments, Vol. 31, Nos. 2 and 3, 2000). We did not include “isolated”
other papers published in this or another journal or in another book. In total, our database contains
more than 200 entries from these sources. The conmplee database will be aovessible at the Word
Wide Web at the time of the workshop.
In the system dynamics fidd two literature reviews on simulators exist. Both, the work of
Sterman (1994) and Hsiao and Richardson (1999) provided valuable sources of inspiration. Our
work, however, differs from their studies regarding soope of the review (as measured by number of
literature sources taken into account) and focus of investigation: on the one hand, this paper has a
nnalrower perspective because it only considers literature from a system dynamics context; on the
other hand, it is broader because it not only examines papers where simulators were used as
instruments to conduct research in decision making but also evaluation studies, general discussions
on simulator design and mere presentations of simulators. Thus, we split the total sample of papers
into four main categories according to their primary focus:
(A) Presentation of simulator only;
(B) Evaluation studies that explore the effectiveness of a simulator (effects beyond the simulator
context on eg. transfer of leaming to daily work);
(C) Investigations into dynamic decision making with the help of simulators (effects within a
simulator on eg. performance);
(D) General discussions of simulator usage, design and utility without intensive referee to one
ormore specific simulators;
After some remarks on the definition and the expected value of simulators, the paper presents a
oad sample of studies from the system dynamics literature; descriptive statistics from this sample
are shown. In the section after that a summary of empirical findings from the literature sample is
given. The paper closes with the discussion of issues connected to the usage of simulators in research
about rationality.
General description of the database
The results preserted here very much represent work in progress and preliminary outcomes are
based on 214 collected papers. The following figure orders the papers by their year of publication.
number of papers
il
As can be seen from the figure, there has been an interest in simulators from the middle of the 1980s
alalall
TOOZ
roe
ou S &
]
a
zor
sset
zset [4
eset |O
test
Z66T
€66T
vO6T
S66T
966T
L661
866T
666T
000z
986T
686T
O66T
on. Although the number of publications generally increased over the years, the last two years have
5
seen a decline from 33 papers in 2000, to 13 in 2001 and 17 in 2002. Of the total number of 214
papers, at present 30 papers have not been placed in one of the categories A to D described above.
Tn most cases the uncategorised studies are presentation handouts only and do not include a full
paper. Of the 184 papers which are categorised, the majority (80) concems presentations of a
simulator without further reference to evaluations. These are the papers categorised as A. Categories
B and C ae of main interest here and include 12 and 49 papers respectively. Category D includes
43 papers.
General description of research into dynamic decision making (category C)
Independent variables Effect on performance
Positive effect Negative effect Mixed/ no effect
Model characteristics delay | high frequency (Bakken, 1992) | misperception time dday - no effect (Diehl, 1989)
(Stearman, 1989) ~ physical easier than reporting
ddays (Brehmer, 1989)
- continuous easier than discrete
ddays (Batlas and Ooevin,
2001)
‘strength feedback ~ misperception feedback
decision to environment
(Sterman, 1989)
~ increased negelive or positive
feedback (Diehl, 1989)
- Langley, Paichand Staman
(1998)
- Young, Chen, Wang and Chen
(1997)
structural complexity - change in compditor behavior | - step increase exogenous input no|
(Langley, Paich and Sterman, effect (Barlas and Ooevin, 2001)
1998)
~ turbulence decreases
performance (Schultz, Dutta
and Johnson, 2000)
Simulator characteristics -no effect (Barlas and Ocevin,
Jength of decision interval 2001)
transparency - Groer(1998) - CLDs confusing (O’ Neill, - graphical mare useful than.
- Gréler, Maier and Milling 1992) mumerical (O'Neill, 1992)
(2000)
interface design ~ Howie, Sy, Ford and Vicente
(2000, reduction
misperception)
- screen design impacts
‘performance (Y oung, Yang and
Wang 1992)
cognitive feedback ~ decision mules and structure - - outcome feedback is not
behavior relationships sufficient (Diehl, 1988)
(Langley, 1995) - teaching how to make decision.
- threshold wamings effective tules explicit has no effect
(O'Neill, 1992) (Langley and Morecroft, 1996)
Player characteristics ~ feedforward better than feedback
decisonstrtegy strategy (Pack, Kim, Yi and Jun
(1996)
- idem (Schultz, Dutta and
Johnson, 2000)
menial modd/ cognitive _| - highersimilarity menial moda‘ - short tam goals negativdy —_| - neither GCSI, nor MBI explain
style to simulator increases affect performance (Y ang, variation in scores; The two
performance (Ritchie Dunham, | 1996, 1997) abstract components of the GSD
2001) (ASand AR) seamto explain
- financial moddss better than simulator scores (Soott-Trees,
health care models (Schultz, Doyle and Ratzicki, 1996)
Dutta and Johnson, 2000)
group composition. - pairs better than individuals
(Pak, Kim Yi and Jun, 1996)
The literature also suggests a number of control variables that can influence dependent variables in
addition to the variables manipulated in the research: number of hours played, simulator (complexity
simulator), instructions, presence of a facilitator, practice (Diehl, 1989) presence of monetary
rewards (Sterman, 1989: 310).
Other dependent variables, apart from performance in the table above, are the following:
- perfonmanoe, transfer of leaming (between simulators),
- insight dynamic understanding or mental models (Doyle, Ratizicki and Scott Trees, 1996),
knowledge (Vennix 1990), knowledge (Gréfer, 1998), causal understanding (McComack
and Ford, 1998), number of infonvation terms, inspecting time, mental model correctness
(Young, Yang and Wang, 1992), mental model development (Shields 2001, 2002).
- thinking skills: systemic thinking (Cavalai and Thompson, 1995), presence seven system.
thinking skills (Maani and Maharaj, 2001)
- others: parveived usefulness dements (Cavaleri and Thompson, 1996), number of tials
before equilibrium achieved (Jensen, 2002), satisfaction (e.g. Kim, 1989), collective versus
“egotistic’ decisions (o’Naill, 1992)
Finally, the literature also suggests combinations of independent variables or intermediate variables:
- combinations of independent variables and decision rules Stemman (1989), Langley, Paich
and Sterman (1998, in the form of simple CLD), Barlas and Ocevin (2001)
7
- transparancy to knowledge to performance (cf. Gréfder, Maier and Milling, 2000)
References
Bakken, B.E., 1992. “Frequency and Experiential Leaming in Unstable Markets”, Proceedings of the
1992 International Conference of the System Dynamics Society, pp. 52-64.
Barlas, Y. and Ocevin, MG., 2001. “Testing The Decision Rules Used In Stock Management Models”
Proceedings of the 19” International Conference of the System Dynamics Society 2001.
Brehmer, B., 1989. “Feedback Delays And Control In Complex Dynamic System’, Computer-based.
Management of Complex Systems -- Proceedings of the 1989 International Conference of
the System Dynamics Society, pp. 189-196.
Brehmer, B., 1992. “Dynamic Decision Making: Human Control of Complex Systems”, Acta
Psychologica, 81, 211-241.
Brehmer, B. and Démer, D., 1993. “Experiments With Computer-Simulated Microwords: Escaping
Both the Narrow Straits of the Laboratory and the Deep Blue Sea of the Field Study”,
Computers in Human Behavior, 9, 171-184.
Buchner, A., 1995. “Basic Topics and Approaches to the Study of Complex Problem Solving”, in P.A.
Grensch and J. Funke (eds.): Complex Problem Solving- The European Perspective pp. 27-
Cavalei, S., and Thompson, 1996. “Assessing the Efficacy of Microwords for Promoting Systems
Thinking, System Dynamics '96 -- Proceedings of the 1996 International System Dynamics
Conference, pp. 94-97.
Cavalei, S., Raphael, M. and Filletti, V., 2002. “Evaluating the Performance Efficacy of Systems
Thinking Tools”, Proceedings of the 20° International Conference of the System Dynamics
Society 2002, p. 57.
Davidsen, P.I., 1996. “Educational Features of the System Dynamics Approach to Modelling and.
Simulation’, Journal of Structural Learning, 12(4), 269-290.
Diehl, E.W., 1988. “Participatory Simulations as Training Tools - A Study Based on the Market
Growth Model”, Proceedings of the 1988 International Conference of the System Dynamics
Society, pp. 52-64.
Diehl, EW., 1989. “A Study On Human Control In StockAdjustment Tasks”. Computer based.
Management of Complex Systems. Proceedings of the 1989 International Conference of the
System Dynamics Society, pp. 205-212.
Doyle, J.K., Radzicki, M. and Scott Trees, T., 1996. “Measuring the Effect of Systems Thinking
Interventions on Mental Modes” Proceedings of the 1996 International System Dynamics
Conference, pp. 129-132.
Ford, D.N. and McConmack, D.E.M., 2000. Simulation and Gaming, Vol. 31, No. 3 (2000), pp.309-
330.
Forrester, J.W., 1961. Industrial Dynamics, Cambridge: MIT Press.
Forrester, J.W., 1968. “Market Growth as Influenced by Capital Investment”, Industrial Management
Review, (2), pp. 83-105.
Funke, J., 1991. “Solving Complex Problems: Exploration and Control of Complex Systems”, in RJ.
Stemberg and P.A. Frensch (eds.): Complex Problem Solving - Principles and Mechanisms,
pp. 185-222.
Funke, J., 1995. “Experimental Research on Complex Problem Solving”, in P.A. Frensch and J. Funke
(eds.): Complex Problem Solving- The European Perspective, pp. 243-268.
Goodyear, P, Njoo, M, Hijne, H and van Beckum, JJ.A., 1991. “Leaming Processes, Leamer
Attributes and Simulations’, Education and Computing, 6, 263-304.
Grier A., 4998. “Structural Transparency as an Element of Business Simulators”, Proceedings of
the 16 International Conference of the System Dynamics Society 1998, p. 39.
Gréfler A., 2001. “Musings about the Effectiveness and Evaluation of Business Simulators”, in J.H.
Hines et al. (eds.): Proceedings of the 19" International Conference of the System
Dynamics Society, p. 72.
Gre, A., Maie, FH. and Milling, P.M, 2000. “Enhancing Leaming Capahilities by Providing
Transparency in Business Simulators”, Simulation and Gaming, 31(2), pp. 197-229.
Howie, E,, Sy, S., Ford, L., Vicente, KJ., 2000. “Human-Computer Interface Design Can Reduce
Misperceptions of Feedback”, System Dynamics Review, 16(3), pp. 151-171.
Hsiao, N. and Richardson, G.P, 1999. “In Search of Theories of Dynamic Decision Making: A
Literature Review”, in RY. Cavana et al. (eds.): Systems Thinking for the Next Millennium -
Proceedings of the 17'" International Conference of the System Dynamics Society, CD-
ROM.
Jensen, E,, 2002. “Is Explicit Information Important for Peformance in Dynamic Systems?”,
Proceedings of the 20° International Conference of the System Dynamics Society 2002, p.
82.
Kim, D., 1989. “Leaming Labarotories: Designing a Reflective Leaming Environment’, Computer
based Management of Complex Systems -- Proceedings of the 1989 International
Conference of the System Dynamics Society, pp. 327-334.
Kienmumtz, D.N., 1987. “Human Decision Processes: Heuristics and Task Structure’, in P.A.
Hancock (ed.): Human Factors Psychology, pp. 123-157.
Langley, P.A., Morecroft, J.D., 1996. Leaming From Microword Environments: A Summary of
Research Issues, System Dynamics '96 -- Proceedings of the 1996 International System
Dynamics Conference, pp. 300-303.
Langley, P.A., 1995. “Building Cognitive Feedback into a Microworld Leaming Environment: Results
from a Pilot Experiment’, System Dynamics '95 -- Proceedings of the 1995 International
System Dynamics Conference, pp. 628-637.
Langley, P.A., Paich, M, Sterman, J.D., 1998. “Explaining Capacity Overshoot and Price War
Misperceptions of Feedback in Competitive Growth Markets’, Proceedings of the 16"
International Conference of the System Dynamics Society 1998, p. 3.
Maani, KE, Maharaj, V., 2001. “Systemic Thinking and Complex Problem Solving. A Theory Building
Enmnitical Study”, Proceedings of the 19” International Conference of the System Dynamics
Society 2001, p. 101.
Machuca, J.A.D., 1992. “Are We Losing One of the Best Features of System Dynamics?”, System.
Dynamics Review, 8(2), 175-177.
Maier, F.H. and Grier A., 2000. “What Are We Talking About?—A Taxonomy of Computer
Simulations to Support Leaming”, System Dynamics Review, 16(2), 135-148.
McConmack, DEM, Ford, D.N, 1998. “Impacts of Dynamic Decision Making and Policy
Devdopment Modes on the Causal Understanding of Management Flight Simulators’,
Proceedings of the 16° International Conference of the System Dynamics Society '98,
p. 59.
Morecroft, J.D.W. and Sterman, J.D., 1994. Modeling for Learning Organizations, Portland:
Productivity.
O'Neill, K., 1992. “Real-Time Tragedies: A Simulated "Commons Leaming Laboratory”, Proceedings
System Dynamics 1992- Supplement, pp. 25-34.
Park, Kim, Yi and Jun, 1996, “Enhancing the Performance in Dynamic Decision Making: The Adaptive
Model Reconstruction Using Feedforward vs. Feedback Decision Strategy”, Proceedings of
the 1996 International System Dynamics Conference, pp. 413-416.
Richardson, G.P. and Andersen, D.F., 1995. “Teamwork in group model building’, System Dynamics
Review, 11(2), 113-137.
Ritchie- Dunham, J.L., 2001. “Informing Mental Models for Strategic Decision Making with ERPS and
the Balaced Scorecard: S SimulationBased Experiment’, Proceedings of the 19
International Conference of the System Dynamics Society 2001, p. 126.
Schultz, F., Dutta, P. and Johnson, P., 2000. “Mental Models and Decision Making in a Dynamic
Health Care Environment’, Sustainability in the Third Millennium -- Proceedings of the 18"
International Conference of the System Dynamics Society, p. 186.
Scott Trees, T., Doyle, J.K. and Raczicki, M,, 1996. “Using Cognitive Styles Typology to Explain
Differences in Dynamic Decision Making in a Computer Simulation Game Environment’,
System Dynamics '96 -- Proceedings of the 1996 International System Dynamics
Conference, pp. 557-560.
Senge, P.M., 1990. The Fifth Discipline - The Art and Practice of the Learning Organization,
New York: Currency/Doubleday.
Shields, M., 2001. “An Experimental Investigation Comparing the Effects of Case Study, Management
Fight Simulator and Facilitation of These Methods on Mental Model Development in a Group
Setting” Proceedings of the 19” International Conference of the System Dynamics Society
2001, p. 139.
Shields, M.,, 2002. “The Role of Group Dynamics in Mental Model Development’, Proceedings of the
20" International Conference of the System Dynamics Society 2002, p 131.
Sterman, J.D., 1989. “Misperceptions of Feedback in Dynamic Decision Making’, Organizational
Behavior and Human Decision Processes, 43, pp. 301-335.
Sterman, J.D., 1994. “Leaming in and about Complex Systems”, System Dynamics Review, 10(2/3),
291-330.
Vennix, J.A.M., 1990. Mental Mbdels and Computer Models, Unpublished doctoral dissertation,
Nijmegen University.
Vennix, JAA.M, 1996. Group Model Building: Facilitating Team Learning Using System
Dynamics, Chichester Wiley.
10
Wolfe, J., 1985. “The Teaching Effectiveness of Games in Collegiate Business Courses”, Simulation
& Ganes, 16(3), pp. 251-288.
Yang, J., 1996. “Facilitating Leaming Through Goal Setting in a Leaming Laboratory”, System
Dynamics '96 -- Proceedings of the 1996 Intemational System Dynamics Conference, pp. 593-
596.
Yang, J., 1997. “Give Me the Leverage, I Will Be a Good Dynamic Decision Maker”, 15°
International System Dynamics Conference -- Systems Approach to Learning and
Education into the 21st Century, pp. 709-712.
Young, SH, Chen, C.P., Wang, S.-W. and Chen, CH, 1997. ‘The Landmine Structure and the
Degrees of Freedom of Decision-Making’, 15” International System Dynamics Conference -
- Systems Approach to Learning and Education into the 21st Century, pp. 15-20.
Young, Yang and Wang, 1992, “Enhancing the Leaming Effects of Dynamic Decision Game on
Systems Thinking”, Proceedings System Dynamics 1992, pp. 847-856.
11
Back to the Top
