Mental Models and Dynamic Decision Making: An
Experimental Approach for Testing System Methodologies
Rajat Dhawan, Marcus O’Connor and Mark Borman
Discipline of Business Information Systems, School of Business
The University of Sydney, NSW 2006
Telephone: +61 2 9036 9432, Fax: +61 2 9351 6638
E: {r.dhawan,m.oconnor,m.borman}@ econ.usyd.edu.au
Abstract
The effect systems thinking and system dynamics on a person’s dynamic decision-making
abilities is yet not fully known. To explore this relationship, a controlled experimental study
was conducted. Results of the study show that most participants initially had poor
understanding of basic dynamic situations. However, the completeness and accuracy of
their mental models improved considerably with system interventions. Specifically
participants’ ability to discern between stocks and flows, identify causal relationships and
feedback improved by around 27% after systems thinking intervention. These abilities
further increased by around 4% after participants underwent a system dynamics
intervention. Interestingly, in complex tasks, systems thinking hardly made any positive
effect on participants’ decision-making. However, for the same tasks, participants’ mental
models improved significantly after system dynamics intervention. This study contributes to
the long-standing debate on the links between system methodologies and dynamic decision-
making: particularly the relative contribution of systems thinking and system dynamics to
decision-making.
Keywords: dynamic decision-making, intervention, psychology, experimental, systems
thinking, system dynamics
Introduction
Systems thinking and system dynamics are widely used methodologies for studying and
managing complex systems. Their focus is to study the relationships between components
in the system, especially feedback loops and the patterns of behaviour generated by them.
Though both these system methodologies are widely accepted as aids for making decisions
in complex systems, surprisingly the relationship between these and dynamic decision-
making is yet to be fully explored (Cavaleri and Sterman (1997), Huz et. al. (1996) and
Doyle (1997)). There have been some studies conducted in the past to explore participants
understanding of dynamic tasks like those by Sweeny and Sterman (2000) and Kainz and
Ossimitz (2002). Other studies which address similar issues include those by Maani and
Maharaj (2004) and Gary and Wood (2005). However, there isn’t any literature on the
relative contribution of systems thinking and system dynamics on understanding dynamic
situations.
This study seeks to explore the relationship between decision making in dynamic tasks and
systems methodologies in a pre-test/post-test design. The aim of this experiment is to test
participants’ ability to perform in dynamic situations with a variety of skills.
(i) With systems thinking skills alone
(ii) | With a combination of systems thinking and system dynamics skills
With this aim the following hypotheses are proposed:
1. The understanding (completeness and accuracy of mental model) of a complex
system is enhanced when participants use systems thinking as a decision aid as
compared to a control group not using any decision-aid.
2. The understanding (completeness and accuracy of mental model) of a complex
system is enhanced when participants use a combination of systems thinking and
system dynamics as a decision aid as compared to those who use mere systems
thinking and controls.
The paper consists of five sections including this introduction. The second section discusses
the method that was followed to conduct the experiment. The subsequent section is devoted
to results and discussion. The paper concludes with the major finding of this research, its
limitations and avenues for further research.
Method
Subjects
Thirty-one graduate students enrolled in the School of Business at the University of
Sydney, participated in the study. All students were either in their penultimate or final
semester at the University and some had prior work experience. Students had to undergo
the study in order to fulfill course requirements and were graded. Five out of the thirty-one
students failed to satisfy the criteria of the experiment as they missed either one of the
lectures or one of the tests. These were excluded from the data set.
Design
The experiment is structured around a scenario of a firm that provides consulting and IT
services. The scenario describes in sufficient detail the operations of the firm. It then
describes a problem of periodic oscillations in revenue over time, the measures taken by the
top management and other parties involved as well as includes the response of the
employees towards those corrective policies. The task for the participants was to analyse
the situation and assess the cause of the periodic oscillations.
All participants were administered three tests (one pre-test and two post-tests termed post-
test 1 and post-test 2). Students were distributed the case study a week prior to the pre-test
and had ample time to read and re-read the scenario. In the following week they were
administered the pre-test. In between pre-test and post-test1, the participants were taught
systems thinking. Participants were taught system dynamics modelling between post-test1
and post-test2.
The tests on all three occasions were similar. However, the questions were not provided to
the participants after the any test, nor were the students aware that similar questions would
be repeated in subsequent tests. Many questions were re-phrased and the data and context
was changed for quantitative questions. Keeping in mind that there is a possibility of
learning of questions from one test to the other, future experiments have been designed to
address this issue.
Procedure
The experiment was spread over 5 weeks and took place either in a lecture hall or a
computer lab. All tests were individual and participants were seated at sufficient distance
from each other.
While conducting the experiment, eight goals, as specified by Doyle et.al. (1998) were
adhered to a large extent.
The detailed schedule of the experiment is given below:
Week 1: Participants were provided with the case study at the end of the lecture. They were
asked to read the case during the week and they were also told that they would be quizzed
on the scenario in the next week’s lecture.
Week 2: All participants underwent the pre-test in the first hour. After the test, students
underwent a short course of systems thinking.
Week 3: The systems thinking lecture continued from the previous week and concepts were
revised. After a short break, all participants underwent the post-test1 in the first hour. For
the remaining period they were introduced to the concept of system dynamics modeling.
Participants were asked to familiarize themselves with the software during the week.
Week 4: The lecture on system dynamics modeling was continued from the previous week.
All participants devoted significant time to model in Powersim, analyze the outputs and
conduct sensitivity tests. At the end of the session, participants were given questions to
practice at home.
Week 5: The first two hours was a revision class where all the concepts from week 2 to 4
were reinforced. During the last hour, participants were administered the post test 2.
The total time spent by students on systems thinking was 10 hours and that on system
dynamics was 13 hours. This included time spent in the class as well as time taken to
complete related tasks at home. The lectures, related handouts and exercises were drawn
from standard systems thinking/system dynamics texts and covered majority of the
concepts of these two methodologies.
Data Analysis
The data collected from participants was both qualitative and quantitative. The method of
their analysis is described below.
Quantitative Analysis
Quantitative data included participants’ response to questions relating to discerning
between stocks and flows, calculating the values of certain variables (based on the data
provided) and measuring the change in confidence after and before the intervention. Part of
this analysis was based on the framework utilized by Kainz and Ossimitz (2002).
Qualitative Analysis
Most of the data collected from participants was in the form of narratives. Analyzing these
narratives to measure change in participants’ mental models required a coding criteria. The
criteria utilized in this study are based on two previous studies (Doyle et.al. (1998) and
Maani and Maharaj (2004)). An expert model of the problem was constructed before hand.
Each answer of the participant was then matched to this model and the percentage of
correctness was calculated by identifying correct causal relationships. This helped in
measuring the completeness of the participants’ mental model pre and post intervention.
Additionally the completeness of the mental models was also measured by measuring the
type and number of relationships identified by participants.
Results and Discussion
This study attempted to explore the links between two system interventions (systems
thinking and system dynamics) and dynamic decision making. Figure 1 below presents the
results of this study.
90
—? 85
80 4-8
70 a
60 Ys
we 38
50 Za
‘age of participants who got it
correct
wis af
10
4 ES A
Control ST sD
Level of Intervention
—@-simple 1 (stocks and flows)
——complexl (suggest a solution)
—m- medium (whatis causing oscillations)
—<— complex 2 (predict the workforce)
Figure 1: Results
During the course of analysis of the results, various things came up. First and foremost, as
found by previous studies (Sweeny and Sterman (2000), Kainz and Ossimitz (2002)) the
dynamic decision-making abilities of participants were surprisingly low. The authors were
surprised to see such poor results from graduate students, all of whom had a mathematical
background. More than 40% of the participants could not find the maximum value of a
stock, given the in and outflows. Our results were very similar to those conducted by Kainz
and Ossimitz (2002). However, results improved to a great extent by a short course on
systems thinking and explanation of stocks and flows. Results further improved (though not
as much as before) once participants were given further exposure to stock and flow
modeling
The results of the question “what in your view is causing the periodic oscillations in
revenue?” were then analysed. Without any systems intervention, more than 50% of the
participants could only identify one-to-one relationships. Most of these were quite clearly
mentioned in the case study. Furthermore participants were provided a week’s time to read
the scenario. Participant responses were mostly superficial and lacked any in-depth analysis
of the relationship between key variables. Few recognised the “big picture” by identifying
the relationship between the decision, its short-term effect and the long-term unintended
consequences. One participant wrote that “I think the most important thing which causes
these periodic oscillations is the problem of hiring process. They shouldn't hire new
employees only when they have [got] not enough people and fire them when have not [got]
too many projects”. Another simply stated that “an ineffective management on human
resources aspect”. Some merely copied parts of text from the case study showing their
inability to analyse the problem. However, post-intervention most participants improved in
their ability to recognize the key relationship. They could trace relationships as well. Many
did an in-depth analysis of the problem and some even attempted to compare the situation
correctly with well-known system archetypes. One participant wrote “from the short-
period, more project in hand, more new employees hired, and relieve the work pressure,
make revenue increase. However, from long-period, this lead to increase in quit-rate,
experienced employee decreased, that makes the project fail, this make the revenue fall
down. This is a repeatable situation’. This participant not only identified most
relationships, but traces variables with their effects on each other, identifies the short-term
versus long-term effects and also that this is a cyclic process. Another participant wrote
“since employee pressure increase, company will try to increase hiring rate to decrease
employee pressure, but there is delay in their process, and other unintended consequences
happen faster than this delay, so employee pressure is accumulated, which cause
performance to slow down = revenue slow down”. The intervention, made a significant
impact on the ability of participants to think causally, focus on the delays of the system and
analyse the situation by tracing causal links. Many students were able to identify the
feedback loop that was the cause of the oscillations.
The third question “advise a long-term solution to the problem” required participants to
utilize the knowledge that they had gained from their analysis of the problem and suggest
measures that would help in alleviating the problem. Without any intervention, most (58%)
participants focussed their suggestions on ‘planning’ for the project before hand. An equal
number of participants (27%) suggested that the current retrenchment of employees be
stopped and recruitment should commence. Some (15.38%) participants suggested that the
delay in hiring be reduced. Most participants provided more than one suggestion. After
undergoing the systems thinking intervention, unlike previous questions, response hardly
improved. Participants who thought reducing the delay would be helpful, increased by
19%. However there was no change whatsoever in the number of participants who
suggested retrenchment to reduce. Surprisingly there was a reduction in the number of
participants who indicated that recruitment be increased (by 4%) and that there was a need
for planning of the project (by 12%). After participants underwent a system dynamics
modelling session, they modelled the entire scenario in Powersim. The suggestions that
they provided subsequently told a different story. A huge amount of participants (50% more
when compared to the after the systems thinking intervention) suggested that recruitment
should increase as long-term solution. 8% more participants thought that reducing the delay
was one of the strategies to alleviate the problem and 15% more thought that impeding
retrenchment was a good strategy. 7.7% more participants thought that an improved
planning was required. Most participants were able to model the scenario correctly and
perform sensitivity tests on it. The improved perception of the underlying cause of the
systemic problem might be expected since participants interacted much more with a formal
model and simulated it. This indicates that though systems thinking alone does help in
understanding the relationship between key variables and delays and to discern between
stocks and flows, it is not very useful in situations where the scenario is very complex. In
this case, computer modelling, simulation and sensitivity tests give a much deeper insight
on the existing problem.
Conclusion
This study was conducted to explore the links between system interventions and dynamic
decision making. The findings suggest that both systems thinking and system dynamics are
useful methodologies in understanding complex systems. Specifically, participants’ ability
to take decisions in simple tasks such as discerning between stocks and flows, identify
causal relationships and feedback improved by around 27% after a systems thinking
intervention was introduced. These abilities further increased by around 4% after
participants underwent a system dynamics modeling intervention. Interestingly, in a more
complex task that involved numerical data to be processed, required an in-depth analysis
and subsequently involved suggesting a solution to alleviate a systemic problem, systems
thinking hardly had any positive effect on the participant’s decision-making. However, for
the same situation, participants’ mental models improved significantly after system
dynamics modeling intervention. The results of this study confirm the results of some of the
previous studies done in this area (Sweeny and Sterman (2000), Kainz and Ossimitz (2002),
Gary and Wood (2005)) and give a deeper insight on the impact of system interventions on
dynamic decision-making. The study has its own limitations. Given that the study was
conducted in a University setting and not with real-world practitioners, these results cannot
be generalized to a great extent. Also, the interventions provided to the participants were
short in duration due to the structure of the course. The authors are involved in further
experiments to explore these relationships in a more rigorous experimental setting.
References
Cavaleri, S. and Sterman, J.D. (1997). “Towards evaluation of systems thinking
interventions: a case study” System Dynamics Review Vol. 13 (2): 171-186
Doyle, J.K. (1997) “The cognitive psychology of systems Thinking” System Dynamics
Review Vol. 13 (3): 253-265
Doyle, J. K., Radzicki, M.J., Trees, W. S. “Measuring Change in Mental Models of
Dynamic Systems: An Exploratory Study” Report No. 14 May 26, 1998 accessed from
https://www.wpi.edu/A cademics/D epts/SSPS/Research/Papers/14.pdf on 1st March 1, 2006
Doyle, J. K. and Ford, D.N. (1999). “Mental models concepts revisited: some clarifications
and a reply to Lane” System Dynamics Review Vol. 15 (4): 411-415
Gary, M.S. and Wood, R. (2005) Mental Models, Decision Making and Performance in
Complex Tasks. Proceedings of the 2005 International System Dynamics Conference.
System Dynamics Society: Albany, NY.
Huz, S, Andersen, D.F., Richardson, G.P., and Boothroyd R. (1996) “A framework for
evaluating systems thinking interventions: an experimental approach to mental health
system change”
Kainz D. and Ossimitz G. (2002). Can students learn stock-flow thinking? an empirical
investigation. Proceedings of the 2002 International System Dynamics Conference.
System Dynamics Society: Albany, NY.
Maani, K. and Maharaj, V (2004) “Links between systems thinking and complex
decision making” System Dynamics Review Vol. 20 (1): 21-48
Sterman, J.D. (1994). Learning in and about complex systems. System Dynamics Review
10(2/3), 291-330.
Sweeny, L.B. and Sterman, J.D. (2000) ” Bathtub dynamics: initial results of a
systems thinking inventory” System Dynamics Review Vol. 16 (4): 249-286
