Enhancing the Learning Effects of Dynamic Decision
Game on Systems Thinking—An Experimental Study
Showing H. Young
Jenshou Yang
Sy Feng Wang
Department of Business Management
National Sun Yat-Sen University
Kaohsiung, Taiwan, R.O.C.
ABSTRACT
This study attempts to examine the effectiveness of our proposed learning environment
for systems thinking, and the effects of different kinds of task's screen design for the
interactive dynamic decision game to enhancing the learning effect. Two experiments
were implemented for the two investigation purposes. In the first experiment, we found
that the proposed learning environment was viable for learning resulting from the ‘Positive
effects of challenging goal setting and causal feedback on the increase of participants’
motivation and understanding of the game. In the second experiment, the effects of
causal feedback was examined directly by the comparisons of three different kinds of
task’s screen design including causal, hierarchical, and departmental types. We found
causal type of screen design induced more analytical cognitive type just as the prediction
of the inducement principle (Hammond, 1988) and outperformed the other two screen
design as predicted by the correspondence-accuracy principle (Hammond, 1988). But the
effect of causal type on performance improvement was not significant. The insignificant
effect of causal relations task's screen design on performance i improvement revealed that
the learning of systems thinking relied mainly on “by doing" or “by failure", not on "by
knowing". In conclusion, we suggested that the design of dynamic decision | game aided
systems thinking learning environment should take the motivation factor into account to
lead participants make more efforts to learn systems thinking by doing through failures.
Although causal relations type could not improve learning effect significantly, however, it
induced.corresponding causal analytical cognitive type that is beneficial to the learning of
systems thinking.
INTRODUCTION
Decision-makings are common in everyday life. Nobody knows what will be the results
of decisions before time is up. Not only because it is hard to predict precisely under such
complex and dynamic decision environment, but also because the rationality of human
beings is bounded to acquire perfect information to make unbiased judgments and optimal
decisions (Simon, 1957). For the purposes of enhancing decision-makers’ rationality,
system dynamicists have exerted themselves to the design of learning laboratory for
systems thinking combined with the aid of the dynamic decision game such as People
Express (PE) Management Flight Simulator (Sterman, 1988a) to cultivate the ability of
business managers in handling complex system (e.g., Senge, 1989, 1990; Graham and
Senge, 1990). How to design an effective learning laboratory has become an important
issue for system dynamicists. The purpose of this study attempts to examine the
effectiveness of our proposed learning environment for systems thinking, and the effects
- 847 -
of different kinds of task's screen design for the interactive dynamic decision game to
enhance the leaming effect. Two experiments were designed for these two purposes.
The proposed learning environment we designed is based on the goal setting theory
(Locke and Latham, 1990) and related cognitive theory in Social Judgment Theory
(Hammond, et al., 1975). The common results of empirical studies of the goal setting
effects indicate that if there is commitment to the challenging goals, adequate feedback,
high self-efficacy (and ability), and suitable task strategies, high performance will be
expected (Latham and Locke, 1991; Locke and Latham, 1990). Empirical evidences
from the studies in Social Judgment Theory indicate that effectiveness of the provision of
cognitive feedback is superior than that of the provision of outcome feedback on multiple-
cue probability learning tasks (Hammond, et al., 1973). Therefore, we tried to
manipulate high challenge in the form of specific goal and provide participants causal
relations feedback in the proposed learning environment in order to enhance the learning
effect of systems thinking in the first experiment . For the purpose of examining the pure
effects of the provision of causal relations feedback, we implemented the second
experiment to compare the effects of different kinds of task's screen design, including
causal, hierarchical, and departmental for dynamic decision game. ‘
With respect to the issues in designing interactive dynamic decision games,
Anderson; et al. (1990) suggested a number of general considerations including the
definition of the game's purpose, the assumptions and approaches taken toward user
psychology and gaming technique. On the issues of user psychology, Anderson, et al.
(1990) also suggested the comparison of two kinds of gaming screens, i.¢., designed
from lens model perspective (Brunswick, 1956) and system dynamics, as an important
topic for future research. Unfortunately, there still lacks of empirical works to examine
the effects of different kinds of task's screen design. To do this, we’ comply with
Anderson, et al.'s suggestions that the designed screens for comparison must base on
cognitive theory. In 1988, Hammond proposed a theory of task systems from the
cognitive point of view. There are two major principles for the predictions of decision-
maker's cognitive modes. The first is the inducement principle which means that the task
systems induce, but not compel, the subject to employ a form of cognition compatible
with the system. The second, the correspondence-accuracy principle, when cognitive
activity is found to correspond to task properties, judgmental accuracy will be greater
than cognitive activity that dose not correspond to task conditions (Hammond, 1988).
Based on the inducément principle, causal type of cues representation, which shows
subjects causal relations among cues, will induce causal loop analysis which belong to
more analytical cognitive type. In addition, causal feedback dynamic decision tasks call
for causal analysis and detail computation. Therefore, it can be classified as more
analytical cognitive type. Owing to the analytical characteristics of dynamic decision
task, the causal type of cues representation will perform better than the other two designs
based on the correspondence-accuracy principle. For measurement, Hammond (1988)
proposes the cognitive continumm index to judge what kind of cognitive types (more
analytical or more intuitive) used by subjects. The cognitive continumm index are
calculated with six dimensions including cognitive control, rate of data processing,
conscious awareness, organizing principle, errors, and.confidence. On a dynamic task! ,
rate of data processing is the only suitable dimension to judge the cognitive type used by
subjects. The usé of more analytical cognitive type leads to slower rate of data
processing.
1 The cognitive continumm index is developed based on static tasks.
- 848 -
eae
EXPERIMENT 1
In the first experiment, we manipulated challenging goal and the provision of causal
relations feedback in the proposed learning environment to examine the manipulation
effects on the enhancement of the learning of systems thinking.
Method
Subjects and Task
Subjects were the 86 MBA students (from two classes) at National Sun Yat-Sen
University attending courses in Decision Analysis and Management Information System.
All subjects were divided into two groups by class. We use People Express Management
Flight Simulator developed by Sterman (1988a) to serve as decision task.
Procedure
The experiment was a two days session (three hours each day). In the first day three
steps were taken: to introduce what the decision task was, how to operate the dynamic
decision game and then subjects were led to play the People Express Management Flight
Simulator. Subjects were asked to play the game after class. A week later, subjects
joined another three-hour sub-session which included four steps. First, subjects were
asked to finish the first questionnaire to evaluate the achievement of systems thinking that
learned from the game playing. Second, experimenter led subjects to present their
understanding about the underlying structure of the task with the tool of causal diagram.
Experimenter then éxplained the correct structure of the task after the discussion to finish
the program.
Design
For the purpose of finding a better procedure for the learning of systems thinking, the
second group was asked to finish an assignment as manipulation. To finish the
assignment, the participants were asked to achieve Don Burr’s? dramatic growth as
challenging goal to increase their motivation. When subjects attained the goal, they might
Jead the company go through quick growth and dramatic decline as the actual case of PE.
Under such circumstances, subjects must try their best to rescue the company, thus they
could learn something from the experience of failure or success. Besides, subjects in the
second group acquired a causal diagram of the task after the first day session. As a
result, this manipulation contained the effects of goal setting and causal feedback.
Subjects' performance was measured by stock price. It is from the financial
management point of view that maximizing stock price is the major goal of CEOs. The
other dependent variables included the number of game trials, the use of high ticket price
strategy, and the achievement of systems thinking. Whether subjects used high ticket
price strategy was analyzed from the collected data of the questionnaire. Achievement of
systems thinking was measured by the collected data of questionnaire with the
recognition of delay (Bakken, 1989) and three other indicators which were designed
based on Sterman’s work (Sterman, 1988b) as measures.
2 Don Burr was the founder of the People's Express Airlines who led the company with a record high
growth rate of 100% per year for several years (see Sterman, 1988b).
= 849 -
Results
As presented in Table 1, the number of game trials, the use of high ticket price strategy,
the achievement of systems thinking, and the performance of the first group were
significantly worse than that of the second group who accepted assignment manipulation.
These results showed that the effect of the assignment treatment including goal setting and
causal relation feedback was highly significant to motivate subjects to play more trials and
stimulated them to realize the underlying structure of the system and thus to improve their
performance.
Table 1. The comparison between the first and the second groups
Dependent Variables
Performance Systems Thinking Price-Strategy No. of trials
mean (SD) mean (SD) mean (SD) mean (SD)
Group 14 96.7(147.0) 1.0(1.0) .13(0.34) 5.1(24)
Group 2 383.3(270.5) 1.90.9) -66(0.48) 51.1653.7)
F Value 16.65** 8.20* 15.44** 11,29*
4 group 1 is the contrast group; group 2 is the manipulation group *
* p<0.01; ** p<0.001
Further tests focused on the effects of moderators of performance including the
number of game trials, the use of high ticket price strategy, and the achievement. of
systems thinking. As shown in Table 2, there existed significant positive relations
between performance and the number of trials, the use of high ticket price rule , and the
achievement of systems thinking (F(3,34)=42.36, p<0.0001; F(3,34)=18.03, p<0.0002;
F(3,34)=5.86, p<0.003, respectively). The multiple R square was 0.77. The results
showed that performance was affected by the number of game trials, the use of high ticket
price strategy, and the achievement of systems thinking. The use of high ticket price
Strategy was the most significant factor.
Table 2. Performance and its affecting factors
No. of Trials? Price Strategy Systems Thinking
Tevel 1 78.07 142.880 123.67
evel 2 318.00 438.26 243.21
level 3 335.79 378.15
level 4 $70.11 401.27
4 the number of trials is divided with quartiles into four levels
Devel 1 represents the use of high ticket price strategy
We then tested the effects of the number of game trials on the achievement of
systems thinking and the use of high ticket price strategy. As presented in Table 3, the
relationship between number of game trials and systems thinking was positively
significant, and so was price strategy. The results suggested that, the more the subjects
played the game, the more they might understand the underlying structure of the system
and found the price strategy to use.
Table 3. The effects of the number of game trials on systems thinking and the use of high
ticket price strategy
‘Dependent Variables
number of trial Systems Thinking Price Strategy
mean (SD) mean (SD)
level 1 1,09(1.22) 0.13(0.35)
level 2 1.50(1.07) 0.50(0.53)
level 3 1,93(0.73) 0.71(0.47)
level 4 2,220.83) 0.67(0.50)
F Value FG,41)=2.71 FG,45)=4.71
Prob.>F 0.0583 0.0064
Those results described above showed the effects of our proposed learning
environment. The challenging goal setting motivated participants to play the game more
times. As participants played the game more times under the aid of causal relations
feedback, they could realize the underlying operation rule of the system through the
learning from failure. Consequently, by learning the systems thinking skill subjects
might find the high ticket price strategy to improve performance.
EXPERIMENT 2
In the second experiment, we designed three types of task's screen, including causal,
hierarchical, and departmental types, to examine the pure effects of the provision of
causal relations feedback on the enhancement of the learning of systems thinking.
Method
Task
The decision task used in this experiment simulated a business management decision-
making problem. The macrodynamic decision environment we designed from
Forrester's "Market growth" system dynamics model (Forrester, 1961) and implemented
in Chinese on a Macintosh computer using HyperCard software. Subjects were
presented information related to financial, personnel, manufacturing, and marketing
departments of the simulated company for a single quarter year or historical time series
.data, numerically or graghically. Two decisions were made each quarter year including
how many salesmen to hire and how much investment to make on manufacturing. The
decision goal was to maximize total assets throughout 32 quarter years. Three principles
should be complied to make correct decisions. The first principle was to match the sales
volume with the manufacturing capacity. The second principle was to match the cash and
expenditure to avoid bankruptcy for the shortage of cash and utilize available cash
completely. The third was the consideration of delay in decisions.
Design
The experiment utilized a 2*3*3 factorial design, with one manipulation that related to the
task's screen design, to examine the effects of different kinds of cues representation. The
other two manipulations were not reported here. There were three kinds of task's screen
design including causal, hierarchical, and departmental for the dynamic decision game.
- 851 -
The causal type of screen was designed like the causal diagram used by system
dynamicists to represent the relations of variables. The hierarchical type of gaming
screen was designed with lens model's perspective. The example shown in figure 1
represented that total assets was correlated with total cash and total value of capital, and
so on. The departmental type of screen represented cues with the classification of
departments of simulated company. Since the departmental type is the most common
way to represent company's information, it can served as the contrast of the other two
conditions.
total cash
total assets
total value of capital
Figure 1. A demonstration example of hierarchical type of screen design /
The dependent variables included (1) induced information searching behavior,
measured by the number of searched information term and mean inspecting time on each
searched information term, (2) subjects’ mental model correctness, to measure the
performance by cognitive system, and (3) decision performance, to measure the
performance by decision outcomes. The number of searched information and inspecting
time on each searched information term were acquired through an automatic recording
system of our computer program. Subjects’ mental model correctness were analyzed by
the descriptions of their own decision strategies as protocol analysis inputs. Then the
reported decision strategies were compared with the three principles for correct decisions
to code the degree of subjects' mental model correctness. Decision ‘performance was
measured by the mean total assets and order growth rate averaged over 32 decision
quarter years.
Procedure
Subjects were provided with a five-page set of written instructions with the initial
conditions of the simulated company, the relationships among all variables in the task
with listed equation, the reward structure, and the goal of decision task (maximize the
assets). Throughout the decision process, the subjects' mental model which related to
how to manage the simulated company and their decision strategies were recorded.
Subjects could refer any feedback information before making each decision. Each trial
consisted of 32 quarter years. Subjects were asked to finish three complete trials in
addition to bankruptcies. It took subjects about 1.5 to 3 hours to finish the experiment.
Subjects
The subjects were volunteers and all were undergraduate students with the department of
business management at National Sun Yat-Sen University in Taiwan. Seventy two
participants were randomly assigned to the 18 cells of the factorial design.
Results
Induced Information Searching Behavior
The number of searched information term represents the range of information searching.
The mean inspecting time on each searched information term represents the rate of
information processing. The mean number of searched information term for causal,
hierarchical, and departmental types of screen design were 269, 218, and 195
respectively (F(2, 50)=4.07, p<0.03). The mean inspecting time for causal, hierarchical,
and departmental types of screen design were 1.28, 0.82, and 1.06 seconds respectively
(F(2, 50)=2.74, p<0.08). The results illustrated causal type of screen design induced the
information searching behavior that characterized with wider range of information
searching and slower information processing rate. That is, causal type induced more
analytical cognitive type compared with the departmental condition. Moreover, wider
range of information searching and quicker information processing for hierarchical type
indicated that hierarchical type induced more intuitive Cognitive type compared with the
departmental condition.
Moreover, correlational analysis showed significant relations between information
processing rate and both of the performance measured by subjects’ mental model and the
decision outcomes. The mean correlation coefficients averaged over three trials were
0.37 (p<0.05) and 0.50 (p<0.001) respectively. The results complied with the prediction
of the correspondence-accuracy principle that more analytical cognitive type, with slower
information processing rate, leads to better performance on the causal feedback dynamic
task which belong to the more analytical task.
Performance of Subjects' Mental Model Correctness
The subjects’ mental model correctness is presented in Table 4. Table 4 revealed that the
initial correctness of subjects’ mental correctness for three groups were nearly the same,
the result of ANOVA indicated no significant differences either. In the three finished
trials, the correctness for causal type was higher than hierarchical and departmental types.
Nevertheless, mean differences among three types were significant only in the first trial
(F(2,49)=2.83, p<0.07). Though there was significant trial effect, the interaction effect
between trial and manipulation was not significant. The results showed that there did
existed learning effect, as shown in the mean raw of Table 4, but the size of the learning
effect was not influenced by the screen design.
Table 4. Subjects' mental model correctness
initial Ist 2nd 3nd
causal 0.58 0.92 1.25. 1.29
hierarchical 0.58 0.54 1,04 1,25
departmental 0.33 0.54 0.92 1.00
mean 0.50 0.67 1.07 1.18
Decision Performance
As presented in Table 5, the causal type outperformed the other two conditions and
hierarchical type outperformed departmental type both on total assets and order growth
rate. However, the results above were not statistically significant.
With respect to the learning effect, decision performance on total assets and order
growth rate were improved from trial to trial as shown in the mean column of Table 5.
The trial effect tested by MANOVA were significant both on total assets and order growth
rate (Wilks' Lambda=0.79, F(2,48)=6.21, p<0.005; Wilks' Lambda=0.73,
F(2,48)=8.71, p<0.002). However, there was no significant i interaction effect between
- 853 -
trial and manipulation. The results demonstrated that there did exist learning effect, as
shown in the mean column of Table 5, but the size of the learning effect was not
influenced by the screen design.
With respect to the correlation between two types of performance measures, the
results of correlation analysis supported the positive relation between understanding and
decision outcomes. The mean correlation coefficient between mental correctness (i.e.,
understanding) and total assets was 0.57 (p<0.001), mental correctness and order growth
rate was 0.54 (p<0.001). The results supported the consistency between two types of
performance measured by cognitive system and decision outcomes.
Table 5. Decision performance
causal hierarchical departmental mean
total assets Tst 2.69E+05 4.04E+04 3.67E+05_2.25E+05
2nd” 3.97E+08 2.08E+06 7.94E+05 1.33E+08
3rd 7.87E+10 151E+07 2.566+06 —-2,62E+10
mean 2.64E+10 5.74E+06 1.24E+06 —8.78E+09
order growth Ist 031 025 02 § 027
rate 2nd 0.43 035 0.34 0.37
3nd 0.49 0.46 0.36< 0.44
mean 0.41 0.35 0.31 0.36
DISCUSSION AND CONCLUSION
In the first experiment, we found that the proposed learning environment was viable for
learning resulting from the positive effects of challenging goal setting ‘and causal
feedback. The challenging goal motivated participants to play the game more times. As
participants played the game more times, under the aid of causal relations feedback, they
could realize the underlying operation rule of the system through the learning from
failure. Consequently, subjects learned the systems thinking skill might discover the high
ticket price strategy to improve performance.
Furthermore, the findings of the present study supported the positive relationship
between understanding (measured by systems thinking in the first experiment and by
mental model correctness in the second experiment) and performance argued by Bakken
(1989). However, the use of high ticket price strategy was the most significant affecting
factor of performance suggested that the positive relationship might be interfered with the
use of price strategy under the condition when the strategy was just acquired by try and
error but not by understanding. That is, understanding may just be a necessary condition
for high performance.
In the second experiment, the effects of causal feedback was examined directly by
the comparison of three different kinds of task's screen design including causal,
hierarchical, and departmental types. We found causal type of screen design induced the
information searching behavior that characterized with wider rarige of information
searching and slower information processing rate. That is, causal type of task's screen
design induced more analytical cognitive type as the prediction of inducement principle.
Referring to the prediction of correspondence-accuracy principle, the causal type indeed
outperformed the other two screen design although was not statistically significant. The
results also arised an interesting question: why the provision of a clear underlying causal
structure of the task could not lead to "significant" improvement of performance?
We think there might be four probable causes of this learning disability. First of
all, subjects ignored the importance of delay. We found that information delay was
- 854 -
evaluated as the least important considerations by subjects in post-experiment
questionnaire. Secondly, subjects failed to recognize the gap between expected goal and
actual state. We found that about 90 percents of the bankruptcies occurred within 10
quarter years resulting from the ignorance of initial conditions. The initial total cash of
the simulated company could not support too much growth. Thirdly, subjects were
subject to using functional relation seeking strategy but not pattern seeking strategy. For
example, they often sought for the functional relation between order growth rate and the
number of hired salesman, yet ignored the other affecting factors, e.g., reputation of the
company. Fourthly, subjects preferred to find a static decision rule such as the optimal
ratio between hiring and investment. However, static rule was hardly ever the optimal
one on dynamic decision tasks. Unfortunately, although the provision of causal relations
let subjects know what was the relations among variables, it was useless for the
avoidance of the four causes of this learning disability. That is, subjects could learn
systems thinking adaptively mainly "by doing" or "by failure", not "by knowing".
In conclusion, we suggest that the design of a leaning environment with the tool of
dynamic decision game should take the motivation factor into account, such as, to set
challenging goals, to lead participants put moré efforts to learn systems thinking by
doing. Although causal relations type of task's screen design can not improve learning
effect significantly, however, it can induce corresponding analytical cognitive type to
increase the cognitive efforts on the inspection of information that is beneficial to the
learning of systems thinking.
REFERENCES
Anderson, D. F., I. K. Chung, G. P. Rechardson, and T. R. Steward. 1990. Issues in
Designing Interactive Games Based on System Dynamics Models. Proceedings of
the 1990 International System Dynamics Conference , Mass, USA: 31-45.
Bakken, B. E. 1989. Learning in Dynamic Simulation Games: Using Performance as a
Measure. Computer-Based Case Studies in Management Education and Research.
‘in Computer-Based Management of Complex Systems, in P. M. Milling and E. O.
K. Zahn Eds., New York: Springer- Verlag: 309-316.
Brunswick, E. 1956. Perception and the Representative Design of Psychological
Experiments. Berkely, California: University of California Press.
Forrester, J. W. 1961. Industrial Dynamics. Cambridge, Mass: MIT Press.
Graham, ‘A. K., P. M. Senge, J. D.Sterman, and J. D. W. Morecroft. 1989. Computer-
Based Case Studies in Management Education and Research. in Computer-Based
Management of Complex Systems, in P. M. Milling and E. O. K. Zahn Eds., New
York: Springer- Verlag: 317-326.
Graham, Alan K. and Peter M. Senge. 1990. Computer Based Case Study and Learning
Laboratory Projects. System Dynamics Review 6: 100-105.
Hammond, K. R. 1988. Judgment and Decision Making in Dynamic Task. in A. P. Sage
Ed., Information and Decision Technologies 14. New York: Elsevier Science
Publishers B. V.
Hammond, K. R., T. R. Steward, B. Brehmer, and D. Steinmann. 1975. Social
Judgment Theory. in M. Kaplan and S. Schwaetz Eds., Human Judgment and
Decision Processes. New York: Academic Press.
Hammond, K. R., D. A. Summers, and D. H. Deane. 1973. Negative Effects of
Outcome-Feedback in Multiple-cue Probability Learning. Organizational Behavior
and Human Performance 9: 30-34.
Latham, G. P. and E. A. Locke, 1991. Self-Regulation Through Goal Setting.
Organizational Behavior and Human Decision Process, 50: 212-247.
Locke, E. A. and G. P. Latham. 1990. A Theory of Goal Setting and Task Performance.
New York: Prentice-Hall.
Senge, P. M. 1990. Catalyzing Systems Thinking in Organizations. Sloan School of
Management, Massachusetts Institute of Technology, Cambridge, MA 02139,
USA, Report No. D-3877-9.
Senge, P. M. 1989. Organizational Learning: New Challenge for System Dynamics.
Sloan School of Management, Massachusetts Institute of Technology, Cambridge,
MA 02139, USA, Report No. D-4023.
Simon, H. A. 1957. Rationality and Decision Making in Models of Man. John Wiley,
New York. e
Sterman, J, D. 1988a. People Express Management Flight Simulator. Sloan School of
Management, Massachusetts Institute of Technology, Cambridge, MA 02139,
USA.
Sterman, J. D. 1988b. Strategy Dynamics: The Rise and Fall of People Express. Sloan
School of Management, Massachusetts Institute of Technology, Cambridge, MA
02139, USA, Report No. D-3967. é
- 856 -
