1994 INTERNATIONAL SYSTEM DYNAMICS CONFERENCE
Overcoming the Learning Barriers of M: t Flight Simulators
Task Sali and the Dissociation between Performance and Learning
Showing H. Young Sy-Feng Wang
D of Business M: D of B
National Sun Yat-Sen University National Sun Yat-Sen University
Kaohsiung, Taiwan Kaohsiung, Taiwan
Jenshou Yang
Dep of Business A
National Yunlin Institute of Technology
Yunlin, Taiwan
Abstract
Recent experimental studies in flight simul showed a dissociation between task
performance and learning: subjects’ performance was Significantly improved through practice, but
very little deeper learning was detected. A th ‘k is developed to explain the
dissociation. That is, the cognitive strategies really used by subjects, e.g., situation matching,
feedback control and feedforward control, are different from the normative cognitive strategy of
mental model simul d by t Methods to overcome the dissociation are
d and d d by two experi studies. Based on the discussions and the
experimental results, we found that the considerations of cognitive strategies and task salience are
very imp for di ing effective learning environment of management flight
simulators.
Microworlds, page 101
1994 INTERNATIONAL SYSTEM DYNAMICS CONFERENCE
Overcoming the Learning Barriers of M t Flight Simul S:
Task Sali and the Dissoci between Performance and Learning
Introduction
The tool of Management Flight Simulator (MFS) has become a significant focus in the
system dynamics field. However, the popularity of these simulators has far outstripped the
research on their effectiveness. A number of MFSs have been developed to propagate the
thought of system dynamics (Sterman and Meadows, 1985), convey the understanding of specific
systems (Meadows, 1989; Graham et al., 1992), cultivate CEO's systems thinking, and further,
aid organizational learning in busi Senge and Sterman, 1992). Yet, there are little scientific
evidences to support the superiority of MFS on the learning of systems. Academic effort on
effi icacy of MFS is now ‘more important than to design new games.
Phi of d iation between per and learning i in MFS's has been found to
be a problem to prove the effecti of MFS. The that practice
proved subjects’ per igni ly but had no effect on the i inquiry of task knowledge
(Berry and Broadbent, 1988; Berry and Diencs, 1991; Sanderson, 1989). Paich and Sterman
(1992) found that subjects’ performance were improved resulting from the familiarity to the task
and the use of specific decision rule found when practicing but not the learning of task
knowledge. Wang and Young (1992) had similar findings that performance was dissociate with
task specific knowledge. These results demonstrate that to possess the ability to control a
management game is not equal to learn the task system. Thus, finding the causes leading to the
ion between per and learning in MFS and their solutions is very important.
This study aimed at the investigation of the dissociation phenomenon, particularly
focusing on the underlying cognitive processes behind the phenomenon, and on the methods to
overcome the dissociation.
The findings by Berry and Broadbent
The findings by Berry and Broadbent are ive to the und ding of
dissociation phenomenon in MFS. A series of studies by nee and Broadbent (e.g., Berry, 1991;
Berry and Broadbent, 1984, 1987, 1988) have demonstrated a dissociation between task
performance and associate verbalizable knowledge. They showed that practice significantly
improved ability to control the task, but had no effect on ability to answer post-task written
questions. In contrast, verbal instruction on how to reach and maintain the target value
significantly improved ability to answer questions but had no effect on control performance.
Moreover, there was an overall significant negative correlation between task performance and
question answering. The findings were similar to those found by system dynamicists, except for
the tasks used by system d: were more
Two possible cognitive processes were adopted to explain the dissociations (for more
detail, see Sanderson, 1989). The first lies in the distinction between explicit and implicit modes
of learning. That conscious self-report task specific knowledge is not available, because some
information p ing is done iously. This is related to the long-standing idea that
cognitive activity takes place in parallel at multiple levels. Another explanation lies in the idea
of production-system that verbal knowledge might decay in the process of cognitive skill
acquisition (Neves and Anderson, 1981). As learning progresses, simple productions are replaced
by more complex, incl productions through the k process. However,
the simple productions can support verbalizable knowledge about performance, but the more
complex one can't, because the latter compresses a large number of initiating conditions and
resulting actions. This explanation is similar to the idea that human cognitive capacity is limited,
thus only the most salient information will be processed and reported.
Different cognitive strategies may be a cause to lead to the foregoing two cognitive
processes. Broadbent, et al. (1986) proposed two kinds of cognitive strategies, namely, model
manipulation and situation matching. When using model manipulation, subjects have known
relations among variables so that they can forecast the performance of alternatives and choose
the best one. when situation matching is used, subjects remember the relations among the
Microworlds, page 102
1994 INTERNATIONAL SYSTEM DYNAMICS CONFERENCE
situation, decision, and performance to make the best decision. Model manipulation strategy is
based on task knowledge, then explicit learning occurred. Subjects can modify their task
knowledge through the comparison between f and While the und ding
about task systems is not necessary for situation matching, explicit learning does not occur. The
situation matching process may be done iously, so that i If-report of task
specific | ledge is not available or the p of situation matching become too complex
to support verbalizable knowledge.
In short, the relations between performance and task knowledge depend on the cognitive
strategy used. The is prehensive to the dissociations in MFS and will be discussed
later.
Rather than simply demonstrating dissociations, an alternative approach has been to look
at conditions that give rise to either implicit or explicit learning. Berry and Broadbent (1988)
propounded that "salience" of task could affect the used cognitive strategy. They found low
salience led to implicit learning, and the relation between performance and task knowledge is
vague or even negative, vice versa. Task salience, defined by Berry and Broadbent, is the
probability that, if a person learns by the explicit rather than the implicit mode, the key variable
will be chosen. There are three ways to increase level of task salience as follow:
(1) To reduce irrelevant factors in situation (Broadbent, et al., 1986); For example, to
reduce the number of relations of variables to be processed in a decision.
(2) To make the key events act in accordance with general knowledge from outside the
task; For example, to remove the delay between actions and outcomes (Berry, 1991;
Berry and Broadbent, 1988), or to add a positive feedback loop to increase the impact
of actions on outcomes (Broadbent, et al., 1986)
(3) To give an explicit verbal direction as to which are the key variables; For example, to
instruct subjects what kind of variables are relevant (Berry and Broadbent, 1988)
A dingly, for system dy ici it is possible to lead subjects to use the expected
Cognitive strategy through the manipulation of task property in order to overcome the
dissociations in MFS. Nevertheless, the manipulation of task salience should be modified, because
task properties in MFS are different from those in the research by Berry and Broadbent.
Cognitive strategies in dynamic complexity task
The difference between tasks used by Berry and Broadbent and MFS research lies in task
property and learning objectives. The typical tasks used by Berry and Broadbent are combined by
a set of linear equations. The task knowledge to be learned is the relations of polarity and
quantity between decision variables and objective variables. Tasks of MFS are characterized with
nonlinearity, delay, and multiple causal feedback loops, and are more complex than the former.
Furthermore, the polarity and quantity relations between variables are generally provided in MFS.
A holistic understanding about system structure is the objective in MFS.
For the situation matching strategy, since the interdependence and the shift of dominance
loops in MFSs' dynamic complexity task, using the situation matching strategy in MFSs task is
not so effective than used in Berry and Broadbent's task. However, we still find that the
existence of the situation matching strategy in our recent experiment.
Feedback and feedforward control were found to be used often in MFS tasks (e.g., Paich and
Sterman, 1992; Wang and Young, 1992). For the feedback control strategy, system structure is
treated as a black box when subjects use feedback control. While using feedback control, no more
than the knowledge of polarity relations between decision and objective variables is needed to
approach the goal. The pattern of decision behavior in the use of feedback control is similar to a
goal directed negative feedback loop. The efficacy of feedback control strategy depends on
whether the decision negative feedback loop can dominate the system.
Feedforward control is similar to feedback control that system structure is treated as a
black box. To use feedforward control, forecasting based on historical data, theory, or expert's
experience is the base to make decision rather than on outcomes in feedback control. For
example, the pattern of production life cycle was used by subjects in Paich and Sterman's study
(1992); Books' law was used by subjects in the study of Abdel-Hamid (1993); forecasting by
Microworlds, page 103
1994 INTERNATIONAL SYSTEM DYNAMICS CONFERENCE
experience in Wang and Young's study (1992). These ways of control need lower level of
cognitive effort comparing to mental simulation where understanding about system structure is
necessary (Brehmer, 1990).
Although these three cogniti ies (situati hi dback, and feedforward
control) are preferred by subjects and ad for the imp! of perfec they
are not helpful for learning in MFS.
In contrast, the expected cognitive strategy by system dynamicist is mental model
i for its th ef for learning. When using mental model simulation,
subjects produce a mental model to represent the task system based on their information and
knowledge. Subjects formulate decision policy from the model and test it on the MFS, and they
can modify their mental model based on decision outcomes (Isaace and Senge, 1992). Then, the
learning about the task system occurs.
To use model manipulation strategy is difficult in MFS. The distinction between model
manipulation and mental model lation lies in the ion of task where the former
task with th I type, the latter with a way which is comparable with rule of
human thinking. Subjects can simulate policies for a long-term period with mental simulation but
just one period decisions with model manipulation in MFS because of the complexity of task. In
fact, subjects could hardly use model manipulation i in MFS because subjects can not compute the
high order and nonlinear equations in MFS. Therefore, model manipulation is ignored in the
following discussion.
The di i d that dissociati in MFS resulted from the
cognitive strategies ‘chosen by subjects are not the expected ones by system dynamicists. There
are two reasons for subjects tend not to use mental model simulation. First, it needs more
cognitive inputs to use mental model simulation than situation matching, feedback and
feedforward control. Second, human beings have poor ability to represent dynamic feedback
systems (Brehmer and Dorner, 1993; Forrester, 1975; Senge, 1990; Sterman, 1989a, 1989b).
Therefore, how to evoke subjects to use the expected cognitive strategy is the proposition to
improve the effectiveness of MFS.
Task Sali for dy i lexity task
The analysis of task-induced cognitive strategy is helpful for the prediction of what kinds
of design of MFS is advantageous to learning rather than performance only. The effect of
Sengupta and Abdel-Hamid's (1993) design was ambiguous from the point of induced cognitive
strategy, though they claimed cognitive feedback provided in their study has induced mental
model simulation strategy. It is possible that subjects use feedback control strategy to approach
decision goal based on the provided "indicated workforce level" which was an indicator of
experts’ knowledge. For induced cognitive strategy was not measured in Sengupta and Abdel-
Hamid's (1993) study, it is hard to make conclusion.
The manipulation of task salience to induce mental model simulation strategy in MFS is
possible. First, to provide subjects reference mode of the task system can increase task salience,
because key variables and their pattern of behavior are given. Second, to provide causal loop
diagram can eliminate redundant information and hint subjects the polarity relations and delay
between those key variables. Furthermore, causal loop diagram can instruct subjects how to
represent a complex dynamic system, and decrease the barrier of using mental model simulation.
Third, partial model test proposed by Morecroft (1985) divides a whole complex system into
several controllable parts, and then increase the salience of task. This design is similar to that in
the study by Broadbent, et al. (1986) where subjects were instructed to test the relations.between
variables one a time.
Examples to manipulate task salience
Two examples of manipulating task salience are given as follow. They are all tested by
experimental methods. The details of the experimental results can be found in Young, Wang and
Yang (1992).
Microworlds, page 104
1994 INTERNATIONAL SYSTEM DYNAMICS CONFERENCE
Example 1: assignment as manipulation
The first ple used assi: to ipulate task salience in the task of People
Express Management Flight Simulator (Sterman, 1988). Task salience was manipulated by the
aids of reference mode, partial model test , and causal feedback diagram. For the aid of
reference mode, it was manipulated in the following question, wherein three reference modes
(customer growth rate, turnover rate and service quality) were announced.
Q1...Using the following policies until quarter 4 of 1984: (a) Fare=0.09, (b) Target service
scope=0.6, and (c) Aircraft annual growth rate = 100%. Your aircraft must be more than
72. Please answer the following questions:
(1) Why the growth rate d d after d ic growth?
(2) Why more hiring induced higher turnover rate?.
(3) Why did high service quality gradually decline?
For the aid of partial model test, it was manipulated in the second question of the
assignment, wherein subjects were asked to solve three problems. Moreover, subjects were asked
to treat the problems one by one, that is, one problem a time .
Q2 ...Using the policies in Q1 until quarter 4 of 1984, then solve the problems of
declining customer growth rate, high turnover rate, and declining service quality. Please
treat the problems one by one, that is, one problem a time . Records every policy and
associated outcomes which you have tried, then explain "why".
Finally, for the aid of causal feedback diagram, subjects acquired the causal feedback
diagram of PE constructed by Sterman and Kim (1988) without verbal description.
The experimental results show that, the treatment of task salience not only have positive
effect for decision performance, but also for the learning of the underlying structure (for more
details, see Young, et al., 1992).
Example 2: screen design
The second example di d methods to ipulate task salience in MFS's computer
screen. There were three kinds of simulator's screen design includi I-loop, hi hical.
and departmental in the experiment (Young, et al., 1992). As shown in Figure 1, the causal-loop
type of screen was designed like the causal loop diagram used by system dynamicists. The
hi ical type of simulator’s screen was designed like a hi i I-tree diagram, as
shown in Figure 2. Both the screen d ( I-loop and hi hical) have offered the on-
line causal relationships among variables. However, the departmental type of screen, as shown in
Fgure 3, only represented variables without relationships.
The experimental results show that subjects provided with the causal-loop type screen have
best performance and best learning effects, followed by those provided with the hierarchical type.
Subjects provided with departmental type have worst performance and worst learning effects,
although the effects were not statistically significant. Moreover, in the analysis of subjects’
cognitive strategies. it is found that the causal-loop type screen induced a more analytical
cognitive type compared with the departmental design. However, the hierarchical screen design
induced more intuitive cognitive type than the departmental condition (for more details, see
Young, et al., 1992).
Microworlds, page 105
1994 INTERNATIONAL SYSTEM DYNAMICS CONFERENCE
Eee ee]
total asset
Figure
ee
[20s
—-
=_
delivery f
2 therarchical type screen design
Finance
revenue 2
cost
IMVESt cost
Wage cost
prone
interest ?
cash
capatals’ cash value
total aysets ou
Mar
MARR
20 ae ba
20
\ orca
sales 20
aces diay 0 Production
ser qualtty i
2
| Human Resource Shipping capatal oO
capatal 10
hue ftre ’ capacity 20
salesman 10 depreciation 0.25
Microworlds,
Figure
pave 100
3: Departmental type screen design
1994 INTERNATIONAL SYSTEM DYNAMICS CONFERENCE
Summary
The design and evaluation of the effects of flight simul: is an imp
and challenging topic for the field of system dynamics. Recent experimental studies in
flight simul: showed a dissociation between task performance and learning:
subjects’ perfc was signi: ly improved through practice, but very little deeper learning
was detected. A th ical fr k is developed to explain the dissociation. Methods to
the dissociation are d and d d by two experi I studies. Based on
the discussion and the experimental results, we found that the considerations of cognitive
strategies. and task salience are very important di i for designi ffective learning
nvi of flight simul:
Reference
Berry, D. C. 1991. The Role of Action in Implicit Learning. Quarterly Journal of Experimental
Psychology, 43A(4), pp.881-906.
Berry, D. C. and D. E. Broadbent. 1984. On the Relationship between Task Performance and
A iated Verbali K ledge. Quarterly Journal of Experimental Psychology, 36A,
pp.209-231.
Berry, D. C. and D. E. Broadbent. 1987. The Combination of Implicit and Explicit Learning
Processes, Psychological Research, 49, pp.7-15.
Berry, D. C. and D. E. Broadbent. 1988. Interactive Tasks and the Implicit-Explicit Distinction.
British Journal of Psychology, 79, pp.251-272.
Berry, D. C. and Z. Dienes. 1991. The Relationship between Implicit Memory and Implicit
Learning, British Journal of Psychology, 82, pp.359-373.
Brehmer, B. 1990. Strategies in Real Time Dynamic Decision Making. in R. M. Hogarth (ed.)
Insights in Decision Making, The University of Chicago Press, pp.262-279.
Brehmer, B.and D. Doerner. 1993. Experiments with Computer-Simulated Mi Ids: Escaping
both the Narrow Straits of the Laboratory and the Deep Blue Sea of the Field Study.
Computers in Human Behavior, 9, pp.171-184.
Broadbent, D., P. FitzGerald, and M. Broadbent. 1986. Implicit and Explicit Knowledge in the
Control of Complex Systems, British Journal of Psychology, 77, pp.33-50.
Forrester, J. W. 1975. Counterintuitive Behavior of Social Systems. in Collected Papers of Jay W.
Forrester, Cambridge, MA: Productivity Press, pp.211-244.
Graham, A. K., J. D. W. Morecroft, P. M. Senge, and J. D. Sterman. 1992. Model-Supported Case
Studies for Management Education. European Journal of Operational Research, 59 (1):
151-166.
Isaacs, W. and P. M. Senge. 1992. Overcoming Limits to Learning in Computer-Based Learning
Environment. European Journal of Operations Research, 59(1)- 183-196.
Meadows, D. L. 1989. Gaming to Implement System Dynamics Models. in P. M. Milling and E.
O. K. Zahn (eds.), Computer Based Management of Complex Systems, Berlin: Springer
Verlag, p.p.635-640.
Morecroft, J. D. W. 1985. Rationality in the Analysis of Behavioral Simulation Model.
Management Science, 31: 900-916.
Neves, D. M. and J. R. Ander: 1981. Knowledge C ilation: Mechani for the
Automatization of Cognitive Skill," in J. R. Anderson (ed.), Cognitive Skills and Their
Acqusition, NJ: Erlbaum, pp.57-84,
Paich, M. and J. D. Sterman. 1992. Boom, Bust, and Failures to Learn in Experimental Market.
MIT Sloan School of Management Working Paper 3441-92-BPS.
Sanderson, P. M. 1989. Verbalizable Knowledge and Skilled Task Performance; Association,
Dissociation, and Mental Model, Journal of Experimental Psychology: Learning Memory
and Cognition, 15, pp.729-747.
Senge, P. M. 1990. The Fifth Discipline: The Art and Practice of the Learning Organization. New
York: Doubleday.
Senge, P. M. and J. D. Sterman. 1992. Systems Thinking and Organizational Learning: Acting
Locally and Thinking Globally in the Organization of the Future. European Journal of
Operations Research, 59 (1): 137-150.
Microworlds, page 107
1994 INTERNATIONAL SYSTEM DYNAMICS CONFERENCE
Sengupta, K. and T. K. Abdel-Hamid. 1993. A ive Conceptions of Feedback in Dynamic
Decision Environments: An Experimental Investigation. Management Science, 39(4),
pp.411-428.
Sterman, J. D. 1988. People Express Management Flight Simulator. Sloan School of
Management, Massachusetts Institute of Technology, Cambridge, MA 02139, USA.
Sterman, J. D. 1989a. Modeling managerial behavior: Misperceptions of feedback in a dynamic
decision making experiment. Management Science, 35 (3): 321-339.
Sterman, J. D. 1989b. Misperception of feedback in dynamic decision making. Organizational
Behavior and Human Decision Process, 43: 301-335.
Sterman, J: D. and D. L. Meadows. 1985. STRATAGM-2: A Microcomputer Simulation Game of
the Kondrative Cycle, Simulation and Games, 16(2), pp.174-202.
Sterman, J. D. and D. H. Kim. 1988. Strategy Dynamics: The Rise and Fall of People Express.
Report No. D-3967, Sloan School of M husetts Insti of
Technology, Cambridge, MA 02139, USA.
Wang, S. and S. H. Young. 1992. A Preliminary Experiment on Examining Thinking in a Meta-
Dynamic Decision Making i In The P dings of the 1992 International
System Dynamics Conference, Utrecht, Netherlands, pp.57-766.
Young, S.H., J. Yang and S. Wang. 1992. Enhancing the Learning Effects of Dynamic Decision
Game on Systems Thinking: An Experimental Study," The Proceedings of the 1992
Int ional System Dy ics Conf , Utrecht, Netherlands, pp.847-856.
Microworlds, page 108
