Improving Dynamic Decision Making Through Debriefing: An
Empirical Study
HASSAN QUDRAT-ULLAH
York University
School of Administrative Studies
4700 Keele St. Toronto, ON, M3J 1P3, Canada
Phone: 1-416-5363957
Fax: 1-416-7365963
Email: hassanq@yorku.ca
Abstract
Empirical evidence suggests that people perform poorly in dynamic tasks. The thesis of
this article is that dynamic decision performance can be improved by helping people to
develop more accurate mental models of the task stems through training with debriefing
supported computer simulation-based interactive learning environments (CSBILEs). I
report a laboratory experiment in which subjects managed a dynamic task by playing the
role of fishing fleet managers. One group of participants used a CSBILE with debriefing,
whereas another group used the same CSBILE but without debriefing. A comprehensive
model consisting of four evaluation criteria is developed and used: task performance,
structural knowledge, heuristics, and cognitive effort. It is found that debriefing was
effective on all four criteria; debriefing improves task performance, helps the user learn
more about the decision domain, develop heuristics, and expend less cognitive effort in
dynamic decision making.
Key Words: Computer simulation, Interactive learning environments; Dynamic decision
making; Debriefing; Feedback; Renewable resource management
INTRODUCTION
Overexploitation and mismanagement of renewable resources abounds.
Examples, where such resources have been more or less mismanaged, include fish and
whales, forests, soil, ground water reservoirs, as some reindeer pastures (Moxnes 2000).
There are two main theoretical themes for why overexploitation of renewable resources
happens. One prominent theory is known as the “tragedy of commons” (Hardin 1968).
On this reading, open access to a resource whereby users of the resource compete to
maximize their individual benefit (the commons problem) causes overexploitation of
common renewable resources. The other explanation of why renewable resources are
poorly mismanaged, drawing on the emerging literature on dynamic decision-making, is
the misperception of feedback (MOF) hypothesis (Sterman 1989a) or misperception of
the bioeconomics of renewable resources (Moxnes 1998, 2000). From this perspective,
people’s poor performance in renewable resource management tasks can be attributed to
a general tendency for people systematically to misperceive the dynamics of bioeconomic
systems. Even when the commons problem is eliminated, overexploitation occurs. On
this reading, interpretation of the MOF hypothesis is that people mismanage the
renewable resources because their knowledge of renewable resource system structure is
less than perfect. Appropriate management could perhaps be achieved through training by
helping people to develop more accurate mental models of renewable resource systems.
Computer technology together with the advent of new simulation tools and
methods provides a potential solution to the need for the training tool for complex
dynamic systems such as renewable resource systems. For instance, computer simulation-
based interactive learning environments (CSBILEs) are often used as decision support
systems to improve decision making in dynamic tasks. CSBILEs allow the compression
of time and space and provide an opportunity for managerial decision making in a non-
threatening way. Despite an increasing interest in CSBILEs, recent research on their
efficacy is inconclusive. The increasing urge to improve the efficacy of CSBILEs in
promoting decision making in dynamic tasks has led the researchers to suggest
improvements. One such way to improve the efficacy of a CSBILE is to incorporate
debriefing. Debriefing is the processing of simulation-based learning experience from
which the decision-makers are to draw the lessons to be learned (Ledrman 1992;
Stienwachs 1992). With the help of the facilitator, debriefing activity allows the decision
makers to reflect on their experiences and overcome the misconceptions they had with
the dynamic task. Thus, debriefing influences decision makers’ learning in such a way
that it may enhance their task performance and learning in dynamic tasks.
Although the literature on dynamic decision-making (DDM) and learning in
CSBILEs (e.g., Davidsen and Spector 1997; Issacs and Senge 1994) has embraced the
concept of debriefing its effectiveness has rarely been evaluated empirically. Does
debriefing improve performance in dynamic tasks? Does debriefing influence subjects’
structural knowledge? Does debriefing help users to improve their heuristics knowledge?
How does debriefing affect decision makers’ cognitive effort? This study attempts to find
answers to these questions. The findings should be valuable to CSBILE designers, users,
and researchers. If supported, CSBILE designers will be able to enhance the efficacy of
CSBILEs by explicitly building in debriefing. CSBILEs users will get more effective
support in understanding the dynamic tasks such as renewable resource management.
CSBILEs researchers will be able to use this research as a base to examine other
important issues, such as the influence of debriefing on group decision making, the value
proposition of debriefing, and the impact of debriefing on time-critical tasks.
DEBRIEFING IN CSBILEs AND DYNAMIC DECISION MAKING
How well people perform in dynamic tasks? The empirical evidence on DDM
suggests it is “very poorly” (Brehmer 1990; Diehl and Sterman 1995; Moxnes 1998;
Sterman 1989b). In dynamic tasks, where a number of decisions are required rather than a
single decision, decisions are interdependent, and the environment changes (Edwards
1962), most often the poor performance is attributed to subjects’ misperceptions of
feedback. That is, people perform poorly because they ignore time delays and are
insensitive to the feedback structure of the task system (Diehl and Sterman 1995).
Debriefing may help reduce the misperception of feedback.
In dynamic tasks, outcome feedback alone is not effective (Paich and Sterman
1993). Decision makers have to be able not only to examine the results of their decisions
but also the causes of these results. Debriefing provides an opportunity to reflect on their
experiences with task the (Cox 1992; Davidsen and Spector 1997; Spector 2000). Such
discussions, necessarily held after accomplishing the task, purport to foster learning by
making the decision makers aware of their entrenched beliefs and provide them with
information which facilitates their eventual re-interpretation (Issacs and Senge 1994).
There are many experiences that the decision-makers can have with a decision-
making environment. They initially have no way of knowing which are important and
useful in the real world (Elsom-Cook 1993). Debriefing may provide this knowledge and
hence aid learning.
The behavior of dynamic systems is most often counter-intuitive (Forrester 1961).
Decision makers have their own views about what types of decision strategies are needed
to achieve desirable behavior in the system as a whole. But, when they play out their
assumptions and strategies, often the outcome is not what they perceive, even though
they thought they understood the interactions in the task system (Sterman 1994).
Debriefing may effectively fix the inadequacies of their mental models — preconceived
conceptions about the task system. The decision-makers can discover gaps between their
expectations of system behavior based on their understanding of system interactions and
the behavior that actually arises (Issacs and Senge 1994). This discovery may help the
decision makers to have an “adequate model” of the system and hence improve learning.
These previous studies provide an insight into the effectiveness of debriefing. However,
several gaps in the understanding of the effects of debriefing still remain:
e We still do not know the incremental effects of debriefing. Only one study,
(Davidsen and Spector 1997), analyzed the combined effects of CSBILE and of
debriefing rather than the effects of debriefing alone.
e In the context of CSBILE, the definitions and measures of effectiveness of
debriefing lack a comprehensive framework.
This study bridges some of these gaps and advances of previous research by
proposing and using a comprehensive research model, discussed in the next section,
to evaluate the effectiveness of the collaborative discussion type of debriefing in
CSBILEs on multidimensional performance outcomes.
RESSEARCH MODEL AND HYPOTHESES
We use “CSBILE” as a term sufficiently general to include microworlds,
management flight simulators, learning laboratories and any other computer simulation-
based environment — the domain of these terms is all forms of action whose general goal
is the facilitation of decision making and learning. Based on this conception of CSBILE,
we propose a comprehensive evaluation model (Figure 1) that evaluates the influence of
debriefing on the decision, the decision maker, and the decision-making process.
Figure 1: The Model: The Effects of Debriefing on Decision Making
CSBILE H1
Decision Task Performance
& D H2
<
2 Décision Structural Knowledge
5 Maker
a 3
3 a + Heuristics Knowledge
Decision
Ld Making Cognitive Effort
H Process
« H4
Measurement Variables
In our model, we view learning as a progression towards expertise (Davidsen and
Spector 1997; Issacs and Senge 1994; Sternberg 1995). To better capture the range of
expertise development, we propose that CSBILEs effectiveness should be evaluated on
four criteria: (1) task performance, (2) structural knowledge, (3), heuristics knowledge,
and (4) cognitive effort.
Task Performance
Research on DDM and learning in CSBILEs has focused on improving
managerial performance (Bakken 1993; Dérner 1980; Langley and Morecroft 1995;
Sengupta and Abdel-Hamid 1993). CSBILEs provide a practice-field for managerial
decision making. After having experiences with the simulation environment, debriefing
may help the decision makers in at least two ways: (i) expanding the limits of their
bounded rationality, and (ii) reducing their misconceptions about the task system. When
CSBILEs are used as tools to expand the cognitive capacity, a decision maker is able to
accomplish the task more thoroughly by processing more information and evaluating
more alternatives. As a result, the decision maker makes a better decision.
In addition to the expansion of cognitive capacity, debriefing activity allows the
decision makers to discover the gaps between their expectations of system behavior and
the actual behavior. This clarity about the misconceptions about the task system leads to
better understanding of task system. Improved understanding helps the decision makers
to analyze similar task problems in future. Therefore, users of debriefing should perform
better in dynamic tasks.
H1; Users of CSBILE with debriefing show relative improvements in task
performance more than users of CSBILE without debriefing.
Structural Knowledge
Structural knowledge pertains to knowledge about principles, concepts, and facts
about the underlying model of the decision task. Subjects improved both their structural
knowledge and their task performance when the task was changed to one with relatively
more salient relations among the task system variables (Berry 1991). Provision of
structural information about the underlying task system results in improved task
knowledge and task performance (Grobler, Maier, and Milling 2000).
Debriefing provides an opportunity to the facilitator to elaborate on the relations
between key variables of the task system. As a result, decision makers may develop better
understanding about the facts and principles (e.g., which variables are related to goal-
attaining behavior of the task system) about the underlying task system. Consequently,
the users of debriefing should acquire better structural knowledge.
H2: Users of CSBILE with debriefing show relative improvement in structural
knowledge more than users of CSBILE without debriefing.
Heuristics Knowledge
Heuristics knowledge concerns how decision makers actually control or manage
the task. Dynamic decision makers may well know the strategies to achieve better task
performance (heuristics knowledge) even though they can not show improvement on
declarative knowledge (Hsiao 2000). In a debriefing session instructing subjects with task
properties, strategies and decision rules and their lagged effects may enhance their
capability for dynamic decision-making. Thus, the users of CSBILEs with debriefing
should acquire better heuristics knowledge.
H3: Users of CSBILEs with debriefing show relative improvement in heuristics
knowledge more than users of CSBILE without debriefing.
Cognitive Effort
CSBILEs, like any support tool, affect the decision-making process by making it
more (or less) efficient and therefore, enable decision makers to make decisions faster (or
slower). Using decision time as a surrogate provides an estimate of the cognitive effort
employed by a decision maker in performing a task (cf. Jarvenpaa 1989). Provision of
cognitive feedback actually lengthens the decision-making process (Sengupat and Abdel-
Hamid 1993). Debriefing can reduce the decision-making time by helping users make
judgmental inputs and avoid spending extra time for searching for which variables of the
task systems give rise to specific system behavior. In the absence of debriefing users have
to turn solely to their cognitive resources to infer the relationships between key task
variables, leading to more time to digest the information and hence to make a decision.
Therefore, users of CSBILEs with debriefing should make decisions faster.
H4: Users of CSBILEs with debriefing make decisions relatively faster than users of
CSBILE without debriefing.
METHODOLOGY
Research Design
To test the hypotheses we designed a single-factor, completely randomized design
involving one control group and one experimental group. Each participant in the control
group used a CSBILE without debriefing and each participant in the experimental group
used a CSBILE with debriefing.
Experimental Task
The research questions were explored in the context of a CSBILE, FishBankILE
(Qudrat-Ullah et al., 1997) in which subjects played the role of fishing fleet managers
making fleet capacity acquisition and utilization decisions over the life of fishing area
contract: 30 years. The task simulation is based on system dynamics model of Dennis
Meadow’s board game built around the “tragedy of the commons” phenomenon.
This task embodies the essential characteristics of a dynamic task — a series of
multiple, interrelated decisions made over a period of time and in a dynamic environment
(Edwards 1962). Each year, as fishing fleet managers, subjects were required to order
new ships and decide on the percentage utilization of the fleet. Figure 2 exhibits the
“Decision Panel” of FishBankILE. In the task, there are delays between ‘action’ and
their ‘consequences’, e.g., it takes three years for new ship to be built and added to the
fleet. Uncertainty surrounds the ‘carrying capacity of the fishery area’ and the ‘fish re-
generation threshold’. Subjects had to ‘infer’ the nature of relationships between task
variables, e.g., between ‘fish harvested’ and ‘fish re-generation threshold’. Performance
was measured by cumulative profits made by each firm over a period of 30 years plus the
remaining resource (fish) value in the final year.
Participants
The experiment was conducted with senior undergraduate students, 47% males
and 53% females, at the National University of Singapore. They were on average 23
years old. There were eighty-seven student subjects who signed up for the FishBankILE
task experiment. Seventy-eight of them showed up, signed the consent form, and
completed the whole experiment. Participation in the experiment was entirely voluntary.
US $25, 15, 10 respectively was paid to the top performers in each group.
Procedure
Two separate electronic classrooms in the faculty of business administration at the
National University of Singapore, where forty IBM PC compatibles have been set up,
were reserved for the experiment, one for the control group and other the for
experimental group. The task program, FishBankILE, was installed and pilot tested one
academic term prior to the main study. In addition to the task program, all subjects were
supplied with a folder containing the consent form, instructions to lead them through a
session, training materials (which were also available on the computer screen), as well as
notepads and pens (as they were encouraged to take notes along the experiment). The
task training materials were also emailed to all the subjects two days prior to their
experiment session and they were asked to go through it thoroughly. The experiment
started with each participant returning the signed consent form and taking a pre-test on
FishBankILE task knowledge. Both the groups received the same general instruction.
Two goals were made clear to the participants: to obtain a good result when controlling
the simulation task (i.e., maximize the total profit) and to learn as much as possible about
system structure and behavior.
co Cy eo.
EA Decision Panel
Click Here for
the Number of
INew Ships You
Want to Order
Click Here to
Start the Game
0 20 40 60 80 100
Click Here and Drag
for the Percentage of
Fleet You Want to
Send to the Sea
Click Here to
Restart the Game
eports
[mmr Financial | —
[ Click Here For Summary ]
[ Click Here for Financial Report
Figure 2: The Decision Panel of FishBankILE
All the subjects completed a training trial by going through the computer screens
of FishBankILE, making decisions in each period, accessing and observing the feedback
of their decisions via graphs and tables. Then, all the subjects completed two formal trials
interceded by either a small break of about 10 minutes for the control group (the
experimenter prepared the computers for the next trial and the subjects were told to relax
without any communication with the fellow subject(s)) or a debriefing activity of about
40 minutes for the experimental group.
In the debriefing activity, not only ‘what happened’ but also ‘what did not
happen’ was discussed. Subjects’ performance charts were shown and discussed to relate
the structure of the system to its behavior. Structure-behavior graphs (Davidsen 1996) of
all the main feedback loops of FishBankILE system were explicitly elaborated on.
Subjects were free to ask questions and were answered accordingly.
Each participant took a post-test, which was identical to the pre-test, following the
completion of the 2™ formal trial. It was conducted to measure any possible structural
and heuristics knowledge gain or loss during the experiment. The experiment took about
80 minutes for the experimental group and about 60 minutes for the control group.
Measurement
Dependent Variables: For H1, user task performance is the dependent variable.
The task performance metric for each subject was chosen so as to assess how well each
subject did relative to a benchmark rule. The task performance measure for subject s, TP;
has the following formulation:
ny * ip
where n, is the number of task performance variables, mr is the number of trials the task
has to be managed, bj; is the benchmark’ value of task performance variable i at time ¢,
and yi is the empirical value of task performance variable i at time t. Every decision
period, the benchmark’s performance variables’ values are subtracted from those of the
subjects. The subject’s final performance, 7P, is the accumulation over 30 periods of this
difference averaged over the number of task performance variables and number of trials.
For H2, user structural knowledge about the task system is the dependent variable.
A post-test measured structural knowledge reflected by fourteen closed-ended questions
on the relationships between pairs of the task variables and six questions on the algebraic
relationships of the variables. The answer to each question was evaluated as right (one
point) or wrong (zero points). The difference between pre-test and post-test scores was
used in the analysis.
For H3, user heuristics knowledge about the task system is the dependent
variable. On the post-test, two open-ended questions asked the subjects about their
strategy for ordering new ships and ships utilization. Two independent domain experts
scored the answers. The average score measured heuristic knowledge in this study.
* Details about the benchmark rule are available to interested readers from author upon request.
For H4, cognitive effort in decision-making is the dependent variable. Using
decision time as a surrogate provides an estimate of the cognitive effort employed by a
subject in performing a task (Sengupta and Abdel-Hamid 1993). Decision time was
measured as the time spent by a subject making decisions in each of the decision periods.
The CSBILE we used in the experiment was designed to record the decision time for all
thirty periods period in a complete trial of the decision task.
Independent Variable: For all hypotheses (Hl, H2, H3, and H4), the
independent variable is the availability of debriefing in a CSBILE. Two self-reported
scales (Usefulness of debriefing in arousing Interest in the Task (UIT) and Usefulness of
debriefing in Understanding the Task (UTT)) were used to assess the subject’s reaction to
the debriefing.
RESULTS
Table 1 summarizes demographic characteristics of the participants. The results of
various ANOVA and chi-square test suggest that the group distributions were statistically
identical for age, gender, academic background, structural knowledge, and heuristics
knowledge. Therefore, it is safe to assume that even if these demographic factors were to
have any effect on the subject performance, the effect was same for both the groups and
did not materially affect the results of this study.
Table 1: Participant Demographics
Control Group Experimental Test Critical p-
(without Group statistic value —_- value
Debriefing) (with Debriefing)
Participants 39 39
Age 23.1 23.4 F=.26 3.97 0.611
Gender
Male 19 18 y2=.97 28.87 1
Female 20 21 7° =3.66 31.41 1
Academic
Background?
FAE 2.2 2.2 F=.03 3.97 0.864
CMS 6 5 F=.07 3.97 0.798
BL 6 6 F=.05 3.97 0.818
Structural 10.15 10.05 F=.11 3.97 0.739
Knowledge”
Heuristics 3.92 4.13 F=.93 3.97 .338
Knowledge”
*. FAE: Finance, Accounting, & Economics group (score is 1 if course is taken & 0 if no course
is taken in the area; max is 3 and min is 0); CMS: Computer Science, Mathematical Modeling,
and System Dynamics group (max is 3 and min is 0); BL: Business Strategy and Logistics
group (max is 2 and min is 0).
average score on pre-test with a max score of 20 and min score of 0.
Table 2 summarizes the results of four ANOVA tests for task performance for H1,
structural knowledge for H2, heuristics knowledge for H3, and cognitive effort for H4.
Since the calculated F-values are higher than the critical F-values, all four hypotheses are
supported. Subjects who received debriefing outperformed those without it.
For H2, there is overwhelming support. Subjects in the debriefing group scored
significantly higher on post-test on structural knowledge. H3 is also supported very
strongly. Subjects in the debriefing group developed better heuristics knowledge than
subjects without debriefing aid. Finally, H4 is immensely supported. Subjects with
debriefing support made decisions very efficiently. They spent less than half the time
subjects without debriefing spent.
Table 2: Effects of Debriefing
Hypotheses Group Performance F-Value Critical F- p-value
Value
H1 Without TP? = -.63 353.09 F95,1,76= 0.000
Debriefing TP =-.45 3.97
With Debriefing
H2 Without SK’ = 12 141.53 F.95,1,76=- 0.000
Debriefing SK = 16.32 3.97
With Debriefing
H3 Without HK® = 3.85 726.37 F95,1,76= 0.000
Debriefing HK = 13.08 3.97
With Debriefing
H4 Without DT°= 64 852.81 F95,1,76= 0.000
Debriefing DT = 28.03 3.97
With Debriefing
*: TP: Task Performance is the average performance (relative to the benchmark) in Trial 2. A
score of 0 means that subjects performance is at par with the benchmark rule. A score of > 0
would mean better performance and a score of < 0 means subject performed poorly as
compared with the performance of the benchmark rule.
: SK: Structural Knowledge is score on post-test, a max score of 20 and min score of 0.
°: HK: Heuristics Knowledge is also a score on post-test, a max score of 20 and min of 0.
*: DT: Decision Time is the average time in seconds, per decision period, in Trail 2.
One could argue that improvement in performance in Trial 2 might have come as
a result of practice of Trial 1. We analyzed the effects of practice separately utilizing the
data obtained from control group. Table 3 summarizes the results of the analyses. We
find significant effects of practice on structural knowledge and decision time of the
subjects in the control group, but not so much as to represent a real threat to the support
of the hypotheses.
Based on separate ANOVAs, significant effects for debriefing manipulation were
found for UIT (Usefulness of debriefing in arousing Interest in the Task) (F=18.66,
p=0.000) and UUT (Usefulness of debriefing in Understanding the Task) (F=22.91,
p=0.000).
10
Table 3: Effects of Practice
Performance * F-Value Critical F-Value p-value
TP1 = -.64 vs. TP2 = -.63 1.01 F 95,1,76=3.97 0.319
SK1 = 10.15 vs. SK2 = 12 27.29 F 95,1,76=3.97 0.000
HK1 = 3.92 vs.HK2 = 3.85 17 F 95,1,76=3.97 0.682
DT1 = 59 vs. DT2 = 64 11.85 F 95,1,76=3.97 0.001
*: TP1 refers to Task Performance of Control Group (CG) in Trial 1; TP2 refers to Task
Performance of CG in Trial 2; SK1 refers to Structural Knowledge of CG on pre-test and SK2
refers to Structural Knowledge of CG on post-test; HK1 refers to Heuristics Knowledge of CG
on pre-test and HK2 refers to Heuristics Knowledge of CG on post-test; DT1 refers to average
decision time spent (in seconds) by subjects in CG in Trial 1 and DT2 refers to average
decision time spent (in seconds) by subjects in CG in Trial 2.
DISCUSSION/ CONCLUSION
Limitations
The main limitations of this study center on the simulation-based experimental
approach and the participants. First, the experimental setting was computer simulation-
based laboratory experimentation. Despite the general concern of external validity
associated with the experimental approach, computer simulation-based laboratory
experimentation is advantageous in the study of complex dynamic decision-making
environments such as renewable resource management. In such environments, much of
the complexity in the tasks is the result of the interaction among the flow of information,
actions, and consequences during the performance of the task (Sterman 1989a). In
CSBILEs, experimental simulations can capture some of the dynamic intertemporal
aspects of tasks which are characteristics of natural settings but absent from the
preprogrammed stimuli in a controlled laboratory experiment (Abdel-Hamid and
Sengupta 1999). Moreover, laboratory experimentation is an established method of
scientific inquiry within psychology/organizational behavior literature. A review of this
literature has found remarkable similarities between research findings obtained in
laboratory and filed settings (Lock 1986). A priori, there is no reason to believe that use
of experimental approach should be less successful to study dynamic decision making
than to study judgment and choice in traditional cognitive psychology discipline (Moxnes
2000).
The use of students as participants raises the question of the generalizability of
results. However, student subjects have been used in many decision making studies
including software project environments (Abdel-Hamid and Sengupta 1999), production
scheduling task (Brehmer 1992), market strategy task (Paich and Sterman 1993), and
fisheries management task (Moxnes 1998) based on the general conclusion that the
formal properties of task are much more important determinants of decision making than
subject profiles (Ashton and Kramer 1980). Although the task of managing fishing fleet
is somewhat different from all of these tasks, it is similar enough to assume that student
subjects are acceptable surrogates in this experimental investigation.
11
Key Findings
This study demonstrates that debriefing is an effective decisional aid for dealing
with complex, dynamic tasks. In CSBILEs, debriefing is useful because it improves
subjects’ task performance, helps user learn more about the decision domain, develop
heuristics, and expend less cognitive effort in decision making. Subjects with debriefing
aid performed better because of their access to what Schén (1938) called “reflective
conversation with the situation” gave them greater adaptability in recognizing system
changes and updating their mental models about the task system. Debriefing helped
decision makers to understand better the task system and employ an effective capacity
acquisition and utilization heuristic in managing the fishing fleet as well as the fishery
stock (Puts-Osterloh, Bott, and Koster 1990; Moxnes 1998). Also, subjects with
debriefing aid took significantly (F=852.81, p=0.000) less time, and therefore expended
less cognitive effort in making their decisions than those subjects who were without any
debriefing aid. As indicated by self reports, subjects provided with debriefing aid were
more motivated and understood the task better than those without debriefing. These
results provide evidence that improved performance of the subjects was due to
debriefing. This reduces the likelihood of a potential alternative explanation that it was
simply the provision of more information, as opposed to debriefing, that caused the
improvement.
Implications for the Understanding of Renewable Resource Management
These findings are important for renewable resource management because they
suggest that misperceptions of bioeconomics can be reduced through training with
debriefing supported CSBILEs. Debriefing activity allowed the decision makers to reflect
on their performance and examine their conceptions about the task system in a
constructive way. Improved and ‘corrected’ understanding about the dynamic task system
helped the decision makers to employ effective heuristics to manage the fishing fleet.
Morte specifically, if debriefing support can improve human performance in a CSBILE,
then the same debriefing support could perhaps also be used to redesign pedagogical
instruments including CSBILEs being used for training of the public policy makers in the
domain of renewable resource management. CSBILE users will get more effective
support in understanding the dynamic tasks such as renewable resource management task.
The challenge for further studies is to re-assess the earlier studies pertaining to
dynamic decision making, especially those supporting Sterman’s misperception of
feedback hypotheses and Moxnes’s misperceptions of bioeconomic dynamics. Performed
rigorously, these future experimental studies may help generalize the efficacy of
debriefing-supported CSBILEs in renewable resource management.
Implications for DDM Research
One of the findings of this study that debriefing helped the subjects to acquire
more heuristics knowledge, a score of 13.08 for subjects with debriefing and 3.85 for
subjects without debriefing, is of immense importance to DDM researchers. Specifically,
heuristics knowledge is believed to support transfer learning-how well the decision
makers learn from the previous task by making them attempt another task either in the
same domain (Huber 1995) or in a different domain (Hayes and Broadbent 1988). Only a
12
future study, where debriefing is interceded by two different dynamic tasks, can better
evaluate the effects of debriefing on transfer learning.
This study proposes and uses a comprehensive model to evaluate debriefing
effectiveness. The model comprises four evaluation criteria-task performance, structural
knowledge, heuristics knowledge, and cognitive effort. This research model is an all-
inclusive instrument that can be used by other researchers in future studies. For instance,
researchers will be able to use this research as a base to examine other important issues,
such as the influence of debriefing on group decision making, the value proposition of
debriefing, and the impact of debriefing on time-critical tasks.
Finally, task complexity affects the decision behavior of a decision maker, as a
result, CSBILEs effectiveness (Brehmer 1990). This study focused on only one decision
task. There is need to extend this research to dynamic task with varied degrees of
complexity.
REFERENCES
Abdel-Hamid, T., and Sengupta, K. “The Impact of Goals on Software Project
Management: An Experimental Investigation,” M/S Quarterly (23:4), 1999, pp.
531-555.
Ashton, R., and Kramer, S. “Students as Surrogates in Behavioral Accounting Research:
Some Evidence,” Journal of Accounting Research (18), 1980, pp. 1-15.
Bakken, B. E. Learning and Transfer of Understanding in Dynamic Decision
Environments, Unpublished doctoral dissertation, MIT, Boston, MA, 1993.
Berry, D. C. “The Role of Action in Implicit Learning,” Quarterly Journal of
Experimental Psychology (43A), 1991, pp. 890-906.
Brehmer, B. 1990. “Strategies in Real-time Dynamic Decision Making,” in Jnsights in
Decision Making, R. M. Hogarth (ed.), University of Chicago Press, Chicago,
1990, pp. 262-279.
Brehmer, B. “Dynamic Decision Making: The Control of Complex Systems,” Acta
Psycologica (81), 1992, pp. 211-241.
Cox, R. J. “Exploratory Learning from Computer-based Systems,”in Instructional
Models in Computer-Based Learning Environments, S. Dijkstra, H. P. M.
Krammer, and J. J. G. van Merrienboer (eds.), Springer-Verlag, Berlin, 1992, pp.
405-419.
Davidsen, P. I. “Educational Features of the System Dynamics Approach to Modeling
and Simulation,” Journal of Structural Learning (12:4), 1996, pp. 269-290.
Davidsen, P. I., and Spector, J. M. 1997. Cognitive complexity in system dynamics based
learning environments. /nternational System Dynamics Conference, 757-760.
Istanbul, Turkey: Bogazici University Printing Office.
Diehl, E., and Sterman, “Effects of Feedback Complexity on Dynamic Decision
Making,” Organizational Behavior and Human Decision Processes (62:2), 1995,
pp.198-215.
Dérner, D. “On the Difficulties People have in Dealing with Complexity,” Simulations
and Games (11), 1980, pp. 8-106.
13
Edwards, W. “Dynamic decision theory and probabilistic information processing,”
Human Factors (4), 1962, pp. 59-73.
Elsom-Cook, M. T. “Environment Design and Teaching Intervention,” in Simulation-
Based Experiential Learning, D. M. Town, T. de Jong, and H. Spada. (eds.),
Springer-Verlag, Berlin, 1993, pp. 165-176
Forrester, J. W. Industrial Dynamics, Cambridge, Productivity Press, MA, 1961.
Groébler, A, Maier, F. H., and Milling, P. M. “Enhancing Learning Capabilities by
Providing Transparency in Simulated Tasks,” Simulation & Gaming (31:2), 2000,
pp. 257-278.
Hardin, G. “The Tragedy of the Commons,” Science (162:3), 1968, pp. 1243-1248.
Hayes, N. A., and Broadbent, D. E. “Two Modes of Learning for Interactive Tasks,”
Cognition (28), 1988, pp. 249-276.
Hsiao, N. Exploration of Outcome Feedback for Dynamic Decision Making, Unpublished
doctoral dissertation, State University of New York at Albany, Albany, NY, 2000.
Huber, O. 1995. “Complex Problem Solving as Multistage Decision Making,” in
Complex Problem Solving: The European Perspective, P. Frensch and J. Funke
(eds.), Lawrence Erlbaum Associates Publishers, NJ, 1995, pp. 151-173.
Issacs, W. and Senge, P. “Overcoming Limits to Learning in Computer-based Learning
Environments,” in Modeling for Learning Organizations, J. Morecroft and J.
Sterman (eds.), Productivity Press, Portland, OR, 1994, pp. 267-287.
Jarvenpaa, S. “The Effect of Task Demands and Graphical Format on Information
Processing Strategies,” Management Science (35), 1989, pp. 285-303.
Langley, P. A. and Morecroft, J. D. W. “Learning from Microworlds Environments: A
Summary of the Research Issues,” in System Dynamics’ 96, G. P. Richardson and
J. D. Sterman (eds.), System Dynamics Society, Cambridge, MA, 1995.
Ledrman, L. C. “Debriefing: Towards a Systematic Assessment of Theory and Practice,”
Simulations & Gaming (23:2), 1992, pp. 145-160.
Lock, E. A. Generalizing from Laboratory to Field Settings: Research Findings from
Organizational Behavior and Human Resource Management, Heath, Lexington,
MA, 1986.
Moxnes, E. “Not Only the Tragedy of the Commons: Misperceptions of Bioeconomics,”
Management Science (44), 1998, pp. 1234-1248.
Moxnes, E. “Not Only the Tragedy of the Commons: Misperceptions of Feedback and
Policies for Sustainable Development,” System Dynamics Review (16:4), 2000,
pp. 325-348.
Paich, M., and Sterman, J. D. “Boom, Bust, and Failure to Learn in Experimental
Markets,” Management Science (39:12), 1993, pp. 1439-1458.
Putz-Osterloh, W. Bott, B., and Koster, K. “Modes of Learning in Problem Solving — Are
They Transferable to Tutorial Systems,” Computers in Human Behaviors (6),
1990, pp. 83-96.
Qudrat-Ullah, H., Saleh, M. M., and Bahaa, E. A. Fish Bank ILE: An interactive learning
laboratory to improve understanding of ‘The Tragedy of Commons’; A common
behavior of complex dynamic systems, Proceedings of 15" International System
Dynamics Conference, Istanbul, Turkey, 1997.
Schon, D. The Reflective Practitioner, Basic Books, New York, 1938.
14
Sengupta, K., and Abdul-Hamid, T. “Alternative Concepts of Feedback in Dynamic
Decision Environments: An Experimental Investigation,” Management Science
(39:4), 1993, pp. 411-428.
Spector, J. M. “System Dynamics and Interactive Learning Environments: Lessons
Learned and Implications for the Future,” Simulation & Gaming (31:4), 2000, pp.
528-535.
Sterman, J. D. “Modeling Managerial Behavior: Misperceptions of Feedback in a
Dynamic Decision Making Experiment,” Management Science (35), 1989a, pp.
321-339.
Sterman, J. D. “Misperceptions of Feedback in Dynamic Decision Making,”
Organizational Behavior and Human Decision Processes (43), 1989b, pp. 301-
335.
Sterman, J. D. “Learning in and Around Complex Systems,” System Dynamics Review
(10:2-3), 1994, pp. 291-323.
Sternberg, R. J. “Expertise in Complex Problem Solving: A Comparison of Alternative
Conceptions,” in Complex Problem Solving: The European Perspective, P.
Frensch and J. Funke (eds.), Lawrence Erlbaum Associates Publishers, NJ, 1995,
pp. 3-25.
Stienwachs, B. “How to Facilitate a Debriefing,” Simulation & Gaming (23:2), 1992, pp.
186-195.
15
