1,
Improving Decision Making and Learning in Dynamic Tasks Through Structured
Debriefing-based Interactive Learning Environments: An Experimental Study
Abstract
The thesis of this article is that decision making and learning in dynamic tasks can be
improved by helping individuals develop more accurate mental models of dynamic tasks
through training with system dynamics-based interactive learning environments (ILEs)
that include systematic debriefing. A laboratory experiment is reported in which participants
managed a dynamic task by playing the roles of fishing fleet managers. It was found that
process-oriented debriefing improved subjects’ task performance, helped users learn more
about the decision domain and develop heuristics. Groups with outcome-oriented debriefing
and process-oriented debriefing did not differ on both decision time and decision strategy
used by the subjects.
Introduction
Successful decision making is raison d'étre of today’s managers and policymakers.
However, most of the managerial tasks are increasingly complex and dynamic in nature-a
number of decisions are required rather than a single decision, decisions are interdependent, and
the environment in which decision is set changes either autonomously or because of the decision
made or both (Brehemer, 1990; Edwards, 1962; Sterman, 2000). For instance, managing a
business firm, controlling the money supply, and achieving a sustainable use of renewable
resources are all dynamic tasks. Improved decision making in these task would enhance the lives
of individuals and the performance of organizations (Blazer et. al, 1989, Sterman, 2000).
In dynamic tasks, decision makers need ways to test their decision strategies before a
costly and often irreversible implementation follows. ILEs provide a potential solution. For
instance, ILEs are often used to improve decision making in dynamic tasks. We use “ILE” asa
term sufficiently general to include micro worlds, management flight simulators, learning
laboratories and any other computer simulation-based environment - the domain of these terms
is all forms of action whose general goal is the facilitation of decision making and learning. ILEs
allow the compression of time and space and provide an opportunity for managerial decision
making in a non-threatening way (Issacs and Senge, 1994). Despite an increasing interest in
ILEs, recent research on their efficacy is inconclusive (Benbasat, and Nault, 1990; Bell et al.,
2008; Davidsen, 2000; Faria, 1998; Plate, 2010). The increasing urge to improve the efficacy of
ILEs has led the researchers to suggest improvements. One such way to improve the efficacy of
an ILE is to incorporate structured and systematic debriefing.
For effective learning to occur, most of the learning activities with or without simulations
require feedback. Prior research has shown that simple multiple-cue probability learning tasks
can be learned by outcome feedback, complex cognitive tasks are not (Blazer et al., 1989).
Debriefing is a special kind of feedback process whereby the decision makers are provided with
an in-depth facilitation and reflection on their decision making experiences to improve their
decision making skills in, cognitively intensive, dynamic tasks (Dreifuerst, 2009; Fanning and
Gaba, 2007: Lederman, 1992).
In the context of dynamic tasks, a debriefing is a time to reflect on the learning
experiences gained from an ILE. Debriefing is the processing of simulation-based learning
experience from which the decision-makers are to draw the lessons to be learned (Lederman,
1992; Stienwachs, 1992). Debriefing is delivered in different forms and methods. Oral
discussions, written notes, debriefing games are the most common variants (Lederman, 1992;
Stienwachs, 1992; Vissers and Peters, 2004). In an oral discussion, learners and debriefer engage
in a question and answer session designed to guide leamers through a reflective process about
their learning. In written notes, a passive form of debriefing, the learners are provided with hand-
outs that present “expert solution” to the task they had in the ILE and examples of potential
applications of their learning. Debriefing games are interactive strategies, played through
computer or board games where the learners are encouraged to reflect on earlier events
(Thiagarajan, 1992). Debriefing sessions can be organized in two ways: (i) where participants are
presented with a sort of “expert solution” to the task in the ILE and are asked to recall, reflect,
and compare their “own” solutions (Lederman, 1992; Stienwachs, 1992; Peters and Vissers,
2004; Qudrat-Ullah, 2010), and (ii) where participants are led through a process that illustrates
the underlying structure of the task systems and how it relates to the behavior of the task system
(Cox, 1992; Crookall et al., 1987; Spector 2000; Qudrat-Ullah, 2007). We term former as
“outcome-oriented debriefing” and later as “process-oriented debriefing”. This distinction is
important as for well-structured and well-learned tasks, outcome-oriented debriefing alone may
be sufficient to stimulate performance improvements. When a task embodies uncertainties,
process-oriented debriefing should help the learners to overcome the misconceptions about the
task. Also, debriefing plays fundamental role in helping the participants connect the knowledge
and skills developed in a simulation session to the corresponding real life situation— transfer
learning (Peters and Vissers, 2004; Dreifuerst, 2009; Fanning and Gaba, 2007; Gonzales and
Cathcart, 1995; Lane and Tang, 2000). Therefore, we assert that learners in our debriefing-based
ILE will have the opportunity to develop such transfer learning skills.
In summary, previous studies provide an insight into the effectiveness of debriefing to
decision making and leaning in ILEs. However, with the exception of a single study (Qudrat-
Ullah, 2007), prior studies, have explored, only theoretically, how debriefing may contribute to
decision making processes. Several gaps still remain, for instance, (i) measures of effectiveness
of debriefing lack a comprehensive framework, (ii) the effects of various forms of debriefing
(e.g., outcome-oriented, process-oriented) are unknown, (iii) no empirical evidence on how
debriefing effects the decision making process. This project aims to bridge some of these gaps
and advance previous research by proposing and using a comprehensive research model aimed at
evaluating the effectiveness of structured and systematic debriefing on both the decision making
process and the decision outcome in an ILE.
2. Theoretical Premise and Hypothesis Development
To perform better in dynamics tasks, decision makers need to develop an adequate model of
the task (Conant and Ashby, 1970; Sterman, 2000; Qudrat-Ullah and Karakul, 2007). Outcome-
oriented debriefing does not provide enough information to the participants to enable them form
a suitable model of the dynamic task (Blazer et al, 1989; Sengupta and Abdel-Hamid, 1993;
Sterman 1989). Individuals need to understand both the delays and the feedback structures
underlying the task. Process-oriented debriefing, however, has the potential to impart this crucial
knowledge: the debriefer identifies the feedback structures and their relation to the outcomes,
delays are examined, and uncertainties are discussed.
Sengupta and A bdel-Hamid (1993), on the other hand, found that subjects provided with
cognitive feedback-information provided to the decision makers to improve their decision
making capabilities by enhancing their comprehension of the task structure, employed consistent
decision strategies and performed better than those provided with outcome feedback alone.
Process-oriented debriefing, with the potential to aid the decision makers develop a suitable
model of the task (Conant and Ashby, 1970; Zydney, 2010), should induce decision makers to
adopt consistent decision strategies and perform better. However, this increased understanding of
the task system comes at the expense of increased cognitive effort expended (e.g., in systematic
exploration and testing of hypotheses regarding the relationship between systems variables)
(Kirlik et al., 1995). On the other hand, outcome-oriented debriefing where expert solution is
presented, subjects might mimic and use the presented heuristic and become efficient in decision
making.
3.0 Methodology
We designed a single factor, completely randomized design involving one control group
and two experimental groups. Each participant in the experimental group used FishBankILE with
either process-oriented or outcome-oriented debriefing. Debriefing was delivered in a scripted
discussion between the debriefer and the participants, after the participants have completed 1“
formal trial of the task. We conducted the experiment with 93 to 99 executive-MBA program
participants, recruited from three local Canadian universities. A pre-test questionnaire was used
to control subjects’ background education, knowledge, and demographics. The computer
program embedded in FishBankILE allowed the automatic capture of users’ decisions data and
task performance.
The Task. In the dynamic task, subjects played the role of fishing fleet managers making
fleet capacity acquisition and utilization decisions. Each year subjects was required to order new
ships and decide the utilization of the fleet. Task performance is measured by cumulative
profits. The dynamic behavior in the model arises due to two fundamental accumulation
processes: accumulation of ships and stock of the common resource-fish.
Catch per ship drives the profitability for the firm. The increased profits provide
incentives for fleet expansion. A diminishing rate of fish catch may trigger the lay-up of the
existing ships. The catch per ship is a function of the fish density of the fishing area. The
relationship between the fish density and the fish catch per ship is non-linear. The current stock
of fish determines the fish density. Fish catch depletes the fish stock, while fish generation adds
to the stock.
Procedures. All subjects were supplied with a folder containing the consent form,
instructions to lead them through a session, training materials for the task, notepads, and pens as
they will be encouraged to take notes along the experiment. The experiment started with each
participant returning the signed consent form and taking a pre-test on task knowledge. Then the
experimenter provided an introduction to the task system and the experiment. All the groups
received the same general instructions.
All the subjects completed a training trial, making decisions in each period, accessing and
observing the feedback of their decisions via graphs and tables. Then, all the subjects completed
two formal trials interceded by either a small break for the control group (no discussions was
allowed) or a debriefing activity for the experimental groups.
Independent Variable is the “availability of debriefing” in an ILE. Dependent
Variables are task performance, structural knowledge, heuristics knowledge, and cognitive
effort. The task performance metric is chosen so as to assess how well each subject did relative
to a benchmark rule.
. Task performance, TP, is assessed in the following way. Every decision period, the
benchmark’s performance variables’ values are subtracted from the subject’s. The subject's final
performance, TP, is the accumulation over 30 periods of this difference, averaged over the
number of task performance variables and number of trials. A post-test questionnaire measured
the structural knowledge through fourteen closed-ended questions on the relationships between
pairs of the task variables. Two open-ended questions asked the subjects about their general
strategy for ordering new ships and ships utilization. Two independent domain experts graded
the answers. The average scores on the two questions measured the heuristic knowledge.
Decision time was measured as the time spent by a subject making decisions in each of the
decision periods (excluding the time it took to run the simulation).
4, Results
There were no significant differences across treatments (i.e., one control group and
two experimental groups) with respect to gender (p = 0.870 both for males and females),
age (F (2, 96) =.23, p = 0.798), priorstructural knowledge (F (2, 96) =14, p =..657)
and heuristics knowledge (F (2, 96) = 0.876, p = 0..878) about the task system.
Likewise, all the groups did not differ in terms of their background education.
There was significant difference among the three groups (i.e., Group 1: No Debriefing,
Group 2: With Outcome-oriented Debriefing, and Group 3: With Process-oriented Debriefing)
when considered jointly on the variables of decision making (i.e., task performance, decision
time, and decision strategy) and learning (structural knowledge, and heuristics knowledge),
Table 2: Between-Subjects Effects
Dependent Variable p-value
Task Performance 0.000
Decision Time 0.000
Decision Strategy 0.000
Structural Knowledge 0.000
Heuristics Knowledge 0.000
We conducted planned contrast analysis among all the three groups on all the
dependent variables. Compared with the group with no debriefing, all the treatment groups
performed significantly better on task performance, so the hypothesis Hla is strongly
(p=0.000) supported. Also, the hypothesis H2a, group with process-oriented debriefing
achieved better task performance than the group with outcome- oriented debriefing. However,
on decision time, H2a is supported but H2b is not supported: contrary to the hypothesis,
group with process-oriented debriefing, where the subjects were expected to spend more
time say in focusing on structure-behavior patterns of the key variables, spent less time than
those with outcome-oriented debriefing.
On task knowledge performance, the hypotheses H4a, H4b, H5a, and H5b were
strongly supported too. The process-oriented debriefing group performed the best both on
structural and heuristics knowledge. Group with no debriefing did not show any statistically
significant improvement in their task knowledge.
5 Concluding Remarks
It is interesting to note that despite the overwhelming evidence of the effects of
debriefing on subjects’ task performance, none of the groups did (statistically) better that the
bench mark rule. In fact, it would be naive to think that subjects will become expert on
dynamic decision making as result of couple of trials. This implies that we should look at
broader measures of task performance in dynamic tasks. Overall, we find positive impact of
debriefing on subjects’ decision making and learning about a dynamic task.
Acknowledgements
We would like acknowledge the funding for this project by SSHRC via SSHRC Insight Grant-
2014, administered through Y ork University, Toronto, Canada.
References
Abdel-Hamid, T., Sengupta, K., & Swett, C. (1999). The impact of goals on software project
management: An experimental investigation. MIS Quarterly, 23(4), 1-19.
Argyris, C. (1993). Knowledge for action: A guide to overcoming barriers to organizational
change. San Francisco: Jossey-Bass.
Bakken, B. E. (1993). Learning and Transfer of Understanding in Dynamic Decision
Environments. Unpublished doctoral dissertation, MIT, Boston.
Barnett, S., & Ceci, S. (2002). When and Where Do We Apply What We Learn? A Taxonomy
for Far Transfer. Psychological Bulletin, 128(4): 612-637.
Bell, B. S., Kanar, A. M., & Kozlowski, S. W.J. (2008). Current issues and future directions in
simulation-based training in North America. The International J ournal of Human
Resource Management, 19 (8), 1416-1434.
Benbasat, I., & Nault, B. R. (1990). An evaluation of empirical research in managerial support
systems. Decision Support Systems, 6, 203-226.
Blazer, W.K., Doherty, M.E., & O'Connor, R. (1989). Effects of cognitive feedback on
performance. Psychological Bulletin, 106(3), 410-433.
Brehmer, B. (1990). Strategies in real-time dynamic decision making. In R. M. Hogarth (Ed.),
Insights in Decision Making. Chicago: University of Chicago Press, 262-279.
Briggs, P. (1990). Do they know what they are doing? An evaluation of word-processor user’ s
implicit and explicit task-relevant knowledge, and its role in self-directed learning.
International J ournal of Man-Machine Studies, 32, 385-298.
Conant, R., & Ashby, W. (1970). Every good regulator of a system must be a model of the
system. International Journal of System Science, 1, 89-97.
Cox, R. J. (1992). Exploratory learning from computer-based systems. In S. Dijkstra, H. P. M.
Krammer, & J. J. G. van Merrienboer (Eds.), Instructional Models in Computer-Based
Learning Environments. Berlin, Heidelberg: Springer-Verlag, 405-419.
Davidsen, P. I. (1996). Educational features of the system dynamics approach to modelling and
simulation. J ournal of Structural Learning, 12(4), 269-290.
Davidsen, P. I., & Spector, J. M. (1997). Cognitive complexity in system dynamics based
learning environments. International System Dynamics Conference, Istanbul, Turkey:
Bogazici University Printing Office, 757-760.
Davidsen, P. I., (2000). Issues in the design and use of system-dynamics-based interactive
learning environments. Simulation & Gaming, 31(2), 170-177.
Diehl, E., & Sterman, J. D. (1995). Effects of feedback complexity on dynamic decision making.
Organizational Behavior and Human Decision Processes, 62(2), 198-215.
Druckman, D., & Bjork, R. A. (1994). Learning, remembering, believing: Enhancing human
performance. Washington, DC: National Academy Press.
Edwards, W. (1962). Dynamic decision theory and probabilistic information processing. Human
Factors, 4, 59-73.
Ericsson, K. A., Krampe, R. T., & Tesch-Romer, C. (1993). The role of deliberate practice in the
acquisition of expert performance. Psychological Review, 100, 363-406.
Faria, A. J. (1998). Business simulation games: Current usage levels - an update. Simulation &
Gaming, 29, 295-308.
Ford, D. N., & Mccormack, D. E. M. (2000). Effects of time scale focus on system
understanding in decision support systems. Simulation & Gaming, 31(3), 309-330.
Forrester, J. W. (1961). Industrial Dynamics. Cambridge, MA: Productivity Press.
Gegenfurtner, A., Veermans, K., & Vauras, M. (2013). Effects of computer support,
collaboration, and time lag on performance self-efficacy and transfer of training: A
longitudinal meta-analysis, Educational Research Review, 8: 75-89.
Gonzalez, M., Machuca, J., & Castillo, J. (2000). A transparent-box multifunctional simulator of
competing companies. Simulation & Gaming, 31(2), 240-256.
Grobler, A., Maier, F. H., & Milling, P. M. (2000). Enhancing learning capabilities by providing
transparency in transparency. Simulation & Gaming, 31(2), 257-278.
Halpern, D. F. (1998). Teaching critical thinking for transfer across domains. American
Psychologist, 53, 449-455.
Hayes, N. A., & Broadbent, D. E. (1988). Two modes of learning for interactive tasks.
Cognition, 28, 249-276.
Hofsted, J. de Claluwe, L., & Peters, V. (2010). Why simulation games work- In search of the
active substance: A synthesis. Simulation & Gaming, 41(6), 824-843.
Hogarth, R. M., (1981). Beyond discrete biases: Functional and dysfunctional aspects of
judgmental heuristics. Psychological Bulletin, 9(2), 197-217.
Huber, O. (1995). Complex problem solving as multistage decision making. In P. Frensch & J.
Funke (Eds), Complex Problem Solving: The European Perspective. NJ: Lawrence
Erlbaum Associates Publishers, 151-173.
Issacs, W., & Senge, P. (1994). Overcoming limits to learning in computer-based learning
environments. In J. Morecroft and J. Sterman (Eds), Modeling for learning organizations.
Portland, OR: Productivity Press, 267-287.
Jansson, A. (1995). Strategies in Dynamic Decision Making: Does Teaching Heuristic Strategies
By Instructors A ffect Performance? In J. Caverni, M. Bar-Hillel, F.Barron, & H.
Jungermann (Eds). Contributions to Decision Making-I. Amsterdam: Elsevier, 213-253.
Kayande, U., De Bruyn, A., Lilien, L., Rangaswamy, A., & van Bruggen, H. (2009). How
incorporating feedback mechanisms in a DSS affects DSS evaluations. Information
System Research, 20(4), 527-546.
Kelly, H. H. (1971). Attribution in social interaction. Morristown, NJ: Learning Press.
Kintsch, W. (1988). The use of knowledge in discourse processing: A construction-integration
model. Psychological Review, 95, 163-182.
Klabbers, J. H. G. (2000). Gaming & Simulation: Principles of a science of design. Simulation &
Gaming, 34(4), 569-591.
Kleinmuntz, D. (1985). Cognitive heuristics and feedback in a dynamic decision environment.
Management Science, 31, 680-701.
Kolb, D. A. (1984). Experiential Learning: Experience as the Source of Learning and
Development. New Jersey: Prentice-Hall.
Kottermann, E., Davis, D., & Remus, E. (1995). Computer-assisted decision making:
Performance, beliefs, and illusion of control. Organizational Behavior and Human
Decision Processes, 57, 26-37.
Kriz, W. C. (2003). Creating effective learning environments and learning organizations through
gaming simulation design. Simulation & Gaming, 34(4), 495-511.
Lane, D. C. (1995). On a resurgence of management simulations and games. Journal of the
Operational Research Society, 46, 604-625.
Lane, M., & Tang, Z. (2000). Effectiveness of simulation training on transfer of statistical
concepts. Journal of Educational Computing Research, 22(4), 383-396.
Ledrman, L. C. (1992). Debriefing: towards a systematic assessment of theory and practice.
Simulation & Gaming, 23(2), 145-160.
Leemkuil, H., & De Jong, T. (2012). Adaptive advice in learning with a computer-based
knowledge management simulation game. Academy of Management Learning &
Education, 11(4): 653-665.
Limayem, M., & DeSanctis, G. (2000). Providing decisional guidance for multicriteria decision
making in groups. Information Systems Research, 11(4), 386-401.
Locke, E. A. (1986). Generalizing from laboratory to field settings: Research findings from
organizational behavior and human resource management. Lexington, MA: Heath
Lexington
Mayer, W., Dale, K., Fraccastoro, K., & Moss, G. (2011). Improving transfer of learning:
Relationship to methods of using business simulation. Simulation & Gaming, 42(1), 64-
84.
Meadows, D. L., Fiddaman T., & Shannon, D. (1993). ‘‘Fish Banks, Ltd.: A Micro-computer
Assisted Group Simulation that Teaches Principles of Sustainable Management of
Renewable Natural Resources,’’ 3rd edition, Laboratory for Interactive Leaming, Hood
House, University of New Hampshire, Durham, NH.
Moxnes, E. (2004). Misperceptions of basic dynamics: The case of renewable resource
management. System Dynamics Review, 20, 139-162.
Niranjan, T., Wagner, S., & Bode, C. (2011). An Alternative Theoretical Explanation and
Empirical Insights into Overordering Behavior in Supply Chains. Decision Sciences, 42
(4), 859-888.
Njoo, M. & de Jong, T. (1993). Supporting exploratory learning by offering structured overviews
of hypotheses. In D. M. Town, T. de Jong, & H. Spada. (Eds). Simulation-Based
Experiential Learning. Berlin: Springer-Verlag, 207-223.
NTSA, (2011). President's notes. Training Industry News, 23(4), 2-2.
Paich, M., & Sterman, J. D. (1993). Boom, bust, and failures to learn in experimental markets.
Management Science, 39(12), 1439-1458.
Plate, R. (2010). Assessing individuals’ understanding of nonlinear casual structures in complex
systems. System Dynamics Review, 28(1), 19-33.
Qudrat-Ullah, H. (2007). Debriefing can reduce misperceptions of feedback hypothesis: An
empirical study. Simulations & Gaming, 38(3), 382-397.
Qudrat-Ullah, H. (2010). Perceptions of the effectiveness of system dynamics-based interactive
learning environments: An empirical study. Computers & Education, 55, 1277-1286.
Qudrat- Ullah, H., Saleh, M. M., & Bahaa, E. A. (1997). Fish Bank ILE: An interactive learning
laboratory to improve understanding of ‘The Tragedy of Commons’; a common
behaviour of complex dynamic systems. Proceedings of 15” International System
Dynamics Conference, Istanbul, Turkey.
Schon, D. (1938). The Reflective Practitioner, New Y ork: Basic Books.
Sengupta, K. & Abdel-Hamid, T. (1993). Alternative concepts of feedback in dynamic decision
environments: An experimental investigation. Management Science, 39, 411-428.
Sfard, A. (1998). On two metaphors for learning and dangers of choosing just one. Educational
Research, 27(2), 4-12.
Sing, D. T. (1998). Incorporating cognitive aids into decision support systems: The case of the
strategy execution process. Decision Support Systems, 24, 145-163.
Spector, J. M. (2000). System dynamics and interactive learning environments: Lessons learned
and implications for the future. Simulation & Gaming, 31(4), 528-535.
Sterman, J. D. (1989). Modeling managerial behavior: Misperceptions of feedback in a dynamic
decision making experiment. Management Science, 35, 321-339.
Sterman, J. D. (1994). Learning in and around complex systems. System Dynamics Review, 10
(2-3), 291-323.
Sterman, J. D. (2000). Business Dynamics: Systems Thinking and Modeling for a Complex
World. McGraw-Hill: New Y ork.
Sternberg, R. J. (1995). Expertise in complex problem solving: A comparison of alternative
conceptions. In P. Frensch and J. Funke (Eds.), Complex Problem Solving: The European
Perspective. NJ: Lawrence Erlbaum Associates Publishers, 3-25.
Te’eni, D. (1991). Feedback in DSS as a source of control: Experiments with the timing of
feedback. Decision Sciences, 22(3), 644-655.
Wilensky, U., & Resnick, M. (1999). Thinking in levels: a dynamic systems approach to making
sense of the world. Journal of Science Education and Technology, 8(1), 1-24.
Yi, S. & Davis, F. (2001). Improving Computer Training Effectiveness for Decision
Technologies: Behavior Modeling and Retention Enhancement. Decision Sciences, 32(3),
521-544.
