Constructing an Interactive Learning Environment (ILE)
to Conduct Evaluation Experiments
Andreas Gréfler*, Ingo Notzon**, Aamir Shehzad**
*Industrieseminar der Universitat Mannheim, **University of Bergen
*D - 68131 Mannheim, Germany, **N - 5020 Bergen, Norway
*Phone: (+49 621) 292-31 40
*Fax: (449 621) 292-52 59
*E-mail: agroe @is.bwl.uni-mannheim.de
Abstract
Evaluation research on interactive learning environments (ILE) is still in its infancy.
Experiences made with other instructional media (for instance, interactive video) and
with non-computer-based simulation games suggest that this is to a great share
caused by asking the wrong questions. In the first part of this paper, it is briefly
discussed why questions about the absolute efficacy of learning environments cannot
be answered generally. In the same way, comparisons with other teaching methods
are not fruitful. The only evaluation approach open for rigorous experimentation is
testing of ILE which are systematically varied in one factor. The second part of this
paper presents a prototypical ILE that was planned to allow for controlled variation
of features. This ILE can be used for evaluation purposes very easily.
The effectiveness and efficiency of interactive learning environments (ILE) is still an
issue of evaluation research. In order to build a comprehensive body of knowledge,
common methods and research concepts are needed. For this purpose, different
definitions of effectiveness and different forms of interactive learning environments
have to be discussed and evaluated.
However, more technical questions need to be solved, too. These instruments,
which are used to make people learn, are not equally suited to measurement of their
effects on learning. For example, business simulators usually do not allow the tracking
of user input; they do not contain validated scores that represent game performance;
they do not provide possibilities for adapting them to different research questions and
experimental settings.
In a course project a prototypical ILE was constructed with the above mentioned
issues in mind. The domain of the ILE surrounds the management of different kinds of
dynamic resources of a company. The program allows the game facilitator to vary
certain experimental conditions online. For instance, the experimenter can change the
levels of difficulty of the learning environment by setting values of model variables.
Furthermore, the structure of the ILE is flexible, that is, components can be switched
on or off and various game settings are possible.
The paper starts with a brief description of different forms of effectiveness of
simulation tools and possible experimental designs for obtaining empirical data. In the
next section, the prototypical interactive learning environment is presented. The focus
of interest lies in the opportunities for experimentation. Finally, a sample
experimental setting is described.
The Effectiveness of Business Simulators and Its Evaluation
Questions of the effectiveness and efficiency of business simulators or interactive
learning environments (ILE) are still an issue of research.! Although there are some
hints that these programs promote learning (Milling 1995, p. 106) their validity has
not been proven yet. Further evidence is needed, if and how business simulators can
enhance decision making through learning (Bakken, Gould and Kim 1994, p. 251).
Even anecdotes and observations collected over decades can be misleading and are
open to systematic errors (Doyle 1997, p. 7). Accumulation of reliable results is a
precondition of scientific progress (Richardson 1996, pp. 145-147). It needs common
research concepts, definitions, methods and procedures.
For this purpose, a clear definition of the objects of evaluation research (i. e., the
business simulators) is needed. For a primary step in this direction, see Maier and
GréBler 1998. The constituting, interwoven parts of business simulators are three:
underlying model, human-computer interaction component, and functionality as
shown in Figure 1.
model
business
simulator
functionality
human-computer
interaction
Figure 1: Three aspects of business simulators
In addition to the definition of the objects of evaluation, conceptual and technical
issues concerning the rather vague term “effectiveness” must be clarified. This paper
addresses the latter topic.
Business simulators are mostly used to promote learning.” Generally speaking,
this means that they are effective if users learn something. In most cases, it is not
clearly defined what users are supposed to learn when they fly a simulator. This is true
in two ways: it is neither made explicit for the users nor have the developers or
facilitators of simulators it clarified for themselves. In other words, usually interactive
learning environments do not have clear learning objectives. Nevertheless, it is often
stated that users should learn about the domain and about general behavior modes of
complex, dynamic systems (Funke 1991, p. 208). In terms of Andersen et al. (1990,
p. 38) these goals, however, would only be “meta-purposes”. Considering evaluation
purposes, we agree with Andersen et al. when they continue with an appeal to make
“meta-purposes” more concrete (p. 40). Without a clear statement of learning goals
any evaluation becomes superfluous.
Without elaborating this point here, it is stated that knowledge acquisition has to
be evaluated by examining changes in cognitive structure and performance. Together,
these two are the measures in order to account for a complete learning process
(Spender 1998, p. 15). To consider only performance or only cognitive changes is not
enough (Doyle 1997, p. 8).
As far as performance is considered, business simulators are effective in the short
term. If simulators are internally valid users can be supposed to perform better in
controlling these instruments after a certain period of time using them. Thus,
performance and behavior have changed. But gains in explicable knowledge usually
are difficult to prove because the connection between this form of knowledge and
controlling often is delayed and distorted. This issue led to the discussion about
implicit or tacit knowledge (Broadbent, FitzGerald and Broadbent 1986; against the
assumption of implicit knowledge argues, for instance, Haider 1993). The term “tacit
knowledge”, however, indicates that some kind of knowledge acquisition occurs.
Usually, carry-over effects to other complex systems are intended; the transfer of
insights is a learning goal. Users should not only be able to control a specific
simulator better, but should also perform better controlling a certain range of complex,
dynamic situations. The question is, if insights, which were acquired through
simulation gaming, can be transferred to real life situations (Graham and Senge 1990,
p. 103-104). Chances for a learning transfer to take place increase when the individual
manages to create explicit, domain-independent knowledge. Whether, on the other
hand, implicit knowledge can be transferred seems questionable. Because the
acquisition of explicit (or even domain-independent) knowledge could not be proved
in many cases, the capability of business simulators to stimulate a learning transfer has
not been confirmed yet.
Connected with the question of transferability is whether users perform better in
the long run.’ There are only few studies regarding this because methodological (and
ethical) problems are enormous (Funke 1995a, p.211). For example, a better
performance over ten years after using a business simulator can certainly not solely be
accounted as an effect of this use. (See Norris and Snyder 1982 for an exploratory
longitudinal study.)
As another important question remains whether business simulators are more
effective than other training measures. For a lot of training methods can be assumed
that they influence people in the way intended. They make people learn. Nevertheless,
it is obvious that some methods are better suited in certain situations (in particular, for
specific learning objectives) than others. The construct of effectiveness therefore is
reduced to a relative meaning.
However, the question of the advantage of one training measure over another
cannot be answered in general and, most of the time, even not for a specific learning
situation. For example, let us assume three groups of students: one group is using a
business simulator, the second is reading a textbook, and the third is attending a
lecture about the very same topic. When conducting an experiment it can be assessed
which training group learned the most in relation to the learning goals set. This
comparison needs a simple pre and post test experimental design (Cook and Campbell
1979, pp. 103-118) and some statistical inference (Hayes 1994).4 Additionally, let us
assume that the business simulator group performed better than the lecture group, and
this group better than the reading group.> Can now be concluded that one of the
methods applied is generally more effective than another?
It cannot, because too many disturbing and interfering variables simply cannot be
controlled, for instance, the quality of the lecturer (Schulmeister 1996, pp. 374-376).
The only thing which can be concluded is that in this specific situation, with the given
objectives, the given training methods, the given environment, and the given people a
business simulator yielded better results than a lecture or a book. Most probably, not
all parameter that determine this specifics are known to the experimenter. Therefore,
results must not be generalized to all uses of that simulator, book, and lecture. And it
must not at all be generalized to all conceivable business simulators, books, and
lectures.
The problem of such comparative studies is that the methods under observation
consist of a sample of characteristics which can be varied in essentially infinite ways
(Stolurow 1962, p. 520).° The easiest way to obtain data about a comparison of
different training methods—asking users about their opinions and impressions—can
lead to objectively wrong and biased results (Marchionini 1989, pp. 607-608).
Hawthorne-effects must be assumed for many comparison studies (Roethlisberger and
Dickson 1939).
Also, efficiency matters. For example, do people learn faster (with less cost)
when using a business simulator instead of reading a book about a subject? The
argumentation can be equivalently transformed applying the universal rule of
efficiency: given the same cost of two learning measures, does one measure yield
better results than the other?’
Back to the example, let us assume the students used the business simulator for
one hour, read the book for half a day, and attended the lecture lasting 45 minutes.
Given the same ranking of effectiveness as above, it is not clear which method is more
efficient. Imagine that the simulator group outperformed the lecture group just one
percent in relation to the learning goals. Does this small advantage justify the cost of a
longer learning time? Statements about efficiency can only be made if equal cost for
the three methods are supposed. That is, one hour usage of simulator costs as much as
half a day reading and as attending a 45 minutes lecture.* In this case, the business
simulator indeed would be more efficient than the lecture and the textbook.
However, because of the problems of measuring relative effectiveness, this
statement is limited to the specific situation, with the given objectives, the given
training methods, the given environment, and the given people. Again, these results
must not be generalized in a way that the simulator is more efficient than book or
lecture in any case. And, once more, it must not at all be generalized to all conceivable
business simulators, books, and lectures.
In summary, the (absolute) effectiveness of busin imulators is trivial in a
short-term view, given that suitable learning goals are specified. The (relative)
effectiveness and the efficiency of these tools in comparison to other training methods
can hardly be measured out of methodological reasons (Clark 1983, p. 450). The same
holds true for longitudinal studies. What remains is the evaluation of different features
of business simulators or different ways of the use of simulators. In this case, forms of
simulation tools can be compared that differ in just one variable. This allows rigorous
experimentation.
Systematic variations of exactly one characteristic of business simulators permits
to find effective characteristics by which business simulators can be improved. (These
characteristics are located in all three aspects of business simulators; see Figure 1
above. See also Wolfe 1985, pp. 275-281, for a comprehensive list of features of
business games, which can easily be applied to interactive learning environments.)
Design-guidelines for simulators can be verified using this experimental approach (if
the guidelines are operational. See, for instance, Thurman 1993, p. 87, for a list of
design issues and implications.) It becomes possible to identify features of business
simulators which make them effective in a particular learning situation.
Thus, comparisons of interactive learning environments that differ in one feature
are meaningful and useful. Such research uses business simulators as objects of
interest and methods to obtain empirical data. At this point, one can also consider the
double function of the used simulation tool in many experiments: it serves as a
treatment as well as being a means to measure the results of such a treatment.
Methodological problems concerning this point have not been discussed widely so far.
See also Funke (1993, pp. 315-316), for a criticism aiming in the same direction and
the claim for a clear distinction between knowledge acquisition and knowledge
application.
In conclusion, the search for characteristics of business simulators which are
effective for specific groups of users in specific situations can take over the place of a
search for the absolute value of simulators. Because, as Bosco (1986, p. 15) pointed
out in an analogy, “the question ‘Are books effective in providing instruction’ does
not lead to a categorical answer. Rather, the answer to this question depends upon
content of the book, the way it is being used, the objectives of instruction, etc.”
Therefore, consequences of the use of business simulators depend on various factors.
When conducting any kind of evaluation research not only characteristics of the
learning tool business simulator have to be taken into account. What also needs to be
considered are characteristics of the users and situational determinants business
simulators are used in (Funke 1995b, pp. 250-251). See Figure 2 for an extension of
Figure 1.
user
model
effectiveness
learning
situation
instrument
functionality
human-computer
interaction
Figure 2: Components of evaluation research about the effectiveness of business simulators
Are Interactive Learning Environments Ready for Experimentation?
The rest of this paper deals with the more technical questions which need to be solved
in order to conduct evaluation experiments as described above. Business simulators,
which are used to make people learn, are not equally suited to measurement of their
effects on learning. They do not provide possibilities for adapting them to different
research questions and experimental settings. Usually, simulation tools are not build
to allow an easy variation of variables, which are of interest to the researcher. Only the
variation of exactly one factor permits controlled experimentation.
Following is a list of possible features of an “experimentation-friendly” business
simulator: It allows
= to manipulate model variables which determine the difficulty of a scenario (for
instance, initial values of levels, delay times, etc.),
= to set arbitrary time intervals (over-all gaming time, time per decision period, etc.),
= to select different levels of transparency of the underlying structure of the model,
= to let or let not user test their decisions in a “strategic mode”,
= to have roles in competitive games played by humans or the computer (following
different decision rules),
= to display different performance scores,
= to switch on or off different kinds of user-interfaces (that is, different modes of
human-computer interaction, various modes of presentation of results and
background information), and
= to track all user input.
Certainly, not in every evaluation study all of these features are needed. Nevertheless,
a business simulator having these features would be a powerful instrument. It could
serve for experimentation about effectiveness. But it could also be used for
experiments in psychological research. In combination with better documented and
validated models as the basis of a simulator, some of the issues mentioned against the
use of simulators for this purpose (for instance, SiiB, in print) would be less serious
any more. For example, through experimentation with distinct kinds of scores valid
and reliable ones could be found. Such a powerful tool would combine chances to
study dynamic decision making, knowledge mediation, and psychological processes
(Doyle 1997, p. 9).
Although, in general, these forms of experiments are possible with any business
simulator it has hardly be done until today. One reason for this is a lack of interest in
and knowledge about the topics described. Another reason, however, are technical
difficulties in the implementation of such features. The rest of this paper shows the
design and prototypical implementation of an interactive learning environment with
some of the above listed features. It could serve as a guideline and reminder for other
business simulators.
A Business Simulator to Manage the Resources of a Company
In a course project, the task was to create a prototypical ILE with the evaluation issue
in mind.” The program should allow the game facilitator to vary certain experimental
conditions in an easy way. In the rest of this section, the ILE is described from a user’s
point of view and the managerial background is presented. The next section contains
examples and further possibilities how this ILE could be used to conduct evaluation
experiments and shortcomings of the prototype.
Research in the field of strategic management suggests that firms obtain
competitive advantages by implementing strategies that exploit their internal
strengths, while neutralizing external threats and avoiding internal weaknesses. For a
long time, management has tended to focus too narrowly on the position of products
and market share as a basis for competitive advantage. On the contrary, the resource-
based approach emphasizes that resources within the company have to be deployed to
protect and to consolidate the company’s market position. It goes back to 1984 when
Wernerfelt’s article ,,A Resource-Based View of the Firm‘ was published.
“Resources” refers to financial, physical, human, technological, and
organizational resources of the company. They can be divided into tangible resources
(for instance, equipment and raw material) and intangible resources (for instance,
reputation, staff morale, and staff loyalty). To be a strategic asset and a sustainable
source of competitive advantage, the company’s resources have to be difficult to
imitate and substitute.
Resources as seen by Dierickx and Cool (1989) are accumulated and depleted
over time. Morecroft illustrates this idea in his article ,,The Rise and Fall of People
Express: A Dynamic Resource-Based View“ (1997) and shows that the described
approach can be well represented by a system dynamics stock and flow structure as
shown in Figure 3.
Resource_in_Development Full_Productive_Resource
Oo ; xO
Corrective
Action
Assimilation Attrition
Gap
Desired Apparent.
Condition condition
or Goal
Other_Resources
Figure 3: Stock and flow diagram of a strategic resource (after Morecroft 1997, p. 582)
Furthermore, Morecroft joins Dierickx and Cool’s point of view that ,,a key
dimension of strategy is the task of making appropriate choices about strategic
expenditures with a view to accumulating required resources and skills” (p. 580) and
classifies resources in managed and unmanaged resources. Unmanaged resources are
often difficult to locate within the firm because they are invisible at the operating
level. Furthermore, most unmanaged resources are intangible and therefore difficult to
measure.
The purpose of the presented ILE is to improve the management skills of the user
and to support learning about the resource-based view. In the business simulator four
different sectors and resources have to be managed; each of them with different
characteristics. (See Figure 4.)
Production
Workforce
Infinite and
tangible
Figure 4: Resources used as examples in the ILE
“Raw material”, “workforce” and “organizational knowledge” can be directly
influenced by investments. The intangible resource “staff loyalty” is changed due to
the state of the resources “workforce” and “organizational knowledge”. The number
of goods produced serves as performance indicator.
To broaden the users’ view of resources, the simulation also includes a finite
natural resource (raw material). Users have to pay attention to the depletion of this
resource and decrease the dependency of production on raw material over time.
Learning objectives connected with the resource-based view are:
= to know about the accumulation and depletion of resources,
= to distinguish between tangible and intangible resources,
= to learn about finite and infinite resources,
= to understand that a “dominant logic” (that is, concentration on some resources)
can lead to unforeseen results over time (Prahalad and Bettis 1986),
= and to manage resources in a competitive environment.
The business simulator presented here was created using Powersim®. Figure 5 shows
the main screen of the application with some information about the meaning of
different parts of the screen.
The Layer Panel
It shows the number of The Information Screen
the actual layer and It changes depending on
makes it easier to. the chosen layer and
navigate through the sector.
model.
Raw Material Knowledge
*100000,00
[20.694,92
y Purchase
Money (8) 4.905.256
Organisational
| Money tet to invest (8) 4.943.096 Knowledge
"Raw Material Purchase (tone)
Usage of Raw Material (tons)
==
Workforce Hiring (Persons)
—ery |
Workforce Fling (Poréons) Workforce Staff a
Social Expenses (8)
_ 7 coyality
» Months
s&s
The Remote Control \L the simulation Control Panel
The six sliders allow to put in These buttons start and stop the simulation.
investment decisions for the The actual simulation time is displayed.
model sectors. Above the Furthermore, buttons to enter the Strategic
sliders financial information Mode, to see information about the
about the company can be competitors and to modify the Network Object
found. of Powersim@ are provided.
Figure 5: Main screen of ILE
The human-computer interaction component of the business simulator compris
three different layers of detail. The first layer (depicted in Figure 5) contains general
information and an overview of the model. It is useful to gain insights about the
relationships of the different model sectors. The second layer serves to provide a
deeper understanding of a sector. It displays all information that is required to
understand how the sector works. The third layer goes down to the smallest detail. The
user can gather information about the utilised system dynamics structure and all
contained variables. Figure 6 illustrates the different layers.
Tangible infinite Resource:
Workforce
CLICK and GO >> (salary) [Human Resources} [BACK]
@ Feciitators input
ME
rkforce | Lie <osta DechugRatromentTin
Material ~
yaity
Production
Ti knowiedge
\AveRetiemeniTine
i
Raw Material Knowledge
Purchace
Organisational
Production
Workforce Staff Loyality \
Worktoee _ Loyalty
Figure 6: Three levels of detail in business simulator
The business simulator presented here is a networking game, that is, it can be
used in a local area network. Up to four players (minimum one) can play and compete
for common resources. It is possible to simulate one to three players by the computer.
Players have the opportunity to try their decisions in a “strategic mode”. Thus, they
can conduct if-then-analyses without advancing the actual simulation game.
The business simulator is embedded in a learning environment which contains
additional material, for example,
= acover story as a motivational factor,
= instructions how to handle the simulation,
= questions to guide reflection before and after using the simulator,
= information about the domain modeled,
= and links to additional sources of information (for example in the WWW).
Examples of Experimental Settings Using the Resources ILE
In addition to the up to four players, a facilitator can influence and observe the
game. For this purpose, a special interface was created. Technically, the facilitator
plays the role of a fifth player. Figure 7 depicts the main facilitator screen.
Efficiency (Costs per produced Unit) Facilitator Player Information
3.000
aa Effiency Player 1
2.000 2
z 3 a Effiency Player 2 TD simulate
1.000 1 3a 2au8
ceugtpaa'zaazas 294 % Effiency Player 3 Fi simulated
ae
O12 a 3 48 60 72 es a6 ta 120
Effiency Player 4 ‘ml
na y Play Pi simulated
Production
200. Production Player 1
= eet
5 2001 | \ + Production Player 2 Ti simulated
= 4 2
Broold \ = Production Player 3
2 rool d oduct
Be ca a Pi simulated
fade, aged
4 Production Player 4
Mi simulated
Month
Total Costs
— Total Costs Player 1
a1 Total Costs Player 2 D simulated
Ce a a a ae
‘ _ Total Costs Player 3 simulated
=~
ae Total Costs Player 4 M simulated
wee 72 9 108 120
Month
STOP for all Players! [Send Message|
Figure 7: Main screen for facilitator
In the main screen, the facilitator can start and stop the simulation for all players.
Messages can be sent to each player and the most important performance variables be
monitored. However, the game facilitator/experimenter has more possibilities to
influence the simulation and observe the proceeding of the game. Buttons on the left
hand side allow to observe detailed behavior of the four players.
An experimenter can also lay down the value of many variables of the underlying
system dynamics model. Delay times are the most prominent ones. For example,
training time for new employees can be set. See Figure 8 for all input fields to
influence model variables.
Further influence on the functionality of the business simulator—in this
prototypical version—can be achieved by manipulating the network-game
initialization files of Powersim® (*.psn and *.pss files) or through direct
manipulation of Powersim® simulation controls. Through this mechanism the
availability of a “strategic mode”, the number of processing periods per decision step,
initial values of level variables, and the automatic progression of the simulation can be
effected. In a next, more elaborate version of the ILE these functions are planned to be
built-in the facilitator’s interface, too.
Let us describe one application of such advanced _possibi ‘s for
experimentation. The three layers of detail in this business simulator (as discussed
above) are not only meant to allow easier user navigation. They also permit to conduct
experiments about different levels of transparency. Therefore, the experimenter can
conduct gaming session with or without Layer 2 or Layer 3, so that users have only
coarse information, detailed information or comprehensive information available. This
provides the possibility for research on transparent-box business simulators (Machuca
et al. 1998; GréBler 1998).
Variables (Double click for Input)
Workforce
Education Time Human Resource
Initial Human Resource (Number of Persons)
Initial Trainees (Number of Trainee:
Average Starting Salary Train
General Settings
Monthly Budget ($)
Figure 8: Screen to configure model variables
With the help of these features, it becomes easier for a facilitator to conduct
evaluation experiments following the classical pre test/post test control group design.
With a suitable knowledge test to as: changes in knowledge of the users, this
experimental design looks as depicted in Figure 9.
experimental a
group: Zz Ta
pre knowledge test
“ie fo
control \
group:
Figure 9: Pre test/post test experimental design with control group using ILE
post knowledge test
time
Pre and post knowledge test are equal for all groups. They are used to evaluate
changes in the cognitive structure of the subjects. Performance is evaluated with the
ILE (for example, efficiency of production), which also serves as treatment.
Therefore, in the control group the ILE runs with a known, fixed variable setting (for
instance, default values). The experimental group uses an ILE that is modified in one
variable. This variation can be easily done with the help of the facilitator’s interface."
Differences between experimental and control group in performance or knowledge are
presumably due to the variation of this variable—if they are statistically significant.
However, only an initial share of the points listed above for an experimentation-
ready ILE could be achieved with this prototype. From its concept, the ILE provides
opportunities for experimentation. As just mentioned, many features of the model and
the user interface can be changed, but not in an as easy way as we wished. Still
necessary in some cases are, for example, changing of initialization or configuration
files. We have not touched the important point of different, validated performance
indicators. There are mainly two reasons for these issues:
1. A limitation in resources for development of the ILE (three persons, five weeks).
2. Technical problems and inflexibility of the used simulation and development tool.
In this area, the re-programming of the interface components and simulation
controls in an all-purpose programming language would definitely offer big
potential. The development of the system dynamics model could still be done
with Powersim®.
Thus, only explorative studies have been conducted with the help of the ILE presented
so far. Nevertheless, the impact of it lies in two points: Firstly, it is a paradigm for
future ILE, which should be created with evaluation purposes in mind. Secondly, an
improved and enhanced version of the ILE can definitely be used in experiments about
the effectiveness of business simulators as laid down in the beginning of this paper.
The first part of this paper showed that questions of absolute or relative
effectiveness of interactive learning environments can hardly be answered. Like with
other teaching methods, the efficacy of simulation games depends on situation, tool,
and learner. Statements about a simulator’s effectiveness are only reasonable in such a
specific context. However, in order to improve business simulators, these tools can be
assessed using an experimental approach. Thus, two simulations are compared which
differ in exactly one feature. Usually, business simulators are not built to support
experimentation. Therefore, a prototypical ILE was presented that allows an easy
variation of important model variables.
References
Andersen, D. F., Chung, I. J., Richardson, G. P. and Stewart, T. R. (1990). Issues in
Designing Interactive Games Based on System Dynamics Models. In Andersen,
D. F., Richardson, G. P. and Sterman, J. D. (eds.), Proceedings of the 1990
International System Dynamics Conference, Chestnut Hill, pp. 31-45.
Bakken, B. E., Gould, J. and Kim, D. (1994). Experimentation in Learning
Organizations: A Management Flight Simulator Approach. In Morecroft, J. D. W.
and Sterman, J. D. (eds.), Modeling for Learning Organizations, Portland,
pp. 243-266.
Bosco, J. (1986). An Analysis of Evaluations of Interactive Video. Educational
Technology, May 1986, 7-17.
Broadbent, D. E., FitzGerald, P. and Broadbent, M. H. P. (1986). Implicit and Explicit
Knowledge in the Control of Complex Systems. British Journal of Psychology
77, 33-50.
Clark, R. E. (1983). Reconsidering Research on Learning from Media. Review of
Educational Research 53(4), 445-459.
Cook, T. D. and Campbell, D. T. (1979). Quasi-Experimentation. Design and
Analysis Issues for Field Settings, Boston et al.
Dierickx, I. and Cool, K. (1989). Asset Stock Accumulation and Sustainability of
Competitive Advantage. Management Science 35(12), 1504-1510.
Doyle, J. K. (1997). The Cognitive Psychology of Systems Thinking: An Agenda for
Collaborative Research. In Barlas, Y., Diker, V. G. and Polat, S. (eds.), /5th
International System Dynamics Conference: Systems Approach to Learning and
Education into the 21st Century, Istanbul, pp. 7-10.
Funke, J. (1991). Solving Complex Problems: Exploration and Control of Complex
Systems. In Sternberg, R. J. and Frensch, P. A. (eds.), Complex Problem Solving
— Principles and Mechanisms, Hillsdale/Hove, pp. 185-222.
Funke, J. (1993). Microworlds Based on Linear Equation Systems: A New Approach
to Complex Problem Solving and Experimental Results. In Strube, G. and
Wender, K. F. (eds.), The Cognitive Psychology of Knowledge, Amsterdam,
pp. 313-330.
Funke, J. (1995a). Erforschung komplexen Problemlésens durch computerunterstiitzte
Planspiele: Kritische Anmerkungen zur Forschungsmethodologie [Research on
Complex Problem Solving with Computer-Based Planning Games: Critical
Comments on the Research Methodology]. In Geilhardt, T. and Miihlbradt, T.
(eds.), Planspiele im Personal- und Organisationsmanagement, Gottingen,
pp. 205-216.
Funke, J. (1995b). Experimental Research on Complex Problem Solving. In Frensch,
P. A. and Funke, J. (eds.), Complex Problem Solving — The European
Perspective, Hillsdale/Hove, pp. 243-268.
Graham, A. K. and Senge, P. M. (1990). Computer-based Case Studies and Learning
Laboratory Projects. System Dynamics Review 6(1), 100-105.
GréBler, A. (1998). Structural Transparency as an Element of Business Simulators. In
System Dynamics Society (ed.), Proceedings of the Sixteenth International
Conference of the System Dynamics Society, Québec City, p. 39.
Haider, H. (1993). Was ist implizit am impliziten Wissen und Lernen? [What is
Implicit with Implicit Knowledge and Learning?]. Sprache und Kognition
12(1), 44-52.
Hayes, W. L. (1994). Statistics, 5h edition, Fort Worth et al.
Kerres, M. 1998. Multimediale und telemediale Lernumgebungen [Multimedia and
Telemedia Learning Environments]. Miinchen/Wien.
Klein, R. D. and Fleck, Jr., R. A. (1990). International Business Simulation/Gaming:
An Assessment and Review. Simulation & Gaming 21(2), 147-165.
K6obberling, A. (1971). Effektiveres Lehren durch Programmierten Unterricht?
Weinheim/Basel/Berlin.
Machuca, J. A. D., del Castillo, J. C. R., Carrillo, M. A. D. and Zamora, M. M. G.
(1998). Our Ten Years of Work on Transparent Box Business Simulation. In
System Dynamics Society (eds.), Proceedings of the Sixteenth International
Conference of the System Dynamics Society, Québec City, p. 5.
Maier, F. H. and GréBler, A. (1998). A Taxonomy for Computer Simulations to
Support Learning about Socio-Economic Systems. In System Dynamics Society
(ed.), Proceedings of the Sixteenth International Conference of the System
Dynamics Society, Québec City, pp. 57-58.
Marchionini, G. (1989). Making the Transition from Print to Electronic
Encyclopedias: Adaptation of Mental Models. International Journal of Man-
Machine Studies 30(6), 591-618.
Milling, P. (1995). Organisationales Lernen und seine Unterstiitzung durch
Managementsimulatoren [Organizational Learning and its Support through
Management Simulations]. Zeitschrift fiir Betriebswirtschaft, Ergdnzungsheft
3/95: Lernende Unternehmen, 93-112.
Morecroft, J. D. W. (1997). The Rise and Fall of People Express: A Dynamic
Resource-Based View. In Barlas, Y., Diker, V. G. and Polat, S. (eds.),
Proceedings of the 15" International System Dynamics Conference, Istanbul,
pp. 579-586.
Norris, D. R. and Snyder, C. A. (1982). External Validation of Simulation Games.
Simulation & Games 13(1), 73-85.
Prahalad, C. K. and Bettis, R. A. (1986). The Dominant Logic: A new Linkage
between Diversity and Performance. Strategic Management Journal 7, 485-501.
Richardson, G. P. (1996). Problems for the Future of System Dynamics. System
Dynamics Review 12(2), 141-157.
Roethlisberger, F. J. and Dickson, W. J. (1939). Management and the Worker,
Cambridge.
Schulmeister, R. (1996). Grundlagen hypermedialer Lernsysteme: Theorie, Didaktik,
Design [Foundations of Hypermedia Training Systems: Theory, Didactics,
Design], Bonn et al.
Spender, J.-C. (1998). The Dynamics of Individual and Organizational Knowledge. In
Eden, C. and Spender, J.-C. (eds.), Managerial and Organizational Cognition,
London/Thousand Oaks/New Delhi, pp. 13-39.
Stolurow, L. M. (1962). Implications of Current Research and Future Trends. Journal
of Educational Research 55(9), 519-527.
Sii®B, H.-M. (In print). Intelligenz und komplexes Problemlésen [Intelligence and
Complex Problem Solving]. To be published in Psychologische Rundschau,
Themenheft “Komplexes Problemlésen”.
Thurman, R. A. (1993). Instructional Simulation from a Cognitive Psychology
Viewpoint. Educational Technology, Research and Development 41(4), 75-89.
Wernerfelt, B. (1984). A Resource-Based View of the Firm. Strategic Management
Journal 5, 171-180.
Wolfe, J. (1985). The Teaching Effectiveness of Games in Collegiate Business
Courses. Simulation & Games 16(3), 251-288.
Notes
1. Both terms, “business simulators” and “interactive learning environments” are
synonymously used in this paper. For a more detailed definition see, for instance,
Maier and GréBler 1998.
2. As the term “business simulators” already implies, the domain of these tools are
business and economics. Most of the statements in this paper, however, can be
applied to simulators from other domains as well.
10.
Klein and Fleck (1990, p. 156) argue that classical pre and post tests measure what
they call the “recall effect”, not a learning effect. However, this seems to be
dependent on the validity of the tests used.
A validated test to measure learning success is assumed.
According to Clark (1983, p.453) educational media are just “vehicles for
instruction”. What matters is content. Thus, any teaching measure can lead to any
learning objective in principle. However, “certain elements” seem to facilitate
learning. For example, gaming simulations seem to be well suited to teach
dynamically changing matters (Reigeluth and Schwartz 1989, p. 2).
K6bberling (1971, pp. 5—7) discusses the possibility to achieve reliable statements
about the relative effectiveness of training methods in specific situations. This
ought to be done through a substantial number of replicated “evaluative
experiments.”
. Hays and Singer (1989, p. 140) briefly describe different concepts for cost analysis
of training measures.
Neglecting at this point different cost of developing the discussed measures of
education. In particular for computer-based methods, the development cost are
relatively high: Kerres (1998, p. 122) gives an estimation of 10,000 to 30,000 Euro
per one hour of Computer-based Training. However, cost estimation for the
development of business simulators have not been given in the literature yet.
Course on “Interactive Learning Environments” as part of the master program in
system dynamics at the University of Bergen, Norway. Lecturer was Pal Davidsen
whose support is gratefully acknowledged.
Of course, every experimentation should be initiated by theoretical thoughts and
be guided by well-formulated hypotheses.
