On the Effectiveness of Corporate Gaming Environments
and Management Simulators
Peter M. Milling and Frank H. Maier
Industrieseminar der Universitat Mannheim
D - 68131 Mannheim, Germany
Phone: (+49 621) 292-55 25/27 Fax: (+49 621) 292-52 59
e-mail: pmilling@is.bwl.uni-mannheim.de
ASSUMPTIONS ABOUT THE BENEFITS OF MANAGEMENT SIMULATORS
Gaming is a highly effective tool for management training and education. Management games work as catalysts in the processes of group
decision making. They counteract narrow specialization, lead to improved communication between different corporate functions, and
encourage the identification and the pursuit of shared values and overall objectives. Management practitioners and students confirm that
these learning environments increase the motivation to try new ways of behavior and to improve the understanding of complex decision
problems. However, especially with respect to the effectiveness of management simulators, objective data about their effectiveness are
missing. In a series of investigations it was tried to get a more reliable picture, how performance and the use of management simulators
correlate with each other. The research design varies between a simple questionnaire and an elaborate investigation, how the participants
react under several sets of circumstances.
To test the hypothesis of a positive relation between the availability of a management simulator and the achieved performance it was first
investigated, how knowledge about systems in general and System Dynamics in particular play a role with respect to the intensity of use of
the management simulator LEARN! and the quality of the insights gained from it.
EXPERIMENTAL DESIGN AND DATA COLLECTION
Besides LEARN! typical management flight simulator (see e.g. Graham/Morecroft/ Senge/Sterman 1992 for a definition)-a multi-person
Corporate Planning Game called LOBSTER (an acronym from Leaming Organization By Simulating The Economic Reality) is used in
academic education and corporate management training since many years at Mannheim University. At the beginning of the game, all
corporations have a product with the same level of technological sophistication and with the same market share.
The game deals with the processes of Research and Development, the time-to-market, and the time-to-volume for new products. It focuses
on the substitution between different generations of innovative products. It requires decision inputs for all classical fields of corporate
management, i.e. budgeting and resource allocation for R&D and advertisement, investment in production capacity, and the way to finance it,
personnel recruitment, etc. The timing of market introduction, investment and production planning, cost management, pricing policies,
product quality, and delivery delays are key control variables in these processes (Milling 1996).
During the game, each group of players is confronted with interrelated decision making requirements. It is difficult to understand intuitively,
how the decisions interact with each other, how the competitors and the whole system will react. To improve their market performance, the
group must identify and collect relevant information. The team members must derive alternative courses of action and evaluate their
expected consequences. A feeling for complex system behavior should be gained (Vennix 1990) and our experience supports this
expectation.
The simulator LEARN! (for Learning Environment in an Artificial Reality Network) is used to investigate the particular market dynamics,
especially to support decision making in the artificial reality of our corporate gaming environment. In a man-machine dialog it allows to test
the market response of different courses of actions. Similar to LOBSTER, LEARN! deals with the processes of Research and Development,
the time-to-market, and the time-to-volume for new products. It requires decision inputs for the same classical fields of corporate
management as LOBSTER. The difference to the LOBSTER-Game-which in the context of the flight simulator is interpreted as the ,reality,-
are the missing direct actions and reactions of the competitors. They are included as endogenous model variables.
Two groups (out of four) can use the simulator to investigate the expected consequences of their actions. The other two rely for their
analyses only on conventional tools like spreadsheets. Different behavior modes and different performance of the groups with and without
simulator should be expected. This hypothesis, however, is only partly confirmed. Significantly different modes of behavior or profit
performance could not be confirmed (see Maier and Strohhecker 1996). In addition to this research design the students had to answer in each
gaming period several questionnaires because we wanted detailed information about their aspirations, motivation, contentment, and intensity
in using LEARN! as a decision support tool.
USE OF MANAGEMENT SIMULATORS - EXPECTATIONS AND ACHIEVEMENTS
Although LEARN! can be seen as an ideal decision support system for the virtual reality of LOBSTER, it was not used very intensively during
the sessions or in preparation for the sessions. In the contrary, on an average the students used LEARN! only for 1 hour, and during this time
they did only one single simulation run. As the main reason for such a rare use, it was frequently mentioned, that the reality, i.e. LOBSTER,
is complex and requires most of the time for analyzing the results and discussing future decisions. No time was left, to apply other means of
analysis or support.
It could be assumed, that the players did obviously not recognize the effectiveness of LEARN! as a transitional object for argument and
debate. They might have failed to appreciate the power of such a tool, to improve the understanding and to analyze the consequences of
their decisions. However, the analysis of the filled questionnaires showed a somewhat different picture.
In the first round of the game, the students were asked for their expectations about the potential of LEARN! and in the last round they had to
evaluate its effectiveness on the basis of the experience they gained. As shown in Figure 1, most of the students expected, that the use of
LEARN! would lead to a better understanding of LOBSTER, enhance insight in the interrelations of this complex system, allow the
investigation of decision consequences. All this would cause improved decisions.
[The students had to give their view of the following
Istatements:
Please indicate to what extent you agree or disagree:
(1 = | fully disagree; 6 = | fully agree)
\Statistics: ‘Std.
Statements: Mode Mean Dev. N
|l expect better understanding ig 5.25 [50 [4
| did get a better understanding ig 4-56 |-70 [18 |
ll expect enhanced abilities to analyze 6 5.75 50 4
Iconsequences me |i
LEARN! did enhance my abilities to analyze lo 3.22 {1.17 he
consequences ! :
Fig. 1: Expectations and Achievements of the use of LEARN!
Students strongly agreed, that they got an improved understanding by using LEARN! But they did not see it as a tool
for the analysis of decision consequences. They expected the simulations of LEARN’ to start with the actual initial
conditions of the management game, i.e. to act as a decision tester, based on the current data of their virtual reality,
which clearly is not the intention of the simulator. The use of LEARN! should lead to a more general, learning in and
about complex systems, and therefore to improved decision effectiveness.
To investigate whether a better understanding of the complexity and the dynamics of the systems was actually
achieved, they were asked each period to estimate the development of some of the model variables like next
quarter's sales volume, profits, and market share. These estimations were compared with the actual results. The
goodness of fit between expected and achieved outcome serves as an indicator whether the groups with access to
LEARN!had an improved understanding of system reaction.
Due to the small sample size, the results, the validity and the reliability of the statistical results estimations have only
preliminary character. They seem, however, to indicate a trend. Figure 2 shows some of the values: the mean error
of the groups estimations, the standard deviation and the standard error for the variables sales volume and market
share together with the t-statistics. The results show a slight tendency that company 1, which did use the
management simulator LEARN! for more than 14 hours, has a better understanding of the consequences of their
decisions. Their estimation errors are significantly smaller than those of company 2 and 3.
Compared with company 2-which was the best performing company and did not have access to LEARN-the first
company's estimations are only insignificantly better. However, since the second company had access to detailed
data from the management game of the year before, this group can not serve as a reference group. Company 2 used
the information and just applied the strategy which has been proven successful in an earlier round of the game. In
some sense, this group learned from failures of their ancestors, but it is doubtful, that they managed to utilize their
data for a better understanding of the game's dynamics. The goodness of fit between estimated and realized sales
volume and market share supports this hypothesis although the measures are not significant.
measures: T-Statistics
company 1 vs.
company x
z|_2 | 38] 3
Else&s| 22] 2s
SIES S| $3 | SS [tVaue| p
sales yolume | 1 [0.0168 | 0.0942 | 0.0298 : -
2 | 0.0495 | 0.0981 | 0.0310 | -0.088 | 0.390
3 7 -0.1078 | 0.1353 | 0.0451 | 2.254 | 0.038
4 [0.1137 | 0.1764 | 0.0557 | 1.982 | 0.068
market share} 1 | 0.0090 | 0.0886 | 0.0281 - -
2 10.0311 | 0.1080 | 0.0341 | -0.498 | 0.625
3 | -0.0879 | 0.0667 | 0.0226 | 2.666 | 0.016
4 [0.1204 | 0.1743 | 0.0551 | 2.093 | 0.056
Significant values are printed bold (p=0.1)
Fig. 2: Goodness of fit of estimations
Due to the small sample, the results are not statistically valid and reliable, and the ongoing research efforts try to get
a broader sample size and improve the acceptance of LEARN/ Management flight simulators could improve
decision effectiveness, and tools that allow insight into model structure-like Vensim's causal tracing facility-might
lead to even better understanding.
REFERENCES
Graham, A. K.; J.D.W. Morecroft; P. M. Senge and J. Sterman 1992: Model-supported Case Studies for Management Simulators. In:
European Journal of Operational Research, Vol. 59, pp. 151-166.
Maier, F. H. and J. Strohhecker. 1996. Do Management Flight Simulators Really Enhance Decision Effectiveness? System Dynamics ‘96.
eds. George P. Richardson and John D. Sterman. Volume 2, Cambridge, MA. pp. 341-344. Also available through: http://iswww.bwl.uni-
mannheim.de/pr/sd96/index_ab.htm.
Milling, P. M. 1996. A Management Simulator to Support Group Decision Making in a Corporate Gaming Environment. System Dynamics ‘96.
eds. George P. Richardson and John D. Sterman. Volume 2, Cambridge, MA. pp. 369-372.
Vennix, J. A. C. 1990. Mental Models and Computer Models. Design and Evaluation of a Computer-Based Learing Environment for Policy-
Making. Diss. University of Nijmegen.
ISDC'97 CD Sponsor Bagel bed
