Cognitive biases, modeling and performance: an experimental analysis
Nicholas C. Georgantzas, Fordham University, New York, NY 10023, (212)841-5478
Abstract
Producing (or constructing) strategic decisions entails numerous cognitive and other bounds on human
rationality, which often cause systematic errors and biases. Yet among the economic and management
models used in strategic planning, few try to explain why decision makers remain so stubbomly and
extravagantly irrational, ignoring logic, principles of optimization, and even postulated self-interest. One
explanation may be the difficulty of extending methods used to study individual choice and decision-
making behavior to dynamic group settings. This experimental analysis assessed the impact of cognitive
simplification processes on the performance of 118 graduate business students who worked in a simulated
strategic context. Randomly assigned to twenty-four teams, the subjects run international conglomerates
with multiple actors, feedback loops, non-linearities, and time lags and delays. The teams' interaction,
expectations, choice and model selection produced results that systematically diverged over time. Within a
crossed factorial design, these results support the hypothesis that cognitive biases interact with strategic
management models to influence performance. Poor performers chose models that reinforced their
cognitive limits and bounds. Conversely, good performers constructed models which helped them
recognize and overcome the negative effects of cognitive simplification processes. They produced
effective decisions, not by optimizing functions, but through searching for recognizable patterns when
they received feedback.
Introduction
Writers in the field of strategic management generally recognize that human cognitive limitations affect
strategic decisions. Those responsible for an organization's strategic decision situation face a task of
extreme complexity and ambiguity. The complexity of the task appears almost infinite, while the matching
human and organizational information processing capabilities are usually limited. One of the central
features of strategic decision situations is their lack of structure (Mintzberg, Raisinghani & Thedret, 1976).
The strategic decision-making process entails novelty, complexity and openendedness. Strategy and
policy makers usually begin with little comprehension of the situation and their understanding deepens as
they work on it (Mintzberg et al., 1976: 265). The complexity of strategic decision situations makes
strategic decision-making an ill-structured process (Mason & Mitroff, 1981: 10-13). Because strategic
decision situations have no clear formulation, it is extremely difficult to either describe the nature and
structure of a situation, or determine the criteria by which a certain course of action should be chosen.
Complex strategic decision situations involve uncertainty and ambiguity for the decision makers.
Michael (1973) suggests that when environmental uncertainty cannot be minimized by organizational
action, managers may alter their perceptions of the environment so that it appears more certain. This
happens because the psychological state of uncertainty regarding important decisions is very painful. As a
result, decision makers may repress awareness of the uncertainty and act on a simplified model of reality
which they construct. Morecroft sees this phenomenon enabled by five information filters surrounding the
strategic decision-making process (1988: 306). The first filter represents people's cognitive limits and
Simon's (1976) notion of bounded rationality: people are unable to process all the information that a
strategic decision situation may present; they decide on the basis of a few sources of information processed
according to quite simple rules of thumb. The outer four filters in Morecroft's framework represent the
ways in which an organization actually conditions the information made available to its constituents.
These four filters provide a "psychological environment" which limits the range of factors considered and
supplies only "relevant" information.
in a manager's day-to-day work, the lack of pertinent and well-structured information generates potential
pitfalls for strategic thinking and makes planning and learning from experience difficult. To the rescue
come some important developments in "problem-forming" methods that managers increasingly view as
"sources of new knowledge" or as "tools for learning" about business and social systems (De Geus,
1988). These are improvements in the symbols and software used for mapping and model structuring;
behavioral decision theory ideas which help to capture the knowledge of decision makers into models;
improvements in methods of simulation and scenario analysis that enable modelers and model users to gain
better insight into dynamic behavior; and emphasis on small transparent models, on games and on dialogue
410
System Dynamics '90 411
between "mental models" and simulated scenarios. For example, system dynamics are now used by
management teams to create "microworlds" or "incubators for knowledge" (Papert, 1980) that allow
structuring informed debates about strategic change, in a process where models and simulated scenarios
become an integral part of management dialogue (Morecroft, 1988).
As a result of these developments, business educators face increasing demands in the "capstone"
business policy course. The fast pace of change in the global business environment dictates that business
policy teachers integrate problem-forming with conventional case studies in order to adequately prepare
students for the world of work. A conventional business policy case provides general information on a
selected company: the problems it faces, the industry it competes in, its products and markets, its history,
its organization and administration, and the personalities of its leaders. This information initiates debate
and dialogue, which lead to the situation's clarification and, eventually, to recommendations for action
(Christensen, Andrews, Bower, Hamermesh & Porter, 1987). What the conventional approach to case
analysis (by argument) provides is a context of drama and realism where the interplay between
information and debate produces a consensus for action. However, focusing on mere observations
involves a possibility of substantial misinterpretation (Copeland, 1958). The risk is even greater if the
conventional approach to case studies remains the still point in a complex, dynamic, and rapidly changing
global environment. Conversely, problem-forming methods which include descriptions of policy
functions, algebra, and simulated scenarios provide discussants with additional, more pertinent, and better
structured information on strategic decision situations.
Mintzberg et al. (1976) and more recently Nutt (1984) show that some managers do not formally and
explicitly diagnose strategic decision situations. Indeed, the most popular organization behavior model
which legitimizes the return to basics and sticking to the knitting (Peters & Waterman, 1982) is "logical
incrementalism" (Quinn, 1980). This same model was called "the science of muddling through"
(Lindblom, 1959) a few years back, and it can still be used by managers who assume their firms will
remain immune to the turbulent forces of global competition. Through the use of problem-forming
methods, such as Scenario-driven Strategic Information Mapping (S-dSIM), decision makers who do not
escape to and reaffirm the dogmas of the quite past will be able to consider most of the important factors
that influence a firm's problematic. This should also lead to improved performance and a better strategic
posture for their organization.
This study examined the impact of nine cognitive biases (CBs) and the effect of S-dSIM on the relative
performance of 118 graduate business students who worked under the experimental conditions of a
simulated strategic context. Randomly assigned to twenty-four teams, the subjects run international
conglomerates with multiple actors, feedback loops, non-linearities, and time lags and delays. The
interaction, expectations, choices and teams' model selection produced results that systematically diverged
over time. Within a crossed factorial design, these results support the hypothesis that cognitive biases
interact with strategic management models to influence performance. Poor performers chose heuristics
that reinforced their cognitive limits and bounds. Conversely, good performers constructed models which
helped them recognize and overcome the negative effects of cognitive simplification processes. They
produced effective decisions, not by optimizing functions, but through searching for recognizable patterns
when they received feedback.
There are those who argue that inferences cannot be drawn about executives’ performance at real world
decision making from students and laboratory decision making tasks (Ungson, Braunstein & Hall, 1981).
However, laboratory research investigating the effects of cognitive biases using tasks more representative
of the ill-structured situations encountered in strategic decision making has been advocated as the most
fruitful approach for several questions in strategic management (Schwenk, 1984 & 1982). Many call for
renewed investigations designed to provide data at the micro level that provide direct evidence about the
behavior and performance of decision makers and the ways in which they go about making their decisions
(Coleman, 1987; Simon, 1984; Sterman, 1989),
Cognitive Biases (CBs)
Cognitive psychologists and behavioral decision theorists have identified a wide range of cognitive
processes which serve to simplify decision makers' perceptions of strategic decision situations and render
the strategic decision-making process manageable (Hogarth, 1980; Hogarth & Makridakis, 1981; Slovic,
Fischoff & Lichtenstein, 1977; Taylor, 1975; Tversky & Kahneman, 1974). The processes used are
cognitive biases, but some researchers prefer the term "heuristics", because the term "biases" suggests that
these processes generally have a negative impact on strategic decisions. Tversky & Kahneman (1974),
412 System Dynamics ‘90
Winkler & Murphy (1973) and other behavioral decision theorists observe that these processes may
actually improve decisions as organizations display effective decision-making despite people's cognitive
limits and over-abundance of information (Morecroft, 1988; Simon, 1976) . However, as useful as these
heuristics may be, sometimes lead to severe and systematic errors (Tversky & Kahneman, 1974; 1125).
Drawing on this literature, Schwenk (1984) conjectures about cognitive simplification processes
frequently encountered in problem-forming and decision making under uncertainty. Assuming that the
formulation of strategic decision situations begins with the recognition of gaps between expectations or
standards and performance (such standards may be based on past trends, projected trends, standards of
global competitors, expectations of internal and external stakeholder groups, or even theoretical models),
Schwenk (1984) selected nine groups of cognitive biases (CB1 through CB9) and, according to the
decision-making stage they may affect, he proposed the following classification:
Stage I: GOAL/PROBLEM-FORMING
CB1 Prior hypothesis bias and adjustment & anchoring. Under their influence strategic
decision-makers tend to perceive fewer gaps than their data indicate.
CB2 ~~ Escalating commitment. Under its influence strategic decision-makers tend to minimize the
significance of gaps, and they tend not to make full use of these gaps as a basis for changes in
strategy. Strategic decision makers may even become more committed if they receive feedback
indicating failure than when receiving feedback indicating success of a change in strategy.
CB3 —- Reasoning by analogy. Even if the significance of a gap is recognized, strategic decision-makers
tend to define the factors causing the gap through an analogy to a simpler situation.
Stage Il: ALTERNATIVES GENERATION
CB4 Single-outcome calculation. In searching for a solution to a strategic decision situation, strategic
decision-makers tend to generate and bolster a single alternative rather than several alternatives.
CB5 Inferences of impossibility and denying value tradeoffs. Strategic decision-makers tend to deal
with non-preferred alternatives by denying that they serve any values better than the preferred
alternative, and by over-estimating the difficulty in implementing them.
CB6 Problem set. Under its effect and that of unchallenged assumptions, strategic decision makers
who attempt to generate more than one alternative tend to generate very few.
Stage III: EVALUATION AND SELECTION
CB7 Representativeness. Under its influence, strategic decision makers tend to overestimate the
accuracy of their predictions of the consequences of alternatives.
CB8 _ Illusion of control. Under its influence, strategic decision-makers tend to over-estimate the
importance of their own actions in ensuring the success of alternatives.
CB9 Devaluation of partially-described alternatives. Strategic decision makers tend to exhibit a
preference for alternatives described in great detail, even though partially-described alternatives
score higher on their evaluation criteria.
Schwenk's purpose of discussing the possible operation of these processes in strategic decision making
was not to criticize the quality of strategic decisions but, rather, to generate ideas about the ways decision
makers actually deal with complexity, ambiguity, and uncertainty. However, his focus on processes
which have been encountered both in laboratory and/or field settings does offer a basis for selecting those
biases which have some probability of affecting decision making in strategic decision situations.
Scenario-driven Strategic Information Mapping (S-dSIM)
be problem storming method tested in this paper is Scenario-driven Strategic Information Mapping
(S-dSIM). S-dSIM consists of several interdependent components and its purpose is to help managers
and/or business students formulate a firm's strategic decision situation; its "mess" of problems or
"problematic". The components include the method of rational argumentation (Mason & Mitroff, 1981),
the nominal group structure (Van de Ven & Delbecq, 1974) -slightly modified to incorporate Rapoport's
(1967) suggestions, the Strategic Assumption Surfacing and Testing (SAST) approach advanced by
Emshoff, Mitroff & Kilman (1978) and Mitroff & Emshoff (1979), and comprehensive situation mapping
(Acar, 1983; Acar, Chaganti & Joglekar, 1985). The method of rational argumentation teaches
participants how to present claims (with implications that lead to recommendations) on the nature and
structure of a strategic decision situation to other group members during a case discussion. It forces
System Dynamics '90 413
participants to provide the underlying data and reasoning processes (warrant) that support their claim, and
a safety valve: conditions under which their claim will not be valid (Brightman, 1987). The nominal group
procedure requires that each person within a team independently generate his/her analysis of a case or
problematic. Each person then presents his/her case analysis to the other group members. No discussion
is permitted during these round-robin opening arguments. After everyone has made an opening
presentation (with claim and implications, data, warrant, and safety valve), each of the protagonists in the
debate presents the other person's claim on the nature and structure of the problematic, to the satisfaction
of that person. This forces each debater to acknowledge and understand the opposing view, a critical
ingredient to a dialectical interchange aimed at unearthing critical strategic assumptions (Figure 1).
DEFINITIONS TIONS
IMPLICATIONS
(SCENARIOS )
L ft Why?
i
; po.
4 5 - 10 YEARS
- A INTO
| NOW THE FUTURE
i
i
i
Because... So what?
How the DATA support the CLAIM... Are you sure?
Unless...
CRITICAL SAST |
oe
_.
ss .dchLULLCCrCC.
Fig. 1. The Scenario-driven Strategic Information Mapping (S-dSIM) problem-forming method.
Through the process of specifying conditions under which the claim and its implications will not hold
(safety valve), participants unearth and specify assumptions implicit to their understanding of the nature
and structure of the strategic decision situation (e.g., assumptions about events, attributes, and stakeholder
objectives and values). This S-dSIM component exploits the SAST process which can handle a) strongly
different strategic options; b) forces of continuity, tradition, and organizational inertia (i.¢., resistance to
change); and c) in-depth analyses so a team is less vulnerable to non-penetrating thinking, and does not fall
victim to unconsciously-made yet critical implicit assumptions.
Each participant provides his/her claim on the nature and structure of the problematic along with the
claim's implications (i.¢., scenarios) using comprehensive situation mapping (CSM). CSM is a
sophisticated extension of influence diagramming (ID) that overcomes its limitations without introducing
undue complexity. An influence diagram offers a graphic map of the web of qualitative interrelationships
bearing on a business problematic (Diffenbach, 1982; Maruyama, 1963; Weick, 1979). Its purpose is to
make the dynamics of the interrelationships more visible, more explicit, and thus more comprehensible. It
is a desk-top tool that can be used individually or collectively. CSM is also a desk-top tool that, in
addition to providing a graphic representation of the network of causal influences, allows tracing
interrelationships quantitatively in the form of change scenarios appended to the map itself.
414 System Dynamics '90
In CSM, a short name is written for each variable that represents a possible source of change in itself or
in the level of other variables. Factors which may prevent the transmission of change under certain
conditions are also included. The elements of the problematic (variables and factors) are connected with
three types of arrows: a double-line arrow is used to connect a sender and a receiver of change when the
sender of the arrow is sufficient, by itself, to transmit a change to its received; single-line arrows are used
when two or more senders have to vary in order to generate a change in a receiver; when the sender is a
factor which could prevent the receiver from changing under any influence, a dotted-line arrow is used to
express this condition. When there is a time lag involved in the transmission of change, or when the
change induced in the receiver of an arrow is not comparable in sign or proportion to the change generating
it, the extent of time lag and the change equivalence coefficient are written next to the arrow. Thus, in
CSM, change transmittance coefficients express the ratio of the induced percentage change to the one
generating it (Acar, 1983).
Methods
This section describes how the effects of the nine CBs and the S-SIM problem-forming method on the
relative performance of student teams were assessed in a controlled class experiment followed by statistical
analysis. While tests using real life settings are highly desirable, the systematic control of experimental
data makes it easier to identify situations and determine when a particular method can be expected to
perform better than another. The experiment described here was plausible because it was carried out with
graduate business students recruited from a Business Policy class, and they participated in the study in
partial fulfillment of a course requirement.
Experimental Design
The purpose of the experiment was three-fold. First, to test if prominent cognitive simplification
processes can substantially influence the performance of student teams. That is, to test whether the
differences in the performance of teams that encountered a particular set of cognitive biases was
significantly different from the performance of teams that did not experience that set. Second, to examine
if the performance of student teams that used Scenario-driven Strategic Information Mapping (S-dSIM)
was significantly different from the performance of teams that did not rely on such processes in currying
out their decision-making task. Third, to test for possible interaction effects between prominent cognitive
biases and the use (or not) of S-dSIM on the relative performance of the 24 teams.
The task was a business strategy simulation package in which each company was managed by a team of
four to five students (Smith & Golden, 1989). A total of 118 graduate business students from a university
in New York City participated in the experiment. The participants were assigned to a total of 24 teams,
each a multi-divisional firm competing in the dynamic information systems industry. There were two
levels of decision making: each team had to make decisions for both the corporation as a whole and each of
its three strategic business units (SBUs). Each team had to make pricing, marketing, research and
development (R&D), human resources, and capacity decisions for each SBU. The corporation decisions
entailed acquiring debt (loans and bonds) and equity financing, paying dividends, buying and selling
business units, acquiring new ventures, determining types of needed market research, and responding to
management incidents. Throughout the simulation teams were expected to establish objectives, formulate
strategy, and make the required decisions dictated by their plans. These decisions were submitted
periodically and they were input into the computer, which produced a report for each team concerning the
firm's sales and profits.
Before they assumed their decision-making roles, the students were exposed to the uaiege management
models in Hatten & Hatten (1987) and in Smith & Golden (1989), the components of the S-dSIM
problem-forming method, and Schwenk's (1984) work on cognitive biases CB1 through CB9. During
this phase of preliminary training, which lasted eight weeks, eight case studies from the Hatten & Hatten
text were used to demonstrate, discuss and clarify the differences between cognitive biases CB1 through
CB2, the intricacies and application of strategic management models, and the S-dSIM components. These
training sessions were closely monitored and controlled so that participants did not voice their judgement
or preference on any of the models, problem-forming components, or cognitive biases involved.
To reinforce learning through feedback (Kopelman, 1986; Locke Cartledge & Koeppel, 1968), each
team had to compose and submit structured reports on two of the eight cases covered. Each report
contained three interrelated sections: an executive summary of the team's work with recommendations, a
System Dynamics '90 415
problem-forming section (with claim and implications, data, warrant, and safety valve), and a section
which diagnosed cognitive biases encountered by the team in their analysis of each case. Before applying
S-dSIM to the cases in the text, the students were first taught how to interact with comprehensive situation
mapping (CSM), so they could generate their own causal graph of a strategic decision situation. The CSM
mechanics were demonstrated through "John Farmer", an up-to-date mini-case which is widely used in
American colleges and universities (Alvarez, 1980). Moreover, each student had to apply CSM and run
strategic change scenarios to show the implications of his/her claim on the nature and structure of the
"Baskin Robbins" problematic, another mini-case adapted from Bovée & Thill (1989: 221).
The purpose of all these pre-test exercises was to create homogeneous groups in terms of their ability to:
use strategic management models in case analysis, implement the S-dSIM problem-forming method, and
diagnose cognitive biases CB1 through CB9 when encountered. Thus, the subjects were well informed
and had some experience with complex decision situations prior to their participation to the business
strategy simulation of Smith & Golden (1989). Throughout the pre-test exercises students were able to
learn from their mistakes and improve their performance because they were required to make numerous
conjectures on which they received outcome and process feedback soon after they submitted their work.
During the simulation experiment students were also able to learn from their mistakes and improve their
performance because they were required to make numerous predictions based on more clearly identified
data and received outcome feedback soon after the submission of their decisions, this time from the
computer. The one difference from the pre-test exercises was that the teams could now choose whether
they would use the S-dSIM components or not in competing against other teams. It was made clear to the
participants that they could treat the simulation experiment either as an opportunity for the continuous
practice of S-dSIM, or as the unique opportunity to simply get a "hands on" experience with manipulating
Strategic variables in a dynamic setting. Based on the guidelines by Smith & Golden (1989), the end
results of the simulation experiment could not affect anyone's course grade by more than five percent.
After six rounds of the simulation, each student was asked to complete a final questionnaire assessing
prominent cognitive biases encountered by his/her team during the simulation experiment, and the team's
emphasis on S-dSIM components. After completing the questionnaire, subjects were debriefed on the
simulation experiment and thanked for their participation.
This design is a variation on the classic post-test-only control group design (Campbell & Stanley, 1963).
‘Teams that rated a cognitive bias as not prominent (CBi=1, i=1,...,9) provided a control against which the
performance of teams that rated the same cognitive bias as most prominent (CBi=2) could be contrasted.
In addition, teams that chose not to use the S-dSIM problem-forming method (PROBFORM=1) provided
a control against which the performance of Scenario-driven Strategic Information Mapping users
(PROBFORM=2) could be contrasted. Random assignment to groups and the relatively short duration of
the experiment controlled for primary threats to validity, such as maturation, regression toward the mean,
and mortality, but not selection bias.
A controversy exists among information comprehension researchers with respect to individual
characteristics that may cause selection bias. Clark (1975), Kolb (1974), and Levie & Lentz (1982)
suggest that factors such as age, experience, education, visual literacy, verbal ability, and learning style
can have a profound impact upon message reception and processing. They point out that individual
differences may compound the difficulty of understanding the general effect of a causal graph (such as
CSM) upon message reception: a message transmitted by symbols and software used for mapping and
model structuring may be redundant or incongruous for one person but not so for another. Allen (1978),
Freedman & Stumpf (1980), Smeltzer & Vance (1989), Stumpf & Freedman (1981), and Wexley (1984)
find a lack of interaction effects between information processing and individual differences. In this study,
each team had to diagnose which cognitive biases it encountered, and had to decide whether or not to use
the S-dSIM problem-forming method. Thus, the proposed interaction effect between information
processing and individual differences could not be ignored.
Subject Characteristics
The characteristics of the 118 participants are given in Table 1. The sample is biased in favor of part-time,
male students, majoring in Finance, and working full-time in middle management positions. The brain
orientation of the subjects was assessed using Raudsepp's (1981) left brain/right brain inventory. This
inventory determines whether subjects are right brain oriented, left brain oriented, or use both hemispheres
when dealing with facts, ideas and issues. Based on the inventory, the sample is biased in favor of double
416 System Dynamics ‘90
dominant students who are capable of both processing verbal and numerical information sequentially in a
linear fashion, and of grasping complex images of holistic relational configurations and structures.
Table 1. Student characteristics (n=118)
Standard
Characteristic Frequency Percentage Mean Deviation
age _ 28.522 4.818
Brain Orientation
Left 40 33.9%
Double Dominant 59 50.0
Right 19 16.1
GMAT Score 571.296 75.950
Gender
Female 35 29.7%
Male 83 70.3
Job Title
Full-time student 19 16.1%
First-line Supervisor 18 15.3
Engineer 7 5.9
operator 6 5.1
Technician 10 8.5
Middle Manager 54 45.8
Upper Manager 4 3.4
Major
Accounting 1. 9.38
Finance 75 63.6
Information Systems 4 33
Management Systems 8 6.8
Marketing 20 16.9
Managerial Problem Solving Styles
Information Gathering Method (Sensation-Intuition) -8.137 18.414
Information Evaluation Method (Feeling-Thinking) 4.814 16.653
Social Desirability (SD) Index (0-4) 1.661 1.023
Work Experience 6.475 4.006
The managerial problem-solving styles of the participants were assessed using the Myers-Briggs Type
Indicator test (Myers & McCaulley, 1985). This test has been validated with a large number of studies and
provides reasonably accurate scores that remain relatively constant for some time (Hellriegel, Slocum &
Woodman, 1989: 91-92). It determines a subject's orientation in terms of four psychological functions
involved in information gathering and evaluation: sensation, intuition, thinking, and feeling. According to
Jung (1923), individuals gather information either by sensation (S) or intuition (N), but not by both
simultaneously. These two functions represent the orientation extremes in information gathering.
Similarly, the feeling (F) and thinking (T) functions represent the orientation extremes in evaluating
information. Each individual's dominant function is normally backed up by one (and only one) of the
functions from the other set of paired opposites. Though the sensation-thinking (ST) combination
characterizes best the people in today's Western industrialized societies, Jung also believed that individuals
tend to move toward a balance, or integration of the four psychological functions. With a few notable
exceptions, and a slight but expected bias toward the sensation-thinking (ST) type, the study's subjects
were well balanced among the four psychological functions (Figure 2).
System Dynamics '90 417
MANAGERIAL PROBLEM-SOLVING STYLES
Thinking then nn pen nathan nn then eto nto nn nto nnn ten nate neato nen te ncn ten nc tt
60+ sT * | NT +
| | |
| | |
| | |
| * | |
40+ | * +
| * * *] os |
| * * \* |
| * a | |
| | |
20+ + oe * kok ke +
| te ee ** * * |
| * es |
| * ween | ake |
Information | * Pe ee ae a ees * |
Evaluation +
Method |
|
| |
* |
| +
| | * |
| | |
| | |
| * | |
~40+ | +
| | i |
| | |
| | |
| | |
-60+ SF | NF +
Feeling Fon nnn n ita nto nnn ten nn tenn tenn nto nn nto nnn tenet enn nto nett
-60 -40 -20 0 20 40 60
Sensation Information Gathering Method Intuition
Fig. 2. The managerial problem-solving styles grid of the 118 graduate business students.
In addition to the above assessments of individual differences, a small set of items measuring social
desirability (SD) was included in the study. The purpose of these items was to test for any possible
selection bias effects owing to the SD set. Included in the survey were the following four SD items taken
from Smith (1967: 91): 1) Do you like everyone you know? 2) Have you envied the good luck of others at
times? 3) Have you taken advantage of someone at times? 4) Have you ever felt you were being punished
without justification? It is assumed for these items that the "true" response is known in each case: that
everyone dislikes at least some other people (s)he knows, that everyone has envied or taken advantage of
another person, and that every-one has felt unjustly punished at some time or another. Moreover, it is
assumed that the reason for not admitting the truth of these items is a tendency to make oneself appear
more socially acceptable than one is in fact, or a reluctance to admit negatively evaluated facts about
oneself. Thus, the number of items on which an individual fails to admit the "truth" is taken as an index of
his or her social desirability set. The average score of the 118 individuals in the study's sample is shown
in Table 1.
Experimental Analysis
The purpose of the statistical investigation was also three-fold. First, to assess the impact of cognitive
biases CB1 through CB9 on team performance. That is, to test whether the performance of teams that
rated a cognitive bias as not prominent (CBi=1, i=1,...,9) was significantly different from the performance
418 System Dynamics '90
of teams that rated the same cognitive bias as most prominent (CBi=2). Second, to asses the impact of the
Scenario-driven Strategic Information Mapping on performance. That is, to test whether the performance
of teams that chose not to use the S-dSIM problem-forming method (PROBFORM=1) was significantly
different from the performance of teams that used S-dSIM (PROBFORM=2). Third, to test for possible
interaction effects between the cognitive biases CB1 through CB9 and the use (or not) of S-dSIM on the
relative performance of the 24 teams.
The performance points on the administrator's report at the end of the sixth trial period represented the
relative performance (PERFORM) of each team. This was the team's common stock price multiplied by
10 and rounded up to be used for ranking. According to Smith & Golden (1988), the stock market price
in their simulation is indicative of a team's operating profits, but does not overly reflect short-term gains
from the disposal of assets (viz., the sale of an SBU). Other performance measures suggested by Smith &
Golden are return on investment (ROD), and return on sales (ROS): the ratio of net operating profits (NET)
to sales volume (SALES). Based on their suggestions, the impact of cognitive biases CB1 through CB9
and S-dSIM on the performance of the 24 teams was assessed on the criterion values of five dependent
variables: SALES, NET, ROS, ROI, and PERFORM (relative performance points).
The study's participants used nine-point Likert-type scales to evaluate the prominence of CB1 through
CBS, and their teams' emphasis on S-dSIM during the simulation experiment. It turns out that nine-point
scales give a more accurate representation of the way in which people think and compare similar elements
(Saaty, 1977). In general, qualitative distinctions are meaningful in practice and have an element of
precision when the items being compared are of the same order of magnitude or close together with regard
to the property used to make a comparison. If this last condition is satisfied, and if the items are slightly
different from each other, then the psychological limit of 7+2 items in a simultaneous comparison suggests
that nine points are needed to distinguish these differences (Miller, 1956).
Although the prominence of each cognitive bias was assessed on the basis of a single-item scale (1 = not
prominent, 9 = most prominent), PROBFORM was a three-item scale with a Cronbach's a=0.7257. The
three items questioned each team's emphasis on: strategic assumption surfacing and testing (SAST),
structuring strategic decision situations through CSM, and running strategic change scenarios (1 = no
emphasis, 9 = most emphasis). The average inter-item correlation for the three scale items was 0.1257.
By breaking the teams’ average responses at the mean value of each scale, it was possible to distinguish
between teams that rated a cognitive bias as not prominent (CBi=1, i=1,...,9) and teams that rated the
same cognitive bias as most prominent (CBi=2), and between teams that chose not to use the S-dSIM
problem-forming method (PROBFORM=1) and teams that used S-dSIM (PROBFORM=2).
Although the experiments were properly designed and conducted with randomization procedures
throughout, the experimental data obtained from the 24 student groups had to be checked for possible
confounding effects. In order to remove any extraneous variation from the dependent variables, to
increase measurement precision, and based on the three-stage classification of Schwenk (1984), a series of
analysis of covariance (ANCOVA) designs was used. In ANCOVA designs, the term covariate is used to
designate a metric independent variable (viz., age, GMAT score, work experience, etc.), and the term
factor is used to designate a nonmetric, categorical independent variable (viz., brain orientation, gender,
major, etc.), as in the simpler analysis of variance (ANOVA) context. In this study, covariate and factor
effects were of equal interest without any priority established between them. Thus, the decomposition of
explained variance in the dependent variables (SALES, NET, ROS, ROI, and PERFORM) was quite
similar to a regression analysis involving both metric and dummy variables as predictors.
Thus, the statistical tests were based on the variability of these five dependent variables and its respective
decomposition. In this context, three classes of hypotheses were tested at the conservative level of
significance =0.05. The three classes of hypotheses are listed as follows:
Hi: There is no difference in the average performance (measured in SALES, NET, ROS, ROI, or
PERFORM) of teams that rated a cognitive bias as not prominent (CBi=1, i=1,...,9), and teams
that rated the same cognitive bias as most prominent (CBi=2).
H: There is no difference in the average performance (measured in SALES, NET, ROS, ROI, or
PERFORM) of teams that did not emphasize the S-dSIM problem forming method
(PROBFORM=1), and teams that emphasized S-dSIM (PROBFORM=2).
H3: The interaction effects of the cognitive biases CB1 through CB9, and the interaction effects of
cognitive biases and PROBFORM on the average performance of the 24 teams are all equal to
zero.
The SPSSX sub-program MANOVA (Notusis, 1985) was used to implement the series of analysis of
covariance designs, and to contrast those criteria for which a significance difference in the means of the
System Dynamics '90 419
student groups was determined by analysis of variance. The ANCOVA designs allowed controlling for
possible confounding effects (viz., the GMAT score and other individual differences). The contrasts
resulted in sets of one-tailed t-tests. It must be noted that contrasts should be applied only if significant
differences in group means have been demonstrated by analysis of variance (Neter, Wasserman & Kutner,
1985). If the null hypothesis of equal means cannot be rejected by analysis of variance, and contrasts are
applied to the group means, spurious significant contrasts may occur. Moreover, the tests of the means
could not have been made by using simple t-tests; by using the method of contrasts, the (1-a.)100 percent
level of confidence (95%) was maintained for each set of contrasts by increasing the level of confidence
for each contrast in the set.
Results
The primary focus of the investigation was to test for any significant differences in the profiles of the five
performance criteria: SALES, NET, ROS, ROI, and PERFORM. Upon completion of the simulation
experiment, the mean values assigned to these criteria (dependent) variables for each team, clearly
constituted an experimental data set with unequal samples for which the random assignment method was
used (rather than random sampling). SPSSX caught the linear dependence of ROS on NET and SALES
and indicated that multivariate tests should not be used. The preliminary univariate F-tests with df=23/94
revealed significant differences (p<0.0001) between the 24 teams in terms of SALES, NET, ROS, ROI,
and PERFORM. These preliminary tests were also used to control for possible confounding effects due to
the individual difference covariates and factors listed in Table 1.
Following the procedure outlined in Smith (1967: 89), the Pearsonian product-moment correlation
coefficients were calculated between each individual's total SD index score and each opinion item in the
questionnaire. The resulting set of correlation coefficients was then scanned to see if any of them were
significantly related at the 0.01 or 0.05 levels of probability. No items satisfying either condition were
found. As a result, it. was assumed that the SD set had no significant influence on the student responses to
any of those items and hence no corrections for SD were necessary. Similarly, the preliminary analyses
did not reveal either any significant main effects or any significant interactions between the information
processing covariates and the individual difference factors of Table 1. Based on the lack of main and/or
interaction effects between information processing and individual differences, the independent variates and
factors of Table 1 were excluded from subsequent analyses.
Table 2. Significant results of analysis of variance (@ = 0.05).
Dependent Covariates Computed Significance
Variable and Factors dtf* F-ratio of F-ratio R?
SALES cB9 1/22 6.509 0.018 0.228
NET PROBFORM 1/22 4.665 0.042 0.175
ROS PROBFORM 1/22 5.530 0.028 0.201
PERFORM PROBFORM 1/22 4.486 0.046 0.169
CB3*PROBFORM 1/20 5.009 0.037 0.392
PROBFORM 1/20 6.844 0.017
cB3 1/16 8.093 0.012 0.563
CB1*CB3 1/16 9.479 0.007
NET 1/15 6.021 0.021 0.670
+ df = 1/22 = univariate design; df = 1/20 > two-factor ANOVA design; df= 1/16 = three-factor
ANOVA design; and df = 1/15 = three-factor ANCOVA design. These factorial designs were based on
the three-stage decision-making classification of Schwenk (1984).
420 System Dynamics '90
Table 2 shows the significant results of the ANCOVA designs used to test the classes of hypotheses H,
through H3. With the level of significance set at a=0.05, the F-ratio value in the table indicates that there
is a significant difference in SALES between teams that rated devaluation of partially described alternatives
as not prominent (CB9=1) and teams that rated the same cognitive bias as most prominent (CB9=2). The
corresponding one-tailed contrast verified that the average SALES of teams that encountered devaluation
of partially described alternatives was significantly lower than teams that did not. Similarly, the F-ratio
value in the table indicates that there is a significant difference in PERFORM between teams that rated
reasoning by analogy as not prominent (CB3=1) and teams that rated the same cognitive bias as most
prominent (CB3=2). The corresponding one-tailed contrast verified that the performance points of teams
that encountered reasoning by analogy were significantly less than teams that did not.
The F-ratio values in Table 2 show that the use of the S-dSIM problem-forming method caused
significant differences in the performance of teams in terms of NET, ROS, and PERFORM. The
corresponding significance values attest to the stability of the univariate factorial designs. Subsequent
one-tailed contrasts indicated that teams that emphasized Scenario-driven Strategic Information Mappin
(PROBFORM=2) performed much better than teams that did not (PROBFORM=1) in terms of net profit
(NET), return on sales (ROS), and performance points (PERFORM). The sharp positive slopes of the
dummy regression lines in Figures 3(b), 3(c), and 3(d) demonstrate the positive effect of S-dSIM on the
average performance of teams in terms of NET, ROS, and PERFORM, respectively.
1000000 ; (a) : 50000 + (b)
‘ .
900000
Cy e
s . 0 |
A n
A 800000 $ N ! :
E ’ = 7 n
§ 700000 ' :
-50000
600000 ) .
‘
500000 -100000
ry 2 1 2
PROBFORM PROBFORM
SALES=519151.643+121205.858*PROBFORM NET=-43956 .809+21508.164*PROBFORM
0.1 (ec) 6007 (a) ry
°
P
0.0 i z
* R
R $ F
° fe)
8 ‘ R
* M
-0.1 ’
-0.2 Ente ene: | 0
1 2 1 2
PROBFORM PROBFORM
ROS=~-0.073+0.036*PROBFORM PERFORM=114.767+113.378*PROBFORM
Fig. 3. Effects of S-dSIM (PROBFORM=2) on (a) SALES, (by NET, (6) ROS, and @) PERFORM.
System Dynamics ‘90 421
Perhaps the most interesting among the results of Table 2 are the interaction effects of: (a) prior
hypothesis bias and adjustment & anchoring and reasoning by analogy (CB1*CB3), and (b) reasoning by
analogy and emphasis on the S-dSIM (CB3*PROBFORM), on the performance points of teams
PERFORM). The CB1*CB3 interaction effect on PERFORM was found significant in a three-factor
OVA design, where 56.30% of the variability in performance points (PERFORM) was accounted for.
This strong interaction effect on PERFORM is shown by the mean curve plots of Figure 4(a). The figure
attests to the fact that, overall, teams that did nor encounter either CB1 or CB3 were the best performers.
Among teams that did not reason by analogy (CB3=1), the performance of teams with prominent prior
hypothesis bias and adjustment & anchoring (CB1=2) was inferior to the performance of teams without
(CBi=1). However, among teams that reasoned by analogy (CB3=2), the performance of teams with
prominent prior hypothesis bias and adjustment & anchoring (CB1=2) was superior to the performance of
teams without (CB1=1).
(a) 600 -
500 -
400 -
300 - cBl = 2
BWM ne
cB
4
e
v
(b) 600 -
cB3
EpPOmM mY
»
cB3
a
N
°
N
PROBFORM
Fig. 4. The interaction effects of (a) CB1*CB3, and (b) CB3*PROBFORM on PERFORM.
The CB3*PROBFORM interaction effect on PERFORM was found significant in a two-factor ANOVA
design, where 39.20% of the variability in performance points was accounted for. This strong interaction
effect on PERFORM is shown by the mean curve plot of Figure 4(b). The figure attests to the fact that,
overall, teams that emphasized the S-dSIM problem-forming method (PROBFORM=2) performed better
than participants who chose not to use Scenario-driven Strategic Information Mapping (PROBFORM=1).
Among teams that used the S-dSIM problem-forming method (PROBFORM=2), those that did not reason
by analogy (CB3=1) performed better than those that did (CB3=2). Conversely, among teams without
SM (PROBFORM=1), those that reasoned by analogy (CB3=2) performed better than teams that did
not (CB3=1).
422 System Dynamics '90
Conclusion
The study examined both the impact of nine cognitive biases (CB1 through CB9) and the S-dSIM
problem-forming method on the relative performance of 118 graduate business students who worked
under the experimental conditions of a simulated strategic context. Randomly assigned to twenty-four
teams, the subjects run international conglomerates with multiple actors, feedback loops, non-linearities,
and time lags and delays. The interaction, expectations, choices and teams' model selection produced
results that systematically diverged over time. Within a crossed factorial design, these results support the
hypothesis that cognitive biases interact with strategic management models to influence performance. Poor
performers chose heuristics that reinforced their cognitive limits and bounds. Conversely, good
performers constructed models which helped them recognize and overcome the negative effects of
cognitive simplification processes. They produced effective decisions, not by optimizing functions, but
through searching for recognizable patterns when they received feedback.
The results of the study indicate that in the absence of a problem-forming method, reasoning by analogy
is at least one process that can help define a problematic. Yet decision makers who reason by analogy in a
complex and dynamic environment will not perform as well as those who don't. According to Dijkstra,
one of the characteristics of the Middle Ages was that reasoning by analogy was rampant, and "...b
developing a keen ear for unwarranted analogies, one can detect a lot of medieval thinking today..." (1989:
1399).
The data also indicate that decision makers who use problem-forming methods of the S-dSIM variety,
and who work with causal graphs (ala CSM) should outperform those who don't. The S-dSIM problem-
forming method helps build consensus for action. Its components lead to a negotiated perception of a
situation that decision makers in a group can live with. Yet S-dSIM is likely to correct for cognitive
simplification processes and the development of "groupthink" through its safety valve that unearths and
challenges critical and divergent assumptions. CSM can be used individually or collectively as a desk-top
tool to map the web of interrelationships bearing on a strategic decision situation or context. Its purpose is
to make the dynamics of the interrelationships more visible, more explicit, and thus more comprehensible.
As a sophisticated extension of ID, CSM allows representing and distinguishing between "necessary" and
"sufficient" causal relationships. CSM possesses modeling flexibility allied with computational capability
that allow capturing the causalities involved in the transmission of change and presenting them to higher
authorities (Acar et al., 1985).
The study's findings add insights to the effect of text illustrations on information processing. The first
guideline in the Levie & Lentz (1982) review was that pictorial embellishments will not necessarily
enhance the comprehension of information. However, the results of this study suggest that causal graphs
can enhance the comprehension of information presented in a business decision situation. The substantial
improvements in the performance of teams that emphasized the S-dSIM problem-forming method signify
that problem-forming with causal graphing, scenarios, and strategic assumption surfacing and testing can
have a significant effect on strategic performance. It should be noted that this research is culturally specific
to the Northeast region of the United States. Generally, the use of graphics and visuals differ from one
culture to another. Japanese business people, for example, are much more accustomed to using graphics
in their business environment than are business people in the United States (Gritzmacher, 1987; Smeltzer
& Vance, 1989).
Finally, the study's findings support the research of Allen (1978), Freedman & Stumpf (1980), Smeltzer
& Vance (1989), Stumpf & Freedman (1981), and Wexley (1984) who point out the lack of interaction
effects between individual differences and information processing. Controlling for individual differences
did not alter the significance levels of the computed F-ratio values, which signifies a lack of any
confounding effects attributable to individual differences.
System Dynamics '90 423
References
Acar, W. 1983. Toward a Theory of Problem Formulation and the Planning of Change: Causal Mapping
and Dialectical Debate in Situation Formulation. Ann Arbor, MI: UMI.
Acar, W; Chaganti, R. & Joglekar, P. 1985. Models of Strategy Formulation: The Content-focused and
Process- focused Modes Can and Must Meet! American Business Review 2(2): 1-9.
Allen, TH. 1978. New Method in Social Science Research. New York, NY: Praeger.
Alvarez, J.A. 1980. The Elements of Technical Writing. New York, NY: H.B. Jovanovich.
Bovée, C.L. & Thill, J.V. 1989. Business Communications Today, (2nd ed.). New York, NY: Random
House.
Brightman, HJ. 1987. Toward Teaching Excellence in the Decision Sciences. Decision Sciences 18(4):
646-661.
Campbell, D.T. & Stanley, J.C. 1963. Experimental and Quasi-experimental Designs for Research.
Boston, MA: Houghton Mifflin.
Christensen, C.R.; Andrews, K.R. Bower, J.L.; Hamermesh, R.G. & Porter, M.E. 1987. Business
Policy: Text and Cases, (6th ed.). Homewood, IL: Irwin.
Clark, R.E. 1975. Constructing a Taxonomy of Media Attributes for Research Purposes. AV.
Communication Review 23: 197-215.
Coleman, J.S. 1987. Psychological Structure and Social Structure in Economic Models. In R. Hogarth &
M. Reder (Eds.), Rational Choice: The Contrast between Economics and Psychology. Chicago, IL: The
University of Chicago Press.
Copeland, M.T. 1958. And Mark an Era. Boston, MA: Little, Brown & Co. In particular Chapter IX:
"The Case Method of Instruction".
De Geus, A.P. 1988. Planning as Learning. Harvard Business Review 66(2): 70-74.
Dieta J. 1982. Influence Diagrams for Complex Strategic Issues. Strategic Management Journal 3:
133-146,
Dijkstra, E.W. 1989. On the Cruelty of Really Teaching Computing Science. Communications of the
ACM 32(12): 1398-1404.
Emshoff, J.R.; Mitroff, II. & Kilman, R.H. 1978. The Role of Idealization in Long Range Planning: An
Essay on the Logical and Socio Emotional Aspects of Planning. Technological Forecasting and Social
Change 11: 335-348,
Freedman, R.D. & Stumpf, S. 1980. Learning Style Inventory: Less than Meets the Eye. Academy of
Management Review 5: 445-447.
Gritzmacher, K. 1987. Visual Control Tools: A Hidden Productivity Factor. National Productivity Review
(Autumn): 314-323.
Hatten, KJ. & Hatten, M.L. 1987. Strategic Management: Analysis and Action. Englewood Cliffs, NJ:
Prentice-Hall.
Hellriegel, D.; Slocum, J.WJr. & Woodman, R.W. 1989. Organizational Behavior, (5th ed.). St. Paul,
MN: West Publishing Co., pp. 90-97.
Hogarth, R.M. 1980. Judgement and Choice: The Psychology of Decision. Chichester, UK: Wiley.
Hogarth, R.M. & Makridakis, S. 1981. Forecasting and Planning: An Evaluation. Management Science
27; 115-138.
Jung, C.G. 1923. Psychological Types. London, GB: Routledge and Kegan Paul.
Kolb, D.A. 1974. On Management and the Learning Process. In D.A. Kolb, I.M. Rubin & J.M. Mcintyre
(Eds.), Organizational Psychology: A Book of Readings. Englewood Cliffs, NJ: Prentice-Hall.
Kopelman, R.E. 1986. Managing Productivity in Organizations: A Practical, People-oriented Perspective.
New York, NY: McGraw-Hill.
Levie, WH. & Lentz, G.R. 1982. Effects of Text Illustrations: A Review of Research. ECJT 30(4):
195-232.
Lindblom, C.E. 1959. The Science of Muddling Through. Public Administration Review. 19: 79-88.
Locke, E.A.; Cartledge, N. & Koeppel, J. 1968. Motivational Effects of Knowledge of Results: A Goal-
setting Phenomenon? Psychological Bulletin 70: 474-485.
Maruyama, M. 1963. The Second Cybernetics: Deviation-amplifying Mutual Causal Processes. American
Scientist 51: 164-179 & 250-256.
Mason, R.O. & Mitroff, LI. 1981. Challenging Strategic Planning Assumptions. New York, NY: Wiley.
Michael, D. 1973. On Learning to Plan and Planning to Learn. San Francisco, CA: Jossey-Bass.
424 System Dynamics ‘90
Miller, G.A. 1956. The Magical Number Seven plus or minus Two: Some Limits on Our Capacity for
Processing Information. Psychological Review 63(2): 81-97.
Mintzberg, H.; Raisinghani, P. & Thedret, A. 1976. The Structure of ‘Unstructured' Decision Processes.
Administrative Science Quarterly 21: 246-275.
Mitroff, LI. & Emshoff, J.R. 1979. On Strategic Assumption Making: A Dialectical Approach to Policy
and Planning. Academy of Management Review 4(1): 1-12.
Morecroft, J.D.W. 1988. System Dynamics and Microworlds for Policymakers. European Journal of
Operational Research 35: 310-320.
Myers, I.B. & McCaulley, M.H. 1985. A Guide to the Development and Use of the Myers-Briggs Type
Indicator. Palo Alto, CA: Consulting Psychology Press.
Neier | Wasserman, W. & Kutner, M. 1985. Applied Linear Statistical Models, (2nd ed.). Homewood,
i Irwin.
Norusis, M.J. 1985. SPSS - Advanced Statistics Guide. Chicago, IL: SPSS, Inc.
Nutt, RC. 1984. Types of Organizational Decision Processes. Administrative Science Quarterly 29(3):
414-450,
Papert, S. 1980. Mindstorms. New York, NY: Basic Books.
Peters, TJ. & Waterman, R.H.Jr. 1982. In Search of Excellence: Lessons from America's Best-run
Companies. New York, NY: Harper & Row.
Quinn, J.B. 1980. Strategies for Change: Logical Incrementalism. Homewood, IL: Irwin.
Rapoport, A. 1967. Escape from Paradox. Scientific American 217(1): 50-56.
Raudsepp, E. 1981. How Creative Are You? New York, NY: G.P. Putman, pp. 15-20.
Saaty, TL. 1977. A Scaling Method for Priorities in Hierarchical Structures. Journal of Mathematical
Psychology 15(3): 234-281.
Schwenk, C.R. 1984. Cognitive Simplification Processes in Strategic Decision-making. Strategic
Management Journal 5: 111-128.
Schwenk, C.R. 1982. Why Sacrifice Rigor for Relevance? A Proposal for Combining Laboratory and
Field Research in Strategic Management. Strategic Management Journal 3: 213-225.
Simon, H.A. 1984. The Behavioral and Rational Foundations of Economic Dynamics. Journal of
Economic Behavior and Organization 5: 35-55.
Simon, H.A. 1976. Administrative Behavior, (3rd ed.). New York, NY: The Free Press.
Slovic, P; Fischhoff, B. & Lichtenstein, S. 1977. Behavioral Decision Theory. Annual Review of
Psychology 28: 1-39.
Smeltzer, L.Z. & Vance, C.M. 1989. An Analysis of Graphic Use in Audio-graphic Teleconferences. The
Journal of Business Communication 26(2): 123-141.
Smith, D.H. 1967. Correcting for Social Desirability Response Sets in Opinion-Attitude Survey Research.
Public Opinion Quarterly 31(1): 89-94.
Smith, J.R. & Golden, PA. 1989. Corporation: A Business Strategy Simulation. Englewood Cliffs, NJ:
Prentice-Hall.
Smith, J.R. & Golden, PA. 1988. Corporation: A Business Strategy Simulation - Instructor's Manual.
Englewood Cliffs, NJ: Prentice-Hall.
Sterman, J.D. 1989. Modeling Managerial Behavior: Misperceptions of Feedback in a Dynamic Decision
Making Experiment. Management Science 35(4): 321-339.
Stumpf, S. & Freedman, R.D. 1981. The Learning Style Inventory: Still Less than Meets the Eye.
Academy of Management Review 6: 297-299.
Taylor, R.N. 1975. Psychological Determinants of Bounded Rationality: Implications for Decision-
making. Decision Sciences 6: 409-429.
Tversky, A. & Kahneman, D. 1974. Judgement under Uncertainty: Heuristics and Biases. Science 185:
1124-1131.
Ungson, G.R.; Braunstein, D.N. & Hall, PD. 1981. Managerial Information Processing: A Research
Review. Administrative Science Quarterly 26: 116-134.
Van de Ven, A.H. & Delbecq, A.L. The Effectiveness of Nominal, Delphi, and Interacting Group
Decision-making Process. Academy of Management Journal 17: 605-621.
Weick, K. 1979. The Social Psychology of Organizing, (2nd ed.). Reading, MA: Addison-Wesley.
Wexley, K.N. 1984. Personnel Training. In M.R. Rosenzweig, & L.P. Porter (Eds.), Annual Review of
Psychology. Palo Alto, CA: Annual Reviews, Inc.
Winkler, R.L. & Murphy, A.H. 1973. Experiments in the Laboratory and the Real World. Organizational
Behavior and Human Performance 10: 252-270.
