Eliciting Group Knowledge for Model-Building
George P. Richardson!
Jac AM. Vennix
David F. Andersen
John Rohrbaugh
W.A. Wallace
Abstract
System dynamics models are typically created using multiple streams of information including
quantitative data, written records, and information contained in the mental models of both
individuals and groups. While qualitative sources of information are widely recognized as
important in all stages of the model building process, little systematic research has been
completed on how best to elicit and map this knowledge, In this paper, we survey the existing
literature on mapping and eliciting knowledge for system dynamics modeling and also explore the
literature in the broader fields of cognitive psychology and small group processes. Special
attention is paid to new software advances to support these processes.2
The Problem
System dynamics modelers typically rely on multiple, diverse streams of information to create
and calibrate model structure. Such streams include quantitative data, written records, and
information contained within the mental models of key actors in a system. This last class of
information is typically ‘most helpful to set the system boundary, define the dynamic hypothesis,
postulate detailed structure, and calibrate system parameters.
Commonly, the techniques for drawing out germane and accurate information are informal
and highly intuitive. Accessing the most productive source of information for model- building,
the minds of experts and actors in the system, is largely an art in our field. Rarely does the
academic preparation of modelers include training or exposure to academic literature that helps to
! David Andersen, George Richardson, and John Rohrbaugh are focated at the University at Albany, State
University of New York, in the Rockefeller College of Public Affairs and Policy. Jac Vennix is at the State
University of Utrecht and W. A. Wallace at Rensselaer Polytechnic Institute in Troy, New York.
2 A more extensive version of this work, entitled "Processes for Eliciting and Mapping Knowledge for
Mode!l-Building,” will be presented at the mini-conference on Computer-Based Learning Environments for
Business and Social Systems, London Business School, July 1989, and is available from the authors,
344
build formal skills in eliciting information for model building. Rarely do practitioners have the
time to experiment with different approaches for mapping knowledge. But practitioners know
that the arts of knowledge elicitation and mapping are subtle, and can be particularly complex
when the modeling process cafls for drawing information out of groups of people rather than
individuals.
Yet in other fields more or less related to system dynamics, there already exists considerable
literature that casts light on the modeler's information-gathering task. And increasingly, a
number of system dynamics practitioners have begun to explore variations on the intuitive model
development process described in our literature. This paper explores these developments in an
effort to push forward our understandings of productive processes for eliciting knowledge that
helps define problems, conceptualize structure, and develop formal modets for policy analysis.
To establish a common ground for the discussion, we first review the existing system dynamics
literature on knowledge elicitation and mapping, with special attention to small group processes.
Next we describe several taxonomies for organizing the rest of our discussion, including
distinctions among types of cognitive tasks in model-building and kinds of information sought.
Finally, we discuss the various specific techniques that can help to support the knowledge and
elicitation process for constructing system dynamics models.
Existing System Dynamics Literature on Knowledge Elicitation
While system dynamicists have long recognized the importance of eliciting knowledge from
the mental models of individuals and small groups in the model building process, surprisingly
little literature exists describing exactly how expert modelers elicit and map such knowledge.
The Informal Consulting Approach. The textbook approach to system dynamics modeling
typically begins with a six or seven stage process, often with the stages coupled into an iterative
process with repeated cycling among stages. For example, Richardson and Pugh (1981) define
seven stages as problem identification and definition, system conceptualization, model
formulation, analysis of model behavior, model evaluation, policy analysis, and model use or
implementation. Roberts et al (1983) use an almost identical set of six stages to organize their
pedagogical approach.
This well-defined textbook approach to modeling is virtually silent concerning how the
modeler or modeling team elicits knowledge. However, the textbook approach implicitly assumes
that some group of clients, policy elites, decision makers, or the public at large form an audience
for a modeling effort. In fact, precise identification of the model's audience is an important step
in the problem definition stage.
Consultants and others working with clients over the years have evolved very effective
strategies for working with clients, often in small groups, to. insure that client preferences are
well integrated into the model building process. A modest body of literature documents the
experiences of these seasoned modelers in their interactions with decision makers. In his paper
on implementation, Roberts (1977) stresses the importance of working closely with client groups
and posits a series of informal rules for working with them in various stages of the model building
process. Weil (1980, 1983) continues this line of work by developing a more elaborated process
model for involving groups of decision makers in the mode! building process. Forrester and Senge
(1980) propose a number of specific tests that can be performed to increase confidence in a model.
Implicit in their remarks is the notion that some group of people would systematically evaluate
mode! structure and behavior. However, Gardiner and Ford (1980) have demonstrated that
individual decision makers can and do differ sharply in their evaluations of which policies are
better and which are worse. Working on a related problem, Rohrbaugh and Andersen (1983) have
shown that individual preferences or objective functions can lead to dramatically differing
evaluations of a system's performance over time.
All of this literature has in common an implicit call for the modeler to work with groups to
elicit and map knowledge of various sorts--knowledge about problem definition, knowledge about
how much structural detail to include or exclude from a model, knowledge about model evaluation,
and knowledge necessary to evaluate and rank policy options emerging from the model. Yet this
literature stops far short of suggesting productive ways of working with groups to carry out these
tasks.
Reference Groups and Other Structured Group Approaches. A smaller body of published work
suggests some hints as to how the modeler should work with groups of decision makers in eliciting
and structuring these various types of knowledge. Randers (1977) proposed the use of reference
groups to support the model building process. Working primarily with public sector problems, he
suggésted methods for structuring broadly representative groups who will work with a modeling
team through all of the phases of the modeling work sketched above to achieve consensus.
Stenberg (1980) elaborated on Randers’ earlier work and presented a complete process model for
assembling and working with reference groups to support the modeling process. Both of these
works are noteworthy because they begin to get at the “hows” of working with groups to support
the model building process. However, once the group has been assembled and put on to a specific
task, the implicit assumption is that good modelers are also good group process consultants and will
handle a group skillfully in eliciting various types of knowledge necessary to build, test, and
evaluate a model.
Recently, system dynamics modelers such as Richmond (1987, 1988) and Richardson (1988) have
begun to experiment with the reference group approach by using new software products such as
STELLA to get groups of decision makers to interact more directly with a model's structure and
output as the model is being developed. In his work, Richardson had considerable success in
separating the role of the professional modeler, who sat in the back of the room and operated a
STELLA-based model being projected for review by the group, from that of a professional group
facilitator who managed the group. This group facilitator was familiar with system dynamics
modeling but brought generic group facilitation skill rather than system dynamics modeling skill
to the overall group process.
Finally, Vennix (1989) has proposed a significantly more detailed process for working with
relatively large groups in the public sector to build dynamic simulation models. Using a more
fine-grained appreciation of group process, Vennix proposed that different techniques be used to
support different group tasks in the model building process. He carefully designed small group
exercises to match the group process to the exact task facing the group.
Once specific group process models such as these have been proposed for working with smali
groups, researchers can systematically evaluate the “fit" between the specific group technique
proposed and the outcome of that group process. Fortunately a rich body of small group process
literature already exists, much of its focused on using smal! groups to elicit knowledge for formal
models.
Types of Tasks in Eliciting and Mapping Knowledge
The process of constructing a system dynamics model involves a wide variety of conceptual
activities, For example, the process of "brainstorming" variables that may be included or excluded
from the model's boundary is very different from the more detailed task of agreeing upon specific
parameter values, which in turn is very different from the cognitive task of identifying the
important feedback loops within a system
Psychologists specializing in cognitive processes have commonly distinguished between three
general types of tasks: eliciting information, exploring courses of action, and evaluating
situations. Hackman and Morris (1975; Morris 1966; Hackman, 1968) referred to these “intellective”
tasks as production, problem solving, and discussion. Bourne and Battig (1966) labeled similar
"thinking" tasks as conceptual behavior, problem solving, and decision making. Simon (1960)
identified the three principal activities of management with parallel terms--intelligence, design,
and choice--and attributed the trichotomy to Dewey (1910).
Eliciting Iaformation, The creation, generation, or evocation of information results in the
development of a new data base for a group. Such production tasks typically are accomplished as
individuals pool their ideas, insights, or experience. The terms “brainstorming” or “divergent
thinking" have often been applied to some conceptual behavior of this sort. In the system
dynamics model building process, this type of thinking is often most necessary in the problem
definition or model conceptualization phases where an individual or a group is attempting to
determine what factors or variables to include or exclude from a system's boundary, or in the
model evaluation phase where the group is brainstorming how to design or evaluate a model's
performance. In addition, this eliciting process may also be evoked during some phases of the
model! formulation process where several different formulations need to be considered.
There is considerable evidence that work on elicitation tasks in group settings should be
performed by noninteracting, “nominal” groups, rather than with full discussion and exchange of
ideas in an open forum (Lamm and Trommsdorf, 1973), The implication for modelers is that
elements of problem definition and model conceptualization with groups is best accomplished by
eliciting information from individuals and then pooling the results.
Exploring Courses of Action. Solutions to problems are discovered through devising, specifying,
or following combinations of procedures that might achieve specific objectives. Problem solving
within the context of the system dynamics modeling processes involves tasks such as specifying
the feedback paths to be included within a model or devising a specific rate formulation. In
general, problem solving produces the invention or design of multiple alternative explanations
for the functioning (or disfunctioning) of a partially understood system, or the formulation of
answers by diverging as little as possible from seemingly appropriate and well-known rules.
Often referred to as a form of “convergent thinking” such group activity is thought to be at its
best when organized and highly systematized. However, the paucity of rules specifying what
constitutes key information or what is the essential information to be structured typically makes
this type of a task most puzzling to organize for a group. Deep knowledge of the system being
studied and the nature of the model building task at hand is necessary to structure appropriate
group activities. That is, the most critical phases of model conceptualization and formalization will
be very difficult to support with group techniques unless the group is led by a skilled facilitator
with significant understanding of the model building process.
Lvaluating Situations. The most common modes of evaluation are judgment (assessing individuals,
objects, or events one at a time on some scale) and choice (selecting one or more individuals,
objects, or events from a set), In the process of building system dynamics models evaluation
includes tasks such as selecting parameters, assessing the validity of model output, assessing the
performance of various policies, choosing between alternative structural formulations, or
choosing which policies to investigate within the context of model simulations. In both judgment
and choice, evaluation is based on the explicit and/or implicit use of one or more cues that inform
the group in completing its task. Judgment and choice processes do not necessarily lead to the
same conclusions, however. Preferences expressed in one mode may be reversed in the other
(Lichtenstein and Slovic 1971, 1973). Finally, Hammond et al (1977) and Rohrbaugh (1981) have
proposed using specific techniques such a social judgment analysis to support evaluation tasks.
Table 1: Sources of Knowledge in Model Building and Techniques to Elicit that
Knowledge with Relative Advantages (+) and Disadvantages (-)
Sources of Methods and Advantages (+) and
Knowledge Techniques Disadvantages (-)
Written "Content (-) Often not written for the
Documents Analysis purpose of modeling
(+) Tend to be unambiguous
(+) Can be analyzed repeatedly
Individual Interview (+) Can be quite thorough.
{-) No discussion between
members of management team
Questionnaire (-) Stow in comparison to interview
(+) Less time consuming (for
modeler) than interview
Workbook (+) Allows dealing with more
complex models than questionnaire
{-) Respondent cannot ask
questions about unclear issues
Groups Brainstorming (+) Many different ideas can be generated
(+) Much discussion between participants
{-) No better than individual work
for problem solving
Structured (+)Discussion strongly focused and
Workshops structured
(+) Allows many persons to participate
(-) In general, not good for eliciting
Sources of Knowledge and Methods to Extract Them
While different types of cognitive processes are involved in various stages of the model
building process, Forrester (1980) has noted that a wide variety of sources of knowledge must be
incorporated into. the model-building process. These sources of knowledge range from
quantitative data to written documents to the.mental models of both individuals and of groups.
Table 1 presents in summary form the sources of knowledge with which we shall be
concerned here, techniques to be used to elicit that knowledge, and -the advantages and
disadvantages of these various techniques. As shown in Table 1, the range of techniques that can
be used to elicit knowledge from each of these primary sources is quite varied, ranging from
content analysis of written documents to interviews, questionnaires, and workbooks used by
individuals to brainstorming and structured workshops used with groups. All of these approaches
may be supported by special software tools. In the sections below, we review the range of specific
techniques that may be useful in supporting three different types of thinking in the model
building process by small groups, by individuals, and in the review of written documents.
Extracting Knowledge From Written Documents
A number of informal techniques are commonly used by system dynamics model builders to
capture knowledge about system structure and behavior from written documents. However,
several more formal techniques broadly grouped under the heading of content analysis can
support this process. For example, Axelrod (1976) has proposed a series of specific procedures for
creating "cognitive maps" of policy makers by a formal and critical analysis of documents that
they have written. Following strictly specified rules, researchers code written documents in
search of causal connections. When these coded statements are analyzed within a more general
framework, cognitive maps, strongly resembling causal loop diagrams, result. The content coding
rules have been so fully specified that two independent coders will derive very similar cognitive
maps from the same written documents. Axelrod's basic approach can be applied to reconstruct
from public policy documents what Hoogerwerf (1984) calls a “policy theory”, the total set of
assumptions underlying a specific policy. A disadvantage is that policy documents are generally
not written for the purpose of modeling and frequently contain only partially relevant
information for a modeler.
To overcome this apparent disadvantage, Vennix (1989) has suggested that the method
developed by Axelrod can be used to extract system structure from written documents drafted by
policy makers specifically for modeling the policy system. These policy notes are subsequently
coded to extract the implicit "mental policy models" participants. Next a system dynamics modeling
process is started, which might take these policy notes and the extracted mental policy models as
its point of departure. This procedure not only allows extracting basic knowledge and a
preliminary model structure from participants; it also puts the modeler in a position to establish
the impact of a modeling effort on the participants’ mental models. After the modeling effort is
finished, the participants can be asked again to draft policy notes, which can be coded as before to
extract their mental policy models. By systematically comparing these mental policy models
before and after the model development process, Vennix is in a position to measure, in part, the
impact of the modeling process on the thinking processes of key participants.
wg
Eliciting Knowledge From Individuals
The consulting approach to model building relies heavily on informal discussions with key
participants within a system to elicit a wide variety of information relevant to the model building
process. In addition to informal discussions, there are basically two formal techniques that may
be used--interviews and questionnaires, A great body of literature in the areas of sociological and
ethnographic research (Hyman 1954; Riley 1963, Galtun; 1969, Babbie 1979) treat these two types
of techniques in exhaustive detail, and we shall not review all of it here. Suffice to say, interviews
can cover a wide range from very structured to virtually unstructured. Unstructured interviews
take the form of open-ended conversations. Additional structure is introduced when the modeler
presents certain well thought-through questions to guide the conversation. At an even more
structured level, the modeler might actually try to construct, causal loops of system flow diagrams
and have the interviewee participate in the system conceptualization process.
Questionnaires too can be more or less structured, but open-ended questions are usually more
suitable for focusing on complex structures, Vennix et al (1988) proposed using a “model
construction workbook" as an alternative to an open-ended questionnaire. This type of workbook
can be conceived of as a written interview. When open-ended questionnaires are used in such
workbooks, some kind of content analytic procedure such as those discussed above may be
employed in order to uncover the causal arguments being made by the respondents.
Eliciting Knowledge From Groups
While system dynamicists are interested in using groups to construct, test, and interact with
models, researchers studying smal! group processes have long been interested in many of the
more general properties of smail groups working in problem-solving situations. Some of the
conclusions of this long line of research are summarized below.
General Comments on Group Process. McGraw and Harbison-Briggs (1989) have demonstrated that
the type of knowledge and the quality of judgements acquired from experts in a group setting
differ from information obtained when they are questioned as individuals. Shaw (1932) found that
one advantage of using groups was their ability to recognize and reject incorrect or impossible
solutions and suggestions, Steiner (1972) has found that a group of experts may be better able to
solve a problem than individuals working alone when the task can be subdivided into. related tasks
and the expertise of each matched with a particular sub-task.
The effectiveness of groups seems to be correlated with group size, the structuredness of the
process, and the type of task. Formal brainstorming techniques have been helpful in large
groups but of little use in smal{ groups. Communication among group members decreases as the
size of the group increases. Slater (1958) has found that for tasks involving decisions based on
evaluation of exchanged information, groups of five or fewer are most effective. Bouchard (1969,
1972) indicates that introducing structure in group sessions drastically improves group
performance. Hart (1985) also points out that without structure, participants in a group can
become frustrated and group performance can rapidly deteriorate. Moreover, freely interacting
groups can be swayed by strong personalities and may rapidly narrow their focus to a few
approaches or unduly concentrate on evaluating ideas.
Recognizing the wide variety of structures, tasks and circumstances facing groups, various
350
sets of formal procedures have been developed for eliciting knowledge from groups. Prominent
among these are nominal group techniques (Huseman 1973), Delphi techniques (Linstone and
Turuff 1975), and social judgement analyses (Hammond 1975).
Using Group Process to Support System Dynamics Mode!- Building. \t is thus useful to make a
distinction between strongly or weakly structured group processes for model-building. Less
structured group processes are the approaches used by most system dynamicists working in a
consulting mode, but they have significant disadvantages. More structured approaches can be
approached in at least two different ways. The modeling process can be broken into small
‘sequential steps, or the group can be presented with a preliminary model that can be discussed
systematically one part at atime.
An example of the first type of structured workshop is Duke's technique (Duke 1981, 64) for
designing a gaming simulation. The process begins with a brainstorming session in which
participants write down on smail pieces of paper all kinds of concepts that come to mind when
thinking about the policy problem under study. Duke calls these little pieces of paper snowcards.
The second step is to organize and classify these concepts into broader categories by removing
duplicate concepts, merging similar concepts, and classifying groups of concepts. The third step
involves constructing a diagram of system structure using these broad categories. Differing
specific small group techniques are used to support each of these small steps within a structured
group workshop.
Vennix (1989) and Hart (1985) present examples of structured workshops using the
preliminary model approach. In this approach the modeler first designs a preliminary model,
which is then presented to the client for comment, criticism, and revision. This discussion can
itself be structured, e.g., by first focusing on the concepts in the preliminary model, then
addressing relationships, and finally discussing feedback loops. A clear advantage of the
preliminary mode! approach is that it drastically limits the client's time investment. Whether a
preliminary model can generate unwarranted acceptance or even distort clients’ perceptions of
the real system has not, to our knowledge, been investigated. ‘
Effectiveness of the group process in either of these approaches can also be increased by
having participants do some homework. “Divergent thinking" tasks are best be done by
individuals, perhaps working with questionnaires or workbooks, before the actual group session.
Hardware and Software Supports for Knowledge Elicitation
Even as a large literature is beginning to emerge on how individuals and small groups
approach problems and structure knowledge for problem solving, a wide variety of hardware and
software supports have been developed to support brainstorming, idea sorting, and problem
structuring. These software and hardware innovations may be characterized as those primarily
designed to support an individual working on a personal computer or work station and those
designed to support an interacting group.
Supports Designed for Individual Use. Many software and hardware supports for model building
are primarily designed for use by a single expert or analyst working at a terminal or work station.
It is important to note that some of these software tools are being used with groups by having
output projected for review and discussion by a group as a whole. Hence, many of the visually
351
oriented supports discussed below can be used by groups with the simple addition of an overhead
projection pad for computer output.
Most system dynamics practitioners are by now familiar with STELLA as developed by
Richmond et al (1988). Developed exclusively for Macintosh machines, this very powerful model
building tool allows modelers to create models at a conceptual level very different from what .had
been possible previously using conventional simulation languages such as DYNAMO and DYSMAP.
Using STELLA, analysts work with screen oriented icons that allow them to construct system flow
charts interactively. After users respond to several prompts and queries at key decision points
(usually rates and auxiliaries), the STELLA system automatically creates simulation code and can
then execute a simulation, with a standard animated mode possible. However, as a general rule,
persons expert in a policy problem with little or no background in modeling will not be able to
interact directly with STELLA, at least initially. Typically, when STELLA is being used to structure
group discussions and interactions in the model development process, a modeling expert must be
present to help substantive experts interact with STELLA and the models that are created using
this language.
Dieh! (1988) and Richmond (1989) have developed gaming interfaces for STELLA. Using these
interfaces, modelers may create an animated game-like view of a simulation. Using these
animations, users may interact directly with the simulation model, often without having to come to
grips with or understand the structure of the system under study. (Such a facile ability to interact
with a model has both positive and negative implications.)
Modern versions of DYNAMO contain front end packages that allow users to interact more
easily and directly with a simulation model once it has been created. Using a structured and
menu-driven series of screens, users respond to a series of queries and the package creates a
stream of commands much like the traditional RERUN streams that creates a new model run.
Packages such as these are very useful for allowing users to interact with a model once it has been
constructed. Expert modeling support is needed to construct both the model and to program the
front-end package.
A variety of software packages exist for supporting individual or group brainstorming
sessions. For example MAXTHINK (IBM compatible) or MORE (Macintosh) provide a set of flexible
text processing and sorting utilities than can help to both elicit and organize verbal concepts.
Working in one mode the user can create a list of unstructured verbal phrases. Working in
another mode within the software, these phrases or concepts can be sorted and grouped into
similar “bins” and reworked or ranked. When projected in front of a small group, these software
programs can be used to support group brainstorming, acting as a sort of infinitely flexible
“electronic flip-chart".
Shachter (1986) has developed DAVID, a modeling tool that helps to structure influence
diagrams and representations of probabilistic and deterministic decisions. DAVID can be used as a
software support in the conceptualization or problem definition phases of a modeling project
where causal loops are being either generated or discussed by a group. DESIGN on the Macintosh
can be used similarly. The potential of these software tools for mode! conceptualization in groups
has, to our knowledge, not yet been tested.
352
Hardware and Software Designed to Support lateractiag Groups. Recently, a number of sites have
experimented with multiple, linked work stations or terminals designed to support knowledge
elicitation. While these facilities have never been used to support the construction of system
dynamics models, their existence and general capabilities should be discussed because of possible
future utility of such facilities.
Electronic support for group activities have been carried out in a variety of places. The two
most well known are at the decision and planning laboratory at the University of Arizona and at
Xerox Park's COLAB. Arizona hasa research facility for studying the impact of automated support
for planning and decision-making. It is used by executives, managers, and students for planning
sessions and to address complex, unstructured decision processes. As described by Nunamaker,
Applegate, and Konsynski (1988) the lab has been operational since March 1985 with state-of-the-
art computer hardware and software used in a boardroom. Two of their software tools are used to
support the process of deliberation, electronic brainstorming, and stake-holder identification and
analysis. Electronic brainstorming permits participants to network using micro-computers to
share comments and contributions with other participants. Comments from all participants are
consolidated and an analysis support tool is used to identify common issues or categories. This
computer-based technique is adapted from manual procedures developed in association with
Strategic Assumptions Surfacing and Testing as reported in Mason and Mitroff (1981).
The use of dynamic interactive media at Xerox is part of COLAB. This computer !ab's purpose is
to increase the effectiveness of meetings and to provide a research environment to investigate the
effects of computer tools on meetings. Stefik et al (1987) report that within COLAB a variety of tools
are available to provide participants with a coordinated interface, enabling them to interact
cooperatively. COLAB tools support simultaneous action, allowing group members to work in
parailel on shared objects. Conflicts, (e.g., more than one member attempting to act on the same
image) are handled by a busy signal. There are a variety of software tools to extend the uses of
COLAB.
Both the Arizona and the Xerox labs can be seen as experimental mechanisms for eliciting the
group knowledge useful in mode! building. However, their effectiveness in designing models is as
yet to be assessed.
implications: When to Select Which Technique
Several factors help the modeler to select appropriate knowledge elicitation techniques--the
type of task being performed, the number of persons involved in the process, the purpose of the
modeling effort, the phase of the modeling effort, the time available for participants, and finally
the costs involved in using various techniques.
Type of Task. From a psychological point of view, eliciting, exploring, and evaluating tasks need
to be approached very differently. Eliciting tasks, whether performed by individuals or groups
require divergent thinking. These tasks are additive, that is, the largest list of alternatives can be
generated simply by adding up the contributions of individual contributions. Performing these
tasks in the context of well-structured group interactions will actually decrease the quality of
group versus individual performance. For example, a discussion designed to elicit an exhaustive
list of variables that might be included within a model's boundary should not be performed by a
353
whole group; rather, individuals should make a list working alone and the group facilitator should
merely compile these individual responses.
The literature on evaluation, whether it involves individual or group evaluation of options,
events, or alternative formulations, is quite well-developed. Specific techniques such as the
Delphi technique (Linstone and Turuff 1975), social judgment analysis (Hammond 1975), and
nominal group techniques (Huseman 1973) have well developed theoretical underpinnings and
have been well-explored in experimental settings. When client groups are involved in evaluative
tasks--selecting parameter values, evaluating alternative structural formulations, or assessing
model validity or the policy performance of a model--system dynamics modelers must base their
work on the accumulating research results in the field of individual and group judgment.
The exploring (problem solving) task is both most central to the model building process and
least well- developed in the psychological literature. Some evidence suggests that well-trained or
knowledgeable individuals can perform as well as or even better than groups. A well-trained
model builder can do as well as a group of model builders in tasks such as proposing formulations
or designing feedback structures. Involving a group may have an apparent purpose of designing
model structure, but have as a real purpose developing understanding of the system under study
or of the model-building process.
Number of Persons. The number of persons ultimately to be involved in the modeling project will
dictate the appropriate knowledge elicitation techniques because of two factors. First, the fewer
the number of persons involved, the more unstructured the techniques may be. On the other
hand, the larger the number of involved people (as in public policy modeling), the more
structured the approaches must be to prevent discussions from getting out of hand. Second, as
more people become involved in the modeling process, it becomes necessary to use labor-saving
techniques such as questionnaires, workbooks and structured workshops.
Purpose of the Modeling Effort. The process of eliciting and mapping knowledge to build system
dynamics models is iterative--through successive cycles of refinement the ultimate model
graduaily appears. This indicates that the process of modeling involves considerable learning and
improvement of communication between members of the management team. So knowledge
elicitation and mapping is not simply a process of uncovering a fixed body of knowledge and
representing it. Participants learn, as their mental models are reshaped by discussion and
interaction.
This iterative view of the knowledge elicitation process has profound implications for the
methods and techniques to be used. First, in general the knowledge necessary to model a problem
will not be readily available. Rather the modeling effort often uncovers gaps and inconsistencies
in existing knowledge and mental models. One cannot rely on techniques that aim solely at
capturing and representing knowledge (such as content analysis). Modelers will have to employ
methods that allow interaction and discussion in order to improve mental models and to clarify a
problem.
The second implication, following as a consequence of the first, is that modelers will have
have to employ methods and techniques that will enhance learning and communication among
members of the management team. One cannot rely solely on techniques that use written
354
documents or the individual as the only source of knowledge. In modeling policy problems,
groups as a source of knowledge will almost always have to be included in the modeling process.
Phase of the Modeling Process. Asa general rule, each phase of the model-building process tends
to be dominated by a single type of cognitive task and hence is most appropriately supported with
specific knowledge elicitation techniques. For example, the problem identification and system
conceptualization phases are dominated by elicitation tasks, the model. formulation phase by
exploring (problem solving) tasks, and the model analysis and model evaluation phases by
evaluating tasks. As a general rule, less structured techniques tend to be more appropriate for the
earlier phases of the model building process (where thinking is more divergent) and more
structured techniques more appropriate for the later, more convergent phases.
However, even this general rule of thumb can be deceptive. Consider a model
conceptualization exercise designed to get at the issue of model boundary. A first phase of that
exercise might involve brainstorming variables to be included or excluded from the model's
boundary. This eliciting task would probably be best performed in a nominal group by individuals
working alone, with the group convened to sum up all responses generated. However, as a second
step, the group as a whole might be asked to evaluate which of the variables elicited are most
important and need to be retained as the model is developed. Obviously, this would be a more,
structured evaluative task.
Hence the phase of the model building effort interacts subtly with the type of cognitive task
being undertaken in determining what type of knowledge elicitation techniques are most.
appropriate in a given specific situation. In each phase of the modeling process various
techniques may have to be employed in combination, depending on the type of task that has to be
performed.
Time Available for Participaat Discusstoa, A simple but powerful criteria for determining what
knowledge elicitation techniques to use is how much time does the management team or reference
group have to spend on task. The less time that they have available for active participation in the
modeling effort, the more the process will have to be carefully structured. For example, a group
might begin with a preliminary model rather than attempt to develop a model from scratch.
Cost. Finally, the costs associated with the various techniques must be carefully factored into the
selection of knowledge elicitation and mapping techniques. Costs include participant costs
(usually in terms of time devoted to the modeling process) as well as the costs of time for the
modeling team. Usually costs (both monetary and time costs) will be negotiated at the beginning
of a project and the modeler's task will be to select the best techniques given cost constraints.
Hence cost considerations are most important at the stage where a modeling contract or agreement
is being designed. In one innovative approach of which we are aware, a modeler uses group
facilitation techniques in order to help management teams decide early on how much of their time
and funds they wish to expend on a specific modeling project.
Summary and Directions for Future Practice and Research
A rich body of theoretical and experimental work already exists on how to elicit and map
qualitative knowledge that resides in written documents, as well as the mental models of
individuals and groups. An interesting array of software products is beginning to emerge to
support such model-building exercises. Yet most of these techniques and advances seem not to
have penetrated into the system dynamics literature. It seems clear that those who write about the
system dynamics modeling process are not paying close attention to developments in other fields
that hold great promise for improved system dynamics practice. Similarly, those most experienced
in the art of modeling appear not to have the time or inclination to write down the lessons that
they have learned from years of practice working on knowledge elicitation and mapping.
Asa result, the critical phases of problem definition and model conceptualization appear to be
arrested at the point where they remain true art forms. Simply put, systematic research is not
being conducted that will advance our understanding of how modelers and management teams or
reference groups do or ought to interact in the model building process. This fack is all the more
disturbing because psychologists, ethnographers, management scientists, and software engineers
working in fields closely related to system dynamics are making progress in precisely these fields.
The field of system dynamics needs to begin the work of formulating rigorous research programs
that get at general rules helping to make more precise and less artful the process of eliciting and
mapping knowledge.
References
Axelrod, R. 1976. Structure of Decision: the Cognitive Maps of Political Elites. Princeton, NJ:
Princeton University Press.
Babbie, ER. 1979. Zhe Practice of Social Research. Belmont, CA: Wadsworth Publishing.
Bouchard, T. 1969. Personality, problem solving procedure and performance in small groups.
Journal of Applied Psychology 53: 1-29.
Bouchard, T. 1972. A comparison of two group brainstorming procedures. journal of Applied
Psychology 56: 418-421.
Bourne, L. E., and WF. Battig. 1966. Complex Processes. In J.B. Sidowski, ed., Experimental
Methods and Instrumentation in Psychology. New York : McGraw-Hill.
DESIGN. Meta Software, 150 Cambridge Park Drive, Cambridge, MA 02140. Reviewed in Mac User.
April 1988.
Dewey, J. 1910. How We Thiak. New York: DC. Heath.
Duke, R. D. 1981. A paradigm for game design. In C. S. Greenblat and R. D. Duke, Ariaciples and
Practices of Gaming-Simulation. Beverly Hills: Sage.
Edwards, W. and J. R. Newman. 1982. Multi-Attribute Evaluation. Beverly Hills: Sage.
Forrester, J. W. 1980. Information Sources for Modeling the National Economy. journal of the
American Statistical Association. Voi. 75,371: 555-574.
Forrester, J. W., and P. M. Senge. 1980. Tests for Building Confidence in System Dynamics Models.
TIMS Studies ia Management Sciences. 14 (System Dynamics); 209-228.
Fox, W. M. 1987. Effective Group Problem Solving. San Francisco: Jossey-Bass.
Galtung, J. 1969. Zheory and Methods of Social Research. New York: Columbia University Press.
Gardiner, P.C. and A. Ford. 1980. Which Policy Run is Best and Who Says So: 7/MS Studies in the
Management Sciences 14 (System Dynamics): 241-257.
356
Graham, A. K. 1980. Parameter Estimation in System Dynamics Modeling. In Elements of the
System Dynamics Method , Jorgen Randers, ed. Cambridge, MA: The MIT Press, 143-161.
Hackman, JR. 1968. Effects of Task Characteristics on Group Products. /ouraal of Experimental
Social Psychology 4:162- 187.
Hackman, JR. and CG. Morris. 1975. Group Tasks, Group Interaction Process, and Group
Performance Effectiveness: A Review and Proposed Integration. In L. Berkowitz, ed.,
Advances in Experimental Social Psychology (Vol. 8). New York: Academic Press.
Hammond, K R., TR. Stewart, B. Brehmer, and D.0, Steinmann. 1975. Social Judgment Theory. In
MF. Kaplan and S$. Schwartz (eds.), Humana Judgment and Decision Processes in Applied
Settings. New York: Academic Press.
Hammond, KR., J. Rohrbaugh, JL. Mumpower, and L. Adelman. 1977. Social Judgment Theory:
Applications in Policy Formation. In MF. Kaplan and S. Schwartz, eds., Human Judgment and
Decision Processes ia Applied Settings. New York :Academic Press.
Hart, S., M. Boroush, G. Enk, and W. Hornick. 1985. Managing Complexity Through Consensus
Mapping: Technology for the Structuring of Group Decisions. Academy of Management
Review , 10(3): 587-600.
Hoogerwerf, A. 1984. Beleid berust op veronderstellingen: de beliedstheorie (Policies are based on
assumptions: the policy theory), Acta Politica. 4:493-531.
Huseman, R. 1973. The Role of Nominal Groups in Small Group Communication. In R.C. Huseman,
DM. Logue, and DL. Freshley (eds.), Readings in Interpersonal and Organizational
Communication (second edition). Boston MA: Hollbrook.
Hyman, H., ed. 1954. /aterviewing in Social Research. Chicago: University of Chicago Press.
Lamm, H. and G. Trommsdorf. 1973. Group Versus Individual Performance on Tasks Requiring
Relational Proficiency (brainstorming): A Review. Zuropean Journal of Social Psychology 3:
361-388,
Lichtenstein, S, and P. Slovic. 1971, Reversals of Preference between Bids and Choices in Gambling
Decisions. /ouraal of Experimental Psychology 89: 46-55.
Lichtenstein, S$. and P. Slovic. 1973. Response-induced Reversals of Preference in Gambling: An
Extended Replication in Las Vegas. Journal of Experimeatal Psychology 101: 16-20.
Linstone, H. and M. Turuff. 1975. Zhe Delphi Method: Techaiquesand Applications, Reading MA:
Addison Wesley.
Mason, R., and I. Mitroff. 1981. Challenging Strategic Planning Assumptions, New York, NY:
John Wiley and Sons.
McGraw, K.L, and K. Harbison-Briggs. 1989. Xaowledge Acquisition: Principles and Guidelines.
Englewood Cliffs NJ: Prentice Hall.
Morris, C.G, 1966. Task Effects on Group Interaction. journal of Personality and Social
Psychology 4545-554.
Nunamaker, JF., L.M. Applegate, and BR. Konsynski. 1988. Computer-Aided Deliberation: Model
Management and Group Decision Support. Operations Research 36,6 (Nov-Dec 1988). 826-848.
Randers, J. 1977. The Potential in Simulation of Macro Social Processes, or How to be a Useful
Builder of Simulation Models. Oslo: Gruppen for Ressursstudier.
Richardson, G. P. and S. Schuman. 1987. Medical Malpractice Insurance: Policy Implications and
Evaluations. Decision Techtronics Group, The Rockefeller Institute of Government, 411 State
Street, Albany, NY 12203.
Richardson, G.P., and A.L. Pugh. 1981. Jatroduction to System Dynamics Modeling with DYNAMG.
Cambridge, MA: The MIT Press.
Richmond, B. 1987. Zhe Strategic Forum. Lyme, NH: High Performance Systems.
Richmond, B., S. Peterson, and P. Vescuso. 1988. Academic User's Guide to STELLA, lyme NH:
High Performance Systems.
Richmond, B. 1989. S7ZZZAStack. Lyme NH: High Performance Systems.
Riley, M. W. 1963. Sociological Research. New York: Harcourt, Brace & World,
Roberts, E. B. 1977. Strategies for Effective Implementation of Complex Corporate Models.
faterfaces 8,1, Part I.
Roberts, N., DF. Andersen, R. Deal, M. Garet, and W. Shaffer. 1983. Jaéroduction to Computer
Simulation: A System Dynamics Modeling Approach. Reading MA: Addison Wesley.
Rohrbaugh, J. 1981. Improving the Quality of Group Judgment: Social Judgment Analysis and the
Nominal Group Technique. Organizational Behavior and Human Performance 28:272-288.
Rohrbaugh, J.,and D.F. Andersen. 1983. Specifying Dynamic Objective Functions: Problems and
Possibilities. Dyaamica 9:1.
Shachter, R.D. 1986. DAVID: Influence Diagram Processing System for the Macintosh. Working
Paper, Department of Engineering-Economic Systems. Palo Alto CA: Stanford University.
Shaw, M. 1932. A Comparison of Individuals and Smal! Groups in the Rational Solution of Complex
Problems. American Journal of Psychology 44(1932): 491-504.
Simon, H.A. 1960. 7he New Science of Management Decisioa. New York: Harper and Row.
Slater, P. 1958. Contrasting Correlates of Group Size. Sociometry 25 (1958): 129-139.
Stefik, M., G. Foster, D. Bobrow, K. Kahn, S. Lanning, and L. Suchman. 1987. Beyond the
Chalkboard: Computer Support for Collaboration and Problem Solving in Meetings.
Communications of the ACM. 30,1 (1987): 32-47.
Steiner, I. 1972. Group Process and Productivity. New York: Academic Press.
Stenberg L. 1980. A Modeling Procedure for Public Policy. Hlements of the System Dynamics
Method Cambridge, MA: The MIT Press.
Vennix, J.A.M. 1989. Jeteractive Computer Modeling: Improving Mental Policy Models and Policy
Theories. PhD, dissertation: Catholic University of Nijmegen, The Netherlands.
Vennix, J.A.M., J.W. Gubbels, D. Post, and H.J. Popper. 1988. A Structured Approach to Knowledge
Acquisition in Model Development. Aroceediags of the 1988 International Conference of the
System Dynamics Society, La Jolla, CA. Cambridge MA: The System Dynamics Society.
Vennix, J.A.M. 1989. Jateractive Computer Modeling: Improving Mental Policy Models and Policy
Theories. PhD dissertation: Catholic University of Nijmegen, The Netherlands.
Weil, H. B. 1980. The Evolution of an Approach for Achieving Implemented Results from System
Dynamics Projects. H/ements of the System Dynamics Method: Proceedings of the 1976
International System Dynamics Conference. (Geilo, Norway, 1976). Cambridge, MA: The MIT
Press.
Weil, H. B. 1983, The Dynamics of Strategy Implementation. Dyaaaca 9,1 (summer 1983): 43-47.
CC BY-NC-SA 4.0