The Evaluation and Development of Knowledge Acquisition in System Dynamics Studies
Willard Fey
John Trimble
Georgia Institute of Technology
Industrial and Systems Engineering
Adanta GA 30332, USA
ABSTRACT
An examination of knowledge acquisition techniques and knowledge representation
structures used in expert system development and technology forecasting, helped to determine how to
elicit information from System Dynamics analysts. In this ongoing research, insights from the
literature on knowledge acquisition, combined with knowledge elicited from System Dynamics
analysts, are being used to develop an approach designed to improve the knowledge acquisition
processes and structures used during the problem formulation and model conceptualization activities
of System Dynamics. Also, preliminary insights are presented regarding the selection of knowledge
acquisition techniques and knowledge representation structures.
THE PROBLEM
Limited previous work has been done in investigating the selection and use of knowledge
acquisition techniques and knowledge representation structures for System Dynamics studies. System
Dynamics analysts need to obtain sufficient reliable information about many aspects of their subject
systems quickly, inexpensively and unobtrusively enough so they can study and influence them to
‘improve’ their behaviors. They also should efficiently, accurately, and understandably communicate
useable information about the system and the analysis to the appropriate system participants. A
methodology for improving the knowledge acquisition process used by System Dynamics analysts
should make a significant contribution to the effectiveness of System Dynamics studies by:
a) helping to establish consistency in implementing the System Dynamics process;
b) allowing the increased involvement of experts and decisionmakers, laying the groundwork for
wider acceptance of System Dynamics models;
c) facilitating System Dynamics team efforts by providing standard methodology;
d) combatting the increasing tendency to view System Dynamics as simulation programming,
e) improving the quality, implementability and retention time of System Dynamics recommendations.
RESEARCH APPROACH
The System Dynamics literature was investigated regarding what is written on knowledge
acquisition and knowledge representation. Then a range of knowledge acquisition techniques and
knowledge representation structures from expert systems and technology forecasting were examined
to determine what may be applicable to System Dynamics. There are two methods for measuring the
worth of these techniques and structures. One could apply them in real System Dynamics studies and
try to evaluate the differences these new techniques and structures make. This approach would be
very difficult and time consuming to execute. The second method is to involve System Dynamics
analysts in evaluating the worth of these new techniques and structures. This second approach should
take less time, enhance communication within the System Dynamics profession, and be of more direct
value to System Dynamics analysts. Having selected this second method, the next step in the overall
approach was to select the elicitation techniques most effective in eliciting opinions from System
Dynamics analysts. A range of expert techniques were examined. The decision was made to use the
Delphi with a wide cross section of System Dynamics experts, interview local experienced System
Dynamics analysts, and use a modified Nominal Group Technique with the local System Dynamics
group.. The results of these 3 methods are providing: 1) information on what knowledge acquisition
techniques and knowledge representation structures are currently being taught and used, 2) an
evaluation of these techniques and structures, 3) opinions on new techniques and structures, and 4)
methods for consistent exploration and development of knowledge acquisition in System Dynamics.
KNOWLEDGE ELICITATION TECHNIQUES
Vennix (1990) elicited information from system participants regarding model development
and verification. The System Dynamics team developed a model, and had a group of Healthcare
system participants evaluate, revise, and verify it using a modified Delphi exercise. The exercise
consisted of three parts. First, a Delphi questionnaire was distributed. It had three goals: elicit
participants' opinions on concepts and relationships of the preliminary model, expand the number of
concepts considered relevant to the policy problem, and reduce the number of concepts by having
participants prioritize them. Second, a Delphi workbook was distributed. It contained a number of
hypotheses about relationships between concepts which were visualized using causal diagrams. These
hypotheses were based on the preliminary model and the results of the questionnaires. The
respondents were asked to comment on the hypotheses and the causal diagrams. And finally, based
on the workbook results, structured workshops were held, where the controversial concepts and
relationships were extensively discussed. A final conceptual model was designed based on this
modified Delphi exercise.
Richardson et al. (1989) presented a survey of work done in knowledge elicitation for
System Dynamics modeling. They concluded that "problem identification and system
conceptualization phases are dominated by elicitation tasks, ... less structured techniques tend to be
more appropriate for the earlier phases of the model building process” (p. 354), and the selection of a
knowledge elicitation technique should be based on the task at hand, number of persons involved with
the process, the purpose and phase of the modeling effort, along with time and cost constraints.
Sancar and Cook (1985a) examined a set of ‘cognitive criteria’ to guide study participants in
the problem definition phase of a System Dynamics study. In examining knowledge elicitation, they
concentrated on the group discussion process, and specified criteria to guide in identifying the
perspective, time horizon, and policy choices, establishing reference mode, and defining the basic
mechanisms, Sancar and Cook (1985b) used these criteria to develop a decision support system for
community development consisting of: 1) a generic community development System Dynamics
model, 2) a problem structuring algorithm, and 3) a situational System Dynamics model.
Interpretive Structural Modeling (ISM) is central to the problem structuring algorithm, It
allows the generation of a rich picture of the situation reflecting the variety in perceptions, interests,
and interrelationships, without imposing any preconceived structure. ISM has three major steps:
1) List variables, and establish a relational proposition reflective of the type of variables.
2) Generate the interaction matrix. This can be done by an individual, but a group process is
more effective for System Dynamics modeling. It can be done manually or electronically
using an interactive computer program, Matrix entries may be evaluated by ranking and rating.
3) Software is used to generate ‘directed graphs’ from the interaction matrix.
The use of ISM allows "a collective appreciation of the situational context” (p.750).
While Sancar and Cook (1985a, 1985b) and Richardson et al. (1989) drew from psychology
and expert systems to develop knowledge acquisition in System Dynamics, much additional work is
needed. Expert system development has been one of the most profitable areas of Artificial
Intelligence. It has been widely recognized that the bottleneck to expert system development is
knowledge acquisition. This climate has led to a proliferation of research and literature on knowledge
acquisition and knowledge engineering. System Dynamics analysts should be viewed in part as
knowledge engineers who: elicit knowledge from individuals and groups involved with the system at
various levels; acquire knowledge from the literature on the domain and from organizational data; and
structure the knowledge to most effectively employ the principles of System Dynamics.
In expert system development, knowledge engineers are dominantly concerned with
extracting expert knowledge with the goal of developing the most effective expert system. They are
more concemed with the end product, while System Dynamics analysts are more concerned with the
process. The System Dynamics process not only involves extracting information from experts, but
throughout the study includes conveying the dynamic, causal feedback characteristics, inherent in the
situation, that address the clients’ concerns. The techniques of expert system development will not
address all the knowledge acquisition needs of System Dynamics. However, the collective intensity
applied by expert system knowledge engineers, in extracting expertise and understanding the causality
behind the actions of experts, has produced a range of techniques worth examining.
- 174 -
Interviews are the most popular method of gaining information from system participants in
most system analysis studies, and certainly the most popular method of eliciting information in
System Dynamics studies. Yet, little effort is spent studying, documenting or training analysts how to
effectively interview. An interview can be fully structured, semi-structured, or unstructured
(Cordingley 1989). A fully structured interview is one in which the interviewer has carefully prepared
in advance all of the questions to be asked. Each question is precisely worded. The questions are
asked in a specific predetermined order. When multiple interviews are involved the same questions
are asked, in the same order, using the same words in each interview. Structured interviews are most
appropriate for surveys and large multiple interviews. Fully structured interviews allow analysts to
use forms to document interviews. The potential problem with structured interviews is the limitations
placed on responses and dialog by the rigid structure. In semi-structured interviews, the interviewer
starts with a list of questions to be asked, but the wording of the questions and the order in which they
are covered can vary. This flexibility allows the interviewer to adopt the vocabulary of the
interviewee, and adjust the flow and intensity of the session to the particular conditions, Semi-
structured interviews are harder to document than structured interviews, and place more demands on
the interviewer, who must correctly interpret the situation and make spot adjustments. Unstructured
interviews allow interviewees to cover topics in their own way. The interviewer uses starter or seed
questions to initiate elicitation in a very general fashion. Prompts and probes are used to continue
information flow from the interviewee. Probes encourage the elaboration on a point of interest to
force a more complete answer. Prompts are used by the interviewer to change the course of the
interview. Unstructured interviewing is the most difficult. It requires a skilled interviewer to allow
interviewee flexibility, but prevent useless rambling. There are a number of additional approaches to
classifying interview types. Numerous disadvantages to interviews were pointed out by Forsythe and
Buchanan (1989). Through their tone, choice of words, or body language, interviewers may give
away their opinions, attitudes, biases and expectations. This can lead to concealing of information,
negative responses, and lack of cooperation on the part of the expert being interviewed.
Process tracing techniques allow the analyst to learn how the respondent solves a particular
problem, completes a task, or reaches a conclusion. These techniques are effective in showing what
information is used to make decisions and how this information is processed. Unlike interviews,
process tracing sessions typically are not interactive. The analyst presents the particular problem,
process or task. Then the expert responds until the problem is solved or the task or process is
concluded. Process tracing sessions use one of two basic techniques: concurrent or retrospective
verbalization. With concurrent verbalization the expert ‘thinks out loud’ while solving the problem,
and the analyst documents the session. In a retrospective verbalization session, the expert's procedure
is recorded, and later the expert and the analyst jointly review the session and produce documentation.
"Task analysis’ is a method used to: 1) describe the functions an expert performs, and 2)
determine the relationship of each task of a certain dimension to the overall job. ‘Task analysis‘
information is generated for each task. Typically this includes: task title, description, and type;
knowledge required to complete the task; typical, critical, and/or permissible performance times; and
other tasks related to, depending on, or interacting with this task. Subtasks may be identified for each
task. There are no established rules for applying task analysis. Generally form follows function. In
other words, the application of ‘task analysis’ is domain and context sensitive.
The ‘Job analysis’ technique is used to identify the major responsibilities of a job or, on a
broad level the tasks that a job entails. The knowledge engineer develops a list of ‘task statements’
that describe what someone performing the given job should be capable of doing. Job description
documents, onsite interviews, observation methods, and specialized survey style questionnaires can be
used to gather and compare information for task statements. The ‘task statement’ consist of: the
behavior, conditions for the behavior, and standards against which the behavior is evaluated.
McGraw and Harbison-Briggs (1989) specified alternative guidelines for developing ‘task statements’
in conducting a ‘job analysis’.
The Repertory Grid technique draws on George Kelly's theory of personality, which is based
on the notion that humans are scientists, experiencing events, perceiving similarities and differences
among these events, formulating concepts or ‘constructs’ to order, classify and categorize events, and
hence the world, and using such constructs to anticipate events (Magee 1987, p.66) The repertory
= 175 =
grid is both a technique and a structure. It is used to represent a respondent's organization of basic
concepts in’a specific domain. A domain is established, then the analyst elicits a set of constructs
(bipolar characteristics that typify the domain). For example, if the domain is ‘decisionmakers in a
firm’, a typical construct would be ‘level of experience’. It's bipolar values may be ‘inexperienced’ and
‘very experienced’. After the critical constructs have been established, the analyst elicits a set of
domain examples, called elements, from the respondent. Using the domain 'decisionmakers in a firm’
the element list may include marketing manager and finance manager. The grid structure consists of
the constructs listed along the rows and the elements listed above the columns. The respondent rates
each element according to the constructs. The ratings are placed in the grid.
Protocol analysis refers to the analysis of: 1)transcripts of domain activity, 2)documentation
on standards and policies; and 3) data on domain activity. Knowledge engineers use several
techniques to evaluate protocols. The selection of techniques is both context and domain dependent,
Johnson et al. (1987) developed a technique that is useful for a wide range of protocols. This method
consist of syntactic analysis and semantic analysis. Syntactic analysis involves identifying behavior
in the protocol record by category (operations, episodes, and data cues). This information is used to
assign semantic categories (actions, abilities, goals, conditions, strategies, and solutions). “Syntactic
analysis allows the protocol record to be partitioned into separate categories of behavior, it is the
semantic analysis that forms the basis for a representation of expertise” (Johnson et al. 1987, p.165).
Richardson et al. (1989 p.347) mentioned ‘content analysis’ as a useful method for analyzing written
documents. Content analysis is a form of protocol analysis.
The original Nominal Group Technique was developed by Delbecq and Van de Ven and is
summarized by the six steps listed below based on Porter et al. (1990):
Step 1: Silent Idea Generation . Each participant works silently to list factors believed to be central.
Step 2: Group Round-Robin listing of Factors: Without discussion, each participant reads one of
his/her factors, which is posted for all to see. This process continues until all factors have been listed.
Step 3: Discussion and Clarification of Listed Factors: Each factor on the list is discussed
informally by the group to gain clarity. Factors may be added or combined during the discussion.
Step 4: Individual Written Voting on Priorities: Each participant silently and independently selects
the most important factors from the revised list generated in step 3.
Step 5: Discussion of Voting Results: The factors with the highest vote totals in step 4 are displayed.
The group evaluates the result. New formulations may be produced.
Step 6: Final, Silent, Individual Written Voting: Participants select the most important factors from
the list generated in step 5, without regard to the preliminary ranking.
Committees are the most common group technique in modern organization. The value of
committees depends heavily on the skills of the chairperson in preparing for and conducting the
commitee meetings. Because committees lack anonymity, the relationship between members is a big
factor in productivity. While positive relationships can inspire members, and help forge a group
identity and commitment to committee decisions, a negative group relationship can result in
demoralization, animosity and a totally ineffective committee.
The most common method for soliciting input from groups of experts, when meetings are
impractical, is the survey. It is generally quick, easy and inexpensive. It is most popular when a large
number of dispersed participants are involved. The disadvantages are failure to produce consensus or
provide feedback to participants.
The Delphi procedure is a family of methods that are variations on the approach initially
developed at Rand Corporation. The strengths of the Delphi procedure are anonymity, iteration with
controlled feedback, statistical group response, and the ability to deal with geographical dispersion of
experts. It is a special type of survey involving multiple iterations. The first round is generally more
open ended, The statistical results from a questionnaire are included in the next questionnaire. The
Statistical treatment of responses allows portrayal of differences of opinion within the group and
checking for convergence as rounds proceed. A policy Delphi is a particular variation of the Delphi
method that can be used as a precursor to a committee activity. Its goal is not to obtain a consensus,
but to expose all the differing positions advocated, and the principal arguments for and against each
Position.
- 176 -
ah:
KNOWLEDGE REPRESENTATION STRUCTURES
Knowledge representation has been addressed explicitly in System Dynamics over the years
(Randers 1980; Morecroft 1982, 1992). Every System Dynamics study addresses knowledge
representation implicitly. Knowledge representation is particularly critical in problem formulation
and model conceptualization. The activities during these phases of a System Dynamics study must
examine the mental models of study participants, analyze documentation relevant to the study, when
possible observe the situation under study, and encourage communication among study participants
with the objective of developing a shared understanding of the dynamic feedback phenomenon
inherent in the situation under study that is relevant to the study problem. The visualization of
knowledge is key to this process.
The most basic knowledge structure implicit in all dynamical models is the simple variable.
Graphs of simple variables, as a function of time, are widely used in a range of disciplines to illustrate
the dynamics of a situation. Time histories are line graphs based on real or hypothetical data, that
show the dynamic nature of variables critical to the study. Graphs of variables as a function of
another variable are also useful knowledge representation structures. They give a visualization of
correlation between variables and are helpful in establishing or verifying causality, and in setting
parameters. The generation of variable lists and tables has the advantage that the use of lists and
tables is a common practice easily conveyed to study participants and study audience. Coyle (1971)
used lists in the ‘List extension method’. This is a list building method used to generate an influence
(causal loop) diagram. More recently Wolstenholme (1990) used variable lists. Resources of
importance were identified and listed, then subsequent lists were generated for each resource listing
the states the resource could exist in. This set of lists can be the basis of a resource - state table.
The most popular (and most controversial) knowledge representation structure used in model
conceptualization is the causal loop diagram. The strength, of the causal loop diagram, is it provides a
visualization of the feedback, central to System Dynamics, with a minimum of distinct components,
The drawbacks in causal loop diagrams as a conceptualization tool were pointed out in Morecroft
(1982), and Richardson (1986).
Morecroft (1982) presented two tools for conceptualization: the subsystem diagram and the
policy structure diagram. Subsystems represent the major organizational divisions in the system
under study, and should correspond to the major units in the mental models of key system
participants. The subsystem diagram should communicate an overview of the model. While more
aggregated than a causal loop diagram, the distinction in the linkage makes it a richer diagramming
tool. Six types of flows are used: materials, money, people, capital equipment, orders, and
information. A different symbol is used to represent each type of flow. This can make it a more
valuable knowledge representation structure in conveying model conceptualization to study
participants, allowing participants to adjust it based on their mental models, and serving as the basis
for more detailed model conceptualization.
Each subsystem can be represented by a policy structure diagram. Like a flow diagram, it
identifies the stock and flow network in the subsystem. However, policy structure diagrams are
simpler than system flow diagrams. Instead of including the detail of decision making, policy
structure diagrams only address major policies and decision functions. Morecroft outlines a two step
process for generating the linkage in a policy structure diagram. First, policy symbols are drawn to
delineate the decision making responsibilities of the organization. . Then, an information network is
created using policies as nodes for informational links. The advantage over causal loop diagrams is
“feedback structure is then created from the orderly process of piecing together multiple decision
functions, rather than emerging from the more tenuous and ad hoc methods of postulating causal links
independent of the underlying decision making process" (Morecroft 1982,p.24).
Morecroft (1992) presented two unique knowledge representation structures designed to help
management teams construct System Dynamics models. “The value chain provides a working - space
of boxes and labels to categorize facts. The mapping symbols provide building blocks to assemble
and connect knowledge about the operating policies of a business" Morecroft (1992, p.13)
The second knowledge structure Morecroft (1992) presented was the policy function, where
a set of filters is associated with decision making. “Each filter has a label to signify what process of
the organization is conditioning information flow - operating goals, measurement systems,
- 177 -
organizational structure, or culture and tradition" (Morecroft 1992, p.12), Each information link is
associated with one of these filters. Policy functions can be discussed indepth, one at a time, or linked
together (like a policy structure diagram) to provide a broad mapping of operating structure.
Different names and approaches are associated with ‘flow diagrams’. Sometimes referred to
as DYNAMO flow diagrams, STELLA diagrams, Forrester diagrams, or Stock & flow diagrams, they
are the knowledge representation structure most central to the System Dynamics methodology. While
more complex than causal loop diagrams and more detailed than policy structure diagrams,
structurally, they most closely reflect the model to be formulated and simulated.
Constants are the simplest knowledge representation structure, but their selection is very
critical to determining model behavior. Model constants help to establish the temporal boundaries,
and spatial boundaries, Little has been written concerning the selection of the model time frame.
Generally model delay times are either constant or functions with a time constant as a parameter. The
time interval between data points is another vital time constant. Weekly data points may generate a
quite differently shaped curve than monthly data points. Constants may be established to simplify the
model and help bring it to closure. The process of how and when to establish constants is an area in
itself worth investigation, given the implications concerning knowledge elicitation and boundary
establishment.
A widely used form of expressing knowledge, which has not been discussed is ‘prose’. For
example, a written statement defining the modelling purpose or perspective represents knowledge.
This is generally not considered a ‘structure’, unless certain constraints are placed on the form of the
statement or its contents. The "Dynamic Hypotheses” is a prose description of system structure,
performance and pattern causality that is referred to often in the System Dynamics literature. Logic
statements, exogenous "data" inputs, and deterministic and statistical equations are not covered since
they are usually first encountered (after model conceptualization) during model formulation.
Knowledge representation structures determined by system participants are used in various studies.
Because they developed from the experiential base of the system participants, they tend to be a better
reflection of their mental models.
Rules have been the dominant structure for conveying knowledge in expert systems. A rule
is a conditional statement: IF {a given condition exist} THEN {initiate a particular action, or draw
@ particular conclusion}. Rules are linked together, such that certain rules draw conclusions that then
establish the conditions for other rules. A set of rules can effectively convey a single expert's
decisionmaking process, or a policy implemented in a collective process, Complex IF_THEN_ELSE
conditions can be used to account for all the options covered by a policy. An example of a policy
tule is: IF {deposits > $20,000} THEN {interest = 5% of deposits} ELSE IF {deposits > $2,000}
THEN {interest = 4% of deposits} ELSE {interest = 3% of deposits}.
Tables, one of the most popular knowledge structures, vary in complexity from a simple
single column table, to large multidimensional structures. Any information that lends itself to being
detailed as a list, and then elaborated on can be displayed using tables. For example, a table can be
used to list system resources, their different states and initial values. While tables are structures of
two or more dimensions where the cells have unique values, grids are multidimensional structures
where unique elements or characteristics are listed along the axis and the values in the cells are
relational (rating, rankings, or logical indicators). The repertory grid discussed above is an example.
Scenarios are one type of knowledge structure that allows the analyst to relay the temporal
and causal reality of a process or situation. However, some scenarios only relay the temporal
dimension. Scenarios are a ordered list of statements about a system that convey information on the
state of the system at different points in time. A statement in this ordered list may relate a condition,
an action, or both. Scenarios may be used to document the history of a system or present future
potentialities, Scenarios may be provided by a system analyst as a basis for eliciting information from
respondents, or the analyst may request the respondent to provide scenarios,
McGraw and Harbison-Briggs (1989, p.21-24). categorized knowledge into four groups:
procedural, declarative, semantic, and episodic. ‘Procedural knowledge’ includes the skills an
individual knows how to perform. This type of knowledge involves an automatic response to stimuli
and may be reactionary in nature. ‘Declarative knowledge’ represents surface level information that
can be verbalized. It is what one is aware of knowing. ‘Semantic knowledge’ represents one of the
a ao
oe
two theoretical types of long term memory. It reflects cognitive structure, organization, and
Tepresentation. ‘Episodic knowledge’ is experiential information that oné has grouped or chunked by
episodes. It is temporal and spatial in nature. Different types of elicitation techniques are associated
with each type of knowledge. Different System Dynamics activities reflect different types of
knowledge, and require different knowledge structures. Exhibit 1 gives possible matching of
knowledge type, System Dynamics activity, knowledge structure, and elicitation technique.
Exhibit 1: Knowledge Acquisition Table
Knowledge Activity Structures Techniques
Declarative Identify purpose, checklists, forms, brainstorming,
Knowledge perspective, problems, | variables, constants, interviews, surveys,
and time horizon graphs, prose committees
Semantic Identify policymakers, } Grids, tables, lists, job analysis,
Knowledge resources, and their subsystem diagram repertory grids,
states, and subsystems 7 structured interviews
Procedural Identify decision Policy rules, task analysis,
Knowledge making procedures, Policy functions. | process tracing
Episodic conceptualization of | Scenarios, process tracing, protocol
Knowledge flows Flow diagrams analysis, interviews
APPROACHES USED TO ELICIT INFO FROM SYSTEM DYNAMICS ANALYSTS
Group Process .
The elicitation of knowledge in a group setting allowed the collective development of ideas,
and feedback. It also allowed the presentation of information on knowledge acquisition techniques
and structures used in other fields, and the facilitation of discussion on how they could be used in
System Dynamics. The group sessions were conducted with 5 local System Dynamics practitioners,
all had done some modelling using System Dynamics, but only one had published in the the field.
First, information was presented regarding knowledge acquisition techniques and structures that could
possibly be borrowed from expert systems development, this was followed by discussion, and then a
modified version of the Nominal Group Technique was conducted. This process required two 2 hour
sessions and individual end interviews. The steps in the Modified NGT process are listed below:
Step 1: Silent Idea Generation . Each participant, writes down System Dynamics activities believed
to be central to problem formulation and model conceptualization. A System Dynamics activity is
described by indicating who is involved, and the knowledge acquisition methods and structures used.
The activity lists are given to the group facilitator.
Step 2:Group Round-Robin listing of Activities: Sequentially, without explanation or comment from
the group, the facilitator writes all of the System Dynamics activities on the Board for everyone to
see. Each knowledge acquisition technique and knowledge structure is clearly indicated.
Step 3: Discussion and Clarification of Listed Activities Each activity is discussed informally by the
group to gain clarity. Activities may be added, deleted or combined during the discussion.
Step 4: Individual Written Ratings of Activities: Participants silently and independently rate each
activity based on the effectiveness of its knowledge acquisition techniques and knowledge structures.
Step 5: Discussion of Ratings with Group facilitator: The group facilitator meets individually with
each group member. They discuss the ratings and comments made in step 4,
Step 6: Final Written Rating: Each participant silently and independently rates each activity
Twelve acti were generated. All participants indicated, given more time, they would
have listed more activities. Of the 12, 9 activities included knowledge acquisition techniques other
than the non specific interview. There was no consensus with the first ratings, and no significant
convergence with the second ratings. On a scale of 1 to 10 (10 means ideal way to conduct activity),
8 activities received ratings of at least 8, and 5 were rated 9 or above. Of the 6 activities rated higher
than 8.7, protocol analysis was used in 3 and process tracing 2. Knowledge structures suggested
included simple variables, time histories, scenarios, rules, tables, grids, and causal loops. The group
felt the exercise was insightful regarding knowledge acquisition and knowledge representation. They
also concluded that problem formulation and model conceptualization activities are partially context
and domain dependent.
Interviews
Interviews were the most direct method of eliciting information. The personal contact, and
undivided attention had advantages over group sessions, and contact through mail. Time and
availability constraints limited the number of experienced System Dynamics analysts that could be
interviewed to three. The three analysts interviewed averaged 22 years of experience in System
Dynamics, had published papers on System Dynamics and taught courses involving System
Dynamics, The questions were distributed in advance and a semi structured interview was conducted.
In addition to the questions, each interviewee was given a short document outlining the research. For
the initial interview, the time spent ranged from one session of one hour to three sessions totally seven
hours. Five multiple part questions were asked. Two were dealing with knowledge elicitation and
knowledge representation structures, and one question requested information regarding the
respondent's involvement with System Dynamics. The other two questions dealt with the preparation
needed to conduct System Dynamics studies, and will be addressed in a separate paper. A second set
of interviews was conducted, asking four questions concerning a knowledge acquisition process
presented. These interviews were each one session ranging from one to two hours. Only two of the
experts were available for the second set of interviews. The main hypotheses for the interviews were:
1) Experts will provide information distinctly different from each other, and some experts will be
unaware of significant work done regarding knowledge acquisition in System Dynamics.
2) Experts will provide more indepth information than the group session.
3) Due to time demands on some experts, all questions may not be completely covered.
4) Respondents will give favorable responses to the knowledge acquisition process presented, but will
be reluctant to employ it, until it has been used in one or more studies, and the results presented for
evaluation by other System Dynamics analysts.
While all three respondents used interviews as an elicitation technique, the structuring of the
interviews ranged from very informal to semi structured, The first analyst also relied on interactive
modeling and detailed questionnaires, the third also depended heavily on observation, sitting in on
meetings and being involved in the organization, and all three included literature review as a
knowledge source. The first analyst depends primarily on causal diagrams and flow diagrams. His
position was "if System Dynamics stopped with a causal model, in most cases it would be fine". The
second analyst felt the most useful knowledge representation structures were time histories. He felt
after generating and/or simulating time histories, they should be linked to causal loops. The third
analyst relied heavily on time histories, but felt the most important knowledge representation was
verbal, the use of words in communicating with the client and other system participants. The third
analyst also felt the use of knowledge representation structures varies with the background of the
client, and that an analyst should select structures after they become familiar with the study
participants. Both experts, interviewed the second time, were supportive of using the suggested
approach to problem formulation and model conceptualization. They felt its use should be pursued.
They suggested adding observation as an elicitation approach and considering what preliminary study
is needed by the System Dynamics analysts to become familiar with the domain. The former was
added to the approach. The latter will be handled at a different time. The modified approach appears
below in Exhibit 2. It draws on inputs from the Delphi process, and the group process, as well as the
interviews.
Delphi
The use of the Delphi technique allowed knowledge elicitation from a representative number
of System Dynamics experts from a diversity of locations. The multiple questionnaire nature of the
Delphi, allowed feedback among the experts. Since the Delphi is currently still in the second round,
this paper only deals with results from the pre-Delphi questionnaire and the first round. The
hypotheses included all of the hypotheses stated above for interviews, in addition to the following:
1) Slow response by some respondents will slow down the Delphi process, 2) Due to experts’ busy
schedules there will be a fairly high participant dropout, and 3) Due to uneven concern over
- 180 -
knowledge acquisition, and this Delphi, some responses will be very detailed, and others brief. To
minimize participant dropout, a letter and pre-Delphi questionnaire were sent to 49 System Dynamics
analysts, explaining the purpose and process of the Delphi.
Of the 49 analyst sent pre-Delphi questionnaires, 34 responded, 30 agreed to participant, and
21 responded to the first round. The average rating of the level of importance placed on developing
knowledge acquisition techniques and structures in System Dynamics was 4.3, on a scale of 1 to 5
(S=very important), Twelve different elicitation techniques were suggested for problem formulation
and fourteen for conceptualization. They included structured interview, brainstorming, issuing
questionnaires, interpretive structural modeling, Delphi, and interactive model developing in group
sessions. Most analysts combined more than one elicitation technique in their suggested approach.
The respondents suggested 23 distinctly different knowledge representation structures for problem
formulation and 21 different structures for model conceptualization. Many of the same structures
were suggested for both phases. In addition to the System Dynamics knowledge structures mentioned
earlier in this paper,.a sample of the structures used by analysts include: prototype model output, list
of critical management issues, environmental scenarios, Axelrod's cognitive map, Bullet chart,
hexagon diagrams, and table functions. As expected, some responses were quite detailed, while
others were extremely brief.
A PROBLEM FORMULATION AND MODEL CONCEPTUALIZATION PROCESS
Problem formulation begins with an unstructured interview with the client, or an informal
brainstorming session involving key members of the analysis team and key system participants. A
preliminary problem definition is sketched during this activity.
Collecting and analyzing organizational data, and eliciting the interpretation of this data by
system participants is important in problem formulation. The System Dynamics analysts may use
protocol analysis to restructure some of this information, and present the restructured information to
the system participants for evaluation. Whenever possible, the analysts should use observation as a
source of knowledge.
A series of interviews going from unstructured to semi structured are used to: formulate a
concise study purpose, list the resources relevant to the problem and identify their different states
(from this a ‘resource-state table’ is constructed), and develop the initial dynamic hypotheses with
supporting graphs representing the reference mode. The agenda of each interview is determined by
the results of preceding interviews.
Task analysis is used to identify the policies and generate a policy - policymakers grid,
which indicates which system participants influence each policy.
Starting with the initial dynamic hypothesis and resource/state table, develop a flow
diagram without information links.
Have participants identified through the policies and policymakers grid evaluate the flow
diagram and clarify the policies by developing policy rules. Have the participants use process
tracing to generate scenarios based on their involvement with policies. These scenarios and policy
tules are used by the System Dynamics analysts or in a group session with system participants to
add information links to the flow diagram. Review the flow diagram with participants. This is an
iterative process requiring additional scenarios and revisions to the policy rules and flow diagram.
xhibit 2: The "Modified Approach" to Problem Formulation and Model Conceptualization
CONCLUSIONS
System Dynamics analysts uneven awareness of knowledge acquisition developments in
System Dynamics and other fields, coupled with their interest in further developing knowledge
acquisition techniques and structures (a high interest was displayed in the group.sessions), led to the
conclusion that knowledge acquisition skills should be more explicitly included in System Dynamics
training and more consciously used in practice. An extension of the group process used in this
research would be useful to include in the System Dynamics curriculum.
The diversity in responses, and the position expressed in all three processes that knowledge
acquisition is context and domain dependent, led to the conclusion that there is no ideal approach to
knowledge acquisition. The data collected, particularly through interviews and the Delphi process,
~ 181 -
would be useful in developing an expert system to select knowledge acquisition techniques and
structures based on the context and domain of the situation. This would be a helpful training tool for
students, and a decision support system for novices in System Dynamics studies. A group process
with Delphi participants should be used to collectively clarify what analysts feel should be done to
enhance knowledge acquisition in System Dynamics.
It appeared the positions of the different analysts interviewed directly reflected their stated
view of System Dynamics. One analyst felt System Dynamics is an important tool that must find its
proper place in science. This position is reflected in his tendency to draw on techniques widely used
in other areas such as interactive modelling and the use of questionnaires, and his belief that "in most
cases it would be fine to stop the System Dynamics process once a causal diagram is developed.”
Another analyst felt System Dynamics is the central organizing principle of life, and placed a heavy
emphasis on observation, verbal communication, and the need for the analyst conducting a study to be
a key decisionmaker in the system. To establish a strong correlation between analysts’ views of
System Dynamics, and their approach to knowledge acquisition, more analysts must be interviewed,
‘The reaction to new techniques and structures indicate it is worth pursuing their use in actual
System Dynamics studies to determine under what conditions the new techniques and structures are
useful. Preliminary indications are that group process techniques are more effective, and that
knowledge structures using the participants’ terminology are preferred.
REFERENCES
Cordingley, Elizabeth S., Knowledge elicitation techniques for knowledge-based systems, Knowledge
Elicitation: principles, techniques and applications, Dan Diaper (ed.) Ellis Horwood
Limited, Chichester, U.K., 1989, p.89-172
Coyle, R.G., Management System Dynamics, John Wiley, London, 1977
Forrester, J.W., Industrial Dynamics, Cambridge (Mass.) 1961
Forsythe, Diana E., B.G.Buchanan, Knowledge Acquisition for Expert Systems: Some Pitfalls and
Suggestions, IEEE Transactions on Systems, Man, and Cybernetics, Vol. 19, no.3, p.435-442
Johnson, Paul E., I. Zualkerman, S. Garber, Specification of Expertise, International Journal Man-
Machine Studies, Vol. 26, 1987, p.161-181
Magee, Kevin The elicitation of knowledge from designers, Design Studies, Vol.8, 1987, p.62-69
McGraw, Karen L., K. Harbison-Briggs, Knowledge Acquisition Principles and Guidelines, Prentice
Hall, New York, 1989
Morecroft, J.D.W., A Critical Review of Diagramming Tools for Concepmualizing Feedback System
Models, Dynamica, Vol. 8, 1982, p.20-29
Morecroft, J.D.W., Executive Knowledge, Models and Learning, European Journal of Operational
Research, Spring 1992
Porter, Alan, A. Thomas Roper, Thomas Mason, F.A. Rossini, Jerry Banks, B.J. Wiederholt,
Forecasting and Management of Technology, John Wiley, New York, 1991
Randers, Jorgen, Guidelines for Model Conceptualization, Elements of the System Dynamics Method,
Randers (editor), MIT Press 1980
Richerson, GF e Problems with Causal-Loop Diagrams, System Dynamics Review, Vol. 2, 1986,
p.158-1
Richardson, G.P., J.A.M.Vennix,D.F.Andersen,J.Rohrbaugh,W.A. Wallace, Eliciting Group
Knowledge for Model-Building, P.M. Milling,E.O.Zahn (eds), Computer-based Management
of Complex Systems. Proceedings of the International System Dynamics Conference, 1989,
p.343-357
Sancar, Fahriye, RJ. Cook, Cognitive Criteria for Structuring System Dynamic Models, Proceedings
of the International System Dynamics Conference, 1985a, p. 762-775
Sancar, Fahriye, RJ. Cook, Aids for Customizing Generic System Dynamic Models: A Community
Development Example, Proceedings of the International System Dynamics Conference,
1985b, p. 744-761
Vennix,J.A.M., Mental Models and Computer Models design and evaluation of a computer-based
learning environment for policy-making, University of Nijmegen, Netherlands, 1990
Wolstenholme, Eric F., System Enquiry: A System Dynamics Approach, John Wiley, Chichester, 1990
- 182 -
