An Expert System to Aid in Model Conceptualization
Willard Fey
John Trimble
Georgia Institute of Technology
_Industrial and Systems Engineering
Atlanta GA 30332, USA
Abstract
Using interviews and a Delphi exercise, valuable information was collected
from.experts concerning the problem definition and model conceptualization stages of
system dynamics studies. This research examined when to select and use various
knowledge acquisition techniques and knowledge representation structures. Now this
information is being incorporated into a comprehensive expert system for training
novice SD analysts. A teacher model and a student model are being incorporated into
the expert system to provide pedagogical flexibility and intelligent tutoring. This paper
reports on the initial work on the prototype instructional expert system, and plans to
extend this prototype.
THE PROBLEM
The problem definition and model conceptualization stages of SD are most
difficult for students and novice practitioners. The effective implementation of these
stages are considered to require both art and science. There is a need for educational
materials to train novice SD analysts in conducting problem definition and model
conceptualization. Most system dynamics textbooks, articles, and courses concentrate
on model formulation, verification, and simulation. The development of training
material dealing with problem formulation and model conceptualization is a very
practical research endeavor. Any such material will be most effective if the experience
of expert practitioners is incorporated. Effectively exposing analysts to the diverse
options in applying the SD methodology requires capturing and replaying the
experiences of a range of expert SD practitioners. This can be done best through an
automated training package such as an expert system. Any pedagogical package to
teach these stages should allow for flexibility in teaching approaches and diversity in
student background.
Traditional computer aided instruction (CAI) and computer based tutoring
(CBT) allows no flexibility in adjusting the teacher model or student model. Various
intelligent computer aided instructiorl (ICAI) packages allow for diverse student
situations and dynamic adjustment of student models but do little to allow flexibility in
teacher models. Often CAI models are not.derived from knowledge obtained from
leading experts in the selected domain.
Instructional expert systems incorporate the experience of highly qualified
experts. However, these systems generally pay little attention to the student model.
This ongoing research is focused on developing a comprehensive instructional
expert system that incorporates a flexible instructor model and a flexible student
model.
This expert system will serve as a training tool for students to learn the
concepts, philosophy, and structures involved in SD. This system will also address
112 “SYSTEM DYNAMICS '93
the approaches: and methodologies required to interact with system participants in
conducting a SD study.
An expert system for training problem definition and model conceptualization
provides portability and standardization. The portability allows it to be used in settings
that have no access to experts. The standardization provides the SD community a
model to be checked for validity. After extensive testing and validation, the revised
expert system could serve as a benchmark for SD practitioners. The combination of
Boreability and standardization contribute to make this expert system a useful tool for
D courses.
RESEARCH APPROACH
This is a significant problem which requires considerable resources. It has
been divided into segments. This current paper addresses: (1) the development of a
generic instructional expert system (IES) with flexible teacher and student models, (2)
utilization of expert knowledge from Trimble (1992) to establish a domain specific
teacher model, (3) construction of a skeletal prototype IES that addresses selected
concepts and methods critical to problem definition and model conceptualization and
(4) future enhancements to the prototype expert system.
The research on ICAI was examined. The focus was on the structure and
components of various ICAI packages. The developments in teacher and student
models were studied in detail. The philosophy of education is researched to gain a
basic understanding of how philosophy relates to teaching methods. Several computer
based tools that aid SD analysts were studied.
After examining ICAI and SD literature a-generic IES was developed that
provided flexibility in the teaching process and the student model. Knowledge elicited
from novice practitioners is used in developing the student model. The teacher model
is based on information obtained from the interviews with experts and the delphi
exercise. The conclusions of SD analysts regarding knowledge acquisition in SD from
Trimble (1992) were examined. Protocol analysis was performed on the transcripts of
experts interviewed. These activities generated a domain specific knowledge base that
helped define the teacher model of the IES.
A key subset of SD concepts was selected to develop the initial prototype.
This was done to allow for testing the flexibility of the IES before constructing an
extensive set of tutoring modules. The knowledge base was enhanced to handle test
development and student errors. Finally, the knowledge base was expanded to use
student errors to adjust the student model.
INVESTIGATING PREVIOUS RESEARCH
Pigford and Baur (1990) identify three basic components for any expert
system: inference engine, user interface, and knowledge base. The inference engine
is responsible for deciding when and how knowledge from the knowledge base is
applied to make decisions. The user interface is the communication link between the
user and the expert system software. It takes on adued significance with instructional
expert systems since the form of communication between user and machine has a
significant impact on the success of the pedagogical process. The knowledge base
contains the domain knowledge, which can be represented using a variety of
knowledge structures. The knowledge base can be subdivided by the type of
knowledge representation structures such as facts, rules, and frames. Alternatively,
the knowledge base could be subdivided along topical lines. In the case of an
SYSTEM DYNAMICS '93 13
instructional expert system the knowledge base could be divided into knowledge
related to student, teacher, pedagogical process, and instructional domain.
Corben and Wolstenholme (1992) developed a computer based Delphi method
to assist the development of SD models by providing the ability to create consensus
influence diagrams. Adjustments could make this an innovative group based learning
tool.. Kaupe (1992) describes a German-language computer-based. tutorial
SDAQUIRE developed at Mannheim University. This tutorial consists of twelve
lessons and is oriented to SD model building. The tutorial starts with the basic
concepts of systems research, and steps through the model building process all the
way through validation of model structure, parameter values and model behavior.
While providing an good orientation to SD, this tutorial does not allow for adjustments
in the pedagogical process, or account for student diversity.
Rickel. (1989) in surveying research in ICAI identified eight possible
components of an ICAI package. They are learning scenarios, domain knowledge
representation, student model, student diagnosis, pedagogical knowledge, discourse
management, problem generation, and user interface. A learning: scenario is the
circumstances under which the student's learning is to take place. The balance. of
control between student and tutor is critical. Knowledge representation can vary from
canned presentations to complex causal relationships, scripts, or semantic networks.
The student model maintains information of the student's knowledge and capabilities.
It is adjusted based on the student's actions and responses. The student diagnosis
component evaluates student errors to establish the basis for the misconception.
Pedagogical knowledge deals with instructional strategies... This includes choosing an
effective presentation method and determining how to deal with student errors.
Discourse management is concerned with the determination of what the tutor and
student need to communicate. Problem generation should allow flexibility in testing
the student's competency. The user interface involves the presentation of text,
graphics, audio and video, as well-as acceptance of user input. In many ICAI
packages some of these components are. combined.
Vassileva (1990) surveys student modeling techniques and. presents a
classification of student models. This paper classifies student models based on
Tepresenting student domain knowledge versus general characteristics of the student,
and the techniques used to update the student model. An approach is presented that
includes a model of the student's domain knowledge that is updated by comparison,
and a model of student features, such as learning rate, level of concentration, and
preferred style of presenting material.
Chen (1991) examines teaching in the context of a three layer framework. The
three layers are: (1) bottom layer (physical layer) consisting of various physical media,
(2) middle layer (layer of logical contents) consisting of the materials of a particular
learning unit, and (3) top layer (external view) addressing the teacher's cognition,
selecting from different views indicating different patterns of the teaching behavior.
Educators have studied the relationship between philosophy, teaching methods
and curriculum. In Smith (1990) six schools of philosophy are discussed: idealism,
realism, pragmatism, existentialism, essentialism, and perennialism. Idealists feel
students should learn through contemplation or. the dialectic, and the curriculum
focuses on teaching students to think as opposed to memorizing facts.. The
instructional methods of realists include a wide repertoire of traditional approaches,
such as lecture, discussion, and experiential activities: Pragmatists apply a problem
centered learning approach. The focus in on problem solving, personal experiences,
and interaction between student and environment. Because the philosophy of
existentialism is unique-and individualized, it opposes the standardization of schools.
114 SYSTEM DYNAMICS '93.
The curriculum emphasizes skills and subjects that explain physical reality and social
reality, and depict human choices. Essentialism emphasizes discipline, and diligence
on the part of the student.. The main teaching methodology is. drill and rote
memorization. In perennialism the key purpose of education is to promote
intellectualism, as opposed to learning through problem solving. Teaching methods
include well organized narratives, coaching in basic skills, and. selected use of the
Socratic: model of probing questions. Experts in most domains do not isolate their
teaching strategy to a single educational philosophy. The SD experts that participated
in this study frequently applied different teaching approaches under different
conditions. The approach in developing the teaching model is to emulate different
experts not the idealized teaching philosophies and corresponding methodologies.
A GENERIC INSTRUCTIONAL EXPERT SYSTEM
The instructional expert system in this research consist of five components:
student model, teacher model, domain knowledge model, user interface and inference
engine. The inference engine is established by the expert system development tool
used. Problem generation is handled by the domain knowledge model. The
pedagogical approach. and philosophy is embedded in the teacher model. Discourse
management is handled by the user interface, and student diagnosis is incorporated
into the student model. The learning scenario is developed by the student and teacher
models. 2
For the prototype expert system in this research a three layer framework of the
teacher model is constructed. It is somewhat different from Chen (1991). The bottom
layer is the philosophical perspective of the teacher regarding the domain subject. For
example, how does one regard SD - simply as one of many tools for displaying
causality, a simulation language, or a broad paradigm that explains how the world
operates. G
The second layer is the relevant skills base of the teacher. The skills level of
the teacher model consist of general and domain specific skills. “The general level
addresses the knowledge level of each resource (none, novice, experienced, expert),
and the teaching level (elementary, HS, freshmen, advanced undergraduate, and
graduate level). The domain specific skills information should indicate the teacher
experience regarding studying SD, teaching SD, and practicing SD.. The third layer in
this framework is the resources available. This would include literature, experts,
computer resources, group tools, etc..
In interactively constructing the teacher model the user responses to questions,
establishing a knowledge base regarding the model teacher. This process starts with
the bottom layer:and proceeds to the top.layer. Each layer establishes direction and
constraints for subsequent layers.
In constructing the teacher model, the user is in part establishing the student
model. The resources available layer is identical for student and teacher. This layer is
the basis for communication between student and teacher. The general information in
the resources layer of the teacher and student models consist of an availability list
indicating the availability of interactive multimedia, video clips, audio clips, group
decision making software, interpretive structural modeling, electronic mail contact with
experts, non-computer group sessions and group techniques, and direct contact with
experts. The domain specific resource availability list indicates the level of availability
of Stella models and microworld models.
Additional inquiries of the user are needed to further establish the philosophical
perspective of the student (layer 1), and the skills level (aver 2) of the student model.
SYSTEM DYNAMICS '93, 115
The philosophical perspective layer of the student model addresses the student's view
toward learning using ICAI. Is the student optimistic or pessimistic?, the preferred
style of presenting material, and the level of concentration of the student are aspects of
this layer.
The skills level establishes the skills of the prospective user. This corresponds
to the same two dimensions this layer has in the teacher model: (1) knowledge
regarding the instructional domain, and (2) general knowledge regarding knowledge
acquisition, conveyance, and representation. If the teacher model prescribes the use of
certain resources that the student model has little or no skills, the IES must first
provide training regarding the learning resource, before using it in instruction of the
domain subject. The student model skills level has general information that indicates
the student's general knowledge level (elementary, HS, freshman, advance
undergraduate, and graduate). The domain level student model indicates the student's
knowledge level in SD, problem-solving, causal loops, and differential equations
Layer 3 allows extension of the teaching and learning process beyond the
expert system. This implies interrupting the IES with a partially constructed
knowledge base and returning to this exact point. The IES could activate software that
could aid the learning process, such as a simulation program, a Stella model, or a
short video clip. Upon completion of this external computation, control should
automatically be returned to the IES. Additionally knowledge may be made available
to the IES as.a result of the interrupting program. A non-programmed interruption
requires leaving the IES to complete learning tasks unattached to the IES. It may
involve a literature search, group interaction, correspondence, or experimentation. It
may take hours, days or weeks. The state of the IES is saved when the IES is exited,
and maintained until reactivated by the user.
This layered approach to establishing teacher and student models allows one
user (instructor) to establish certain aspects of the environment and other users
(students) to establish other environmental perspectives. Alternately a single user
could. construct both student and teacher models. All this is done in advance of the
actual tutorial sessions.
INITIAL PROTOTYPE DEVELOPMENT
The initial prototype is based on the generic three layer structure. This initial
prototype will be concerned with teaching: (1) components of problem definition and
model conceptualization, (2) the process of eliciting knowledge from system
participants, and (3) select knowledge representation structures. This early version
will be developed to train users with general knowledge at the level of college
students, and basic knowledge of SD equivalent to a 8-10 week course. Level 5 by
Information Builders Inc. is the expert system development tool used to create the
initial prototype. Level 5 generates an expert system with an inference process of
backward chaining, and a rule based knowledge representation structure.
~ For the initial prototype the interactive computer based access to information in
the resources layer of the teacher and student models will be limited to text and simple
graphics. The rule based system will include limited hypertext capabilities. This
hypertext feature is recognized as a critical interactive learning feature. It allows the
user to inspect information in a non linear fashion. The user can decide when to view
more information on select features. External learning tools such as electronic. mail
contact with experts, non-computer group sessions and group techniques, and direct
contact with experts are available.
116 SYSTEM DYNAMICS '93
In the initial prototype the teaching level and student model skills-will
correspond to the advance undergraduate level.
To simplify the prototype, the student philosophical perspective is limited to
the level of concentration of the student (high, medium, low). The philosophical level
of the teacher model focuses on the significance of the subject matter. In our prototype
it addresses whether the teacher views SD. as (1) one of many tools/techniques for
simulating and forecasting, the SD model is just a theory about a system, a simulation
language, (2) one of a few paradigms that explains socioeconomic reality, the: best
tool for medium to long range planning, or-(3) the single most important paradigm for
explaining and understanding how the world operates.
The information related to an SD concept that an expert feels is significant is
based on his/her view of SD. Evaluation of the transcripts from expert interviews and
the Delphi results from Trimble (1992) led to the identification of a list of information
to be used to convey each concept. This information is used to construct the various
scenarios.. Table 1 summarizes, by teacher model philosophy, the key information
used to construct the set of scenarios that convey the components of problem definition
and model conceptualization.
Table 1: Summary of features for ‘components of conceptualization’ scenario
PHILOSOPHY [COMPONENTS OF CONCEPTUALIZATION
‘Dis the main] -talking with clients about the problem without using feedback
paradigm _for| and closed system terminology.
explaining how | -first identify the objectives and clarify the problem
the world| -the analyst identifies time histories of related variables, and then
functions identify feedback related causes.
-presents the truth & discrepancies to the’client, clarifies client
objectives, historical conditions and client attitude toward
conditions.
SD is the most] -start with time histories and develop verbal scenarios which
powerful tool for} explain the relationships in the time histories.
medium to long| -draw the model out on paper
range planning -identify variable you have data for those you do not have data
for (not collected or uncollectable)
-determine how to deal with missing information.
-formalize causal loop by starting with very broad relationships,
key levels , then key rates, then add auxiliaries.
SD is one of] -identify system or aspect of the system to be studied. by using
many tools, A SD interviews, interactive modeling and detailed questionnaires
model is just a| -develop the final conceptualization model using causal diagrams.
theory about. a ‘
system.
The student and teacher model knowledge bases determines how to construct
the instructional scenario which consist of.a series of screens responsible for
conveying a key concept. The student and teaching model are also the basis for
constructing the tests of the concepts conveyed.
SYSTEM DYNAMICS '93 17
Table 2: Summary of features for ‘components of knowledge acquisition’ scenarios
PHILOSOPHY
COMPONENTS OF KNOWLEDGE ACQUISITIO
APPROACHES
SD is the main
paradigm . for
explaining how
the world
functions
-talking with clients without using feedback and closed system
terminology, in an unstructured interview.
-Observations of the system or situation and its participants.
Tour facilities, sit in on meetings, and record if feasible.
-Check time histories against each other and Tesponses from
system participants interviewed.
SD is the most
powerful tool for
medium to long
range planning
-Unstructured and semi structured interviews
-Academic perspective: read about the problem, what does the
literature say about the problem
-Talk to system participants to get information and for the good
of the group.
SD is one of
many tools, A SD
model is just a
theory about a
system.
-Interviews
-Interactive modeling
~Detailed questionnaires
Table 3: Summary
of features for ‘components of knowledge representation’ scenarios
PHILOSOPHY
COMPONENTS OF SELECT KNOWLEDGE
REPRESENTATION
SD is the main
paradigm _ for
explaining how
the world
functions
-Most important knowledge structure is the choice of verbal
expression used.in communication. with client and other system
participants.
-Time histories are identified and presented to clients.
-The use of knowledge representation structures and tools vary
with the background of the clients.
-The central focus is to capture how people feel and how control
is exercised.
SD is the most
powerful tool for
medium to long
range planning
|-In problem formulation, time histories are most useful. Time
histories are linked to flow diagrams.
-Only after generating and/or simulating time histories are they
linked to causal loops, as a way of explaining the model.
SD is one of
many tools, A SD
model is just a
theory about a
system.
“The main forms of knowledge representation are causal
diagrams and structure diagrams
-Initially difficult to go from causal diagrams to structured
diagrams. Once this is mastered one can easily go from causal
diagrams to equations.
-Causal diagrams may be divided into sectors.
118
SYSTEM DYNAMICS '93
Tables 2 and 3-indicate various components of scenarios conveying knowledge
acquisition approaches and knowledge representation structures as they correspond to
the.different philosophies. These components are expanded as determined necessary
by the student and teacher models. The selection of which components to include in a
given scenario is also determined by the state of the student and teacher models.
‘ After an instructional scenario is completed, the student is tested. The
student's responses to the test are used to update. the student model. Incorrect
Tesponses as well as correct responses to the concept test provide information about
the student's skill level. Also these responses can provide insights on certain student
Philosophical characteristics such as preferred learning style and level of concentration.
is revised student model in conjunction with the teacher model are used to construct
the next instructional scenario. If errors were made on the previous concept test, the
new instructional scenario is used-to. convey the same concept. If the student
successfully completed the concept test, the instructional scenario corresponds to a
new concept.
The student's responses to concept tests does not alter the teacher model. The
teacher model is established by responding to questions at the beginning of the
session. The teaching approach can be altered during the instructional session by
invoking the module which establishes the three layers'of the teacher model. While
this is not the usual mode of operation it is an available option.
There are two techniques for generating a variety of instructional scenarios for
the same concept. Scenarios can be constructed from a library of preformatted
paragraphs and clips. The variety in scenarios is directly dependent on the size of the
library. The second technique employs natural language: generation to construct a
scenario when needed based on the user’s.natural language input and the current state
of the teacher and student models. This prototype system uses the first technique.
CONCLUSIONS
Results of this research to date indicate that the knowledge elicitation process in
SD studies centers on the interview, with data gathering, identifying time histories and
direct observation. close behind. However, the degree of complexity and
sophistication employed in interviews varies as do the processes used to complement
the interviews. Those more versed on elicitation techniques tend to use more complex
interviews and various group processes and techniques.
There was a strong indication that analysts adjust their approach based on
problem domain, decision making constraints, and analysts’ preferences. Experts
most committed to SD as the most significant paradigm view knowledge acquisition in
SD as a complex feedback control process. This process involves eliciting information
from people, gathering information from data bases (which may be fed by people),
getting information from direct observation, and generating information from inferred
logic or theory. This control process is conducted until there is satisfaction with the
results or there is a realization that the additional information attainable is insignificant.
SD experts indicated it was very important to use and develop knowledge
Tepresentation structures relevant to system participants and based on the problem
domain. This supports the conclusion that there is a need for a more comprehensive
an 5 options for generating knowledge structures in the instructional expert system
The extensive nature of this project was underestimated. To thoroughly cover
the domain, using a range of teacher and student models requires enumerating an
SYSTEM DYNAMICS '93 119
extremely large number of teaching scenarios. To address this problem other
approaches to elaborating scenarios will be investigated.
Level 5 had limitations as a tool to develop an IES. A tool or development
language for a more comprehensive IES should be geared more toward intelligent
computer aided instruction (ICAI). It should provide more flexibility in knowledge
Tepreseniation, better output and display capabilities, and an easier interface to other
software.
An increasing number of computerized learning environments utilize
multimedia, distance learning, and computerized group communication. These three
techniques must be incorporated in future ICAI to maximize the benefits to users.
This research is useful to SD practitioners, instructors, and students. Since the
teacher model and instructional scenarios are based on the practice of experienced SD
analysts, a more completely developed IES can’serve as a benchmark and source for
new techniques for. practitioners and instructors. The IES will be a useful
enhancement to existing curriculum in SD.
The process of having experts review, refine, and validate this instructional
expert system will be a crucial aspect of the continuation of the knowledge acquisition
process. It will also serve as a basis for the refinement of this instructional expert
system. An initial test version of the prototype will be available the summer of 1993.
FUTURE DEVELOPMENTS
The prototype will be extended by expanding the computerized learning
resources and expanding the teacher and student knowledge bases. The fully
developed prototype instructional expert system will include a range of topics. It will
be designed to cover the elementary concepts in SD as well as details of several
knowledge representation structures, and knowledge acquisition approaches and
techniques. It will allow graphics, video, and audio in addition to text in the
scenarios.
The teacher model will be extended based on additional interviews and surveys
of SD practitioners and teachers. As the prototype is extended student users will help
determine the most effective student diagnosis and discourse management. This
information will be used to revise the student model and the user interface.
The cost effectiveness of different ‘multimedia, distance learning, and
computerized group software and hardware will be investigated. This effort will look
at both the short and long term potential impact these technologies can have on
instructional tools for SD. ia
The instructional domain will be expanded. Additional concepts will be added
to more thoroughly cover the problem definition and model conceptualization phases
of SD. Concepts that are dependent on the SD study domain will be examined and
possibly added to the IES. The additional educational levels mentioned in the generic
IES will be added. Test generation will be expanded to complement developments in
student diagnosis and discourse management.
Further research in educational philosophy coupled with interviews and group
sessions with SD practitioners and students will be the basis of enhancing the
philosophical perspective layer of both the student and teacher models.
A more comprehensive version of the prototype IES is scheduled for
completion by first quarter 1994. The degree to which the future developments listed
above are included in the 1Q94 release will be based on the funding support received.
120 SYSTEM DYNAMICS '93
REFERENCES
Chen, Z., 1991, From Student Model to Teacher Model: Enriching Our View of the
Impact_of Computers on Society, Computers & Society, Vol. 21, Nos. 2, 3
and 4, October 1991
Corben, D. A., and E. F. Wolstenholme, 1992, A:-Hypermedia based Delphi Tool for
Knowledge Acquisition in Model Building, Proceedings International SD
Conference 1992, Netherlands
Kaupe, Guido, 1992, SDACQUIRE - A Tutorial for Imparting the Methodical
Foundations of System Dynamics, Proceedings International SD Conference
1992, Netherlands
Pigford, d.V. and Greg Baur, 1990, Expert Systems for. Business: Concepts and
Applications, Boyd & Fraser, Boston
Rickel, Jeff W., 1989, Intelligent Computer- Aided Instruction: A Survey Organized
Around System Components, JEEE Transactions on Systems; Man, and
Cybernetics, .. Vol. 19, no.1
Smith, Tom E.C., 1990, Introduction to Education, West Publishers, New York
Trimble, John, 1992, Knowledge Acquisition and the System Dynamics
Methodology, PhD dissertation, Georgia Institute of Technology
Vassileva, Julita, 1990, A Classification and Synthesis of Student Modelling
Techniques in Intelligent Computer-Assisted Instruction, Proc. of ICCAL '90,
Hagen, Fed. Rep. Germany, Lecture Notes in Computer Science, No.458,
pp. 202-211, Springer, Berlin
SYSTEM DYNAMICS '93. 121
CC BY-NC-SA 4.0