Modeling Risk Classification Scheme for System Dynamics
Modeling
Michael J. Schwandt
Virginia Tech
Grado Department of Industrial and Systems Engineering
250 Durham Hall, Blacksburg, Virginia 24061
phone: (540) 231-6656, fax: (540) 231-3322
schwandt@ vt.edu
Abstract
System dynamics modelers face a broad spectrum of risks toward achieving project objectives.
As they gain experience, their risk identification and management capabilities increase. By
applying classification techniques from taxonomy development, the collective knowledge of
previous modelers has been captured in a classification scheme for system dynamics modeling
risks. The classification scheme allows modelers to more efficiently and effectively consider
modeling risks by reducing the variation in their knowledge levels. The classification structure
is focused on the steps of the system dynamics modeling methodology and the achievement of
system knowledge and improvement objectives. As part of a broader modeling risk management
approach, the risk classification scheme assists modelers in identifying and prioritizing the
anticipated sources of modeling risks for a project. With that knowledge, they can more
effectively identify the appropriate techniques for managing risks and then efficiently apply those
techniques in a timely fashion through the entire project cycle.
Key words: system dynamics methodology, modeling risk, taxonomy development
1 Introduction
While system dynamics modeling projects have many similarities, individual projects also
present unique challenges, or risks, to modelers and their clients. The system dynamics
methodology is designed to assist modelers in addressing many of these risks. However, the
complexities of modeling projects can increase the difficulties for modelers to successfully
identify the specific risks on a particular project. This can result in the projects not effectively
supporting their intended objectives.
Researchers have recognized this challenge through the years. Although not necessarily calling
them “risks,” discussions of modeling risks and how to manage them can be found in many
literature sources, focusing on testing methods (Sterman 2000; Forrester 1961). However, the
literature has not revealed focused efforts to capture these risks in a structure that supports
modelers in consistently identify the specific challenges to project success.
The internal auditing profession faced similar challenges in their assessment of business process
risks. In response to these challenges and others, they developed an approach to business risk
management that includes a business risk dictionary, a risk classification structure that strives to
improve the consistency with which risks are identified, discussed, and managed.
By adapting the principles of business risk management to the system dynamics modeling
process, a system dynamics modeling risk management process was developed. To support that
process, a taxonomy development process was applied to develop a system dynamics modeling
risk dictionary. This dictionary, a classification system for modeling risks, allows modelers to
use a common language, drawing upon the experience of other modelers, to efficiently and
effectively identify and source the risks to successfully developing and using system dynamics
models.
As a result of using the resulting system dynamics risk dictionary, modelers are guided to more
consistently consider a broader range of modeling risks. They also have a framework for
discussing modeling risks with clients, process participants, and other modelers. Even if the
system dynamics modeling risk management process is not applied, consistency benefits are
achieved.
2 Business Risk Management
In 1985, the Committee of Sponsoring Organizations (COSO) of the Treadway Commission was
created and chartered with studying the factors that contribute to fraudulent financial reporting.
The commission was charged with developing recommendations for improving financial
controls.
The commission developed an integrated approach to internal controls that focused on risk
management. Risk management was defined as “a systematic approach to identifying,
analyzing, and proactively dealing with risks” (Committee of Sponsoring Organizations of the
Treadway Commission 1994). This approach challenged auditors to enhance their traditional
transactional approach with the more proactive, process-focused approach.
Importantly, the commission suggested that the traditional focus on auditing financial risks be
expanded to focus on managing business risks. A business risk is defined as “the threat that an
event or action will adversely affect an organization’s ability to achieve its business objectives
and execute its strategies successfully” (Economist Intelligence Unit 1995).
Three essential elements were identified for successfully managing business risk and became the
defining characteristics of the business risk management approach (Economist Intelligence Unit
1995):
1. development of a common business risk language,
2. development of an effective organizational control structure, and
3. creation of a process view leading to business process control.
The common risk language is for all members of the organization to use when discussing
business risk. Without a common language, it is difficult to have a common understanding of the
business risks and the methods employed to manage them. Communication is hampered, which
negatively impacts the efficiency and effectiveness of the control environment.
The suggested framework, often referred to as a “risk dictionary,” for classifying the business
risk language organized the risks into these top-level classifications (Economist Intelligence Unit
1995):
e Environment risks - arises from external forces that could either put a company out of
business or significantly change the fundamental assumptions that drive its overall objectives
and strategies.
e Process risks - arises when business processes are not achieving what they were designed to
achieve.
e Information for decision making risks - arises when information used to support business
decisions is incomplete, out-of-date, inaccurate, late, or simply irrelevant
These top-level headings were offered as guidance, not as the only structure, for organizations as
they started down the path of implementing the business risk management approach. The only
strict requirement was that the organization develop a common structure for their risk dictionary
and utilize that structure throughout the organization. However, there is no evidence that the
business risk dictionary process has a foundation in a formal taxonomy development process.
3 Taxonomy Development
To provide a structured approach to the development of the system dynamics modeling risk
dictionary, taxonomy development methods were applied.
To determine if a taxonomy could be developed, the fundamental properties of a taxon, a
classification within a taxonomy, were considered (Ruscio, Haslam, and Ruscio 2006). If the
classes do not display these properties, then the risk dictionary would be more accurately
identified as a typology or an empirical classification. A taxonomy structure is a more rigorously
defined subset of a typological structure, which is a more defined subset of a general
classification system.
Strictly, a taxon has to display properties regarding its latent structure, boundary, and endurance.
A latent structure contains the fundamental nature of the construct and exists regardless of how
various individuals choose to conceptualize or measure it. The cases within the taxon should
share a deep commonality. This can be contrasted with the manifest structure, which utilizes
observable features and depends significantly on the theoretical assumptions and measurement
decisions that form the structural basis. Typically, the latent structure is inferred from
observable relationships between variables in the manifest structure.
Each taxon should identify a category with a distinct boundary. The taxon should have a finite
membership that could theoretically be counted. While distinctions with a taxon can be
identified on a continuous scale, the boundaries between taxa should be non-arbitrary and
objective. This criterion is the one that is most often the determinant of whether the
classification system captures taxa or categories. The criteria might be useful, but not identify
taxa. In addition, the taxa criteria should be objective at the latent level. The boundary criteria
and the taxa should be reasonably enduring, persisting for a timeframe that is significant to the
system being classified.
Since the modeling risk dictionary classification structure does not display distinct boundaries,
the structure is more properly classified as a typology, rather than a taxonomy. The iterative
nature of the system dynamics modeling process contributes to the lack of distinct boundaries.
A generic process for developing a taxonomy within an information technology environment
(Table 1) was used as the basis for developing the modeling risk dictionary (Haris, Caldwell,
and Knox 2003). The process starts by analyzing the existing information and determining the
user needs for the information. Combining the results of these analyses generates the initial
information vocabulary. The vocabulary forms the basis of the taxonomy, which is then applied
against the information and tested with users. This is an iterative process until the users agree on
the value of the taxonomy. Then the taxonomy is implemented for the broader community,
utilizing feedback from the community to guide improvements to the taxonomy.
Table 1. Information Technology Taxonomy Development Process
Steps Actions
Analyze Inventory information assets (current and planned)
Audit end-users’ information needs as well as their usage
and access patterns
Establish information vocabulary
Define, build, test, and Design and populate taxonomy
refine Index, link and cluster a test bed of information assets
Test with users and content managers
Implement, monitor, and Implement taxonomy, index all information assets
maintain Monitor usage and user/content manager feedback
The scope of this research encompasses the analysis step for the modeling risk dictionary. The
scope also includes the definition step through initial testing for both the risk dictionary and the
risk techniques database. Full testing and implementation of the modeling risk dictionary and
the risk techniques database require long-term consideration and is, of necessity, beyond the
scope of this research, but is envisioned as part of future work.
4 Developing the System Dynamics Modeling Risk
Dictionary
The system dynamics literature does not reveal any attempts to classify the risks that system
dynamics modelers could encounter during the planning, development, and utilization of their
models. Possibly, this reflects the focus on building confidence and credibility, which has its
roots in the early days of system dynamics (Forrester 1961). In addition, the business risk
viewpoint was developed in the late 1980’s and has been slowly emerging since then (Committee
of Sponsoring Organizations of the Treadway Commission 1994). The principles are now
available for this research.
Taking the viewpoint that the system dynamics methodology is a process leads to the possibility
of applying the business risk dictionary approach and developing a modeling risk framework, or
typology, that identifies the modeling risks that can be anticipated during the system dynamics
modeling process steps.
4.1 Developing the Modeling Risk Framework
The taxonomy development process (Table 1) was used as the basis for developing the modeling
risk framework. The analysis step generated the framework and led to the definition step which
populated the modeling risk dictionary. The specific research tasks within each of the steps are
described in the following sections.
4.2 Information Analysis and Needs Analysis
The system dynamics modeling process form the starting point for the modeling risk framework.
Applying the taxonomy development process, this framework was built from a review of
foundational system dynamics methodology literature, including theoretical and applied
publications. The methodological literature focused on subject matter expertise and the model
development literature provided the user perspective that the taxonomy development process
requires.
Although there is some variability in the names applied to the key steps in the system dynamics
methodology (Forrester 1961; Randers 1980; Richardson and Pugh 1981), a common iterative
process for developing system dynamics models involves these five key steps (Sterman 2000):
problem articulation,
formulation of the dynamic hypothesis,
formulation of a simulation model,
testing, and
policy design and evaluation.
These steps capture both of the primary objectives, increased system understanding and system
improvement, of system dynamics applications. This framework (Figure 1) is aligned with the
steps of the system dynamics modeling process, supporting a tight integration of the modeling
risk management with the overall method.
In addition to consideration of the system dynamics modeling method, the verification,
validation, and accreditation (VV &A) process often used in the defense modeling community
provides a structure for addressing simulation model credibility. This structure focuses on the
“four P’s” of people, project, process, and product (Balci 2005). Giving consideration to these
focus areas generated the system dynamics modeling framework for the modeling risk dictionary
(Figure 1). Within each focus area in the framework are a variety of potential risks.
Model Development
Problem Qualitative Policy Decision-Making
People |——>]_ | Quantitative Testing | -—>| Implementation
Project
People Process Product
Verification, Validation, and Accreditation Alignment
Figure 1. System Dynamics Modeling Risk Framework
In addition to the process-focused format, risk frameworks are often visualized in a tree format
where the specific risks are indicated where they are most likely to impact the achievement of
process objectives. For the modeling risk framework, the tree format branches are focused on
the achievement of the leaming and behavior improvement objectives (Figure 2). This structure
was finalized during the application of the information analysis steps in the taxonomy
development method.
(e
&, %, % e
%y %5, %. OF
%, Sy e- <A
6 “e
Process
Objectives
. ry
ey s
§ x S &
Ry 2 2 é
& & Re) £
¢ a & v
og &
Figure 2. Modeling Risk Tree Diagram
Starting from the COSO definition of business risk, system dynamics modeling risk is generally
defined as “the threat that an event or action will adversely affect the ability of the system
dynamics modeling project to achieve its system understanding and behavior improvement
objectives.”
The general definition is used as the basis for more specific definitions within each area of the
system dynamics modeling risk framework (Table 2).
Table 2. Top-Level System Dynamics Modeling Risk Definitions
Area | Risk Definition
Project Planning
Problem The threat that events or actions relating to defining the modeling
project and how the system dynamics method can contribute will
adversely affect the ability of the system dynamics modeling project to
achieve its system understanding and behavior improvement objectives.
People The threat that events or actions relating to the people associated with
the project, including modelers, clients, process participants,
subject matter experts, and decision-makers will adversely affect the
ability of the system dynamics modeling project to achieve its system
understanding and behavior improvement objectives.
Model Development
Qualitative The threat that events or actions relating to developing the system
dynamics qualitative model, focusing on identification of
appropriate structural elements and their relationships, will
adversely affect the ability of the system dynamics modeling project to
achieve its system understanding and behavior improvement objectives.
Quantitative The threat that events or actions relating to developing the system
dynamics quantitative model, focusing on the quantitative
relationships between structural elements will adversely affect the
ability of the system dynamics modeling project to achieve its system
understanding and behavior improvement objectives.
Model Analysis
Testing The threat that events or actions relating to how the system dynamics
model is assessed will adversely affect the ability of the system
dynamics modeling project to achieve its system understanding and
behavior improvement objectives.
Policy The threat that events or actions relating to policy alternatives and
their assessment will adversely affect the ability of the system
dynamics modeling project to achieve its system understanding and
behavior improvement objectives.
Model Use
Decision- The threat that events or actions relating to how decisions are made
Making based on knowledge gained from the modeling process will
adversely affect the ability of the system dynamics modeling project to
achieve its system understanding and behavior improvement objectives.
Implementation | The threat that events or actions relating to sustainable utilization of
knowledge gained from the project or implementation of system
modifications will adversely affect the ability of the system dynamics
modeling project to achieve its system understanding and behavior
improvement objectives.
4.3 Populating the Modeling Risk Dictionary
The analysis step of taxonomy development established the framework that provides the basis
for populating the dictionary. Then the literature review identified the risks that modelers have
encountered during modeling projects. These results from the information analysis were then
classified according to the modeling framework. A key classification principle was to identify
risks as early as possible within the modeling process, especially if the effects of the risk could
be felt during multiple steps in the modeling method.
For populating the system dynamics modeling risk dictionary, two primary literature review
approaches were pursued. First, foundational publications focused on explaining the system
dynamics modeling methodology suggested an initial list of risks for consideration. The next
approach was to target research publications focusing on verification and validation techniques.
Analysis of the techniques suggested the risks that they were targeting for discovery,
management, and mitigation.
Each suggested risk was defined and then classified within the modeling risk framework to
provide modelers and clients with consistent definitions of risks so that they are speaking the
same language when they are identifying, sourcing, and measuring modeling risks for projects.
While the modeling risks typology is always subject to review, the dictionary should not change
often, reflecting the durability principle of taxonomy development.
4.3.1 Methodology Focus
An example of a methodology-focused starting point for developing the modeling risk dictionary
can be found in “Questions Model Users Should Ask - But Usually Don’t” (Sterman 2000).
Captured in these questions are suggested risks that could be included in the modeling risk
dictionary (Table 3).
Table 3. Suggested Modeling Risks from Assessment Questions
Question Area
Suggested Risks
Purpose, Suitability, and Boundary
Purpose
Boundary
Endogenous Behavior
Time Horizon
Aggregation
Physical and Decision-Making Structure
Physical Laws
Dimensional Consistency
Stock and Flow Consistency
Endogenous Behavior
Delays and Limitations
Rational Behavior
Information for Decision Making
Robustness and Sensitivity to Alternative
Assumptions
Assumptions
Extreme Inputs
Extreme Policies
Pragmatics and Politics of Model Use
cece ee eo eo ole © oe © © © © © ole eo ow oO
Documentation
Source Data
Testing
Reproducibility
Cost
Revision
Modelers’ Bias
Clients’ Bias
Another source for suggesting modeling risks is captured in a proposed generalized assessment
approach (Randers 1980). This approach focused on consideration of a broad variety of system
behavior characteristics that suggest potential modeling risks (Table 4), some of which were
suggested in the previous example.
Table 4, Suggested Modeling Risks from Behavior Characteristics
Behavior C haracteristics Suggested Risks
Generate multiple behavior modes e Behavior Mode Generation
Plausibility of causal structures e Causal Relationships
Plausibility of parameter values [and e Sensitive Parameters
dimensions] e Dimensional Consistency
Compatibility of individual assumptions e Assumptions
with established knowledge
Internal consistency of the full structure e Consistency
Completeness with which the model e Boundary
includes the mechanisms thought to
Endogenous Behavior
generate the problem addressed.
In addition to these sources, other references focused on methodology were utilized (Forrester
1961; Coyle 1976; Richardson and Pugh 1981). These sources provided similar insights as those
mentioned above. Although not all of the preliminary risks were explicitly included the final risk
dictionary, their fundamental characteristics were considered. For example, Physical Laws risk
was originally identified, as was Spatial Structural risk. Within the final dictionary, they are
included in the Physical Properties risk definition.
4.3.2 Testing Method Focus
To create multiple points of contact for assessing models, the system dynamics methodology
relies on applying a framework of tests to achieve the confidence-building objective.
Commonly, these test frameworks are grouped around structure, behavior, and policy (Forrester
and Senge 1980), structure and behavior from suitability, consistency, utility, and effectiveness
perspectives (Richardson and Pugh 1981), and informally around structure, behavior, and policy
(Sterman 2000). These classifications align well with the modeling risk framework. Analysis of
these tests contributed risks for inclusion in the risk dictionary.
In addition to these testing frameworks, other focused tests have been proposed. While not
always adopted for general use, analysis of these tests also indicated modeling risks. The
objectives of these tests suggested risks focused on parameters (Peterson 1980; Graham 1980),
behavioral characteristics (Barlas 1989), surprise behavior (Mass 1991), sensitivity analysis
(Tank-Nielsen 1980), and structural coherence (Coyle 1976).
Tests are not the only indicators of modeling risks to be managed. Best practices and guidelines
outlined in the literature also suggest modeling risks. For example, the five formulation
fundamentals for decision-making (Sterman 2000) make the case for decision-making structure
risk being part of the modeling risk dictionary.
5 System Dynamics Modeling Risk Dictionary
Based on the literature review, the modeling risks were identified and classified in the modeling
risk framework (Figure 1). The modeling risks are easily considered with the use of the risk tree
diagram (Figure 3). Definitions for each of the modeling risks can be found in Appendix A.
5.1 Model Analysis Risks
Development of the model leads to consideration of risks associated with the analysis of the
model, focusing on model testing and policy analysis.
5.1.1 Testing Risks
The testing risks focus on consideration of the challenges that modelers could encounter relative
to the selection and application of system dynamics testing methods, including basic and
advanced techniques. This continues to consideration of the robustness of the model and range
of conditions under which he can be claimed to be valid. Similar to the other development steps,
consideration of the risks linked to inferences made from testing must be considered.
Qualitative Quantitative Model
Mental Model | ; Sensitive Parameter
Translation Link Polarity cleat - Suanthcatonst aang Development
7 eference Mode xogenous Variable
Piette Mode Loop Definition Quantitative Quantification
Simplicity Loop Coherence Time Horizon Physical Properties
- Dimensional Unit Quantitative Time Step
Aggregation Level \ Consistency Source Data Dimensional Quantification
Qualitative Parameter Data Noise/ Consistency
Source Data Identification Random Effects Model Endogenous
Variables Identification \ Delay Identification Link Quantification Behavior
= 7 Non-Linear Initial Conditions
Auxiliary Variable Boundary : i ome
Identification Exogenous Variable Relationships Quantification _
Co-F low Structure Identification Delay Quantification Initial Conditions
Decision-Making Structure \ Structural Inference Parameter Behavioral Inference .
Quantification Modeling
F Process
Client System . 4
Economic Feasibility Experience Policy Alternatives n Objectives
= Decision-maki
System Knowledge ) Pattern Prediction © eS) ecision-making
Melilé ee o Performance
xperience
System Endogenous Pi Q Model Transparency ¥ Measurement
Behavior . LY
Mental Model/ Model Complexity 0 : ;
Methodology Preconception Testing Methods < oO Point Estimate
System Prediction
Politics Philosophy/ to} ‘
Bias/ldeology & Operating Range
Project Objectives Documentation
Subject Matter A’ / Robustness
System Control Expertise Testing Inference Implementation
Problem People Model Analysis Model Use
Figure 3. System Dynamics Modeling Risk Tree Diagram
5.2 Project Planning Risks
The project planning risks are those anticipated to be encountered early in the project, even
before the project has really been initiated. The risk dictionary identifies risks associated with
the problem on which the project is focused and risks associated with the people who will be
involved in the modeling process.
5.2.1 Problem Risks
The Problem Risks include consideration of project objectives and the anticipated project value,
including both economic value from improved system performance and value from knowledge
elicited from the modeling process.
Consideration of the feasibility for applying system dynamics is addressed through the system
endogenous behavior and methodology risks. In addition, consideration is given to the political
environment in the organization to anticipate how that will affect the project.
The final project risk considers whether the state of control of the system, including components,
parameters, and people, supports successful modeling, analysis, and improvement.
5.2.2 People Risks
The People Risks encourage consideration of the experience levels of both the modelers and
clients. Their experience with both the targeted system and with system dynamics modeling
should be considered. Similar consideration is directed toward other subject matter experts who
contribute knowledge to the project.
Consideration is also given to risks associated with the mental models and preconceptions that
the project participants have. Successful management of these risks mitigates their effects during
the model building and analysis steps.
Giving early consideration to the philosophy or biases of project participants addresses their
capabilities to utilize system thinking skills, techniques, and tools to understand and improve the
target system.
5.3 Model Development Risks
The risks associated with model building are first classified into those closely linked with model
development. Then, the risks focused on model analysis are captured in separate classifications.
Within model development, the risks are focused on either qualitative or quantitative model
building.
5.3.1 Qualitative Model Development Risks
Qualitative model development focuses on the identification of structural elements and their
linkages. System dynamics relies on the elicitation of information in mental models and the
application of reference mode archetypes to guide the qualitative model development process.
Therefore, risks associated with archetypes and mental models are considered. Additional risks
can be encountered with the qualitative data sources that are tapped for model building.
The proper identification of structural elements forms the basis for many of the risks in this class.
These elements include the stocks and flows at the foundation of the model and the auxiliary
elements that are included to capture the relationship structures in the system. The risks
associated with identification of links, delays, and parameters are also addressed here. The
structural elements form the basis for loop identification, and consideration of the risks
associated with assigning polarity and maintaining dimensional consistency.
Receiving special attention are co-flow and decision-making structures. These structures are not
intuitive to many modelers and are not encountered in all models. As a result, consistent
development can be more challenging. Therefore, the risk dictionary encourages explicit
consideration of these risks.
As the model evolves, the boundary definitions become important for enabling the model to
generate behavior in an endogenous manner. The boundary definition leads to consideration of
risks associated with the identification of exogenous variables.
Because the development of qualitative models supports the inference of system performance
based on the structural elements, consideration is given to the risks associated with the
aggregation level of the model. Additional consideration is given to the simplicity of the model,
which contributes to whether the relationship between the system and the model is strong enough
to support inference.
5.3.2 Quantitative Model Development Risks
As its name implies, quantitative model development focuses on the quantification of structural
elements and their linkages. An initial risk to consider focuses on the capabilities of the software
being used to simulate the models. Other risks that modelers often associate with software
include the time step used and the time horizon modeled.
Most of the risks encountered in quantitative model development have relationships with
structural element identification risks in qualitative model development, starting with
consideration of data sources. In addition, the impact of data noise or random effects is
considered. Quantification risks for links, delays, parameters, and exogenous variables are all
addressed. Dimensional consistency must be considered for all quantities.
Special consideration is given to the quantification of non-linear relationships. These
relationships are often especially crucial to effective modeling, leading to explicit consideration.
After the initial quantification of the models, consideration is given to risks focused on
dimensional consistency. In addition, physical properties risks are considered. This
consideration includes spatial and temporal risks. Considering these risks sets the stage for
considering risks associated with identifying and tuning sensitive parameters.
Consideration of the risks associated with the initial conditions contributes to managing the
model endogenous behavior risks. The final risks to be considered during quantitative model
development are those associated with the inferences made from the model behavior.
5.3.3 Policy Risks
The policy risks focus on the development of policy alternatives for consideration and how
understanding is developed from application of the policies. Consideration is given to both the
transparency and complexity of the model structure since both could inhibit understanding. The
risks associated with predicting results are also considered.
5.4 Model Use Risks
The model use risks focus on how decisions are made based on the modeling process and then
how changes will be implemented as a result of the modeling process.
5.4.1 Decision-Making Risks
The decision-making risks focus on how decisions are made based on the modeling process.
Consideration is given to the decision-making process and the performance measures that the
organization uses for assessing system performance.
While the focus of system dynamics is on behavior patterns, rather than specific point
performance, the desire of many modelers and clients to use the modeling process to provide
point estimates creates a risk that should be considered.
Finally, the decision-making risks include the risks associated with how the project is
documented. The documentation can support long-term use of the model and contribute to
greater understanding of how the model operates and can be used.
5.4.2 Implementation Risk
The implementation risk focuses on consideration of sources of risks that will limit the
sustainable structural and policy changes that can be anticipated from the modeling process.
6 Future Work
The development of the system dynamics modeling risk dictionary is a component of research
leading to a system dynamics risk management framework. The risk dictionary supports the
identification and sourcing of modeling risks. Additional research supports the measurement of
likelihood and significance of the risks. Linked to a risk management techniques typology, the
risk measurement information suggests risk management strategies for the risks. Modelers are
then able to develop their modeling risk management plan early in their projects and document
results during the entire process. The ultimate objective of the research is to align risk
management resource levels, including testing, with the anticipated risk levels.
7 Conclusions
By recognizing the system dynamics modeling method as a process, the business risk
management principles can be applied to the development of a framework for managing system
dynamics modeling risks.
A key component of this framework is the modeling risk dictionary, which was developed by
applying the principles of taxonomy development. Modelers can use the modeling risk
dictionary to consistently consider the risks that they might encounter during their modeling
projects. Having this resource available to inexperienced modelers supports increasing their rate
of climbing the learning curve.
By encouraging risk assessment early in the project, modelers increase the likelihood that they
will use appropriate techniques for managing their project risks, either by removing them at the
source, avoiding their occurrence, or discovering them and improving the model.
Acknowledgements
The author wishes to acknowledge the support provided by John Grado through the John Grado
Assistantship.
Appendix A
System Dynamics Risk Dictionary Definitions
1 Project Planning
1.1 Problem
Risk
Definition
Economic Feasibility
Risk
the risk that value to be achieved from improved system performance
does not justify the project costs.
System Knowledge the risk that the value to be achieved from elicited system knowledge
Value Risk does not justify the project costs
System Endogenous the risk that the problem system does not generate behavior
Behavior Risk endogenously.
Methodology Risk the risk that the system dynamics methodology is not a strong
approach for eliciting system knowledge or driving system
performance improvement.
Politics Risk the risk that political environment in a client organization can affect
the decision-making or system improvement implementation.
Project Objectives Risk | the risk that the problem objectives are not clearly defined or are not
achievable
System Control Risk the risk that the level of control in the system, including components
and people, does not allow it to be effectively modeled or improved.
1.2 People
Risk Definition
Client System Experience
Risk
the risk that the experience level of system personnel affects their
abilities to participate effectively in the modeling process.
Modeler System
Experience Risk
the risk that the experience level of the modelers will affect their
abilities to effectively model the system, elicit system knowledge,
and determine system improvement recommendations.
Mental Model/
Preconception Risk
the risk that the mental models of the modelers or the clients are
significantly inaccurate.
Philosophy/Bias/Ideology
Risk
the risk that the perspectives of system participants or modelers will
affect their capabilities to utilize system thinking skills, techniques,
and tools to understand and improve the problem system.
Subject Matter Expertise
Risk
the risk that the subject matter expertise used for modeling the
problem system limited in level, breadth, or availability.
2 Model Development
2.1 Qualitative
Risk Definition
Mental Model the risk that the mental models of the participants will not be captured
Translation Risk effectively and translated into modeling elements well.
Reference Mode the risk that the use of a system modeling archetype from the
Archetype Risk reference mode will guide improper development of the model.
Simplicity Risk the risk that the model structure is simplified so much relative to the
system complexity that the linkage between model and system is
tenuous.
Aggregation Level Risk | the risk that the aggregation level does not effectively model the
problem system or that it creates a level of complexity that obscures
system behavior.
Qualitative Source Data | the risk that valid information necessary to qualitatively model the
Risk system is not readily available.
Variables Identification | The risk that the primary variables included in the model, focusing on
Risk stocks and flows, do not represent the system structure well so that
the behavior reflects the structure.
Auxiliary Variable the risk that the auxiliary variables and links in a qualitative model
Identification Risk are identified incorrectly
Co-flow Structure Risk
the risk that a co-flow situation in the system is not properly
recognized and modeled.
Decision-making
the risk that a decision-making structure is not properly modeled.
Structure Risk
Parameter Identification | the risk that the sensitive parameters will not be properly identified in
Risk a qualitative model.
Link Polarity Risk the risk that the polarity of links in a qualitative model are assigned
incorrectly.
Loop Definition Risk the risk that the polarity of loops in a qualitative model are
determined incorrectly, impacting structural inference.
Loop Coherence Risk
the risk that loops in a qualitative model are not structurally coherent.
Dimensional Unit
the risk that the units applied to structural elements in a qualitative
Consistency Risk model are not consistent.
Delay Identification the risk that the material and information delays in the system will not
Risk be properly identified in a qualitative model.
Boundary Risk the risk that the defined model boundary will affect inclusion of
structural or policy elements to generate the endogenous system
behavior
Exogenous Variable the risk that key exogenous variables are not correctly identified for a
Identification Risk problem system.
Structural Inference the risk that the inferences made from the model structure are not
Risk
valid based on the available information
2.2 Quantitative
Risk
Definition
Software Risk
the risk that the modeling software capabilities, including the
integration method, impact the quality of the model.
Reference Mode
Quantitative Risk
the risk that the theoretical reference mode does not accurately reflect
true system behavior.
Time Horizon Risk
the risk that the time horizon modeled is either too short or too long
to effectively represent the system behavior.
Quantitative Source
Data Risk
the risk that valid information necessary to quantitatively model the
system is not readily available.
Data Noise/Random the risk that random effects in the data will lead to improper
Effects Risk quantification.
Link Quantification Risk | the risk that auxiliary and flow rate links in a quantitative model are
quantified incorrectly.
Non-linear Relationships
Risk
the risk that non-linear relationships will be quantified incorrectly.
Delay Quantification
Risk
the risk that the material and information delays in the system will not
be properly quantified in a quantitative model.
Parameter the risk that the parameters will not be properly quantified in a
Quantification Risk quantitative model.
Sensitive Parameter the risk that the sensitive parameters will not be properly quantified
Quantification/T uning and tuned in a quantitative model.
Risk
Exogenous Variable the risk that key exogenous variables are not correctly quantified fora
Quantification Risk quantitative model.
Physical Properties Risk | the risk that physical properties or flows will not be modeler
correctly, including violations of physical laws.
Time Step Risk the risk that the time step selected for the quantitative model will
improperly reflect the system dynamics.
Dimensional the risk that the values applied to structural elements in a quantitative
Quantification model are not consistent with the units assigned to the elements.
Consistency Risk
Model Endogenous the risk that the model behavior will not effectively demonstrate the
Behavior Risk endogenous behavior of the system.
Initial Conditions the risk that initial conditions are not properly quantified for the
Quantification Risk model
Initial Conditions Risk | the risk that the sensitivity of the system dynamics to the initial
conditions is not properly identified
Behavioral Inference the risk that the inferences made from the model behavior are not
Risk
valid based on the available information
3 Model Analysis
3.1 Testing
Risk Definition
Testing Methods Risk the risk that testing methods are not well understood, poorly applied,
and/or inefficiently used.
System Operating Range | the risk that the model does not demonstrate acceptable behavior
Risk across the full range of realistic operating conditions.
Robustness Risk the risk that the model does not function well for a breadth of realistic
starting conditions and/or parameters.
Testing Inference Risk
the risk that the inferences made from testing methods do not elicit
correct system or model knowledge
3.2 Policy
Risk Definition
Policy Alternatives Risk | the risk that the policy alternatives considered will be too narrow, too
broad, or unrealistic.
Pattem Prediction Risk | the risk that the model does not effectively predict the system
behavior pattern in the presence of structural, or policy, changes.
Model Transparency
Risk
the risk that the model behavior can not be easily linked to the model
structure.
Model Complexity Risk
the risk that the model structure is so complex that understanding is
limited about the resulting model behavior
4 Model Use
4.1 Decision-Making
Risk
Definition
Decision-making Risk
the risk that the decision-making approach will be limited or faulty in
execution or timeliness.
Performance the risk that the actual reference mode does not provide an effective
Measurement Risk indicator of system behavior.
Point Estimate the risk that the model does not effectively and accurately predict
Prediction Risk specific system behavior at a specific time.
Documentation Risk
the risk that the modeling project documentation will affect the level
of understanding or the value achieved from the modeling project.
This risk must be considered for both the short-term and over a longer
time period.
4.2 Implementation
Risk
Definition
Implementation Risk
the risk that the level of implementation of recommendations from
the modeling project will affect the value achieved.
References
Balci, Osman. 2005. Challenges in Credibility Assessment of System Dynamics Models for
National Security. In Workshop on System Dynamics Modeling of Physical and Social
Systems for National Security. Chantilly, VA.
Barlas, Y aman. 1989. Multiple Tests for Validation of System Dynamics Type of Simulation
Models. European J. of Operations Research 42 (1):59-87.
Committee of Sponsoring Organizations of the Treadway Commission. 1994. Internal Control -
Integrated Framework: Executive Summary. New Y ork: AICPA.
Coyle, R. Geoffrey. 1976. Management System Dynamics: John Wiley & Sons.
Economist Intelligence Unit. 1995. Managing Business Risks— An Integrated A pproach, edited
by written in cooperation with Arthur Andersen. New Y ork.
Forrester, Jay Wright. 1961. Industrial Dynamics. Cambridge MA: Productivity Press.
Forrester, Jay Wright, and Peter M. Senge. 1980. Tests for Building Confidence in System
Dynamics Models. In System Dynamics, edited by A. A. Legasto, Jr. and e. al. New
Y ork: North-Holland.
Graham, Alan K, 1980. Parameter Formulation and Estimation in System Dynamics Models.
Paper read at International Conference on System Dynamics, at Geilo, Norway.
Harris, K., F. Caldwell, and R. Knox. 2003. A Process Model for Creating a Taxonomy. In
Decision Framework: Gartner Research.
Mass, Nathaniel J. 1991. Diagnosing surprise model behavior: a tool for evolving behavioral and
policy insights (1981). System Dynamics Review 7 (1):68-86.
Peterson, David W. 1980. Statistical Tools for System Dynamics. In Elements of the System
Dynamics Method, edited by J. Randers. Cambridge MA: Productivity Press.
Randers, Jorgen. 1980. Guidelines for Model Conceptualization. In Elements of the System
Dynamics Method, edited by J. Randers. Cambridge MA: Productivity Press.
Richardson, George P., and Alexander L. Pugh, III. 1981. Introduction to System Dynamics
Modeling with DYNAMO. Cambridge MA: Productivity Press.
Ruscio, John, Nick Haslam, and Ayelet Meron Ruscio. 2006. Introduction to the Taxometric
Method. Mahwah, N.J.: Lawrence Erlbaum Associates.
Sterman, John D. 2000. Business Dynamics : Systems Thinking and Modeling for a Complex
World. Boston: Irwin/McGraw-Hill.
Tank-Nielsen, Carsten. 1980. Sensitivity Analysis in System Dynamics. In Elements of the
System Dynamics Method, edited by J. Randers. Cambridge MA: Productivity Press.
