-102-
System Dynamics Modelling Analysis Techniques
A Pragmatical Appraisal
A.S. Camara, J.A. Fernandes, M.G. Viegas and A.P. Amaro_
College of Sciences and Technology, New University of Lisbon,
Monte da Caparica, Portugal
ABSTRACT
This paper reviews techniques that may assist the system dynamics modeller
in defining variables and functional relationships, parameter estimation,
validation, sensitivity and policy analysis. The evaluation was made in the
context of a water resources management modeling effort for the Guadiana
basin in Algarve and based on scientific, economic and operational criteria.
In general, it was difficult to point out the most appropriate technique
but rather recommend combinations of methods for each modeling stage.
INTRODUCTION
Several techniques have been developed in the past for each of the traditio-
nal system dynamics modeling stages: definition of variables and functional
relationships, parameter estimation, validation, sensitivity and policy
analyses.
This paper attempts to assess those techniques in the context of their
application to a water resources management modeling effort for the Guadia-
na basin in Algarve. The assessment is made following scientific, economic
and operational criteria. The goal is to screen for the most suitable
methods in a pragmatic application of the system dynamics approach.
WATER RESOURCES MANAGEMENT MODELING
Guadiana's model is inspired in a previous work of the authors (see Camara
et al., 1984). It basically consists of three interacting sub-models:
sub~model I defines how much water is available; sub-model II computes
how much water is demanded; and sub-model III, / fdéds-back into sub-models
I and II, through a set of management equations.
Guadiana'’s model has been developed in four interacting stages: (1) defini-
tion of variables and functional relationships; (2) parameter estimation;
(3) validation; and (4) sensitivity analysis. After establishing the model
validity, to serve as a plausible predictive tool, policy analyses were
then conducted. The process is represented in Figure 1.
Two perspectives were considered in the definition of variables: a strictly
system dynamics view; and a management view. From the system dynamics
perspective, variables were divided into level (i.e., precipitation,
population), rate (i.e., runoff rate, potential evapotranspiration rate)
and auxiliary. The latter were either the result of algebraic decomposition
of rate variables or simple counters (i.e., water deficit or superavit
conditions). From the management standpoint, variables were divided into
control (i.e., some rate variables) and impact variables (i.e., some level
and auxiliary variables).
~103-
Definition of variables and
yo functional relationships qj
Sensitivity analysis—_—_—»Parameter estimation
NON Validation Policy analysis
Figure 1 - Guadiana's Basin System Dynamics Modeling Stages
Functional relationships were defined using traditional system dynamics
type equations and applied theoretical, empirical (i.e., rational method
to estimate runoff) and ad-hoc information (i.e., luxury tourism water
consumption).
Parameter estimation foccused essentially on rate equations. Validation
was concerned with the model's variables, functional relationships and
parameters adequacy. Sensitivity analysis was applied throughout the
modeling exercise, guiding parameter estimation and helping validation
and policy analysis. The latter stage consisted of evaluating sets of
valuations of control variables upon a set of impact variables, defining
an objective function.
ASSESSMENT OF SYSTEM DYNAMICS MODELLING TECHNIQUES
In this section, the most common techniques available for each of the
system dynamics modeling stages are presented and basically evaluated
from scientific (reliability), economic (computational costs and data
requirements) and operational (ease of application) standpoints.
Definition of Variables and Functional Relationships
Techniques for the definition of variables and functional relationships
and their application to Guadiana basin modeling are summarized in Table 1.
A general scientific, economic and operational assessment is synthesized
in Table 2. It may be observed that none of the methods is sufficient
to define the model's variables and functional relationships. Rather, the
use of a combination of methods is necessary.
Parameter Estimation
There are two kinds of available data to perform parameter estimation!
(1) disaggregate data (i.e., information about events and items below the
level of aggregation of model variables); and (2) aggregate data, corres-
ponding to level type variables.
For both classes of data there are three available kinds of techniques:
(1) direct techniques; (2) indirect techniques; and (3) probabilistic
techniques. In Table 3, observations on these techniques are included.
Table 4 synthesizes a basic scientific, economic and operational. assessment
of the parameter estimation methods reviewed.
-104-
Table 1
Techniques for the Definition of Variables and
Functional Relationships and their Application to Guadiana Model
Techniques Observations Application to
Guadiana Model
Type I- Define variable
and funct. relationships
Cluster Analysis identification of functional analysis of
(Morrison, 1967) rel. between variables and essential comp.
their magnitude but not their of demand model
representation. Extensive and (sub-model II)
intensive data needs
Analysis of principal quantitative analysis of funct. same as in
components rel. Decomposition and present. cluster analysis
(Morrison, 1967) of funct. rel. components relati-
ve magnitude. Do not enable math.
repres. of funct. rel. Extensive
and intensive data neéds
Adjacency matrices Identification of funct. rel. of preliminary
(Cristofides, 1975) variables direct. or indirectly analysis leading to
connected. May be translated in the causal diagram
digraph form.
Application of identification of the magnitude not utilized.
Kirchoff laws of relationships between var. Sensitivity analy-
(Davis and Kennedy, not directly connected, based sis was preferred
1970) on a weighed model. Like a
pre-sensitivity analysis
Type II- Define specific
funct. relationships
Curve fitting , fitting hypothesized math. used to define
Kolgomor ov—-Smirnov expressions to sampled data tourism growth
tests) sets. Limited to the behaviour equations
(Shannon, 1975) verified in the sampled universe
Harmonic analysis of allowing by analyzing time series used to analyze
time series components components and trends, represent. luxury tourism
(Box and Jenkins, 1976)the behav. of a funct. rel., that evolution
behav. determinant factors and its
type of action. Extensive and
intensive data needs
Time wavelet composi~ derivation of the behaviour of a used to analyze
tion of time series funct. relationship after introd. and represent
(Robinson, 1967) perturb. and/or stimula, by wave~ luxury tourism
let composition evolution subject
(Cont. )
-105-
Table 1 (Cont.)
Techniques Observations Application to
Guadiana Model
to perturb. and
or stimula
Catastrophe theory representing funct. rel. suffer. applyed to the
(Sinha, 1981) sudden quantitative and qualitative components of
changes which are not discontinui- economic growth
ties in dem. model
Table 2
Scientific, Economic and Operational Assesment of
Structure Identification Methods
Techniques Scientific Reliab. Comp. Costs Data Req. Ease of Appl.
Cluster anal. + + ++ °
Anal. prin. co. ++ + ++ O
Adj. matrices + ° ++
Kirchoff laws ++ + ;
Curve fitting + + ++ +
Harmonic anal. ++ + ++ °
Wavelet comp. ++ + ++ °
Catast. theory ia + $+ ©
Criteria:
++ - high
+ - average
o ~ low
-106-
Table 3
Parameter Estimation Techniques and their Application
to Guadiana Model
Techniques Observations Application to
Guadiana Model
Direct Methods used considering applied in both
Algebraic estimation data withouth measurement cases: disagregate
Extended algebraic errors; equation without and aggregate data
estimation errors
(Eyckoff, 1971)
Indirect Methods used considering applied in both
Least-squares data without measurement cases: disaggregate
control methods errors; equation errors. and aggregate
(Peterson, 1976) Simulation reinitialized data
(Graham, 1980) at each data point
(Box and Jenkins, 1976)
‘Probabilistic Method ied i
alistic Methods applied in the
Bayesian estimation used considering 3
Maximum likelihood data and equation errors. case of disaggre-
estimation Simulation reinitialized gate ‘data
Other related methods at each data point aggregate data
(Eyckoff, 1974) was insufficient
(Box and Jenkins, 1976)
Table 4
Scientific, Economic and Operational Assessment
of Parameter Estimation Methods
Techniques Scientific Reliab. Comp. Costs Data Req. Ease of Appl.
Disaggreg.
data
a
Direct methods ° r + ax
Indirect meth. + ¥F ¥ ¥
Probabilistic ++ +F FF o
Aggreg. data
Direct methods z r 2
Indirect meth. ¥ +F ¥ ¥
Probabilistic ++ +E = o
-107-
From Tables 3 and 4, one may observe that there are substantial advantages
in the use of probabilistic methods from a scientific standpoint. They
are however data intensive and extensive methods. Thus, one normally
applies indirect methods, which represent a compromise option between
direct and probabilistic methods.
Validation
There are two typical validation stages:(1) internal—to assure that the
model performs the way it was intended; and(2) external—-comparing the
input-output transformation of the model and the real world system. The
latter stage is obviously not always possible. Most common validation
methods and their application in the Guadiana study are summarized in
Table 5. Their scientific, economic and operational assessment is included
in Table 6.
From Tables 5 and 6, one may see that most validation efforts should tend
to be only Type I plus Turing testing procedures. This was also the case
of the Guadiana water resources model.
Sensitivity Analysis
Sensitivity analysis is usually performed by introducing perturbations in
the parameter and functional relationships integrating the model. Its two
main objectives are: the evaluation of precision required in the parameter
estimation stage; and the design of robust models with functional
relationships plausible even for extreme conditions. Thus sensitivity
analysis may be performed at two levels: parameter sensitivity and noise
sensitivity.
Parameter sensitivity analysis
Parameter sensitivity may be conducted: locally-~to evaluate the behaviour
of the system subject to infinitesimal changes in the parameter values,
being altered isolatedly; and globally--to assess the behaviour of the
system considering finite and simultaneous changes of its parameter values.
Table 7 synthesizes the available methods for local and global sensitivity
analyses and their application to the Guadiana basin model. Table 8
includes a scientific, economic and operational evaluation of these methods.
Despite its computational costs, conventional parameter sensitivity
analysis is still preferable in local sensitivity analysis, specially if
applied after preliminary rational screening of sensitive parameters.
Noise sensitivity analysis
Noise sensitivity analysis is used to assess the validity of the model's
structure. A noise term R is added to the function f(x) (£'(x)=£(x)+R).
R may be continuous, random, wave like or intermittent.
In the conventional method a run of the noise free system and several
runs of the perturbated system are performed. Then, function dx=f' (x)-f (x)
is evaluated.
In the perturbation method, function d(x) is analytically defined as
referred above. eet
-108-
Table 5
Validation Methods and their Application
to Guadiana Model
Techniques
Observations
Application to
Guadiana Model
Type I--Internal
to perform logical checking
of the program
applied in comp.
prog. stage
Sensitivity
analysis
to verify plausibility of
(Mass and Senge, 1976) behaviour under extreme
(Bell and Senge, 1980)
conditions
applied after
comp. prog.
stage
Type II--External
Validation
Statistical testing
univariate or
hypothesis testing. Only
if observations are not
multivariate parame- auto-correlated
tric tests (F, t and Z)
(Shannon, 1975)
(Zeigler, 1976)
not applied.
Insufficient
data available
Turing test
(Shannon, 1975)
consists of asking people who
are knowledgeable about the
system if they can discriminate
between system and model outputs
and why
applied in a
limited extent
due to limitat.
on system data
Spectral analysis
(Shannon, 1975)
comparison of spectra between
model and system output to
construct confidence bands.
Assumes that the time series
are covariance stationary
which is not always true
not applied.
Insufficient
data available
-109-
Table 6
Scientific, Economic and Operational Assessment
of Validation Methods
Techniques
Scientific Reliab. Comp. Costs Data Req. Ease of Appl.
Traces + + ° +
Sensitivity + ‘4 ° +
Analysis
Statistical + ++ ++ ¥
testing
Turing test + ° + +
Spectral ae eS op °
Analysis
-110-
Table 7
Local and Global Parameter Sensitivity
Analysis and their Application to the
Guadiana Basin Model
Techniques
Observations Application to
Guadiana Model
Local sensitivity
analysis
Conventional method
(Sharp, 1976)
parameter are changed in each run
and one compares these outputs used after
with results obtained with the preliminary
perturbation-free system. High screening of
computational requirements. Often sensitive par.
used after a preliminary screening
of sensitive parameters. May also
be applied, submitting the para~
meters to random changes
Perturbation method
(Sharp, 1976)
consists of the analitical not applied due
definition of the function d(x)= to extensive
£(x)-£(x+dx) (dx—the perturba- preliminary work
tion of x) using partial diff. required
and simplification methods. The
analysis is done by computing d(x)
instead of f(x) and f(x+dx), as
described in the conventional meth.
Requires mathematical expertise.
After these preliminary calculations,
computational needs are low
Global sensitivity
analysis
Conventional
Perturbation
(Sharp, 1976)
use of conventional/perturbation not used due to
local sensitivity analysis methods comp. req.
in connection with minimization
routines, allowing for the deter.
of the parameter changes leading
to minimal system alterations
Qualitative stability
analyses
(adapted from May,
consists of a matrix transcription
of causal diagram, where a zero
(Cont.)
-1ll-
Table 7 (Cont.)
Techniques Observations Application to
Guadiana model
1975) coefficient represents the not applied
nonexistence of connection due to the °
between two variables. flaws of the
criteria to evaluate stability method
may be inspired in the work
of May. They are not however
fully operational at present
time
Table 8
Scientific, Economic and Operational Assessment of
Parameter Sensitivity Analysis Methods
Techniques Scientific Reliab. Comp. Costs. Data Req. Ease of Applicat.
Conventional ++ ++ ° ++
Perturbation ++ + ° °
Qual. Stab.
Analysis
eS + ° ++
-112-
The advantages and disadvantages of these methods from scientific, economic
and operational standpoints are similar to the ones pointed out for the
parameter sensitivity analysis case (see Table 8).
Sensitivity analysis appears therefore as a method to be applied ‘before
parameter estimation and as an additional validation test. It. may be used
also in policy analysis, as one can consider a policy as a perturbation
introduced in the system which will change its behaviour. As suggested
further on, sensitivity analysis will help in the assessment of the
"degrees of promise" (in terms of objective achievement) of a control
variable valuation.
Policy Analysis
If the model developed is valid for its purpose (and all the previous
techniques are intended basically to increase and test the model's
validity) there are two essential problems in the policy analysis stage:
1. To define from the possible strategies (strategy=set of valuations for
control variables), those to be tested (i.e., define the number of runs
of the model). This screening has to be made, as the number of valuations
for each control variable is large and enormous the number of resulting
combinations of those valuations.
2. To determine the optimal strategy, for each set of development goals,
based on the impact variables values obtained with the model for each
strategy.
For the first problem, the authors found that there are no adequate
techniques and propose that: (1) a value should be assigned to each control
variable based on its level of "promise" in terms of objective function
achievement. This assessment may be done by a pannel of experts or through
preliminary sensitivity analysis; (2) each control variable valuation will
be then represented as a node with a weight equal to its "promise" level;
(3) a network may then be formed, each path representing a strategy; and
(4) applying a k-shortedt path algorithm to this network, one may thus
eliminate numerous alternative strategies from further analysis and define
a relatively small number to be assessed with the model. This method was
applied in the Guadiana basin modelling effort and proved to be reliable,
inexpensive and easy to use.
To evaluate the strategies, one has to solve a multiobjective programming
problem. This has been done in system dynamics modelling by at least two
authors: Gardiner and Ford (1980) and Camara et al. (1984).
Gardiner and: Ford considered that every policy experimented with ,in
simulation runs has valuesin terms of its impacts on a number of different
dimensions. These dimensions are derived from the analysis of the impact
variables trajectories (i.e., peak values, time lags). A multi-attribute
utilitymeasurement technique is then applied to discover those values, one
dimension at a time, and then aggregate them across dimensions using a
suitable aggregation rule and weighing procedure.
Camara et al method considers that for each strategy one obtains trajecto-
-113-
ries for the different impact variables IV. These trajectories may be re-
presented for a simulation period T as vectors f T. Then:
1. For each strategy j, vectors [xv* Ti are translated into a vector [w}.
This is done by dividing simulation period T into sub-periods tl, t2, t3,
+-.,tn, synthesizing sub-vectors (wv ei into a scalar, by computing the
summation, mean, mode, maximum or minimum for the values of {I tas
depending on the nature of IV, assigning weights wti to the sub-periods
f sess t relative importance, and finally calculating
speci Lave] ti,j, Wk.
2. Using a value path display approach and the calculated [wv alk » define
the non-inferior strategies j. If there is a superior strategy, the
optimization stops.
3. If there is no superior strategy, [yv'h are normalized, weights wIV are
defined for each IV, and then£wIV.IV is computed for each j,.the
largest of these values corresponding to the optimal strategy.
Both methods are simple, inexpensive and have a common flaw: subjective
weighing procedures. Their treatment of the time dimension is however
different. Gardiner and Ford consider trajectories as moving pictures
with a number of dimensions. Camara et al. take trajectories as vectors
that can be aggregated into scalars, using well known compression mechanisms.
In both cases, there are obvious problems: it may be difficult to discover
meaningful dimensions in trajectories (Gardiner and Ford); the assignment
of weights to time periods is highly subjective (Camara et al.). They
should be used therefore depending on the circumstances.
SUMMARY AND CONCLUSIONS
This paper attempted to assess techniques that may assist the system
dynamics modeller in defining variables and functional relationships,
parameter estimation, validation, sensitivity and policy analyses. The
evaluation was made in the context of a water resources management
modeling effort for the Guadiana basin and based on scientific, economic
and operational criteria.It was concluded that:
1. To define variablés and functional relationships none of the methods
reviewed is sufficient; rather the use of a combination of methods is
necessary.
2. For parameter estimation, least-squares and control methods are the most
suitable in practical applications.
3. In validation efforts, sensitivity analysis and Turing tests are +
recommended in conditions of data scarcity.
4. Sensitivity analysis is a method to be applied before parameter estima~
tion, as an additional validation test and also in preliminary stages of
policy analysis. Conventional methods are preferable if applied after a
preliminary screening of sensitive parameters.
5. Approaches to policy analysis in system dynamics are based on interfaces
between the simulation model and multiobjective programming. The methodolo-
gies available have the drawbacks common to many multiobjective decision
-114-
methods: they rely on subjective weighing procedures.
6. Finally, one should note that the different modeling stages are not
isolated compartments. There is rather a continuous interaction, being
some analysis techniques suitable for more than one modelling phase.
REFERENCES
Bell, J.A. and P.M. Senge, "Methods for Enhancing Refutability in System
Dynamics Modelling", TIMS Studies in the Management Sciences. New York:
North Holland, 1980.
Box, G.E. and G.M. Jenkins. Time Series Analysis, Forecasting and Control.
San Francisco: Holden Day, 1976.
Camara, A., J. Meneses, J. Fernandes, A. Amaro and G. Viegas. "Water Resour-
ces Management of Algarve—Interfacing System Dynamics and Multiobjective
Programming", Proceedings of the 1984 International System Dynamics
Conference. Oslo, Norway, 1984.
Cristofides, N. Graph Theory, An Algorithmic Approach. London: Academic
Press, 1975.
Davis, C.F. and M.D. Kennedy. "EPS: A Computer Program for the Evaluation
of Problem Structure", in G.T. Moore, ed., Emerging Methods in Environmen-
tal Design and Planning. Cambridge, Ma.: M.I.T. Press, 1970.
Eykhoff, P. System Identification--Parameter and State Estimation. Chiches-
ter: John Wiley, 1974.
Gardiner, P.C. and A. Ford. "Which Policy Run is Best and Who Says So",
TIMS Studies in the Management Sciences. New York: North Holland, 1980.
Graham, A.K. "Parameter Estimation in System Dynamics Modeling", TIMS
Studies in the Management Sciences. New York: North Holland, 1980.
Mass, N.J. and P.M. Senge. "Alternative Tests for the Selection of Model
Variables", Proceedings of the 1976 International Conference on System
Dynamics, Geilo, Norway, 1976. :
May, R. Stability and Complexity in Model Ecosystems. Princeton: Princeton
University Press, 1975.
Morrison, D.F. Multivariate Statistical Methods. New York: McGraw Hill,
1967.
Peterson, D.W."Statistical Tools for System Dynamics", Proceedings of the
1976 International Conference on System Dynamics. Geilo, Norway, 1976.
Robinson, E.A. Multichannel Time Series Analysis. San Francisco: Holden
Day, 1967.
Shannon, R.E. Systems Simulation: The Art and Science. Englewood Cliffs,
N.J.: Prentice Hall, 1975.
-115-
Sharp,J.A. "Sensitivity Analysis Methods for System Dynamics Models",
Proceedings of the 1976 International Conference on System Dynamics.
Geilo, Norway, 1976.
Sinha, D.K., ed. Catastrophe Theory and Applications. New York: John Wiley,
1981.
Zeigler, B.P. Theory of Modeling and Simulation. New York: John Wiley, 1976.
