SYSTEM DYNAMICS APPROACH TO VALIDATION
Branko Griié Ante Munitié
University of Split, Faculty of Economics, University of Split, Maritime Faculty Dubrovnik
Radovanova 13, Split, CROATIA. Zrinjsko-Frankopanska 38, Split, CROATIA
e-mail: grcic@oliver.efst.hr e-mail: munitic@mvsrce.srce.hr
Abstract: Model validation is a problem that both social and natural sciences have been facing
with for many-years.. During the last decades it became particularly pressing in social
sciences due to the development of contemporary complex tools for the modelling of real
social systems. The system dynamics methodology is one of these new tools. Although
it has been developed through relatively long period of time, it was rather "closed" for
critical opinions especially those referring to the validation of system dynamics simulation
models. This paper presents an insight in the system dynamics approach to model
validation. It takes into consideration all relevant discussions about this matter, as well
as some of the procedures and criteria used so far in the system dynamics models
validation. Moreover, based on the evaluation of their advantages and disadvantages,
certain formal criteria are provided aiming to strengthen the credibility of these models.
1. Bases for sistem dynamics approach to validation of models
Validation of simulation models is one of the most important phases in the process of
modelling real systems. A series of methods and procedures has been developed so far
for the purpose of building confidence in the real system models. Conversely, however,
there is no universal concept or original procedure that could be applied to all cases
when we want to represent the real system with a model. This is particularly true of the
system dynamics. The bases of the approach to building confidence in the system
dynamics simulation models (SDSM) can be expressed under the following tenets:
- The confidence in the SDSM is being acquired through confirmation of validity
of each individual structural or control element in the global model architecture, which
elements represent the analogue parts of the real system. In this case we deal with the
conceptual or structural validation based on the successive, semi-formal, that is, mostly
" qualitative and also the interdisciplinary process of building confidence in the conceptual
structure of a model.
- The confidence in the SDSM is also developed by confirmation of the sufficient
degree of correspondence of the dynamic model-generated behavior of the modelled
system with the observed or expected behavior (depending on whether the system has
or has not its past) of the same system. Here, we speak about the behavior validity tests
relying mostly upon formal, that is, quantitative methods and criteria.
Further in the paper we shall concentrate on these behavior validity tests and, besides
mentioning the so-far utilized formal testing criteria and procedures, we shall propose
another possible concept for shesking the replicative validity of the SDSM.
2. Formal tests of validity of the SDSM
In numerous papers dealing with the problem of validation of the SDSM’s a criticism
has been noted stating that the system dynamics insufficiently utilize formal and
impartial quantitative procedures for testing the quality of models. The application of
formal, quantitative validity tests in the system dynmics has been limited indeed. The
mG
reasons are manifold. One of them is the preference for conditional, "inaccurate"
projection of the general dynamics of the modelled system (so-called behavior pattern
prediction), as opposed to other methods of the simular purpose aiming at more precise
estimate or prediction of the numerical value of a particular variable (so-called point
prediction). Of no less importance is the dominating deterministic character of the
SDSM’s which affects the domain by unabling the application of the standard statistical
representative tests. The application of the statistical tests in the system dynamics has
also been limited by unfulfillment of the basic assumptions for their utilization (Barlas,
1989). One’ of the features of the system-dynamics approach, among others, is in its
aiming to include in a model as much as possible of those information that are estimated
to contribute to the more complete description of global dynamic features of the
modelled system. This often results in an inclusion of the deficiently accurate or even
unmeasurable variables of the subjective character, thus elimanting in advance the
posibility to apply any type of formal, quantitative validity test (Forrester and Senge,
1986). Another reason is related to the information system support: due to the most
frequently long-term character of the SDSM’s it is usually difficult to provide for the
sufficiently longlasting and reliable series of historical empirical data, which also reduces
the posibility of applying the formal validity criteria.
Regardless the aforementioned significant limitations to the application of formal validity
tests of the SDSM’s, the development of the system dynamics methodology has been
accompanied with the constant striving after objectivity and strengthening of the integral
process of building confidence in models by introducing certain formal criteria or
procedures. One of the most remarkable contributions is that of Y. Barlas (Barlas, 1989)
who concentrated on the very general behavior pattern validation of the behavior
generated by the SDSM, which explains our giving him a prominence here. The starting -
point of the Barlas’s approach is the comparison of the model generated behavior
pattern with the actual, that is, the observed behavior of the key variables of the
modelled system. Here, a fact is taken into consideration, that the SDSM-simulation
outputs are indeed the time series of data, the break down of wich creates a basis for
comparison of the main dynamic features of these series and for conclusion making on
the degree of their correspondencem, i.e. on the validity of the behavior pattern
generated by the simulation model. Dynamic features, i.e. the time series components
that are being obtained through the mentioned break down, are the following: trend,
oscillation period, oscillation phase, average value and the cycle variation amplitude.
Of interest for this paper are the application limitations of the Barlas’s multiple validity
test, which the author himself emphasizes (Barlas, 1989). With respect to the steps of
this procedure and to their interdependence it can be concluded that the procedure
presumes the existence of a stationary, of the long-lasting stability and of the cyclic
behavior pattern of the modelled system. On the other hand, this means that the
procedure is inappropriate to validation of behavior during a relatively short period of
time, particularly to the explicitly unstable and nonstationary behavior patterns. In such
cases it is recommended to utilize the comparison of the original with the simulated
behavior based on the graphic representation, that is on the approximate visual estimate.
However, the mentioned limitations otherwise permit and even stimulate the application
of the so-called point-oriented approach whenever possible. Therefore, further in the pa-
per, a particularly prepared "statistical-validation block" - the subroutine adjusted to the
aforementioned point-oriented approach or the replicative validation of the SDSM is
offered.
1 $1 2
3. "Statistical-validation block - STV" of the SDSM
The selected simulation model formal validity criteria constituting the "STV" are of the
known statistical-econometric character: RMS-simulation error, RMS-percent error and
Theil’s inequality coefficient (Koutsoyiannis, 1981), all assigned to estimation of the repre-
sentativeness of each particular endogenous variable in the simulation model on the
basis of the previously mentioned point oriented approach. The fundamental supposition
for utilization of these criteria is that there exists a series of the observed data on the
corresponding endogenous variable trend. The additional requirement at structuring of
the subroutine "STV" is the possibility of its direct incorporation into the structure of any
SDSM. This is provided by programming the subroutine in one of the known system
dynamics programming languages, thus enabling the "STV" to serve not only for vali-
dation but also as a criterion for optimization of parameters within the model structure.
Further on, a general structural model "STV" of the universal subroutine for formal
testing of the so-called replicative validity of the SDSM has been presented (Figure 1):
Yigu Yee Yew nee Vhs Yat We Yosin
Lt © ()
‘OBSERVED SIMULATED.
VARIABLE - Yi_obs VARIABLE - sin
Sag ‘DEVIATION ee
‘SUM OF SQUARES OF
SUM OF SQUARES «J \ :
DEMATION . $KO_¥i STANDARDIZED,
DEMATION - SSKO_Yi
RMS-SIMULATION » RMS-PERCENT »
ERROR - RMS_SE_Yi ERROR - RMS_PE Yi
SUM OF SQUARES OF SUM OF SOUARES OF
‘OBS. VARIABLE - 52_¥ obs SIM, VARIABLE - $2_¥1siw
\Q THEIL'S INEQUALITY -.
COEFFICIENT - THEIL_CO_Yi
Figure 1. Mustration of the general structural Figure 2. Flow-diagram of the "STV"
model "STV"
The "STV"-flow diagram in the POWERSIM notation, as illustrated in Figure 2, inevi-
tably makes the integral part of the flowchart of every SDSM that utilize such approach
to model validation.
In the DYNAMO notation, the "STV" computer form on the example of only one
selected endogenous variable "Y" looks as follows:
SKO_Y.K = SKO_Y.J + DT*DKO_YJK - state of summation of squares of the
deviation of the Yi_obs from Yi_sim
SKO_Y = 0 - initial state of summation of squares of the deviation
DKO_Y.KL = (Y_obs.K - Y_sim.K)**2 - increment of squares of the deviation
RMS_SP_Y.K = SQRT(SKO_Y.K / TIME) - RMS-simulation error for variable Y
SSKO_Y.K = SSKO_Y.J + DT*DSKO_Y.JK _ - state of summation of squares of the
standardized deviation of the Yi_obs from Yi_sim
SSKO_Y = 0 - initial state SSKO_Y
Z PpwAze
198 3
R DSKO_Y.KL = ((¥_obs.K - Y_sim.K)/Y_obs.K)**2 - increment of squares of the
standardized deviation
A RMS_PP_Y.K = SQRT(SSKO_Y.K/ TIME) __ - RMS-percent error for variable Y
L $2_Y_sim.K = $2_Y_sim.J + DT*D2_Y_sim.JK ~ state of summation of squares
of the simulated variable of the variable Y
N S2_Y_sim = 0 - initial state of the squares summation
R DS2_¥_sim.KL = Y_sim.K**2 - increment of squares of the simulated variable Y
L S2_Y_obs.K = $2_Y_obs.J + DT*D2_Y_obs.JK - state of summation of squares
of the observed value of the variable Y
N S2_Y_orig = 0 - initial state of squares summation
R
A
DS2_Y_orig-KL = Y_orig-K**2 - increment of squares of the observed value of variable Y
THEIL_KO_Y.K = RMS_SP. _Y.K / (SQRT(S2_Y_sim-K / TIME) +
+ SQRT(S2_Y_orig.K/TIME)) + Theil’s inequality coefficient
4. Conclusion
Emphasizing the informal, mostly qualitative character of the system dynamics approach
to validation of simulation models on one side, that is, giving minor prominence to the
formal, quantitative tests and procedures, on the other side, have very often been the
subject matter of discussions and even of criticisms in the scientific circles dealing with
the real systems modelling in general. The reactions of the system dynamics experts to
such discussions and criticisms are variant: from those which emphasize the objective
limitations to application of the traditional formal criteria and validity tests in the system
dynamics, to those which, taking into consideration the aforementioned limitations, tend
to the development of certain formal, quantitative procedures applicable to the
validation of the SDSM’s. The Barlas’s multiple test procedure represents one of the
attempts to establish the logical sequence of a certain number of the formal validation
criteria and is adjusted primarily to testing the degree of correspondence of the geberal
model-generated behavior pattern to the original or observed behavior of the real system
(the behavior pattern validation). However, this procedure is suitable for the validation
of models of a limited number of the real systems that have a longterm history of the
stationary and, most often, of the cyclic behavior. Among others, this is the reason why
another formal validation concept has been offered in this paper through the so-called
"STV" validation concept consisting of the three point-oriented criteria-indicators. By
incorporating the special subroutine "STV" directly into the structure of any SDSM it is
not only possible to test efficiently the degree of correspondence of the simulated to the
observed values of the main variables in the model, but also perform the additional,
heuristic optimization of the model parameters utilizing the mentioned indicators as the
optimization criteria.
References:
1. Barlas, Multiple tests for validation of system dynamics type of simulation models, European Journal of
Operational Research, 42(1989), str. 59-87.
2. Y. Barlas and S. Carpenter, Philosophical roots of model validation - two paradigms, u: System
Dynamics Review, No 2/1990, SD-Society, MIT, Massachusetts, str. 148-166.
3. G.E.P. Box and G.M. Jenkins, Time Series Analysis, Forecasting and Control, Holden-Day, San
Francisco, CA, 1970.
4. J. Forrester and P.M. Senge, Tests for building confidence in SD models, u: A.A. Legasto Jr., J.
Forrester and J.M. Lyneis, Eds., System Dynamics, TIMS Studies in the Management Sciences 14,
North Holland, Amsterdam.
. A. Koutsoyiannis, Theory of Econometrics, Second Edition, The MacMillan Press Ltd, London, 1981.
. POWERSIM - The Complete Software Tool for Dynamic Simulation, User's Guide and Reference,
ModellData AS, Norway, 1993.
nun
£4 4
CC BY-NC-SA 4.0