1994 INTERNATIONAL SYSTEM DYNAMICS CONFERENCE
Results of the 1993 System Dynamics Society Benchmarking Study
Greg J Scholl
Booz-Allen & Hamilton Inc
101 Park Avenue
New York, New York 10178
Abstract
The lack of commonly held rules or dards for system dy ics, the ecl of its appl
and the wide variety of modeling and deli lhodologies developed over the past thirty
years present formidable chall for ishing an effectively joning system dy i
community. It is reasonable to suggest that the current structure of the community may hold back the
growth of the field more than obvious obstacles such as the inherent difficulty of developing
insightful models or the counterintuitive nature of nonlinear feedback lessons. In this light, the
structure of the community deserves closer scrutiny.
Benchmarking the System Dynamics Community , a survey sent by the author to System Dynamics
Society members in February of 1993. is the first survey of the system dynamics community and a
first step towards developing a more comprehensive understanding of the structure of the field. The
survey instrument consisted of five sections: Background, Practice, Software and Hardware, Model
Building, and History, Development. and Community. The survey was designed to isolate where
system dynamics is practiced, by whom, and for what; understand how the methodology is applied;
and identify lines of communication across the field.
This paper presents the results of survey responses, structured parallel to the five sections of the
survey instrument. It concludes with a senes of ions for further i igati
System Dynamics : Methodological and Technical Issues. page 226
1994 INTERNATIONAL SYSTEM DYNAMICS CONFERENCE
Results of the 1993 Sy Dy ics Ci ity Benchmarking Study
Introduction
The lack of commonly held rules or standards for system ics, the icism of its appli
and the wide variety of modeling and deli thodologies developed over the past thirty
years present formidable chall for ishing an effectively fu ing system d
community. It is reasonable to suggest that the current structure of the community may hold back the
growth of the field more than obvious obstacles such as the inherent difficulty of developing
insightful models or the intuitive nature of nonli dback lessons. In this light, the
structure of the community deserves closer scrutiny.
‘Benchmarking the System Dynamics C ity”, a survey sent by the author to System Dynamics
Society members in February of 1993, is the first survey of the system dynamics community and a
first step towards developing a more prehensi d ding of the of the field. The
survey instrument consisted of five sections: Back d, Practice, and Hard Model
Building, and History, Development, and Community. The survey was designed to isolate where
system dynamics is practiced, by whom, and for what; understand how the methodology is applied;
and identify lines of communication across the field.
This paper presents the results of survey responses, structured parallel to the five sections of the
survey instrument. It concludes with a series of questions for further investigation.
Who participated
Over thirty-five percent of the 455 members of the System Dynamics Society participated in the
study. Although 28 countries are represented in the study,! country-specific analysis is
inappropriate due to the small sample size for most countries.
Survey results
Background
Respondents consider themselves active system dynamics practitioners. Seventy percent indicate a
level of involvement of "7" or above (out of "10"), while only 13% report their involvement to be
under "5". Indeed, respondents are not newcomers to the field: 53% have been involved with system
dynamics for over ten years, while 21% have been involved for 5 to 10 years, 22% for between | and
5 years, and only 4% for less than 1 year.
As shown in Figure 1, most respondents are male, between the ages of 40 and 55, and hold PhD/DSci
degrees. MBAs constitute 37% of the M.A./M.S. degrees, compared with 20% in an engineering
discipline. Similarly, the fields of business or management comprise 32% of the PhD/DSci degrees,
compared with 15% in an engineering discipline. Of the PhDs, 15% indicate that their degree is
explicitly in system dynamics.
Respondents register strong quantitative backgrounds. Sixty-one percent indicate levels of expertise
of "advanced" or “expert” in both statistics and mathematics, while less than 20% consider their
mathematical or statistical skills to be “basic”.
1
All of the 13 countries not represented in the survey responses have two members or less in the System Dynamics
Society.
System Dynamics : Methodological and Technical Issues, page 227
1994 INTERNATIONAL SYSTEM DYNAMICS CONFERENCE
HIGHEST
AGE DEGREE SEX
RECEIVED
1% 9% 7%
ee 27%
26%
54% 65% 93%
Hi <25 sas. I Mate
BB 25<40 @ mams. Female
0 40<55 OD Phowsci
>55
Figure 1: Profile of survey respondents
While 91% of respondents have graduate-level degrees, only 58% have formally studied system
dynamics. Figure 2 shows the percent of respondents who have formally studied system dynamics
and notes the institutions that sponsored their study. Massachusetts Institute of Technology,
Dartmouth College, and State University of New York at Albany sponsored over three-fourths of all
formal study, with the remaining 25% shared among 11 American and 8 European universities. Of
the top three sponsors, however, only MIT and SUNY—Albany still actively teach system dynamics
modeling.
TYPE OF SPONSORS OF
FORMAL STUDY FORMAL STUDY
4% 3%
¢ 42% 25% 54%
7%
14%
I None Bur
Graduate studies EB Dartmouth
{10 Seminar/summer course oO SUNY—Albany
Undergraduate studies Other
E} other
Figure 2: Summary of formal training
Practice
Table 1 shows the areas of application and 2nvii in which resp apply system
dynamics.? The majority of respondents are academics, and almost half of all respondents work in
the private sector. Most resp apply system to business and corporate policy, and a
large percent also apply system dynamics to public and social policy.
2 Please note that the categories in Table 1 are not mutually exclusive. A Professor, for example, might also const
to the private sector, and could be involved with business policy, social policy, and system dynamics research.
System Dynamics : Methodological and Technical Issues, page 228
1994 INTERNATIONAL SYSTEM DYNAMICS CONFERENCE
ENVIRONMENT
AREA OF APPLICATION Academia Private Sector | Public Sector TOTAL
Business/Corporate Policy 37% 37% 21% 58%
Public/Social Policy 32% 21% 25% 44%
System Dynamics* 28% 11% 13% 31%
Policy 18% 14% 11% 25%
Industrial Policy 14% 9% 9% 18%
Other 18% 11% 10% -25%
TOTAL 66% ~ 48% 35%
“Includes teaching and methodological research
Table 1: Practice areas and environments of application
Forty-five percent of all respondents are Professors, almost 70% of the total academic group.
Management consultants comprise over 18% of all respondents, and an additional 11% practice
system dynamics within a business izati The high proportion of business and corporate
policy practitioners is consistent with the applied (as opposed to methodological) content of the
System Dynamics Review from 1985 to 1990, but the high share of management consultants contrasts
with the near absence of Review articles over the same period that relate system dynamics consulting
experience (Richardson 1991).
Interestingly, the subset of respondents who indicate a level of involvement of "7" or greater are not
concentrated within a specific envii or area of application. When Table 1 is adjusted to reflect
only the “active” modeling subset, the relative distributions do not change by more than 1% in any
category.
Environment
In all, 58% of respondents practice system dynamics wholly within one type of environment. In
Figure 3, the degree of overlap between environments is explored in more detail. Over half of the
66% of respondents in academia practice system dynamics wholly within the academic environment
(34% of all respondents). Therefore, almost fourth of the Pi who ded also practice
system dynamics in the public or private sector.
PRIVATE PUBLIC
ACADEMIA SECTOR SECTOR
52%
Bh Academia only I Private sector only I Public sector only
With private sector With public sector ey With private sector
(11 with public sector ID With academia [1] With academia
With private and With public sector With private sector
public sector and academia and academia
Figure 3: Distribution of envi of resp
System Dynamics : Methodological and Technical Issues, page 229
1994 INTERNATIONAL SYSTEM DYNAMICS CONFERENCE
Areas of application
d are less d in their areas of application, with only 38% practicing system
dynamics wholly within one area. Figure 4 shows the distribution of the single-area group.
Respondents who apply system dynamics to business and corporate policy issues constitute the bulk
of the group, with a 59% share.
Il Business/Corporate Policy
EB Public/Social Policy
[ID system Dynamics*
Environmental Policy
=| Industrial Policy
*Includes teaching and methodological research
Figure 4: Dis ion of single de
Not surprisingly, within the broad area categories, the specific topics to which respondents apply
system dynamics widely vary. Modeling topics range from energy research to pharmaceutical new
product modeling; from religion and p phy to and from issues within
the Vermont Department of Corrections system to how technology affects the rise and fall of
civilizations. However, evidence suggests that Society membership might largely miss some
segments of the system dynamics modeling community.3
Model building
Methodology
Not all respondents consider system dynamics to be an isolated area of practice. As one responden’
notes, closed form equations are a standard tool in science, engineering, finance, and economics. The
majority of respondents, however, seem to agree upon what i system di i deli
As shown in Figure 5, most respondents develop continuous models and feel that models must begin
simply, capturing the top-level feedback structures that characterize the system first, and progressing
towards a lower level of complexity as these structures are disaggregated into more detail.
GREE OF
ere ueruae INITIAL COMPLEXITY
("1" = LOWEST)
6%
14%
80%
WB Continuous Bh t'to'3"
© Moe EB "4" toe"
OD osscrete OD "7" to "10"
Figure 5: Model structure and approach
3 In particular, conversations with Dr. Carl Way of the U.S. Army Corps of Engineers point to a large ecological
modeling community outside of the System Dynamics Society.
System Dynamics : Methodological and Technical Issues, page 230
1994 INTERNATIONAL SYSTEM DYNAMICS CONFERENCE
Stages in model development
R dents agree that ization is the most time-i ive model devel stage,
followed by i i and impl ion. Table 2 shows how respondents allocate
their time throughout the four stages of a modeling effort (cf. Randers 1980). Note that the column
totals do not add to 100%, as some respondents ranked activities equally.
TIME SPENT ("1" = MOST TIME)
MODELING ACTIVITY “" "2" "3" "4" TOTAL
Ce 45% 32% 16% 7% 100%
Formulation 33% 41% 17% 9% 100%
Improvement 17% 24% 40% 19% 100%
Implementation* 10% 11% 23% 50% 94%
*6% of respondents do not implement their models
Table 2: Time-ii i of model develop Stages
C lizati
Respondents primarily use causal loop di: to assist in ptuali
* 82% of respondents utilize causal loop diagrams
* 70% utilize flow diagrams
* 41% utilize substructure diagrams
+ 19% utilize policy structure diagrams.
Formulati d improve
During the for ion and ii stages, resp ” use of tests varies (cf.
Forrester and Senge 1980). As illustrated i in Figure 6, respondents do not appear to apply a “standard”
set of confidence tests to their models. Only two tests — structure and parameter verification — are
used more than 80% of the time, and some i tests, like dii ional i , seem under-
utilized.
100
90
80
70
PERCENT OF ©
TOTAL 50
RESPONSES 49
30
20
wea ageless ome se wae alas
SERSESISEEERT PS SLES RAPES
Ze RPSHPRSESS PERE EEE SEE SE
SESeeR BESS SESE OES EES S Ex
GB Es eVsee es zssrsssggzse525
B55 Se ssest erst esses
gu egeg go 8 estas
Q £16 0
| a
STRUCTURE BEHAVIOR POLICY
TYPE OF TEST
Figure 6: Percent of respondent who use confidence tests
‘System Dynamics : Methodological and Technical Issues, page 231
1994 INTERNATIONAL SYSTEM DYNAMICS CONFERENCE
Implementation
Nine percent of de in academia do not impl their models, d with 2% and 3%
in the public and private sectors respectively. Of the 94% of respondents who implement their
models, most transfer system insights and few transfer a physical model:
* 78% transfer an understanding of the system
* 67% recommend policy changes based upon the model
+ 38% implement operational changes based upon the model
* 26% transfer the model itself.
Model types
As illustrated in Table 3, most respondents develop either general or policy
models, and a smaller number develop detailed imp! ion models (cf. Mead 1980).
FREQ. OF DEVELOPMENT ("1" = MOST OFTEN)
MODEL TYPE “" "2" “3" TOTAL
General it 42% 26% 11% 79%
Policy i 42% 30% 8% 80%
Detailed i 15% 13% 19% 41%
Table 3: Types of models developed by respondents
Differences appear to exist, however, in the thodologit it d by developers of different
model types. First, Figure 7 shows the type of model structure, degree of initial model complexity,
and average number of variables for the subset of total respondents who develop either general
understanding, policy formulation or detailed implementation models most often.
DEGREE OF
MODEL NUMBER OF
INITIAL COMPLEXITY
STRUCTURE (‘1" = LOWEST) VARIABLES
8% 8% 8% 3%
12%
GENERAL ws 22%
UNDERSTANDING
73% 80% 67%
8% 5% 3%
11% o%
POLICY 2% 46%
FORMULATION
71% 84% = 42%,
4% 17% 17% ee 17%
DETAILED 48%
IMPLEMENTATIO! 17%
N 48% ° 66% 22% fren
I Continuous HE 't"to"" HB i <100
EB Mixed EB 4" to"e" EB 100< 500
0 Discrete 1D '7" to "10" 1 500< 1,000
BZ > 1,000
Figure 7: Methodologies by model type developed most often
System Dynamics : Methodological and Technical Issues, page 232
1994 INTERNATIONAL SYSTEM DYNAMICS CONFERENCE
While respondents who develop general understanding models or policy implication models the most
tend to share methodologi spond: who develop detailed implementation models
the most show less consensus about modeling practice.
Next, developers of the three model types utilize confidence tests to varying degrees. Respondents
who develop detailed implementation models the most apply more rigorous standards to their models.
On the surface, this stands to reason ~ detailed implementation models are likely to be more
jive than general und ding models and could require a broader range of confidence
tests. A more careful analysis, however, reveals inconsistencies in confidence testing across model
types.
Respondents who develop general understanding models the most utilize structural tests the least.
Certainly, general understanding models may not require a battery of statistical tests, or point-to-point
fidelity with the reference mode. One could argue, however, that structural tests are perhaps most
important for general understanding models, as the scope, purpose and audience of a general
understanding modeling effort often does not force modeling rigor a priori.
On the other hand, Tespondents who develop general understanding models the most test most
quently for symp’ ion, behavior ch istics, pattern predi
and event prediction. To conjecture, these differences could reflect the fact that it is easier to "hard-
wire" a simple model to produce expected behavior that, in the absence of rigorous structural testing,
might seem robust.
Finally, analysis of critical skill requirements across each of the three model types reveals differences
of opinion about the essential skills necessary for effective system dynamics modeling. Figure 8
presents the minimum skill requirements for each model type as judged by respondents who develop
that model type most often. Respondents who develop general understanding models the most rate
‘hnical modeling ability, math ical and
MODEL TYPE
GENERAL POLICY DETAILED
100 + UNDERSTANDING FORMULATION IMPLEMENTATION
PERCENT —¢q
OF
RESPONDENTS 40
20
0
ot> oto a
2228228 8 § $3 8 228 8
$B EE s BB € 8 5 SEE
o gs 6 2° 5 6 ° 8 5 5
> g 2 gs 88 g > 8
&§ © F X 9 e€ o> § x 5 2D x x
2 £& © oo € FEF &bGEeEE EE ES
€3 sy & €8 6 a £
= £sne¢ 2s £s S Ee R
£E°sB SF © ERDBse RE sg £58:
of 2 s 3 = 8 9 Ss ® o 2 = s @
§ & @ 6g: 2 ge &
zg 3 B 28 8 Bg 8
2
anal og @ rr] for e
CRITICAL SKILL
Figure 8: Critical skill requirements by model type
expertise, and expertise in the problem area as non-essential. In contrast, policy formulation and
detailed impl ion model developers both feel that tect deling ability is ly
System Dynamics : Methodological and Technical Issues, page 233
1994 INTERNATIONAL SYSTEM DYNAMICS CONFERENCE
important, and detailed impl ion model developers also place a great emphasis on
th ical and statistical expertise. The premium placed on “system thinking skills” by gener:
understanding modelers in lieu of more rigorous technical modeling ability, bined with their
inconsistent use of confidence tests, suggests that a possible lack of robustness may exist across the
population of general und ding models.
Software and Hardware
Software
Most respondents use STELLA or iThink to support model development. Figure 9 shows software
use among the respondents.
100 +
90 + I Use regulary
80 + a Use occasionally
70
PERCENT OF © |
TOTAL 50 +
RESPONSES, 40 4
30
20 +
10
o4
= g 2 8 & 2
= Zz se) 3 g 3
5 2 Es 3 s
= SB
5 S Bs &
ira es =
5 a
SOFTWARE APPLICATION
Figure 9: Software used by respondents
Hardware
Sixty-eight percent of respond use Macintost 59% use PCs, and 29% use both.
Minicomputers and mainframes are each used by 9% of the respondents. Respondents indicate that
their choice of a hardware platform is driven primarily by either the requirements of a modeling
language or resource availability, and many are frustrated by the lack of cross-platform compatibility
among software applications.
History, development, community
On the whole, respond: profess a to ive familiarity with the history of system
dynamics, but read little system dynamics material outside of the System Dynamics Review regularly,
as shown in Figure 10. Some respondents argue that the Review is narrowly focused, and indeed,
35% of the journal’s pages are dedicated to methodological issues. However, some maintain that, as
the Review is the professional journal of the System Dy ics Society, this methodological focus is
to be expected (Richardson 1991).
System Dynamics : Methodological and Technical Issues, page 234
1994 INTERNATIONAL SYSTEM DYNAMICS CONFERENCE
KNOWLEDGE OF PERCENT OF
HISTORY OF RESPONDENTS WHO
SYSTEM DYNAMICS READ PUBLICATIONS
100
90
18% 80
308 70
PERCENT OF 60
TOTAL 50
RESPONSES 45
50% 30
20
I None 0
EB slight 2a Dp Q¥
e $€ 2 ge
[I Moderate es 52 ge 22 8 €F
Extensive sc 23 2 & 2 = L
a 5 3 2 3 t
g Ce & a §3
& ° 3°
an a
SYSTEM DYNAMICS PUBLICATION
Figure 10: Knowledge of system dynamics history and percent that read publications
Many resp feel that it is imp to follow P outside of system dynamics an
regularly read journals like M: Science, Europ Journal of Operational R h
International Journal of Systems Science, and the Strategic Management Journal. To that point,
several respondents urge that system dynamics must increase its credibility as a problem-solving
ique in disciplines like socig| political science, and economics.
Only 51% of respondents indicate that they are in frequent contact with other practitioners. A number
feel “isolated” from practitioners, published models and papers and current modeling software, and
stress the need for a broader dissemination of relevant research and development throughout the
system dynamics community.
Table 4 presents the top five models and papers or books that respondents cited as most significant in
the development of the field. Of the more than 3,300 articles and books in the literature of system
dynamics, respondents identified 82 unique works.
CITED MODELS PERCENT CITED PAPERS OR BOOKS PERCENT
1. WORLD models 44.0% 1. Industrial Dynamics 15.0%
2. Industrial Dynamics 21.0% 2. Counterintuitive Behavior of Social Systems 9.0%
3. Urban Dynamics 19.0% 3. Market Growth as Influenced by Capital Investment 8.0%
4. Market Growth 11.0% 4. Tests for Building Confidence in System Dynamics Models 6.0%
5. National Model 9.0% 5. The Fifth Discipline 5.0%
Table 4: Top models and top papers or books
Generally, respondents attribute their selections to the positive contribution of the work to the field.
The notable exception was the WORLD models, about which respondents’ opinions bifurcated. Some
felt that the model put forth a timely and important message, while others felt that the model has given
system dynamics a “notoriety” that the field has yet to overcome.
4 For more information about how to receive a bibliography of system dynamics literature, please contact Kip
Cooper or William Steinhurst at The International System Dynamics Society, Bibliography Committee, 49 Bedford Road,
Lincoln, MA 01773, U.S.A.
System Dynamics : Methodological and Technical Issues, page 235
1994 INTERNATIONAL SYSTEM DYNAMICS CONFERENCE
Jay Forrester was the most frequently cited author by a significant margin, with 80 citations. Peter
Senge was second with 21, followed by John Sterman with 18. George Richardson was fourth wit!
12, and John Morecroft fifth with 8.
Questions for further investigation
Ulti , the 1993 benchmarking study could help the system dynamics community to define
questions that may Prove critical to the continued development of the field and the greater
ion of system d i pts worldwide. Based upon the results of the benchmarking,
questions for further investigation could include:
* What are the characteristics of a “typical” member of the System Dynamics Society? Do these
characteristics meet the criteria that are relevant to problem-solving in the 1990s?
* What is the relationship between system dynamics modeling and systems thinking?
+ What are the critical requirements for the field to continue to evolve (e.g., the installed base
necessary within academic institutions to maintain a pipeline of future practitioners)? What steps
can be taken to meet these requirements? How can the field increase demand for system
dynamics expertise?
+ What are the logical sut ities in system d) ics, and is it imp to
between them?
+ Asan umbrella term, does "system dynamics" do more harm than good?
+ Isacommon methodology shared by system dynamicists? Are methodologies shared at the
subcommunity level or across model types? If not, what are the consequences to the field?
+ How stable has the membership of the System Dynamics Society been over time? Who are the
core members, who are the fringe players, and what effect does each group have on the field?
What role do the System Dynamics Society and the System Dynamics Review play in the
development of the field, and how might this role change?
Acknowledgments
The author would like to thank Louis Alfeld, Andy Ford, Steven Maloney, David Peterson, Julia
Pugh, George Richardson, Jack Scholl, John Sterman, and Carl Way for useful comments and
suggestions, Diane Taylor at John Wiley & Sons for providing ion; and Garrett
Dietz at Towers Perrin General Management Consulting for sporisoring the i initial survey mailing.
References
Cooper, Kip and Steinhurst, William. 1992. International System Dynamics Bibliography.
International System Dynamics Society, Bibliography Committee, 49 Bedford Road, Lincoln,
MA 01773, U.S.A.
Forrester, Jay W. and Senge, Peter M. 1980. Tests for Building Confidence in System Dynamics
Models. In TIMS Studies in Management Science 14: 209 - 228.
Meadows, Donella. 1980. The A Priori Imperative. In Elements of the System Dynamics Method, ed.
J. Randers. Cambridge, MA: MIT Press.
Randers, Jorgen. 1980. Elements of the System Dynamics Method. Cambridge, MA: MIT Press.
Richardson, George P 1991. What Are We Publishing? A View from the Editor’s Desk. System
Dynamics Review 7 (1): 61 -—
System Dynamics : Methodological and Technical Issues, page 236
