Stephens, Craig A.; Graham, Alan K.; Lyneis, James M. "System Dynamics Modeling in the Legal Arena: Special Challenges of the Expert Witness Role", 2002 July 28-2002 August 1

Online content

Stephens, Graham and Lyneis: Expert Witness Role

Go Back

Table of Conte

System Dynamics Modeling in the Legal Arena:
Special Challenges of the Expert Witness Role

Craig A. Stephens, Alan K. Graham, James M. Lyneis
PA Consulting Group, Inc.

One Memorial Drive, Cambridge, Massachusetts 02172 USA
Voice: 617-225-2700, Fax: 617-225-2631
craig.stephens@pa-consulting.com
alan.graham@pa-consulting.com

jmlyneis@alum.mit.edu

Abstract

In the late 1970s System Dynamics models began to be employed in legal disputes as a means of
proving and quantifying damages due from one party to another, and such use seems likely to
increase in the future. But most System Dynamics practitioners are unfamiliar with the role of
expert witness and the obligations and responsibilities associated with it. Admissibility of the
would-be expert's testimony is now more significant a hurdle than ever, under recently
established Supreme Court guidelines that lean heavily on the scientific method.

Best-practice System Dynamics work adheres to the scientific method and should prove
admissible. But many System Dynamics applications are carried out to less stringent standards
that leave testimony based on that work vulnerable to being ruled inadmissible. Beyond
admissibility, the process of preparing to deliver expert testimony based on System Dynamics
presents unusual challenges for the practitioner.

System Dynamics practitioners providing expert testimony in opposition to that of other System
Dynamicists face some special challenges. In the experience of the authors and their colleagues,
the quality of opposition testimony by System Dynamics experts has been poor. Opposing expert
testimony must improve significantly to meet the Supreme Court's new admissibility standards.
The stakes are high, since upcoming cases are likely to set precedents that will significantly
affect future use of System Dynamics in the legal arena. In addition, legal admissibility will
doubtless impact the more general perceived legitimacy and acceptance of System Dynamics in
corporate and other non-legal settings.

Keywords: Legal disputes, expert witness, expert testimony, admissibility, Daubert, scientific
method, hypothesis testing, System Dynamics.

The authors wish to acknowledge the long-term contribution of their PA colleagues who have
devoted so much thought and effort to work in the legal arena, especially Ken Cooper and Tom
Mullen. We would also like to acknowledge the tremendous amount we have learned about the
legal arena and the role of the expert witness from our various attorney friends, especially Jeffrey
Dorman of Freeborn & Peters.

1. Introduction

Ever since its founding by Jay Forrester, System Dynamics has been employed to analyze the
past and potential future performance of dynamically complex systems. Many of these have
been business systems involving customers and markets, companies or business units, product
lines and services, value chains, and so on. In such applications System Dynamics has been
appreciated for its ability to make the performance of such systems more understandable, more
reliably quantifiable, and potentially more controllable.

In the late 1970s System Dynamics models began to be employed in legal disputes as a means of
proving and quantifying damages due from one party to another. _In such cases the damages
are usually alleged to have resulted from some breach of contract. A System Dynamics model is
usually employed to prove that the alleged damages did, in fact, occur and to differentiate them
quantitatively from the performance that would have occurred in their absence. In the past
twenty-five years System Dynamic models have been so applied by PA in 45 legal cases, and to
an apparently lesser extent by other practitioners. In most such cases one party to the dispute
employs a System Dynamics practitioner with the expectation that he or she will conduct
dynamic simulation analysis and provide expert testimony that will help win the case.

Often the opposing party will also employ an expert to provide analysis and rebuttal testimony to
refute that of the System Dynamicist. Increasingly the opposing expert is another System
Dynamicist, pitting professionals from our relatively small field against each other in a head-to-
head contest. With high stakes and pressure from clients and attorneys on both sides, these
“expert wars” can easily become rather extreme. Under the press of circumstances it can be
difficult to remember that the role of expert witness imposes requirements and obligations on the
System Dynamicist. Those requirements and obligations are the subject of this paper.

Most of the legal cases in which System Dynamics has been employed have involved complex
design/construction projects that overran planned schedules and budgets, with the dispute arising
between an owner organization commissioning the project and a contractor organization hired to
carry it out. A few cases have arisen between project owners and regulatory bodies, and others
have arisen between project contractors and suppliers or subcontractors. Most legal applications
of System Dynamics have been in support of the contractor organization as plaintiff in these
cases. As the number and complexity of large projects continues to grow in a variety of
industries, with attendant growth in the number and size of project performance “failures”, it
seems likely that such disputes and dispute applications of System Dynamics will continue to
grow in number.

There has been at least one legal application of System Dynamics that involved loss of profit and
market share in an ongoing business rather than development project overruns, and other such
cases are on the horizon. It appears that future non-project legal applications of System
Dynamics are likely to grow even faster than will project-dispute applications.

The 45 legal cases on which PA Consulting has worked cover the shipbuilding, aerospace, oil

and gas, civil construction, and software systems industries. They involve both court trials and
formal arbitration proceedings in Europe and North America under US law, British law, and

International Chamber of Commerce arbitration rules. Disputed amounts have ranged from
several tens of millions of dollars to several billions of dollars, with the average probably falling
in the range of three hundred to four hundred million dollars. All but three of these cases have
been resolved, all of them through negotiated settlements outside the court or arbitration
proceedings. The three cases that have not been settled are still in process. PA’s involvement in
these cases has spanned the following range of activities:

e Rapid qualitative assessment of the strengths and weaknesses of a case based on
expected dynamics of the organizations involved;

e¢ Rough modeling and simulation analyses to support initial quantification of
damages;

e Refined modeling and simulation analyses to withstand adversarial attack;
e Preparation of heuristics to guide settlement discussions;

e Submission of model and analyses during evidentiary discovery;

e Preparation for and participation in deposition sessions;

e Preparation of materials for delivering expert testimony;

e Delivery of expert testimony.

In the legal arena a System Dynamics model serves a dual quantitative purpose: 1) to determine
what dynamics governed the performance of the subject organization; and 2) to employ those
dynamics, once known, to determine what different performance would have resulted had the
parties behaved differently or had different conditions prevailed. Differences in performance
become the basis for determining damages. In such cases the different behavior and/or
conditions that are the subject of the dispute are usually defined by the terms of some existing
contract between the parties.

In legal disputes involving performance of complex organizations, PA’s experience (and that of
our clients) has been that properly executed System Dynamics analyses offer some powerful
advantages over more conventional analysis methods:

e The model is not a black box -- causality is explicitly identified and quantitatively
represented and can be audited and understood by the factfinder(s) — the judge,
jury or arbitration panel charged with determining the facts in the case;

e Good System Dynamics analysis yields both the quantification of damages and
the dynamic story or explanation of how those damages arose;

e System Dynamics analysis reveals clearly and quantitatively how each party
contributed to the organizational performance that is the subject of the dispute,
thus grounding damage claims in the broader dynamic story of “what happened”;

e A good dynamic model is almost always a much richer and more complete
representation of the organizational performance that is the subject of the dispute
than those provided by more conventional (usually static and open-loop) analysis

tools. This makes a good System Dynamics analysis difficult to attack
successfully;

e A good dynamic model can and should be used not only to prove “what
happened” and to quantify resulting damages, but to disprove the opposition’s
alternative theories of the case and associated quantification of damages. This is
much more difficult to do with conventional (usually static and open-loop)
analysis tools, and is a fundamental strength of System Dynamics when employed
in the legal arena.

In adversarial legal proceedings PA’s work has been attacked by a variety of opposing experts in
a wide variety of ways. Some of these opposing experts were from different disciplines and
were quite unfamiliar with System Dynamics, while others were experienced System Dynamics
practitioners. Regardless of their professional origins, opposition attacks have generally
included some of the following range of assertions:

e It is not possible to build a reliable model of all the factors PA purports to have
included in its analysis;

e Based on the particulars of the case in question, System Dynamics is not an
appropriate analysis method;

e The simulation model developed by PA contains fatal flaws or is fatally biased;
e The data employed in developing the model are fatally flawed or fatally biased;

e The simulation analyses conducted are slanted and inappropriately favor PA’s
client;

e The model’s simulated organizational performance under other-than-historic
conditions is not credible;

e The behaviors and conditions that PA purports to have analyzed were not those
that caused the problems that are the subject of this dispute;

e If the behaviors and conditions that truly caused the problems were properly
reflected in PA’s simulation model, the resulting analysis would not support PA’s
client’s case.

Despite such attacks, the settlements obtained in PA’s cases have almost always been regarded
by clients as quite favorable. Attorneys with whom we have worked tell us that the average
settlement obtained in those cases is nearly twice the size of those typically obtained in similar
cases in which System Dynamics was not employed. In no case have opposition attacks on PA’s
work resulted in what the client regarded as an unfavorable settlement. Invariably, when
opposing experts have been involved their critiques of PA’s work have contained major mistakes
of fact and/or methodology that rendered those critiques unreliable. In several cases the
evidentiary discovery process revealed those mistakes and the credibility of the opposing experts
and their criticisms was seriously undermined as a result.

Whether for the plaintiff or the defendant, whether in support or opposition, the role of the expert
witness imposes requirements and obligations that should be carefully considered by System

Dynamicists, especially because those requirements have changed recently. The following
discussion of those requirements and obligations is based on the latest legal standards in US
Federal courts, but the courts of 19 states have also adopted the Federal standards (Mahle 1999)
and the general issues involved are likely to be similar in other legal regimes.

Section 2 of this paper discusses admissibility standards for expert testimony established in
recent case rulings by the United States Supreme Court. Section 3 discusses selected
characteristics of good System Dynamics work and how those match up against the standards
established by the Court’s rulings. Section 4 discusses the tasks typically involved in preparing
and delivering expert testimony based on System Dynamics; Section 5 discusses the ethical
challenges inherent in the expert witness role and the special challenges System Dynamicists
face when offering expert testimony in opposition to that of other System Dynamics
practitioners. Section 6 concludes.

2; Legal Standards for Admissibility of Expert Testimony

On disputes involving performance of complex organizations, good System Dynamics analysis
work has proven to be an unusually effective and reliable means of proving and quantifying
damages resulting from the actions and omissions of the disputing parties. The effectiveness of
System Dynamics analysis in the legal arena results from its ability to reveal the chains of
causality connecting various acts and omissions of the disputing parties with the many resulting
indirect effects of those acts and omissions on organizational performance — this yields a more
complete and defensible proof and quantification of damages. Legal opponents have found it
difficult to mount effective attacks on System Dynamics analyses when those analyses are
properly carried out. But the potential probative and analytical advantages of System Dynamics
will be useless if expert testimony based on it does not meet the new standards for admissibility
in legal proceedings.

The Role of the Daubert Standards. Until 1993, admissibility of expert testimony in legal
proceedings was governed by the Frye standard, also known as the “general acceptance”
standard because it relied on general acceptance of the expert's analysis methods within the
scientific community. In 1993, in Daubert v. Merrill Dow Pharmaceuticals, Inc. the Supreme
Court ruled that the Frye standard was superceded by Federal Rule of Evidence 702 covering
expert testimony based on “scientific, technical or other specialized knowledge”. The court’s
Daubert ruling focused on expert testimony based on scientific knowledge, and laid down
standards for admissibility of such testimony as evidence. Subsequently, circuit courts split on
whether Daubert also applied to evidence based on “technical or other specialized knowledge”
and whether “soft” sciences (psychology, economics, etc.) counted as science under Daubert. In
1999, in Kumho Tire Company v. Carmichael the Supreme Court extended its Daubert ruling to
cover expert testimony based on other than scientific knowledge. In this paper, unless otherwise
specified the term Daubert will be employed to refer to the ruling in that case and in all
associated cases. In the broadest terms, then, Daubert now requires that all expert testimony be
“scientific”.

The importance of the Court’s Daubert rulings is illustrated by the following (Mahle 2001).

“Even after having lost on liability, lawyers are winning cases by using newly
available techniques, suggested by Daubert and Kumho Tire, to exclude the expert
testimony that links money damages to the act or omission for which their client has
been found liable.”

In other words, when an organization has lost the liability portion of its case and been found
legally liable to pay damages to its opponent, the loser can still achieve an effective win if the
testimony of the opposing expert hired to assist the court in quantifying those damages is ruled
inadmissible. That’s because establishing legal liability for damages is necessary but not
sufficient to actually recover damages — an equally crucial second step is to prove the magnitude
of the damages experienced. This is often done by means of expert testimony, testimony which
is not admissible unless it meets the new Daubert standards. If there is little or no admissible
testimony on the magnitude of the damages, then the factfinder or factfinders are unlikely to
award significant damages. Thus, if the System Dynamics practitioner cannot survive a
Daubert-based admissibility challenge then his or her testimony will not be heard, and the client
may not be awarded any damages despite having previously proven that the opposing party is
liable to pay such damages.

So far as the authors are aware, no case involving expert testimony based on System Dynamics
analysis has yet faced an admissibility challenge under the Daubert rulings. Attorneys familiar
with System Dynamics and with the Court’s rulings believe that expert testimony based on
System Dynamics analyses can meet admissibility standards if those analyses are carried out
using best practices. Until that is tested in court, firms and erstwhile experts contemplating use
of expert testimony based on System Dynamics must measure the characteristics of such
analyses against the specific standards set in Daubert since, in any significant case, legal
opponents are likely to challenge the admissibility of such testimony. The remainder of this
section sets out the Daubert standards, and Section 2 matches the characteristics of good System
Dynamics analyses against them.

The Daubert Standards. Daubert and related rulings oblige the trial court to be the evidentiary
“gatekeeper” who screens proffered expert testimony for admissibility.

“The objective of [the gatekeeping] requirement is to ensure the reliability and
relevancy of expert testimony. It is to make certain that an expert, whether basing
testimony on professional studies or personal experience, employs in the courtroom
the same level of intellectual rigor that characterizes the practice of an expert in the
relevant field [when dealing with similar matters outside the courtroom].”

United States Supreme Court in Kumho Tire Co. v. Carmichael

Daubert made an explicit link between the reliability of an expert’s testimony and the expert’s
use of scientific knowledge derived by the scientific method:

“\..the subject of an expert's testimony must be scientific...knowledge...[it is] the
requirement that an expert’s testimony pertain to ‘scientific knowledge’...[that]
establishes a standard of evidentiary reliability.”

“In order to qualify as ‘scientific knowledge’, an inference or assertion must be
derived by the scientific method.”

“Scientific methodology today is based on generating hypotheses and testing them to
see if they can be falsified; indeed, this methodology is what distinguishes science
from other fields of human inquiry.”

United States Supreme Court in Daubert v. Merrill Dow Pharmaceuticals, Inc.

After the Court’s ruling in Daubert, it became popular to proffer expert testimony as non-
scientific to avoid having to apply the new standards for admissibility. The Kumho Tire decision
closed off this option by extending the Daubert standards to other forms of expert testimony.

“Daubert’s general holding...applies not only to testimony based on ‘scientific’
knowledge, but also to testimony based on ‘technical’ or ‘other specialized’
knowledge.”

United States Supreme Court in Kumho Tire Co. v. Carmichael.

Building on the scientific method, Daubert and associated rulings provided factors for trial
courts to use in evaluating the scientific validity and resulting evidentiary reliability and
admissibility of expert testimony. Experts in the legal community have expanded on these
factors with specific admissibility tests (Mahle 1999, Berger 2000, Black et al., 1994).

1) The proffered testimony should be based upon a testable hypothesis or hypotheses.

a) The hypothesis must have explanatory power relative to the case — it must explain
the how and why of the observed organizational performance. More than merely
descriptive, the hypothesis must also be predictive.

b) The hypothesis must be logically consistent — it must contain no internal
inconsistencies and must not be self-contradictory.

c) The hypothesis must be falsifiable — it must be amenable to empirical testing that
would reveal flaws or shortcomings.
2) The hypothesis must have been tested to determine the known or potential error rate.
3) Hypotheses must have been formed and tested and analyses conducted in accordance
with standards appropriate to the techniques employed, including:

a) Standards regarding consistency with accepted theories — because scientific
knowledge is usually cumulative and progressive, hypotheses should most often
build from existing theories;

b) Standards regarding the scope of testing — “...the more severe and the more
diverse the experiments that fail to falsify an explanation or hypothesis, the more
corroborated, or reliable, it becomes...” (Black et al., 1994);

c) Standards regarding precision — precise results and statements are more readily
testable than are broad generalizations;

d) Standards regarding post-hypothesis testing — hypotheses must be capable of
explaining more than pre-existing data.

e) Adhering to the same standards as in non-legal applications.

4) The techniques employed used should have been peer reviewed.

a) Initial peer review usually occurs in the run-up to publication in peer-reviewed
journals.

b) The techniques employed should also have withstood the broader peer review to
which publication exposed them.

5) The techniques employed should have been generally accepted in the scientific
community.

The first two of these five Court-supplied factors are the basis for determining whether
the proffered expert testimony is appropriately based in the scientific method. The third
factor is the basis for determining whether the putative expert witness employed good
scientific practice. The fourth and fifth factors give confidence that the scientific
methodology employed is generally viewed as sound.

3. How System Dynamics Matches Up Against the New Standards

Let’s march through the Court’s criteria and examine how closely best-practice System
Dynamics work comes to meeting them.

1) The proffered testimony should be based upon a testable hypothesis or hypotheses.

a) The hypothesis must have explanatory power relative to the case.

b) The hypothesis must be logically consistent.

c) The hypothesis must be falsifiable.
Testimony should be based upon a testable hypothesis or hypotheses. In PA’s experience, use of
System Dynamics in the legal arena involves forming and testing hypotheses at two different
levels. The first hypothesis is at the whole-organization level, creating and testing a hypothesis
of the organization’s overall dynamics. The second hypothesis is at the level of the ‘theory of

the case’ or each party’s story of “what happened”, and it reflects only the most relevant subset
of the overall dynamics.

At the highest conceptual level a System Dynamics model constitutes a hypothesis about the
dynamics of the organization(s) involved in the dispute, that is, the elements, interconnections
between elements, and resulting feedback loops driving overall organizational performance. The
hypothesis is stated in terms of computerized mathematical equations that explicitly identify each

element and feedback loop and quantitatively characterize the strength and timing of those loops.
This method yields a clear and unambiguous statement of the high-level hypothesis, one that is
readily auditable and can be understood by anyone who takes a little time to learn some simple
equation conventions.

The second hypothesis is narrower, including just those loops that were active in creating the
damages that are the subject of the dispute, e.g., the impact of delay and disruption. The second
hypothesis is developed from analyses that employ the ‘whole-organization’ hypothesis or
model. This second hypothesis is not a model, rather, it is the dynamic story of ‘what happened’
that legal factfinders need to know to determine damages. While it is theoretically possible to
begin forming and testing hypotheses in the form of models at the ‘what happened’ level, for
reasons explained below such hypotheses are unlikely to be persuasive or to withstand a Daubert
challenge. Good scientific practice calls for forming and testing hypotheses that broadly explain
the performance of the subject organization, and then employing those hypotheses (models) to
answer specific questions about that performance as a basis for determining damages.

Hypotheses must have explanatory power relative to the case. For the hypothesis or model to
have explanatory power relative to the legal case, it must usually include far more feedback
loops than will ultimately be found to have caused the damages that are the subject of the
dispute. In other words, the model must be a hypothesis of the organization’s overall
performance and not just of those aspects of performance that one party believes led to the
dispute. There are two main reasons for this: first, the parties to the dispute will rarely agree on
the existence or the causes of damages, and any hypothesis framed narrowly around the “theory
of the case” offered by one party will necessarily lack credibility as well as the dynamic breadth
necessary to evaluate competing theories offered by the other party; second, relying solely or
even primarily on human judgment (even that of experienced System Dynamics practitioners) to
determine which feedback loops played a part in damages experienced by a complex
organization would be quite inconsistent with the scientific method. Until we have properly
carried out the dynamic modeling and analysis process we don’t know which feedback loops
played a part — we may think we know but we can’t be sure. Through dynamic analysis we can
determine which feedback loops caused the damages that are the subject of the dispute, as long
as the model includes those loops. Since at the start we cannot be certain which loops resulted in
the damages, we have a chicken-and-egg problem that can be resolved by starting with a
hypothesis or model that embraces as many of the potentially important loops as possible for the
subject organization(s). Finding which loops resulted in the damages confirms that the
hypothesis/model has explanatory power relative to the case.

Hypotheses_must_be logically consistent. At the macro level a good System Dynamics
hypothesis/model will, by definition, be logically consistent — it will clearly state the dynamic
logic that is supposed to have driven the performance of the organization(s) involved. A good
System Dynamics model will be logically consistent at the micro level as well — for example, it
will not (without good reason) employ different dynamic structures to simulate similar
operations and relationships in different parts of the organization(s) being simulated. Nor will it
(again, without good reason) include different strengths or timings for similar cause-effect
relationships in different parts of the organization(s) being simulated. It will not (without good
reason) contain table functions defining non-linear relationships that include discontinuities. The

presence of such logical inconsistencies would be grounds for questioning whether the expert’s
testimony is truly founded on the scientific method.

Hypotheses must be falsifiable. A hypothesis that cannot be tested cannot be falsified and hence
cannot be subject to the scientific method. A falsifiable hypothesis that fails tests having the
potential to falsify it has been rejected — obviously such a hypothesis would be an unreliable
basis for proving and quantifying damages in the legal arena. When a hypothesis can no longer
be proven false based on the available information (and when the error rate associated with the
hypothesis is small — more on that shortly) that hypothesis can be considered a reliable basis for
expert testimony. The iterative process of testing, falsification, and improvement of the
hypothesis is described in more detail in (Lyneis 1999, Graham 2002, Ariza and Graham 2002).

Like the hypotheses themselves, hypothesis testing takes place on different levels when
employing System Dynamics in the legal arena. Under the scientific method, one object of
testing is to uncover flaws in the hypothesis so that a more reliable hypothesis can be developed.
The System Dynamics model that constitutes the higher-level hypothesis is generally developed
in steps beginning with diagramming of feedback loops, which leads to writing of equations that
quantitatively characterize those loops, followed by numerical simulation and testing of those
equations against known real-world performance. This sequence of steps is usually repeated
numerous times during the development and refinement of the hypothesis/model, and the
resulting iterative process does not differ from the formation, testing and refinement of
hypotheses that is the foundation of the scientific method.

At the ‘whole-organization’ level the most visible hypothesis test is the ability of the System
Dynamics model to re-create the known history of the subject organization within acceptable
standards of fidelity — a model or hypothesis that cannot re-create organizational history has been
rejected and is unlikely to survive a Daubert challenge. Organizational history includes both
‘hard’ data on organizational performance (data-based records of labor-hours expended,
quantities bought/produced/sold, market share, pricing, etc.) and ‘soft’ information or first-hand
knowledge of direct participants regarding the effects that various elements had on different
aspects of performance. At the macro level the System Dynamics model/hypothesis must re-
create the ‘hard’ data, and at the micro level it must do so for the right reasons — that is,
consistent with available first-hand knowledge regarding the relationships that affected
performance. Other high-level hypothesis or model falsification tests include assessing the
robustness of simulated performance under extreme conditions different from those encountered
historically, and comparing the model against models (hypotheses) for similar organizations
when such models are available.

The more focused second-level hypothesis as to ‘what happened’ must be tested for plausibility
in its own right and often against alternative theories of the case. The hypothesis that bases the
claimant’s plea must be explainable in terms of a sequential chain of quantified causes and
effects that is demonstrably consistent with known organizational history, and each causal link in
that chain must be understood by and make sense to the factfinder(s) who will decide the case.
This level of plausibility is always necessary but may not be sufficient to obtain a fair award for
damages, because the opposition usually offers alternative ‘what happened’ theories that, if
accepted as more credible by the factfinder(s), would reduce any damage award and might even

reverse it. These alternative theories usually appear plausible because they are tied qualitatively
and sometimes even quantitatively (by means of some static, open-loop model or analysis) to the
known history of the subject organization. This is a strong reason for conducting System
Dynamics hypothesis/model forming and testing work at the ‘whole-organization’ level, because
such a model can be employed to conduct falsification tests of the opposition’s alternative
theories of the case. If those alternative theories are incorrect then System Dynamics analysis
should demonstrate that they are inconsistent with known organizational history or with the
patterns of behavior and performance characteristic of such organizations, and show how they
are so. Assuming that the opposition does not succeed in falsifying the System Dynamics model
itself, then the opposition’s theories and the analyses/models supporting them will both have
been rejected. When falsification efforts have ended in the legal arena, the last (presumably
dynamic) hypothesis standing is likely to be the basis on which the award for damages is
determined.

2) The hypothesis must have been tested to determine the known or potential error rate.

At both the ‘whole organization’ and ‘what happened’ levels, hypotheses involve potential
errors. At the higher level the model-as-hypothesis reflects uncertainties regarding the nature,
strength and timing of the feedback loops driving organizational performance — in a strong
modeling effort these uncertainties will be small relative to the analytical issues to which the
model will be applied. Lower-level hypotheses regarding ‘what happened’ will reflect these
higher-level uncertainties in a “band of uncertainty” around the damages quantified via analysis.
For factfinders charged with determining liability and awards for damages in the legal arena, the
error rate is relevant to the extent that it affects the proof and/or the quantification of those
damages. The ideal is a narrow band of uncertainty around the most likely value of the damages
— such an outcome helps to verify the proof that damages occurred and enables factfinders to be
confident in the magnitude of the award for damages.

When System Dynamics models are used as the basis for expert testimony, error rates must be
measured for both the ‘whole organization’ and ‘what happened’ hypotheses. Furthermore, as
explained below, error rates at those different levels are linked in an important way. For a
System Dynamics model, the error rate stems from the fact that neither the model nor the
available data and other information on which it based can ever be perfect. As a result, the
pertinent question is whether the error rate is sufficiently small given the purpose for which the
model has been developed. At the ‘whole organization’ level the relevant error rate is the
statistical measure of the fidelity with which the System Dynamics model re-creates the ‘hard’
data elements of known organizational history (Sterman 1999, Lyneis and Reichelt 1996, Lyneis
Reichelt and Bespolka 1996). At the ‘what happened’ level the relevant error rate is the
likelihood that uncertainties regarding the organizational data and the System Dynamics model
and analyses could result in a significantly different quantification of damages. In other words,
at this level the error rate can be expressed as the width of the band of uncertainty around the
quantified damages resulting from modeling uncertainties.

Best practice in quantifying error rates associated with System Dynamics analyses is thus based
on four elements:

e A database containing the best available data on the historical performance of the
organization(s) involved in the dispute;

e First-hand knowledge regarding the nature of the cause-effect relationships that
drove organizational performance;

e Appropriate statistical fit measures for the degree of fidelity with which the model
independently re-creates known organizational performance, and standards for
applying those measures. It is important to match the particular error statistic
employed to the nature of the organization being simulated, otherwise unreliable
and possibly misleading results will be obtained (Lyneis, Reichelt and Bespolka
1996);

e Fit-Constrained Monte Carlo multi-simulation testing to determine how sensitive
the quantified damages are to uncertainties in the strength and timing of the
simulated feedback loops.

In Fit-Constrained Monte Carlo analysis (Graham, Choi and Mullen 2002, Graham, Moore and
Choi 2002) model parameters characterizing the strength and timing of organizational feedback
loops are randomly varied (each within its own band of uncertainty) and thousands of
simulations are then conducted based on those variations. Each simulation is conducted with a
unique combination of parameters, and each is compared against the known performance history
of the organization. Most of the simulations will not re-create organizational history within
acceptable ‘whole organization’ error rates and will be discarded as demonstrably
unrepresentative of the subject organization (this ‘fit constraint’ is how error rates at the ‘whole
organization’ and the ‘what happened’ levels are linked). The remaining simulations adequately
re-create organizational history, and each one constitutes an alternative ‘whole organization’
hypothesis or model from which an alternative quantification of damages can be obtained.
Taken as a whole these alternative damage amounts constitute a sampling that reveals the
probability that the ‘real’ magnitude of the damages differs by various amounts from the most
likely magnitude quantified by the expert witness.

This approach makes it possible to express results from a System Dynamics analysis in terms of
ranges of confidence rather than one-point answers. Such a range of confidence expresses the
error rate resulting from uncertainties about the data and hypothesis-model in a form that is
particularly useful for the factfinder(s), who can be most confident in the expert’s testimony
when the alternative damage amounts are tightly clustered and when they are significantly
different from the damage amounts resulting from the opponent’s alternative ‘what happened’
hypotheses. PA employs such testing of System Dynamics analyses, which can be used to
support the following sorts of assertions regarding analysis results and error rates.

e “Simulation analysis shows that the Owner’s extra-contractual acts and omissions
are most likely to have caused 70% of the cost and schedule overruns
experienced. Because of residual uncertainties regarding the model, the 95%
confidence interval for the effects of Owner’s acts and omissions is between 66%
and 75% of the cost and schedule overruns experienced on the project. In other
words there is less than one chance in twenty that the Owner’s acts and omissions

caused less than 66% or more than 75% of the project cost and schedule
overruns.”

e “In the face of the Contractor’s extra-contractual acts and omissions, and given
residual uncertainties regarding the data and model, there was only one chance in
about 700 that the project could have been completed on the early date suggested
by the Contractor.”

This approach provides information on “the known or potential error rate” in a form that will be
particularly useful to the factfinder(s) and to the decisions they must reach. It does so in a
manner that would be difficult to match using static, open-loop analysis tools and methods. On
this count, opponents in the legal arena should find it difficult to demonstrate that expert
testimony based on good System Dynamics practice ought to be ruled inadmissible.

3) Hypotheses must have been formed and tested and analyses conducted in accordance
with standards appropriate to the techniques employed, including:

a) Standards regarding consistency with accepted theories;
b) Standards regarding the scope of testing;

c) Standards regarding precision;

d) Standards regarding post-hypothesis testing.

e) Adhering to the same standards as in non-legal applications.

Standards regarding consistency with accepted theories. Good System Dynamics modeling
practice follows principles laid down in works by prominent practitioners including Forrester
(1961) and Sterman (2000). These principles are solidly based on feedback control theory and
the “bounded rationality” theory of management decision-making (Forrester 1961, Cyert and
March 1963, Simon 1979, Morecroft 1985). Thus, at the highest conceptual level System
Dynamics is consistent with widely accepted theories.

At the level of organizational causality and dynamic structure, most System Dynamics models
are assemblages of semi-generic feedback structures previously identified, written about and
employed in simulating many different kinds of organizations. These structures constitute
widely accepted theories of causality and performance for various organizational components,
and extensive previous use in other settings has led to widespread understanding of their
behavior and generated considerable empirical evidence of use without falsification. As an
example, System Dynamics models of complex development projects often draw on well
established theory regarding the processes that drive project performance, as discussed in
writings by a variety of practitioners (Abdel-Hamid and Madnick 1991, Cooper 1980, Cooper
1993, Cooper and Mullen 1993, Ford 1995, Rodriguez and Bowers 1995, Rodriguez and
Williams 1997, 1998). A System Dynamics model containing such structures is, by definition,
consistent with accepted theories.

At the level of individual equations, good System Dynamics modeling practice reflects widely
accepted micro-level theories regarding non-linear cause-effect relationships and standard

representational formulations as discussed in writings by noted practitioners such as Forrester
1961, Graham and Alfeld 1976, Coyle 1977, Richardson and Pugh 1981, Coyle 1996 and
Sterman 2000, to name a few. Such theories are exemplified by the rule that a mathematical
look-up table characterizing the causal relationship between two system elements should not
contain “kinks” or significant first- or second-derivative discontinuities.

Standards regarding the scope of testing. The nature and scope of hypothesis/model testing must
be based on the purpose for which that model is built (Forrester and Senge 1980). There are
many different types of tests that can be considered standard for System Dynamics models,
depending on the nature of the application. The adversarial nature of the legal process naturally
demands the most rigorous testing and documentation. Discussion of the full range of tests
required in legal applications of System Dynamics is beyond the scope of this paper, but that
range is illustrated by the following list of tests performed by PA and by an independent expert
in conjunction with a recent case:

Structural logic testing

Causal feedback testing against independent experts’ views
Equation-level dimensional consistency testing

Parameter sensitivity testing

Behavioral sensitivity testing under ‘normal’ conditions
Behavioral robustness testing under extreme conditions
Historical fidelity testing against ‘hard’ data

Historical fidelity testing against first-hand organizational knowledge
Table-function excursion testing

Confidence testing of results given parametric uncertainties
Testing of alternative causal theories

Many of these include test-specific standards, some of them quantitative. All of these tests are
applicable to and have often been employed in conjunction with applications of System
Dynamics outside the legal arena.

Standards regarding precision. The early applications of System Dynamics were primarily
academic and practitioners placed little emphasis on the precision of results. With increasing
commercial use precision and reliability of results gained steadily in importance, and new
techniques of model calibration, validation and testing were developed to meet more stringent
demands. It is becoming more common to measure error rates, and standards of precision have
gradually emerged (Lyneis and Reichelt 1996).

In the legal arena standards of precision are relevant to hypotheses at both the ‘whole
organization’ and the ‘what happened’ level. At the ‘whole organization’ level the primary
standards are those pertaining to the fidelity with which the model is able to independently re-
create known organizational history. The appropriate numerical standards of precision depend
significantly on the nature of the organization being simulated, the characteristics of the data
concerning that organization, and the type of model variables involved (Lyneis, Reichelt and
Bespolka 1996). PA has developed statistically-based standards of fidelity for models of project
organizations (the type of organization involved in most of the legal disputes in which System

Dynamics has been employed), and the application of these standards has improved the model
calibration process and resulted in more reliable models.

Fidelity standards at the ‘whole organization’ level also make it possible to assess the precision
of damage quantification and other analytical results at the ‘what happened’ level, by providing a
foundation for filtering out non-applicable alternative hypothesis-models based on Fit-
Constrained Monte Carlo analyses (Graham, Choi and Mullen 2002, Graham, Moore and Choi
2002). This enables the expert witness to be more precise in his or her statements regarding the
case. For example, instead of testifying that “Under the specified conditions the project would
have been completed around the third week of April”, he or she can testify that “Under the
specified conditions there was a 60% probability of project completion between 12 April and 25
April and a 95% probability of completion between 5 April and 5 May.”

Standards regarding post-hypothesis testing. Post-hypothesis testing is both the raison d'etre
and one of the great strengths of System Dynamics modeling in the legal arena: the model exists
so that the expert witness can demonstrate what organizational performance would have resulted
given different conditions or different behavior by the parties. The different conditions and/or
behavior that are the subject of the legal dispute necessarily require analysis of organizational
performance outside the range of performance that actually occurred. For confidence in analyses
it is necessary to confirm that the simulated organizational dynamics are robust under conditions
different from those observed historically. That is the purpose of the Behavior Sensitivity
Testing and Behavior Robustness Testing listed earlier in this paper, and such testing is standard
in PA’s legal work.

Adhering to the same standards as in non-legal applications. Daubert specifies that the expert’s
testimony must be based on at least the same level of intellectual rigor that characterizes expert
practice when dealing with similar matters outside the courtroom. The majority of PA’s System
Dynamics applications, and, we believe, the majority of all Systems Dynamics applications
conducted to date, have involved matters outside the courtroom. In such work PA employs the
same methods and computer simulation technology that we use in applications on legal cases.
The standards described above apply to and have been used in applications of System Dynamics
outside the legal arena. Where standards differ between legal and non-legal applications, the
difference is one of degree — the standards in legal cases are the most rigorous that we employ in
any application. Applications outside the legal arena are not usually subjected to adversarial
scrutiny and attack, hence the extensive (and expensive) use of all of the most stringent standards
of testing and documentation is rarely justified in such applications. Non-legal applications
usually benefit from the fact that the simulation model structures employed have previously been
tested to the highest standards, and many tests do not need to be repeated for each subsequent
application.

4) The techniques employed should have been peer reviewed.

a) Initial peer review usually occurs in the run-up to publication in peer-reviewed
journals.

b) The techniques employed should also have withstood the broader peer review to
which publication exposed them.

Daubert established peer-reviewed publication as an important means of verifying the scientific
validity of the methods employed as the basis for expert testimony. For the more respected
journals in the scientific and technical communities, publication is the purpose of peer review
and passing peer review is required for publication. Daubert makes three important
presumptions regarding peer-reviewed publication:

e Publication in peer-reviewed journals indicates that the expert’s scientific peers have
sanctioned his or her work and methods as credible;

e Publication exposes the work and methods to further review by the relevant scientific
community, and scientists may show their approval by citing the work as authoritative or by
extending the work;

e Withstanding the scrutiny of the relevant scientific community indicates general acceptance
in that community.

With respect to the “relevant scientific community”, the Court said the inquiry should focus on
“the non-judicial uses to which the scientific techniques are put” -- so the relevant community
consists of those real-world scientists who pursue science for non-litigation purposes. To date all
of the legal applications of System Dynamics of which we are aware have involved the operation
of managed business organizations (rather than biological systems or national economies, for
example). With respect to System Dynamics analysis of managed business organizations there
are two relevant scientific communities, one being a subset of the other. The larger is the
community of those who study the practice of management, which includes practitioners of a
wide range of analysis techniques in academia, large corporations and consultancies. The
community of System Dynamics practitioners analyzing business performance is a subset of this
larger community, and practitioners are likewise found both in academia and, increasingly, in
corporations and consultancies. Since most scientific management work (including System
Dynamics work) is for purposes other than litigation, there is room to judge scientists’
acceptance of System Dynamics business analysis techniques based on peer-reviewed journals
publishing articles on non-judicial applications of those techniques.

There have been many peer-reviewed articles describing analyses employing System Dynamics,
nearly all of them about non-judicial applications. The following are among the journals
publishing such articles, indicating acceptance of the methodology by the management science
community:

e Administrative Science Quarterly e Journal of Economic Behavior and
e American Journal of Physiology Organization
e Behavioral Science e Journal of the Operational Research
« Energy Systems and Policy Society
e Engineering Management Journal of the e Management Science

IEEE ¢ Organizational Behavior and Human
e European Journal of Operational Decision Processes

Research e Project Management Journal
e European Management Journal e Science
e Harvard Business Review e Sloan Management Review
e Interfaces e Strategic Management Journal
e International Journal of Energy Systems e System Dynamics Review
e International Journal of Forecasting e Technological Forecasting and Social
e Journal of the American Statistical Change

Association e TIMS Studies in the Management

Sciences

Among these articles relating to the practice of System Dynamics is a significant subset of peer-
reviewed articles and papers on the application of System Dynamics to analysis of performance
on complex development projects, including the following authors and journals. These articles
are by System Dynamics practitioners and indicate acceptance of System Dynamics for
analyzing project performance among that segment of the management science community.
Since most of the journals reviewing and publishing the articles are devoted to broader topics of
management investigation and application rather than the narrower System Dynamics
methodology, these articles and papers also demonstrate that the broader management science
community accepts System Dynamics for analyzing performance on complex development
projects.

e Akerman, Eden & Williams, Interfaces e¢ Graham, Project Management Journal

e Alfeld, Wilkins & Pilliod, presented at e Howick & Eden, Journal of the
SNAME’s Ship Production Symposium Operational Research Society

e Eden, Williams, Akerman & Howick, e Rodrigues & Williams, Journal of the
Journal of the Operational Research Operational Research Society
Society ¢ Williams, Eden, Akerman & Tait,

e Ford & Sterman, System Dynamics Journal of the Operational Research
Review Society

Kenneth Cooper’s article in /nterfaces illustrates how System Dynamics analysis of complex
projects has been broadly accepted by the management science community. He wrote the article
about PA’s first-ever complex-project application of System Dynamics at one of the leading US
shipyards in the late 1970s. The article was submitted in competition for the Edelman prize,
probably the most prestigious annual contest for management science work in the US. That

contest is conducted by INFORMS, the Institute for Operations Research and the Management
Sciences, one of the leading US professional societies in the field. Following extensive review
by a jury of peers representing a wide range of management science disciplines, the article was
awarded the second-place prize and subsequently published in the Institute’s journal Interfaces.
Since then, that and other peer-reviewed articles by PA consultants have been cited numerous
times in articles by other management scientists. Given the abundance of publication-based
evidence, it is difficult to imagine a successful Daubert admissibility challenge based on peer-
review issues.

5) The techniques employed should have been generally accepted in the scientific
community.

In addition to the strong evidence that such peer-reviewed publications provide, evidence that
System Dynamics is generally accepted in the scientific community can be found in the large
number of colleges and universities that offer regular undergraduate and graduate-level
instruction in System Dynamics. There are at least 80 institutions of higher learning offering
such courses around the world, as listed in the Appendix.

4. Preparing and Delivering Expert Testimony Based on System Dynamics

A detailed discussion of the process of preparing and delivering expert testimony based on
System Dynamics is beyond the scope of this paper. Therefore, what follows is a short summary
of key tasks involved in providing expert testimony based on System Dynamics analysis and
how those tasks relate to possible Daubert admissibility challenges.

Preparation of System Dynamics analyses. The process of model-building and simulation
analysis is well documented and is one with which most System Dynamics practitioners are very
familiar. As has been described earlier in this paper, that process should not change when the
analyses are for application in the legal arena. In fact, due to the deliberately adversarial nature
of most court systems, applications of System Dynamics in the legal arena should be carried out
with the most scrupulous care and use of best practices. This will do much to ensure the
admissibility of resulting expert testimony and its subsequent effectiveness before the
factfinder(s).

Guidance for settlement discussions. As observed in the Introduction to this paper, nearly all of
the legal cases in which PA has applied System Dynamics have been resolved and all of those
have been resolved by means of negotiated settlement outside the formal legal process. This
seems to exemplify the general rule that litigation for recovery of damages usually leads to some
form of out-of-court settlement. Sometimes, System Dynamics work intended to support expert
testimony finds an earlier application in support of negotiations aimed at reaching such a
settlement. Although this might appear to be a very different sort of application, it must be
remembered that in formal legal proceedings the expert witness is there to assist the factfinder(s)
in arriving at the facts in the case, and not to be an advocate for one party or the other. A
negotiation support role is likely to be most effective if it is carried out from a similarly non-

advocative position. It should be noted that what the opposition learns during negotiations about
the expert witness and his or her employment of System Dynamics may provide the grounds for
a subsequent Daubert admissibility challenge, should negotiations fail to yield a settlement.

The evidentiary discovery process. In advance of trial most cases involve a formal discovery
process during which each party is required to furnish the other with the evidence (including
expert testimony) it will employ in court to make its case. In PA’s experience the discovery
process usually requires that the modeler supply the opposing party with the simulation model(s)
employed, the data and other information employed in developing the model(s) and analyses, the
simulation software required to run the model(s), any technical information and possibly
technical support needed to run the model(s), any report(s) prepared on the analysis work, and
the analyses themselves. The opposition can employ this body of information in a variety of
ways including giving it to their own experts to assess its quality and reliability, and this may
require some form of protective order covering the opposing experts’ access to and use of
proprietary information and/or intellectual property. Obviously weaknesses or inconsistencies in
the documentary materials provided to the opposition may provide grounds for a Daubert
challenge and are likely to reduce the expert’s effectiveness in court.

The deposition process. In advance of trial most courts permit each party to question or depose
the other’s witnesses (including expert witnesses) to determine their qualifications to testify and
the basis for and nature of their testimony. An opposing attorney conducts the deposition with a
friendly attorney in attendance to ensure that the rules of the deposition process are followed.
The opposing attorney may be accompanied by an opposing expert who will assist in formulating
deposition questions. As may be imagined, depositions allows each party to judge the strengths
and weaknesses of the opposing witnesses’ testimony in advance of trial, and to plan how to
cross-examine those witnesses in court. The deposition is conducted under oath and a complete
transcription is made of the proceedings. Portions of the transcript may be read back in court as
a cross-check on testimony given. If sufficient weaknesses are revealed during deposition, the
usefulness of the witness and his or her testimony may be so completely undermined that the
party employing the witness will not wish to risk having him or her testify in court. When
preparing for deposition and while being deposed, it is useful to remember that the opposition is
seeking to find or to create the appearance of weaknesses and inconsistencies in the expert’s
testimony and in the foundations of that testimony. Any such weaknesses (whether real or
perceived) may provide grounds for a Daubert challenge and are likely to reduce the expert’s
testamentary effectiveness in court.

Preparation and delivery of expert testimony. In the absence of an out-of-court settlement, and
assuming that the expert witness has survived the deposition process and any Daubert challenge,
the final step is likely to be the preparation and delivery of expert testimony. The difficulty of
these tasks should not be underestimated because the objective is to make dynamically complex
facts understandable to the factfinder(s) who will know little or nothing about organizational
dynamics or dynamic simulation. This usually requires considerable education of the
factfinder(s) in a short period of time, all without boring them or confusing them with
technicalities. This challenge may well be more significant than those of the modeling and
analysis process, and poor testamentary materials or bad delivery of testimony can quickly
destroy the effectiveness of even the best analysis. In creating testamentary materials and

preparing to use them in giving evidence, there is no substitute for repeated rounds of revision
and practice.

5. Handling the Ethical Challenge Inherent in the Expert Witness Role

The Ethical Challenge. In legal cases that revolve around the performance of complex
organizations, the factfinder(s) often need assistance from expert witnesses to understand the
issues and the facts involved in the dispute. An expert witness is there to employ his or her
knowledge and analysis in an impartial manner to assist the factfinder(s). The expert is not there
to act as an advocate or to argue for either party to the dispute. But experts are provided and
paid by the disputing parties, not by the factfinder(s). Naturally, each party will select experts
whose testimony is expected to be favorable to their case. The tension between the expert’s
ostensibly impartial role and his or her employment by a party to the dispute creates a potential
ethical challenge that must be carefully addressed. That challenge forces the factfinder(s) to
guard against partial, biased, distorted or incomplete testimony aimed at producing a particular
legal outcome, and that is the aim of the Supreme Court’s Daubert admissibility standards. That
inherent ethical challenge also suggests that an expert witness should be unusually scrupulous
about his or her work and testimony.

Because of the ethical tension inherent in the role, expert witnesses from a wide range of
scientific and technical disciplines tend to be viewed by the cynical as credentialed charlatans
who will support any position for money. It is a sad fact that the behavior of some (perhaps
many) expert witnesses fuels that reputation. With lots of money being paid to experts and lots
more money riding on the outcome of legal disputes, it is not surprising that some clients and
their attorneys (and even some expert witnesses) feel that the expert should provide whatever
testimony is needed as long as it’s not too obviously wrong to help win the case.

Since PA’s consultants are regularly employed as expert witnesses, we have adopted the
following policies to minimize the ethical tension that would otherwise prevail. We inform the
prospective client of these policies before we are hired:

e Before accepting the assignment PA will learn enough about the case to determine
that the prospective party is probably the party that has been most wronged.
Good System Dynamics analysis will show quite clearly which party bears the
primary responsibility for the damages done, and PA is much less likely to be
pressured to change our analyses if we’re not working for that party;

e PA’s consultants will work to the highest professional standards and will not be
influenced by what the client and the client’s attorneys might regard as a
“desirable analytical outcome”;

e As results from PA’s work become available, no amount of pressure will induce
us to change our analyses to better support a desired legal outcome, and PA will
resign the assignment in the face of any such pressure. PA’s responsibility is to
answer to the best of our ability the questions our client asks us to address, and
those answers are valuable to the client whether or not they support his desired

outcome. PA makes no assurances that our results or testimony will support the
desired outcome, and it is up to the client to decide whether to tender us as
experts;

e Once hired, PA will review and revise our work only in the normal, iterative
course of refining (for increased precision and confidence) the data, the
simulation model, and analyses conducted with that model. We will employ what
we judge to be the best available data and other information, and PA’s modeling
and analysis work will be consistent with that data and information and with our
experience and database from similar past assignments. In reviewing and revising
our work we will not be influenced by what the client or the client’s attorneys
might regard as a “desirable analytical outcome”.

e Ifasked to, PA will provide expert testimony on what our analysis has revealed
about the facts of the case. We will make no assertions that we cannot back up
based on our own work and knowledge. We will not speculate unless asked to do
so in an area where our experience and work reasonably support such speculation.

PA consultants have followed these policies in all of our assignments involving legal
disputes. That such policies are effective is evidenced by the fact that in dozens of
assignments in the legal arena we have never been pressured by a client to alter our
findings or testimony to better support their case.

The Difficult Role of the Opposing Expert. Lawyers generally have come to expect that
different experts, even when working from the same set of “facts”, can conduct different
analyses, reach significantly different opinions and provide very different testimony. So when
one party to a dispute employs an expert to provide important testimony, it is common for the
other party to employ an expert to conduct analyses and provide testimony in opposition. PA has
encountered opposing experts in many of the legal disputes with which we have been involved.

When the testimony to be rebutted is based on a good System Dynamics analysis, the opposing
expert has a tough row to hoe. Because System Dynamics is less well known than some other
analytical methods commonly employed in the legal arena, the opposing expert may not be at all
familiar with System Dynamics. This can result in naive or mistaken pronouncements about the
System Dynamics analyses and the testimony based on them, but such assertions are usually easy
to counter. When the opposing expert’s more glaring mistakes are highlighted, his or her
credibility is often irreparably damaged.

One cannot blame the resulting testimonial ineffectiveness on the opposing expert alone, because
the opposing party and attorneys have usually left the work of evaluating and countering the
System Dynamics analysis until shortly before expert reports must be delivered to the
factfinder(s). This is a common failing by opposing parties and attorneys who are unfamiliar
with System Dynamics, and it is not surprising that an expert who is also unfamiliar with it
would be unable to offer a meaningful critique with just a few weeks to prepare. In most cases
such opposing experts do not even run the simulation model before they comment on it and the
analyses conducted with it. What is surprising is that, even with little time and few if any test

analyses, many opposing experts are still willing to criticize System Dynamics work. Such
criticisms are not likely to withstand a Daubert challenge.

In opposition it can be difficult even for an experienced System Dynamics practitioner to offer
effective counter-testimony against a thorough-going System Dynamics analysis. He or she will
usually be hired late in the game, and further valuable time will be lost before the opposing
attorneys remember to pass along the model and related data files for evaluation. A quick review
of a thorough-going System Dynamics analysis is unlikely to uncover any serious weaknesses,
no matter how much the opposing party is hoping to find such a “smoking gun”. It is impossible
to thoroughly assess in just a few person-days a model and analyses that are the fruits of several
person-years of development and refinement effort.

We have often seen an experienced System Dynamics practitioner placed in a difficult opposing
position by their client’s failure to appreciate what is required for good expert testimony and
what can be expected from it. Under such conditions a careful System Dynamicist will want to
set and manage client expectations regarding what can and cannot be accomplished in the time
available. When those expectations are not properly set from the beginning, the opposing expert
may come under heavy pressure to “find something wrong” with other party’s System Dynamics
analysis and to do so in a hurry. Such pressure may explain some of the surprising testimony
System Dynamics practitioners have offered in opposition to PA’s work. Three typical real-life
examples follow, drawn from the written testimony of experienced System Dynamicists in the
role of opposing experts.

e “System Dynamics is an inappropriate tool for modeling complex development
projects and will give misleading answers when so applied.”

e “[PA] used an automated model tuner in calibrating the model to artificially
maximize the size of their client’s claim for damages.”

e “This formulation serves no useful purpose and appears to exist only to inflate the
size of the damage claim.”

It is interesting to consider how such testimony is likely to fare under the Daubert standards.
The first assertion is a hypothesis regarding the suitability of System Dynamics for a particular
analytical purpose. To be admissible the hypothesis (“System Dynamics is an inappropriate
tool...”) must have been tested and error rates evaluated. Given the fact that PA and other
practitioners have successfully employed System Dynamics in modeling well over 100 complex
development projects, and done so in compliance with Daubert standards even before those were
articulated, the hypothesis is clearly rejected. Even if that were not so, it is difficult to imagine
how an opposing System Dynamics expert could go about testing such a hypothesis, and the
absence of testing reduces it to the level of an unsupported assertion. On either basis, the
proffered testimony leaves the expert vulnerable to a Daubert challenge. Even if the testimony is
tuled admissible, an unsupportable statement of this sort will greatly damage the credibility of
the opposing expert when wielded by a capable cross-examining attorney.

The second assertion, it should be noted, is mistaken on two counts. First, PA is unaware of the

existence of tuning software with the practical capability to automate the calibration of a large
System Dynamics model. If we had employed such a tuner in the case in question, we would

have furnished that tuner to the opposing expert along with our simulation model, as required by
the factfinder(s) in that and most legal cases. Second, as had been explained in detail in our
written testimony in that case, PA’s calibration of that model was carried out against the sole
benchmark of the simulated organization’s historical performance — the size of the client’s
damage claim played no part at all in the tuning process. What could lead the opposing System
Dynamics expert to make such an assertion despite his awareness of PA’s written statement to
the contrary? When asked that question following the conclusion of the case, he replied “Well, I
had heard that [PA] had a tuner.” Offering a hypothesis (“PA tuned to maximize the claim
value...”) without testing or supporting evidence exposes the opposing expert to a Daubert
admissibility challenge before he or she reaches the witness stand, and to a painful cross-
examination experience if their testimony is ruled admissible.

The third assertion is typical of many equation-level criticisms PA has received at the hands of
experienced System Dynamicists serving as opposing experts. Such criticisms are frequently
and appropriately accompanied by disclaimers and qualifiers, usually (as in this particular case)
because the opposing expert has not had time to conduct simulation tests of the model PA
provided. The qualifiers are words like “appears” or “seems” or “is likely to”, which soften the
assertion in the absence of actual testing. These opposing experts have been put in a difficult
position, usually by the failure of their client’s attorneys to allow adequate time for a thorough
evaluation of the work about which the expert is to testify. Without thorough dynamic analysis,
statements about the effect of a particular formulation on the magnitude of computed damages
remain untested hypotheses that are unlikely to withstand a Daubert challenge (especially since,
in our experience, those hypotheses are usually factually incorrect).

Based on considerable experience with legal applications of System Dynamics, we believe that
any practitioner engaged to provide expert analysis and testimony should pay close attention to
the Supreme Court’s new admissibility standards. Specifically, practitioners should carefully
evaluate their work in the legal arena in light of those standards and modify both their own work
and their statements about the work of other practitioners where that is necessary to ensure
admissibility. Upcoming legal cases are likely to yield the first-ever Daubert admissibility
challenges to expert testimony based on System Dynamics, and those cases will set a precedent
that is likely to be referred to in subsequent cases. If expert testimony based on System
Dynamics is ruled inadmissible, few outside the System Dynamics community will stop to
consider whether it was the methodology itself or specific testimony that failed the admissibility
tests. Ideally, when System Dynamicists meet in court as opposing experts their discourse will
be on the highest professional level and their testimony will all be admissible. If they are not,
court rulings may effectively cast System Dynamics into legal disrepute as “junk science”,
painting practitioners as pseudo-experts who trash peers and their own methodology in the heat
of legal battle. We would have no one to blame but ourselves.

6. Conclusions

Expert testimony based on rigorous System Dynamics modeling work can meet the US Supreme
Court’s Daubert criteria for admissibility:

e Best-practice System Dynamics work relies on the processes of hypothesis formation and
testing inherent in the scientific method cited by Daubert as vital for evidentiary reliability
and admissibility;

e Best-practice System Dynamics work includes explicit error testing as required under
Daubert, in forms that are particularly useful to trial courts;

e Best-practice System Dynamics work employs rigorous standards for the formation and
testing of hypotheses;

e Best-practice System Dynamics work in the legal arena employs at least the same level of
intellectual and quantitative rigor typically involved in more numerous applications that do
not involve judicial proceedings, as required under Daubert;

e System Dynamics work by PA and other practitioners has successfully withstood extensive
peer review within the management science community, as required under Daubert;

e System Dynamics is taught in many leading universities around the world, indicating broad
acceptance in the management science community;

e As a result, best-practice System Dynamics work can satisfy the factors laid down by the
Court in Daubert for evaluating the scientific validity and resulting admissibility of expert
testimony;

The expert witness role involves tasks and processes that are unique to the legal arena and may
be unfamiliar to the System Dynamics practitioner. Some of these can provide opponents with
the basis for a Daubert challenge. A considerable investment of time and energy is usually
required to ensure that these tasks and processes do not undermine the effectiveness of solid
dynamic analysis work.

The expert witness role involves inherent ethical challenges that must be surmounted in order for
the expert to be confident of surviving a Daubert challenge. To effectively meet these challenges
we must acknowledge that some System Dynamics work in the legal arena falls short of best
practices and is therefore vulnerable to admissibility challenges under Daubert. This has been
true particularly of expert testimony offered by System Dynamicists in opposition to the work of
other System Dynamics practitioners. Upcoming cases are likely to set precedents for the
admissibility of expert testimony based on System Dynamics, and those precedents may prove
difficult to change. Practitioners should take pains to ensure that their work and testimony meet
the standards set in Daubert.

Appendix — Institutions of higher learning where System Dynamics is taught

(From a web search for schools offering at least one course in System Dynamics)

In the Americas:

American University, DC

Arizona State University

Boston University, MA

Central Connecticut State University
Dartmouth College, NH

Fairleigh Dickinson University, NJ
George Washington University, DC
Illinois State University

Instituto Tecnologico y de E. S. de
Occidente (ITESO), Mexico

Instituto Tecnologico de Sonora, Mexico
ITESM_ Tecnologico de Monterrey,
Mexico

Massachusetts Inst. of Technology
National Defense University, DC
Portland State University, OR

In Europe:

Aristoteles University of Thessaloniki,
Greece

Bogazici University, Turkey

Catedra UNESCO en la Universidad
Politecnica de Catalunya, Spain

Centro Universitario Studi Aziendali, Italy

CERAM and University of Paris Pantheon
ASSAS

City University Business School, UK
Copenhagen Business School, Denmark
Cranfield University, UK

Department d'Organitzacis d'Empreses, ,
Spain

Johannes Gutenberg Universitaet Mainz,
Germany

London Business School, UK

London School of Economics and Political
Science, UK

Universite Laval, Canada

University of Alaska Anchorage
University at Albany, State Univ. of
New York

University of Los Andes, Venezuela
University of California at Davis

University of Illinois at Urbana-
Champaign

Univerdidad Nacional de Columbia
University of Southern Maine

University of Vermont

University of Virginia - Darden

Graduate School of Business
United States Military Academy, NY
Washington State University
Worcester Polytechnic Institute, MA

Luigi Bocconi University, Italy
LUMES, Sweden

Lund University, Sweden

Masaryk University

Nymegen University, Holland
South Bank University, UK
Strathclyde University, UK
Sunderland University, UK
Technical University Delft, Netherlands
Technical University of Denmark
Telinges, Spain

Universidad de Valladolid, Spain
Universitaet Mannheim, Germany
Universitat Stuttgart, Germany
University of Belgrade, Yugoslavia
University of Bergen, Norway
University of Klagenfurt, Austria

e University of Palermo, Italy e University of Sevilla, Spain
e University of Plymouth, UK e University of Split, Croatia
e University of Salford, UK e Westminster Business School, UK

e University of St. Gallen, Switzerland

In Asia / Pacific

e Australian Defence Force Academy, University of New South Wales, Australia
e Bandung Institute of Technology, Indonesia

¢ Chuo University, Japan

e Deaking University, Australia

e De La Salle University, Manila, Philippines

e Fudan University, Shanghai, China

e Indian Institute of Management, Calcutta

e The Institute for System Science, Tokyo, Japan

References

Abdel-Hamid, T. and Stuart E. Madnick 1991. Software Project Dynamics: An Integrated
Approach. Englewood Cliffs, NJ: Prentice-Hall.

Ariza, Carlos A. and Alan K. Graham 2002. Quick and Rigorous, Strategic and Participative:
10 ways to improve on the expected tradeoffs. Proceedings of the 2002 International System
Dynamics Conference. Palermo, Italy. Forthcoming.

Berger, Margaret A. 2000. The Supreme Court’s Trilogy on the Admissibility of Expert
Testimony. In Reference Manual on Scientific Evidence, 2" ed. Federal Judicial Center.

Black, B., Ayala, F. J., and Saffran-Brinks C. 1994. Science and the Law in the Wake of
Daubert: A New Search for Scientific Knowledge. Texas Law Review, March 1994.

Coyle, R. Geoffrey 1977. Management System Dynamics. New York: John Wiley and Sons.
Coyle, R. Geoffrey 1996. System Dynamics Modelling: A Practical Approach. CRC Press.

Cyert, R. M. and March, J. G. 1963. A Behavioral Theory of the Firm. Englewood Cliffs, NJ:
Prentice-Hall .Cooper, Kenneth G 1980. Naval Ship Production: A Claim Settled and a
Framework Built. Interfaces 10(6), 20-36.

Cooper, Kenneth G. 1993. The Rework Cycle: Benchmarks for the Program Manager. Project
Management Journal, March 1993.

Cooper, Kenneth G. and Thomas W. Mullen 1993. Swords and Plowshares: The Rework Cycle
of Defense and Commercial Software Development Projects. American Programmer 6(5).

Ford, David N. 1995. The Dynamics of Project Management: An Investigation of Projects’
Process and Coordination on Performance. Cambridge, Mass.: Massachusetts Institute of
Technology PhD dissertation, Department of Civil and Environmental Engineering.

Forrester, Jay W. 1961. Industrial Dynamics. Waltham, Mass: Pegasus Communications.

Forrester, Jay W. and Peter M. Senge 1980. Tests for Building Confidence in System Dynamics
Models. TIMS Studies in the Management Sciences (North Holland) 14, 209-228.

Graham, Alan K. and Louis Alfeld 1976. Urban Dynamics. Waltham, Mass: Pegasus
Communications..

Graham, Alan K. 2002. On positioning system dynamics as an applied science of strategy.
Forthcoming. Proceedings, 2002 International System Dynamics Conference. Palermo, Italy.

Graham, Alan K., Carol Y. Choi and Thomas W. Mullen 2002. Using Fit-Constrained Monte
Carlo Trials to Quantify Confidence in Simulation Model Outcomes. Proceedings of the 2002
Hawaii Conference on Complex Systems.

Graham, Alan K., Jonathan Moore and Carol Y. Choi 2002. How robust are conclusions
from a complex calibrated model, really? A project management model benchmark

using fit-constrained Monte Carlo analysis. Proceedings of the 2002 International System
Dynamics Conference, Palermo Italy (forthcoming).

Lyneis, James M. 1999. System dynamics for business strategy: a phased approach. System
Dynamics Review 15(1): 37-70.

Lyneis, James M. and Kimberly Sklar Reichelt 1996. Calibration Standards. Cambridge, Mass.:
PA Consulting Group Internal Document. April 11, 1996.

Lyneis, James M., Kimberly Sklar Reichelt, and Carl G. Bespolka 1996. Calibration Statistics
for Life-Cycle Models. Proceedings of the 1996 System Dynamics Conference.

Mahle, S. 1999. The impact of Daubert v. Merrell Dow Pharmaceuticals, Inc., on expert
testimony: with applications to securities litigation. Florida Bar Journal March 1999, 73.

Mahle S. 2002. Daubert and the Law and Science of Expert Testimony in Business Litigation.
Retrieved December 21, 2001 from http://www.daubertexpert.com.

Morecroft, J. D. W. 1985. Rationality in the Analysis of Behavior Simulation Models.
Management Science, 31(7), 900-916.

Richardson, George, Alexander Pugh 1981. Introduction to System Dynamics Modeling.
Portland, OR: Productivity Press.

Rodrigues, A. and John Bowers 1995. System dynamics in project management: a comparative
analysis with traditional methods. System Dynamics Review 12(2), 121-139.

Rodrigues, A. and T. M. Williams 1997. System dynamics in software project management:
towards the development of a formal integrated framework. European Journal of Information
Systems 6, 51-66.

Rodrigues, A. and T. M. Williams1998. System dynamics in project management: assessing the
impacts of client behaviour on project performance. Journal of the Operational Research
Society 49(1), 2-15.

Simon H. A., “Rational Decision-Making in Business Organizations”, American Economic
Review 69(4), 1979.

Sterman, J. D. 2000. Business Dynamics: Systems Thinking for a Complex World.

Irwin/McGraw-Hill: New York.
Back to the Top

Metadata

Resource Type:: Document
Rights:: CC BY-NC-SA 4.0
Date Uploaded:: December 19, 2019

Using these materials

Access:: The archives are open to the public and anyone is welcome to visit and view the collections.

Collection restrictions:: Access to this collection is unrestricted unless otherwide denoted.
Collection terms of access:: https://creativecommons.org/licenses/by/4.0/

About the Archives

Access options

Ask an Archivist

Ask a question or schedule an individualized meeting to discuss archival materials and potential research needs.

Ask a Question Book a Meet

Schedule a Visit

Archival materials can be viewed in-person in our reading room. We recommend making an appointment to ensure materials are available when you arrive.

Schedule a Visit Hours

Collection

System Dynamic Society Records, 1978-2016

Collection ID	ua435
Total components	5492
Items available online	3565
Last indexed	2026-03-07

Schedule a Visit

Collections

Stephens, Craig A.; Graham, Alan K.; Lyneis, James M. "System Dynamics Modeling in the Legal Arena: Special Challenges of the Expert Witness Role", 2002 July 28-2002 August 1

Online content

Metadata

Using these materials

About the Archives

Access options

Ask an Archivist

Schedule a Visit

Collection

System Dynamic Society Records, 1978-2016

Collection contents