Ford, Andrew, "Lessons from Case 1454", 1981

LA-UR-81-2262 178

TITLE: LESSONS FROM CASE 1454

AUTHOR(S): Andrew Ford
Economics Group, MS-605
Los Alamos National Laboratory
Los Alamos, New Mexico 87545

SUBMITTED TO: 1981 System Dynamics Research Conference
The Institute on Man and Science, Rensselaerville, New York LESSONS FROM CASE 1454
October 14-17, 1981 ~~

Andrew Ford
Economics Group
Los Alamos National Laboratory

As a Los Alamos, New Mexico 87545

iN In the winter of 1974, the E17 Paso Electric Company was granted a

tefl Certificate of Convenience and Necessity for the construction of the Palo
EU Le . Verde Nuclear Generating Station near Phoenfx, Arizona. Construction began

ove in 1976, and in February, 1979, Case 1454 before the New Mexico Public Ser-

vice f-onfeston was initiated to consider the company's request to count

Palo Verde Construction Work in Progress in their rate base. The company
argued that its 600 MW share of Palo Verde was necessary if it was to satis-
fy demand at the lowest possible price to the consumer. Representing con-
sumers in the southern part of New Mexico, the energy unit of the Attorney
General's Office took the position that El Paso Electric Company was already
over-invested fn generating capacity and had no need for such a large share
of the nuclear station. The Attorney General's Office suggested that elec-
tricity rates would be lower 1f the company were to sel? part of its owner-
ship in ihe nuclear plant. During February and March, 1979, the Attorney

i Genera? assembled 8 case opposing the company's bid for higher rates. Part
‘Aig © 2 preprint of 8 oaper intended for publication in a

urnal or proceedings. Because changes may be made before
xolication, this preprint i made available with the under-
tending teat it will not be cited oF reproduced without the
veeristion of the author,

of the Attorney General's case rested on calculations from a System Dynamics
model of the electric utility industry. This paper reflects upon the hear-
ings in New Mexico with an eye toward the suitability of System Dynamics
models for such adversary proceedings.

The paper is organized fn three parts. It begins with a brief
review of the substantive exchange of views in the case, including the Com-
pany's position, the Attorney General's position, and the analyses and
counter ‘analyses presented in support of these positions. In Part Two, the
paper describes the participants and the schedule of the hearings. It is
argued that the rapid pace of the hearings and the background of the parti-
cipants are important determinants of usefulness of System Dynamics models
under adversary proceedings. The third part of the paper concludes with a
discussion of the advantages and disadvantages of system dynamics under fast
paced, adversary conditions.

ANALYSIS AND COUNTER ANALYSIS

Alternative Expansion Plans

Much of the analysis given in this case revolved around the relative
merits of the two expansion plans shown in Fig. 1. This figure shows the
capacity construction initfation rate for three units of the Palo Verde sta-
tion with scheduled completion dates of 1982, 1984, and 1986. Plan One, the
plan preferred by the Attorney General, called for the sale of the company
share of units #2 and #3 while Plan Two, favored by the company, called for
continued participation in 200 MW of all three of the Palo Verde units.

179

Palo Verde 41 P, N
(dean 1182) ba ®
(> sar ote rtente to
BU ieretiinteens
Pits Stne inituate

me dese Meltttesten an 2, Slenible
ects turiog this ther

%

Ve 7 mn o © e e2 Bs ty
Gander anal) ‘

Palo Vande #1 Pale Vethe® 2 Bile erdet 3:
(aed ia 192) et ame) (deem eeH)
-
jor Sen ble
Se M2

z hy
i a ee aS re a

Fig. 1. Capacity initiation rates for alternative plans under discussion in
Case 1454,
Notice that the capacity initiation rate in both plans fs assumed to be
flexible after 1982. Consequently, the plans may show differing capacity
initiations in later years depending on the growth in demand fn the E1 Paso
Electric Company service area.
Uncertainty in Demand Growth Projections
It seems that all hearings on the relative merits of alternative

utility expansion plans involve considerable discussion of at least two com-
peting forecasts of the likely growth in electricity demand. The partici-
pants favoring large construction programs present a forecast for rapid
growth in demand and usually arrive well prepared with data bases, econo-
metric modsls, and witnesses to support the forecast. The participants
arguing for a reduced construction program come prepared with their own
forecast, data, models, and witnesses. The inevitable clashing of the two
groups arguing on behalf of different econometric forecasts usually consumes
a good portion of the time and effort of all parties in the hearings.

In this case, the stage was set for such a clash. The company had
forecast that the peak demand for power would grow at 6.7%/year, and the
staff of the Public Utilities Commission in Texas has forecast a 3.9% annual
growth. Rather than jofn in the battle over competing forecasts, my analy-
sis was based on the premise that nefther side can predict the future with
certainty. This view was set out in direct testimony as follows.

Q. What is the underlying premise of your testimony?

A. My testimony is based on the premise that one cannot know
the future. At best, one can only make projections about
the future based on information available about the past.
Projections, of course, can turn out to be incorrect and
actions which are taken in reliance on those incorrect pro-
Jections can tnvolve significant costs or penalties. If an
electric utility planner's demand forecast is too high, for
example, the utility will likely be caught in later years
with an over-investment in base load capacity. On the other
hand, if the forecast 1s too low, the utility will not have
enough base Toad plants and will be forced to rely on other,
more expensive, sources of power. Both of these types of
forecasting errors may result in @ significant price penalty
to the rate payers or a significant profit penalty to the
utility.

Q. Can improved forecasting methods predict the future demand
for electricity with absolute certainty?

A. Ho, the development of more sophisticated forecasting
methods will not eliminate the underlying uncertainty that
accompanies the forecast of the future demand for electric
power. However, the key question which a utility and a util-
ity regulatory commission must ask {s not whether demand
will grow at exactly one rate or another. The important
question fs what steps should the utility take to protect
itself and its rate payers against the unavoidable risk of
forecasting errors while minimizing the cost to the utility
and its rate payers of providing this protection [1].

ation Model

The System Dynamics model used in Case 1454 fs an early version of
the EFPAM (Electric utility Policy and Planning Analysis Model) sequence of

1s¢

models. ‘Full technical documentation of the EPPAM model was made available
to the parties in Case 1454 through @ technical report of the Los Alamos
Sctentific Laboratory prepared for the Environmental Protection Agency and
the Department of Energy [2]. The participants in the case also had a short
paper that described the application of EPPAM to study the capacity expan-
sfon problems of a hypothetical, investor-owned electric utility company
[3]. This paper showed the planning advantages that small coal plants offer
(relative to large coal plants) because of thefr shorter constructfon lead
time. .
EPPAM is a relatively simple System Dynamics model designed to keep
track of the following aspects of the electric utility industry:
© growth in demand for electricity and possible change in the shape of
the load duration curve;
© operation of coal and turbine plants to satisfy the demand for power;
© expansion and retfrement of coal and turbine plants based on internal~
ly generated forecasts of the growth tn demand; -
© accounting of capital and operating costs and price regulation follow-
‘ing the practices of the state regulatory commissions; and
o the financing of construction programs through use of operating in-
come, deprectation expense, and external funding.
As with most System Dynamics models, the key assumptfons are those that
govern the rates. In EPPAM, two fmportant rates are the rate. of capacity
initiation for new coal plants and new turbine plants. Modified DYNAMO flow
dfagrams showing the set of assumptions governing these rates are given in
Figs. 2 and 3.
showing coal capacity initiation rate.

ont

Modified DYNAMO flow diagram

Fig. 2.

>
hy

juph
h

h

181

Fig. 3. Modified DYNAMO flow dfagram showing turbine capacity initiation rate.

eon
Probebsl 4 5. Ptolye
Kaho of Texas uth ness

e 4 2 3 4 Ss é 7 3
Electreity Demand Growth, Role Ctelye)

Fig. 4. Assumed probability distribution for the electricity demand growth rate.

Simulation Analysis
The principal exogeneous input required in using the EPPAM model fs

the indicated growth in electricity denand. In this case, the ."Tow,"
“pean,” and “high” forecasts growth rates noted in Fig. 4 were used to
exercise the model. The probability distribution in Fig. 4 was constructed
by assuming that the company's forecast of 6.7/yr and the Texas conmis-
ston's forecast of 3.9%/yr were equally likely and by assuming a wide range
of uncertainty. The “low" and “high* values were purposely used to empha-
size the large uncertainty inherent in the demand forecasts.

EPPAM was initialized with the same total capacity, capacity under
construction, and load duration curve as the £1 Paso Electric Company and

was used to calculate the average price of electricity that would be charged

182

9

under the alternative capacity expansion plans shown in Fig. 1. The results
of these model calculations are shown in Fig. 5 for mean demand :growth, in
Fig. 6 for low demand growth, and in Fig. 7 for high demand growth.

The EPPAM calculations summarized in-Figs. 5, 6, and 7 were used to
argue that rate payers would benefit from a sale of the company share of the
second and third units of the Palo Verde station. The rate benefits were
shown to be significant under the most likely demand forecast (Fig. 5) and
substantia! under the low demand forecast (Fig. 6). Furthermore, the mode?
calculations showed that ‘the sale of units 2 and 3 would not penalize rate
payers even under the unusually high demand growth rate assumed in Fig. 7.
Company Criticisms and Counter Analysis

The company's criticism of the EPPAM calculations was based primar-
dy on the inclusion of only coal and turbine plants in the model. £1 Paso
Electric Company, on the other hand, has a combination of gas-fired steam
plants, coal-fired steam, and peaking plants in operation as well as the

Pols Wanda =,

vclear units under construction. The company argued that the
large differences between the fuel costs of the Palo Verde units under
construction and the gas-fired units fn operation could not be adequately
represented by a simulation model whose major source of electric power was
coal-fired power plants.

To emphasize the importance of treating natural gas, coal, and
nuclear plants separately, the company presented {ts own model-based
analysts in surrebuttal testimony. The company's director of Energy and
Environmental Affairs presented economic analysis of the company revenue
reqy-rements using @ production costs computer model (PROMOD) to calculate
the fuel costs to operate a system comprised of different amounts of gas-
fired, coal-fired, and nuclear capacity. The company calculations showed
‘04 “Pele Verds)

Plan 2 (all Bunk

yo 92

88

15%,

Face

1990 92

erg

Mo 62

loa

S%

"so fa geome

EPPAM calculation of the price index for

price index for

demand growth of

EPPAM calculation of the
"Tow"

Plans 1 and 2 under the

2.5%/year.

Fig. 6.
demand

Plans 1 and 2 under the "mean"

5.38/year.

Fig. 5.

183

.

Plan 2 Catt 3 units
Of Palo Verde)

?

Pland “a
(only Feat und of
"5% Pale Verde)

ago ga BH MLSS HMO

Fig. 7. EPPAM calculation of the
price index for Plans 1 and
2 under the “hfgh" demand
growth of 8.5%/year.

that total revenue requirements in
the year 1986 would increase by
$13.5 million if they were to sell
their share of the second and third
units of the Palo Verde statton.
This increase arose from calcula-
tions showing fuel system production
costs increasing by about $93.6 mil-
lion with the Palo Verde sale and a
system capital costs reduction of
$80.1 million.
Surrebutal Analysis

The $13.5 millfon advantage
shown in the company rebuttal testf-
mony is reported in Fig. 8 along-
side of the results of nine sensi-
tivity tests of the company revenue

requirements calculations. In the

four revenue requirements labeled as Test 1, for example, the company assump-
tion on the annual increase in the price of natural gas was altered and the
effect on the overall revenue requirement determined. In Test 2, the avafla-
bility factors for the Palo Verde units during their first three years of
operation were lowered. Test 3 involved an increase in the construction
cost of Pato Verde while Test 4 increased the fixed charge rate used to con-
vert total construction cost into an annual increase in the company's reve-
nue requirements. Figure 8 shows that al] nine sensitivity tests showed a

reversal of the company findings. Making any one of the nine changes in
Boe

Lge
AN
ae Ws
egeye E es
Sate =
<THE 5 oi1 a
& 2 G °:
eR 8 € $e. 8 §
5
v
-
4 Pritl
2 3 43\?
g me ie
3c a #23)"
5
8 i 3he
e g3 |e
atale
“Gs n
z =
se ok
3 2 iy
i
cS 2 F Ss
> $ as E
z Bayt ir ae
£ rosdes 8
® € .8 8 g © -¢

25 : ' '
PEM Zp2? |p wed MSZ 40 SONINYS

Results of sensitivity tests of economic calculations performed by Frederic E. Mattson.

Fig. @.

184

3

assumptions would lead to the conclusion that rate payers would be better
off with the sale of the company share of the second and third units of the

Palo Verde station.

Commission Decision
Perhaps the best way to wrap up this short description of the analy-

sis and counter analysis in Case 1454 is to give the commission decision on
‘the company request to count construction work in progress in the company

rate base:

After analyzing the vast amount of testimony regarding E1
Paso's continued participation in the Palo Verde venture,
we believe that serfous questions have been rafsed con-
cerning the prudence of El Paso's reliance upon the Palo
Verde project as the best means available to serve its
Cugtomers in the decade of the 1980s. However, we are
unwilling to support or encourage the Company's continued
participation in the ambitious Palo Verde project at cus-
tomer expense without an exhaustive review of the costs/
benefits of the programs ... In short, E1 Paso's construc-
tion Program and means of financing it needs a thorough

review [4

PARTICIPANTS AND SCHEDULE

Participants
The most important participants in Case 1454 were the three commis-

sioners who rendered a decision at the conclusion of the hearings. In New
Mexico, the commissioners are appointed by the Governor and are assisted by
the commission counsel. In these hearings, ft was not possible for all

three commissfoners to remain in attendance. Often, only one commissioner
4

was present (at least one commissioner must be present to rule on objec-
tions). The commissioners in New Mexico came to the commission with a back-
ground in business and regulatory affairs [5]. ‘

The El Paso Electric Company was represented by three to four law-
yers, a half dozen witnesses employed by the company, and one outside wit-
ness from a major university. The Attorney General's contingent consisted
of a single lawyer, a part-time consultant, and three witnesses. The list
of participants also included a number of intervening groups, usually repre-
sented by a single attorney. These included a housing developer from south-
ern New Mexico, a lawyer representing a large military base, and an offictal
from one of the larger towns in southern New Mexico. In some cases, the
intervenors took no position in the case; and when they did, they. seemed to
play a relatively minor role in the hearings. An important exception, how-
ever, was a consumer group that intervened to oppose the company's bid to
count CWIP in the rate base. Although the consumer group did not bring any
separate witnesses before the commission, the group's lawyer was an active
pértivipaut im questioning witnesses speaking on behalf of the company or
the Attorney General.

Next to the commissioners, the lead lawyers for the two principal
groups are the key fndividuals in the hearings. The lead lawyers for the
company and for the Attorney General were responsible for putting together
the collection of expert testimony to support their positions, for preparing
and executing cross examination of witnesses speaking for the opposite side,
and for preparing the briefs at the conclusion of the hearings. The lead
lawyers remained {n attendance throughout the hearings while many of the
expert witnesses were only aware of the small segment of the hearings in
which they participated directly. ,

15

‘Schedule
To one accustomed to the pace of research in a university and ina

national laboratory, the rapid pace of events in Case 1454 was perhaps the
most unusual feature. Figure 9 shows the sequence and timing of events in
the case. This chronological diagram begins with the issuance of the Palo
Yerde construction permit in 1973 and the opening of the case in February,
1979. cross examination of witnesses before the commissfon ended on May 2,
1979. Nine days later, the company announced a major cost overrun at the
Palo Verde construction site and a one-year delay in the scheduled comple-
tion date for the first of the three units. Briefs were filed at the end of
May, and the commission issued its “Findings of Fact" in June, 1979.

During the months of February and March, the EPPAM model was adapted
to represent the ET Paso Electric Company. Simulation results were obtained
for two expansion plans--one involving all 600 MW of the Palo Verde station,
the other including only 200 Md from the first unit of Palo Verde. The

DECEMBER, ane
FINDINGS.
CERTIFICATE OF FACT

FER.IOTD 6 on
wr

cost
HEARINGS gunn:
SEGIN

NORMAL ANALYSIS ‘TESTIMONY

‘ana as ane ans) f ano

@@® oO

® @
aa? ame snp m= a7 ame ane SN BN BBR
(Tin es, Tees Ly Fel_Set

~ Y
REVISED ANALYSIS ADVERSARY ANALYSIS AND TESTIMOHY

Fig. 9. Schedule of events in Case 1454,
16

feedback loops leading to an endogenous calculation of any further base load
capacity initiations were removed in “the interest of simplicity." That is,
the link between the internally calculated “Planned Coal Capacity Initiation
Rate" in Fig. 2 was eliminated so that the model would reveal the conse-
quences of following an exogenously specified capacity expansfon plan. This
Vine of analysis (called “normal analysis” in Fig. 9) led to shortages of
generating capacity under the high demand growth scenarfo.

By March 27 (Step 5a in Fig. 9) it was decided that the shortages of
generating capacity projected under high demand growth was problematical
because other Attorney General witnesses were unable to Prepare convincing
testimony that purchased power would be available from neighboring utili-
ties. Consequently, the link between the Planned Coal Capacity Initiation
Rate and the rate fn Fig. 2 was returned and the model was allowed to endo-
genously determine additional capacity initiations required after 1982.
Simulation results from this “revised analysis" (summarized in Figs. 5, 6,
an¢ 7* vee reported to the Attorney General on the 28th of March and sub-
mitted in direct testimony on the 30th of March. With the key. feedback
Toops returned to the model, the simulation results showed no problems with
capacity shortages under all the demand growth scenartos tested.

The model-based results were presented to the commission on the 19th
and 20th of April. On the 19th, cross examination by the consumer group law-
yer occurred in a systematic fashion designed to summarize the key findings
for those commissioners who may not have had time to study the testimony.
The opportunity to review and illuminate the model-based findings occured in
this vase vecause the consumer group and the Attorney General both opposed
‘the company request for CWIP in the rate base. On the 20th of April, the
pattern of cross examination turned from a systematic summary designed to

186

Ww

4)luminate the key findings to a highly critical attack designed to dis-
credit them. The purpose of the company cross examination was to weaken the
credibility of the model-based analysis in the eyes of the commission.*

The company attempt to discredit the modeling results shown in
Figs. 5, 6, and 7 did not stop with the cross examination on the 20th of
April, however. By the 27th of April, rebuttal testimony by three witnesses
was filed before the commission. This filing began the portion of the
hearings noted as “Adversary analysis and testimony" in Fig. 9. This por-
tion of the analysis and counter analysis proceded quite quickly. Company
calculations were found to be highly sensitive to changes in parameter input
assumptions on the 28th of April; cross examination to reveal the sensitivity
of the company revenue requirements calculations was planned on the 29th;
cross examination of company witnesses by the Attorney General's office
occurred on the Ist of May (morning and afternoon); surrebuttal testimony
showing the results of the sensitivity tests (Fig. 8) was presented to the
commission and the company at the end of business on the Ist of May; surre-
buttal testimony was presented to the commission in a systematic fashion
under cross examination by the consumer group on the 2nd of May; and the
company's limited cross examination of the surrebutal analysis was completed
‘on the 2nd of May.

*A striking aspect of the lengthy cross examination (to one accustomed to
more academic research) is the ratio of the time in presentation versus the
time available for preparation. In Case 1454, the model-based analysis was
wrepared in the months of February and March while the cross examination
jasted for 1-1/2 days. In academic research, one may be allowed one to two
years for model development and analysis, but only find time for a one-hour
Presentation to the individuals who will judge the model's usefulness.
18

The fast pace of events shown in Fig. 9 is one of the more important
environmental factors in determining the suitability of system dynamics
models under adversary conditions. The features that make system dynamics
useful in university research (where a good number of system dynamics practi-
tioners are employed) may be irrelevant under the accelerated pace of adver-
sary hearings. This and other conclusions are discussed in the final part.

of-this paper.

SUITABILITY OF SYSTEM DYNAMICS
I begin the third part of this paper by referring to several

strengths of the system dynamics technique and noting whether these
strengths proved to be useful under the adversary condfttons in Case 1454.
Inen . win to certain weaknesses sometimes attributed to the system dynam-
fics approach and note whether such weaknesses were relevant. I conclude
with a brief discussion of the relevance of certain "confidence tests" said
to be important in evaluating the usefulness of system dynamics models.
Advantages

The key advantage of the system dynamics approach is the ease with
which feedback loops can be represented and analyzed. This advantage proved
to be crucial fn the analysis conducted in Case 1454. The key feedback
loops governing the rate of initiation of new coal capacity (see Fig. 2)
were found to be crucial when the results of the “normal analysis" conducted

in Februsy end March (see Fig. 9) showed shortages of generating capacity

187

19

under rapid growth (and exogenously specified initiation rates). Returning
the feedback loops to the model and repeating the analysis (“revised analy-
sis" in Fig. 9) eliminated the problemof capacity shortages. 3

The feedback loop structure of the electric utility simulation model
was not only usefu) in the “revised anlaysis" stage, it was also helpful in
the “adversary analysis and testimony" stage of the hearings. Company wit-
nesses givihg rebuttal testimony afmed at discrediting the simulation analy-
sis seemed unaware of the likely pattern of behavior of the capacity initia-
tion rates calculated endogenously in the model. Their criticisms of the
model behavior in rebuttal testimony were often inaccurate, and the inaccu-
racies seemed to arise from a lack of understanding of the. difference be-
‘tween an endogenously calculated capacity expansion plan and an exogenously
specified plan typical of “open system” models.

System dynamics models, when limited to a manageable size, offer an
important advantage in the ease with which structural changes and parameter
changes may be introduced. This advantage also proved to be important in
Case 1454, especially during the “revised analysis” phase, when a large num-
ber of new simulations experiments were performed on the 28th of March (Step
5b in Fig. 9).

In ny opinion, the second most important strength of the system dy-
namics approach 4s the array of descriptive aids that make thorough documen-
tatton possible. Causa? Toop diagrams, DYNAMO flow diagrams, documented
Tistings, and DYNAMO Tistings allow the system dynamics practitioner to pre-
pare several “layers” of documentation. This advantage #s crucial to the
success of system dynamics in the academic world where progress fs achieved
by the reproducibility of simulation experiments by different analysts in

different research institutes. Thorough documentation fs not, however, @
20

useful feature under adversary proceedings. Indeed, it appeared from the
events in Case 1454 that thorough model documentation gives the opposing
participants the opportunity to “come-up-to-speed" rapidly and develop the
most. effective strategy for discrediting the model in cross examination.
Moreover, the use of PROMOD in Case 1454 showed that participants could
deliver testimony based on the results of models that were completely undocu-
mented due to their proprietary nature. The Attorney General objected to
the use of the proprietary PROMOD model in Case 1454, based in part on the
Jack of documentation:

Q. Please summarize your response to Dr. Mattson's rebuttal
testimony.

A. Mr, Mattson's rebuttal testimony invites a more informa-
tive exchange of ideas than Dr. Baughman's because he
attempts to describe the purported benefits of investment
in Units 2 and 3 of Palo Verde in quantitative terms. Un-
fortunately, however, he has used a proprietary computer
code known as PROMOD.. The proprietary nature of PRONOD
does not allow Hr. Mattson's calculation to be examined in
detail. The use of computer models without explanation of
the inner workings of the model fs unscientific and should
not be allowed in a proceeding such as this.

Q. In what sense fs the use of PROMOD unscientific?

A. A key feature of the scfentific method is that results ad-
vanced by one scientist should be reproducible by other
scientists. This principle is most familiar to those en-
gaged in work in the sciences of chemistry and physics. In
these disciplines, results of a chemical or physical ex-
periment are not considered to be "scientific" unless
another chemist or physicist can reproduce the same exper-
iment in his or her own laboratory. Without reproducibil-
ity, the sciences of chemistry and physics would not ad-
vance. Indeed, this principle is so well established that
one publisher publishes a "Journal of Irreproducible Re-
sults” as a parody on the scientific method. Although
more familiar to the physica? sciences, the principle of
Copeais titty is just as important in the economic
sefences.

The ‘economic calculations based on PROMOD and presented
before this Commission cannot be independently reproduced
by either the Commfssfon staff, by intervenors, or, for
that matter, by analysts testifying on behalf of the ET

188

a

Paso Electric Company. None of these potential users have

access to the computer equations of the PROMOD model be-

cause of restrictions placed on its use by Energy Manage-

ment Associates of Atlanta, Georgia [6]. :
Despite the objections to the company use of PROMOD, the commisston ruled
that the company calculation could be entered on the record.
Disadvantages

Now, what about the weaknesses sometimes attributed to the system
dynamics approach? Perhaps the most frequently discussed weakness within
the modeling community is the "informal" fashion with which many parameter
estimates are obtained. Modelers accustomed to the formal estimation proce-
dures used in econometrics have criticized the system dynamics procedure as
lacking an empirical base [7]. From the events in Case 1454, I would con-
clude that this criticism is not likely to be fmportant in adversary pro-
ceedings before state public utility commissions. Very seldom do the comnis-
sioners and their key staff have a background in formal modeling and statis-
tical methods that would make them receptive to the criticisms sometimes
voiced by econonetrictans. Furthermoré, the commissioners have sat through
numerous hearings in which econometrictans battle one another trying to dis-
credit each other's electricity demand forecast. This constant clashing of
econometricians has contributed to a healthy skepticism in the commisstoners
toward the “sctence” of econonetrics.

Another criticism of system dynamics, sometimes: voiced by those
modelers more accustomed to optimization modeling, 1s that the model does
not employ a formalized objective function. This type of criticism was not
relevant in Case 1454 because the interpretation of the simulation results
in Fig. 5, 6, and 7 was not difficult. However, in other cases where simu-
lation results have been presented before state commissions with short-term,
Yong-term tradeoffs, for example, simulation models have been heavily criti-
ctzed for not including a formal objective function.* ‘

Another weakness of the systen dynamics technique 1s the relfance on
continuous approximations to discrete changes. Although Forrester warns the
beginner not “to be carried away by his knowledge of the discreteness of var~
fous decisfons and actions" [10], commissioners may wonder how a major dis-
crete event (like the sale of a large amount of common stock) could be ade-
quately approximated by a continuous simulation model. This type of ques-
tioning did not arfse in Case 1454, but I suspect it will come up in future
hearings if system dynamics models are used to represent individual com-
panies.

Conf: Testing

Forrester and Senge [11] present “confidence-building tests" that
one can use to improve the usefulness of a system dynamics model. Based on
the events in Case 1454, I would label 7 of these tests as irrelevant be-
cause of the fast pace of the proceedings.

© The “extreme conditions" test, the “extreme policy” test, and the

“behavior anomaly” test are helpful to establish self confidence by

the model developer, but they are not likely to be relevant tests

that one could cite in defending model “validity” under cross exam-
ination. Furthermore, I do not think ft is likely that opposing
groups will attempt to discredit a model because it 1s not capable of

*For example, the simulation analysis of the Pacific Gas and Electric
capits? evpansion plan by the Environmental Defense Fund (8) was repeatedly
criticized by the PG&E consultants for not including a formal objective
function (9) in testimony before the California Public Utilities Commission.

189

23

passing one of these tests. Such testing does not appear to be
possible given the pace of adversary hearings. ‘

The “family member“ test is a very appealing test for analysts fnter-
ested in generic structures and their ability to explain problematical
behavior patterns found in many different systems. It fs not likely,
however, that participants in adversary hearings before an electric
utility commission would find this test convincing. Indeed, some
participants might feel that a model was “invalid” if it relied
heavily on theories taken from other members of the “family.”

The “dimensional consistency" test is another test that the mode?
builder should perform to satisfy himself that the model has been
correctly structured, but it fs not likely that participants in adver-
sary hearings will find reference to this test convincing. A pos-
sible exception, however, would be a commissioner with a background
fn engineering since engineers are more accustomed to checking their
calculations for dimensional consistency.

The “surprise behavior" (counter-intuftive behavior?) test fs another
test that participants in adversary proceedings are not likely to
appreciate. Such a test is more likely to help the model builder per-
form his research and analysis with confidence than to help others
gain confidence in the "validity" of the model.

"Other tests," by which Forrester and Senge [Ref. 11, p. 223] mean
the array of statistical tests employed in econometric modeling, are
likely to be unimportant in adversary proceedings. Econometricians
familiar with these tests are more likely to question the ability of
the model to reproduce -historical behavior--a test mentioned below.
28

Other tests discussed by Forrester and Senge may be valuable in de-
fending moael "validity" under adversary proceedings. These include:

© The “behavior reproduction” test was not. Specifically mentioned in
Case 1454, but ft seems that this question could easily be raised by
participants attempting to discredit a model. A reason for not
inqufring about the test, however, 1s that the model developer may
hzve already subjected the model to such a test and be in a position

to speak positively about the results before the commission.

°

The “boundary adequacy" test is another test that could well be impor~
tant because participants in adversary proceedings can easily learn
what has been omitted from a model. Thus, questions about the ade-
quacy of the model boundary come quickly to mind. In Case 1454, for
example, Federal prohibitions on the use of combustion turbines were
outside the model boundary. To check the importance of this omis-
ston, exogenous, time-dependent table functions were introduced to
“ *."" that extra turbines would be on line in time to have each indi-
vidual turbine operate in compliance with the Federal rules. The
model was then used to calculate the incremental increase in the
price of electricity arising from the need to comply with Federal
rules. The fact that this particular test was performed in an exo-
genous fashion did not appear important to the commisston--what was
important was that the comparison of electricity prices in Figs. 5,
6, and 7 did not change with the increased number of turbines.

© The “parameter-verification” test is another test that could well be
‘important, especially {f all the model parameters are documented in @
highly visible fashion. Opposing participants in adversary proceed-
ings could scan the list of parameter values looking for unusual esti-
mates to be discussed fn cross examination.

190

ra

© And finally, the "behavior sensitivity" test is an important test to
perform in the midst of adversary analysis, espectally {f one can
anticipate objections to certain parameter estimates used in the
model. Indeed, one of the most useful functions of a system dynamics
model (or other mathematical models) is to show whether particular
parameters really need to be specified with much precision. From the
events in Case 1454, I conclude that sensitivity testing is one of
the more important of the many tests discussed by Forrester and
Senge. It is particularly helpful in answering questions under. cross
examination, for example, to have performed the proper sensitivity
tests in advance. .

I do not wish to state that only four of the many confidence tests
discussed by Forrester and Senge are important to system dynamics applica-
tions under adversary proceedings. All of the tests serve a valuable role
for the model developer by increasing his confidence in the analysis. How-
ever, not all of them are likely to contribute directly to a defense’ of
model validity or usefulness under questioning by opposing participants in
adversary hearings. Although the list presented by Forrester and Senge fs
one of the longer lists in the literature on model validation, I would add
another test that appears worthwhile under adversary proceedings. This
might be called the “previously-published behavior reproduction" test, for
it involves a comparison of the pattern of behavior presented in the current
hearing with previous projections given in articles or reports available to
opposing participants. It appears that a standard means of discrediting
expert testimony is to search through the witness’ publications looking for
projections or conclusions that conflict with the projections and conclu-

sfons given in the current hearing. To counter this practice, the modeler
26

should review previous projections checking for possible conflicts with the
current testimony. During the course of the review, one should also check
for conflicts with model projections over time intervals for which time

serfes data is now available.

SUMMARY

This paper recalls the analysis, counter analysis, participants, and
schedule of Case 1454 with a focus on the use of a system dynamics model of
the electric utility industry. The principal lesson to be learned from Case
1454 is that system dynamics can contribute to informed decision making
under adversary proceedings despite the rapid pace and adversary nature of
Sa) |v l.g3. Key advantages of the system dynamics approach are the ease
of representing feedback loops and the ease in making structural and param-
eter changes-. The thorough documentation that often accompanies system
dynamics models constructed in the public sector, however, is likely to
serve the opposing participants and decrease the effectiveness of system
dynamic model-based testimony under adversary conditions. The paper con-
cludes with a discussion of numerous tests said to be useful in increasing
one's confidence in a system dynamics analysis. It fs concluded that many
ig these tests are not likely to contribute directly to the defense of model

arty” under cross examination by opposing participants.

191

27

REFERENCES

O)

(2]

[3]

[4]

3]

(6)

7]

(8)

[9]

Direct Testimony by Andrew Ford before the New Mexico Public Service
Commission, Case No. 1454, March 30, 1979, p. 3.

Andrew Ford and Irving Yabroff, "Technical Documentation of the Electric
Utility Policy and Planning Analysis Model,” Los Alamos Scientific
Laboratory report LA-7773-MS (June 1979).

Andrew Ford and Irving Yabroff, “Defending Against Uncertainty in the
Electric Utility Industry," in Ener: stems and Policy, Vol. 4, Nos. 1
and 2 (Crane Russak, New York, .

"Findings of Fact and Order on Revenue Requirements," New Mexico Public
Service Commission, Case No. 1454 (1979), p. 19,20.

National Association of Regulatory Utility Commissioners, "1978 Annual
Report on Utility and Carrier Regulation," (National Assoctation of
Regulatory Commissions, 1979), p. 700.

Surrebuttal Testimony by Andrew Ford before the New Mexico Public Service
Commission, Case No. 1454, March 30, 1979.

A. A. Legasto, Jr. and J. Maciaeriello, "System Dynamics: A Critical

Review," in Studies in the Management Sciences, System Dj mamics, A. A.
Vegeeto, or. J. W. Forrester, and J. M. Lynets, Editors (North-Holland,

W. R. Z. Willey, " Alternative Energy Systems for Pacific Gas & Electric
€o.: An Economic Analysis," the Environmental Defense Fund (1978).

H. Pifer, W. Hieronymus, J. Ristuccfa, and M. Schnitzer, “An Independent.
Evaluation of EDF's Electric Utility Financial Simulation Model,” Joint
Hearing of OI1-26 and OII-a2 before the California Public Utilities
Commission (1978).

[10] Jay We Forres ats Industrial Dynamics (MIT Press, Cambridge,

Massachusetts, 1961).

[11] Jay W. Forrester and Peter H. Senge, “Tests for Building Confidence in

System Dynamics Models," in TIMS Studies in the Management Sciences, Vol.
He System Dynamics, AS A. Legasta, IF, -0- We POrrestays -anECT Re
ets, rs (Nor

gas to,
Holland Press, 1980).