WHAT IS AN ADEQUATE MODEL?
Henry Birdseye Weil
Pugh-Roberts Associates, Inc.
Five Lee Street
Cambridge, Massachusetts 02139
ABSTRACT
Much of the literature on model evaluation focuses on what
amount to absolute measures, that are independent of the con-
text in which a particular model is used. This paper argues in
favor of situation-dependent measures. Whether or not a model
is "good enough" depends on the job it is being asked to do and
the mind set of the people who must use the results. The rela-
tionships between model adequacy and successful implementation
of model-based recommendations are discussed. While rejectin
the classical paradigm, the author emphasizes model realism and
historical accuracy as important determinants of implementa-
tion, The life of a model involves many evaluations of whether
it is “worth the costs", "believable", "useful", and “right”.
Issues surrounding these judgements are explored. How differ—
ences in circumstances can lead to different, but in each case
quite adequate, models is illustrated by contrasting two models
developed five years apart for the same organization. The pa-
per concludes that successful models are persuasive, not simply
to modeling technicians but to high-level decision makers.
281
I. INTRODUCTION
The question “What is an adequate model?" is important to
modeling professionals and their clients. Throughout the sel-
ling, development, and use of any model, both the idea of model-
ing and the specific product repeatedly are evaluated in terms
Of: “is it useful"; "can we believe it"; “is it worth the
costs". flodels have to payoff, in order to build and sustain an
organization's commitment to modeling.
Moreover, there is lack of agreement on this question among
modeling professionals. Some say a "valid" model is one which
very accurately conforms with historical time series. Others
argue that the ability to reproduce "reference modes" of behavior
is key. There is a “consumerist” school, which says that a good
model is one that a user will accept. Still others maintain that
the acid test of any model is whether or not its predictions come
true. Hence, established wisdom does not provide clear answers.
Nor does one usually have the luxury to find the answers
through trial and error. The professional model builder operates
within a tight set of constraints. Perhaps the most obvious of
these are schedules and budgets, but equally important are the
attitudes of the client: initial expectations; patience while
waiting for results; confidence in the model builder and his
technical approach. While there certainly is latitude for "mid-
course corrections", producing an adequate model within the usual
282
time, financial, and relationship constraints requires good
economy of motion and a sense for the jugular. In other words,
the professional model builder's a priori judgement regarding
what constitutes an adequate model (i.e,, at the outset of an
assignment) must be approximately on target.
One is faced with a choice between two fundamentally differ-
ent views of the world. ‘The traditions of classical statistics
emphasize what amount to absolute measures of model adequacy. [1]
While they differ in their technical details, all of these mea-
sures indicate the correlation between simulated and historical
values of model variables.’ The duel messages of this paradigm
are: (1) the correlation measures are to be maximized; and (2)
there is a threshold of significance for these measures below
which a model lacks “validity". The resulting philosophy of
model building and evaluation relies on concepts of adequacy that
are independent of the context in which a particular model is
used. This is what I mean by “absolute” standards or measures.
In essence, the methodology determines the product.
The field of decision theory offers an alternative view. [2]
The concept of the “value of information to a decision maker",
applied to the evaluation of models, leads to situation-dependent
assessments of adequacy. The basic ideas of the paradigm are:
(1) decision situations differ; (2) the specifics of a decision
Situation determine its sensitivity to the quality of information
available to the decision maker; (3) one can quantify the value
to the decision maker of having “better" information (e.g.,
283
additional and/or more accurate inputs); and (4) beyond some
point, diminishing returns argue against continued investment in
the quality of decision inputs. With this approach, the context
determines the product.
I have concluded from nearly twenty years as a professional
model builder and management consultant that the first paradigm
-- absolute measures of model adequacy -- is narrow, rigid, and
misleading. Whether or not a model is “good enough" depends on
the job it is being asked-to do and the mind set of the people
who must use the results. In other words, an adequate model is
the right tool for the job at hand.
This paper argues in favor of situation-dependent measures.
In Chapter II, the concept of model adequacy and model implement-
ation are interrelated, While rejecting the restrictive classic-
al paradigm, the paper in fact emphasizes "realism" and "valid-
ity" as important determinants of implementation. Chapter III
makes the case for situation-dependent criteria of model adequa-
cy. Both results-oriented considerations and technical arguments
are presented. Then, a number of factors which define "the right
tool" for a given job are discussed.
In Chapter IV, the ideas of situation-dependent model eval-
uation are illustrated by contrasting two’ conceptually similar
models that were developed under quite different circumstances.
How different circumstances call for different -- but, in both
cases, adequate -- models is shown via a pair of models developed
five years apart for the same organization. Chapter V concludes
284
that successful models are persuasive, not simply to modeling
technicians but to high-level decision makers.
285
II. MODEL ADEQUACY AND IMPLEMENTATION
A. How These Two Concepts Interrelate
The professional model builder must establish and maintain a
good track record for "success". A private corporation that
hires a consultant generally expects results of immediate value
which can and will be implemented. In public policy analysis,
too, projects that have a definite impact on people's thinking
and actions are more valuable than those which don't. Imple-
mented results are the key to client satisfaction.
I have a highly pragmatic definition of successful modeling:
the tool is used; and the tool produces value many times its cost
of development and use. Success in those terms is not only a
matter of good business, but also of professional satisfaction,
the credibility of our methods, and the field's ability to at-
tract top people.
Working backward from this definition of success, there are
several important tests that a model must meet in order to be
"successful":
1. Are the costs of developing and using the model
consistent with "what's at stake" in the decisions it
is intended to support?
2. Do managers have sufficient confidence in the model
to be willing to base important decisions on it?
286
3. Does the model "speak" to users at a level of speci-
fity consistent with the managerial actions required
to implement indicated decisions?
4. Are the decisions and outcomes indicated by the model
correct?
These are recurring questions as a model is developed and used.
A model which is not “worth the costs" will not be built
and/or used. A model which decision makers do not “believe in"
wlll not be taken seriously. A model which does not produce
“actionable results" will be labeled impractical or academic.
And a model which, over a period of time, does not prove that “it
is right" will be suspect. Unless a model passes all of these
tests, it never will be built, used initially, and, then, insti-
tutionalized.
In very pratical terms, an adequate model is one that passes
the four tests enumerated above. Hence, I believe that "suc-
cess", "implementation", and “adequacy” are highly interrelated
(though not synonymous) concepts. Specifically, model adequacy
is an important cause of implementation success (or failure).
This can be seen best within a comprehensive conceptual framework
of the factors determining implementation.
B. The Role of Model Adequacy in Successful Implementation
Implementation success (or failure) is determined by a
complex feedback system of relationships. Some of these rela~
tionships are shown in Exhibit 1.[3] This conceptualization re~
287
ANyriasnog
3oNgmuMt
ShonLanAssy
‘S39NvHO HIM
S30NvHO 40
Sis. 30
d3038
ANSHNOUIANS.
~~ szomvuo 40
ALFIGVINSNS IAAT
aoNynn0s34
awigniyzinvoue
TNSMINIOAN
Saskivny 30. S3SAqwHy NI
“huvaryya 2
Ae
eeeg|/ eee
seseh
J
sisavyny’nio
‘Anas it
“Sys NY
NouvotsingoS
'S ANVLIMENOD
5110838 40
‘NOLLYATIWAS
INILOMMLENOD
‘suoiays
‘w30
288
_
“WALdSINOD VL LIGIHXS
flects my experience in corporate strategy consulting. For
example, it assumes a defined client, who has substantial power
to implement and some ability to evaluate the effectiveness of
actions taken.
Roberts[4] and Weil[5] have argued that success occurs only
when all of the essential ingredients are present:
1, The results of a project must, in fact, be implement-
able;
2, Those who will have to take action must have a clear
desire to implement; and
3. The environment must be properly receptive.
These factors lie at the center of Exhibit 1.
In my experience, project results -- to be implementable --
must be sufficiently detailed that they can be acted upon within
the client's established management system. By this I mean
within their structure of managerial responsibilities, resource
allocation and control processes, and corporate resources (e.g.,
personnel, facilities, technologies, customer relationships).
Furthermore, if the recommended changes are too extreme, or too
unconventional, or too inconsistent with the client's social/pol-
itical structure, they do not have a high likelihood of being
implemented.
The implementability of recommended changes depends on the
“realism” and "validity" of analyses performed during the pro-
ject. I use the terms as shorthand for the level of detail,
289
practicality, sensitivity to client needs, thoroughness, com-
pleteness, and correctness inherent in the consultant's technical
work. These characteristics are the consequence of:
1, The consultant's overall sophistication and skills,
i.e., his professional experience, his technical
competence (primarily brought to the project in
question as the product of past work, but also en-
hanced by any constructive evaluation of this pro-
ject's results as they are being achieved);
2. The consultant's knowledge of this particular cli-
ent's situation (primarily the result of time spent
with the client -- either during the current project
or its predecessors -- but also aided by the consul-
tant "having séen the problem before"); and
3. The amount of time the consultant spends on the
project's analysis tasks.
Let us return to the center of Exhibit 1. Roberts[6] empha-
sizes that a client's desire to implement is strongly affected by
the urgency of the consulting project's problem focus. Without
this clear motivating force, implementation generally is thwarted
by some combination of indifference, inertia, cost, and fear. An
important feedback loop can exist here. To the extent that the
project's recommendations (or other unrelated factors) improve
organizational performance, the sense of urgency and, hence, the
client's persistence in completing the job of implementation will
lessen.
Given the necessary urgency, a client's desire to implement
recommended changes is the result of his understanding of, ac~
ceptance of, and confidence in those proposals. In several
papers, Weil{7] discusses the critical importance of two factors
in this regard:
290
1. Active client involvement in all aspects of the
project's analytical tasks (so that he understands
the technical approach, agrees with all significant
assumptions, sees "where the results came from", and
takes the lead in formulating the strategy recommend-
ations); and
2. A high level of realism and demonstrable validity in
the analyses (i.e., model structure and behavior
which are highly consistent with all available data
about the client system).
Frohman and Kolb[8] discuss the effect of power on the
relationship between. consultant and client. The consultant's
influence on his client is a type of power that enhances the
client's acceptance of and confidence in the recommended changes.
This influence derives from the consultant's technical/profes-
sional authority, and from the trust ané confidence between
client and consultant which develop during the project.
In my experience, the basis of a consultant's influence
shifts over. the course of a project. Initially, it is more a
function of the compelling logic of the consultant's approach --
his credentials, methodology, problem diagnosis, and project plan
as presented in early meetings and documents. Later in a pro-
ject, the consultant's influence (or lack thereof) is primarily
the consequence of the client's confidence in him. That confi-
dence arises from the time spent working together and, later on,
from client perceptions of the effectiveness of the consultant's
proposed changes.
The consultant's influence, along with the urgency of the
problem focus, determine a client's willingness to invest time in
the project. Here we have another important feedback loop: The
2a
self-reinforcing nature of client participation. Moreover,
client involvement enhances the realism and validity of the
consultant's analyses, thereby strengthening his technical auth-
ority with the client. These related feedbacks are shown on the
left in Exhibit 1.
In summary, model adequacy is an absolutely critical deter-
minant of implementation and project success. It is central to
the development of client understanding, acceptance, and confi-
dence. It reinforces the consultant's influence on his client
(for example, to invest time in the modeling project). Almost by
Gefinition, a more “adequate” model produces results which are
more implementable. The last point refers to both the appropri-
ateness of the results and their actionabililty.
292
III, THE RIGHT TOOL FOR THE JOB AT HAND
A. The Case for Situation-Dependent Criteria
As described above, the life of a model involves may evalua-
tions of whether it is "worth the costs", “believable”, "useful",
and "right". These critical tests must be passed within reasona-
ble time and financial constraints. In light of that, several
points should be considered:
1. Like it or not, such evaluations involve elements of
taste, psychology, and emotion.
2. Different decision situations do have different
sensitivities to the accuracy of information inputs.
3." Efficiency is important, both because urgent problems
demand action and because virtually all clients are
sensitive to the absolute costs of modeling projects.
4, A modeling project is usually undertaken with some
objectives in mind; it is entirely possible to create
a model which is "correct" and at the same time not
very useful.
5. It is unlikely that numerical time series will exist
for all important variables which should be included
in a correct model.
The first point means that each group of people judging a model
is likely to have a somewhat different concept of “adequacy”.
For example, whether one views the world as deterministic or
probabilistic; one's willingness to take intellectual, organ-
izational, and financial risks; and one's commitment to Cartesian
concepts of rationality all come into play.
293
The. second point raises the prospect misjudging the payoff
from making a model better. The mistake can be in either direc-
tion -- either unjustifiable perfectionism or overconfiéence that
a "rough cut" is adequate. The third point underscores the
importance of doing an adequate job, but not an extravagant one.
Most of us work for clients, not patrons. We create tools, not
monuments.
This leads to point four. Models are tools; they are crea~
ted and used for specified purposes. Like any other tools, they
must fit the capabilities and needs of their users. The last
point indicates that a correct model must exploit a wide range of
information, includng data about variables which are important to
the model's purpose but are very difficult or impossible to
directly measure. That necessity raises additional model evalua~
tion issues. [9]
The classical tradition of model evaluation does not address
those points. Any fixed standards are likely to be too high in
some cases and too low in others. If, for safety, they are set
very high, then most modeling will be wastefully perfectionistic.
The fact that clients must have confidence in models before they
will base decisions on them is treated as a technical rather than
a human problem. Concepts of usefulness and usability do not
seem to be part of this paradigm. Perhaps worst of all, the
classical paradigm leads model builders to exclude factors of
obvious importance -for the rather lame reason that "there are no
data available" in the desired form. All too often, the result
of this approach is failure.
In pursuit of successful models; I have rejected absolute
concepts of model adeguacy. That does not mean that I reject or
down-play the importance of historically accurate models. Far
from it! As indicated in Chapter II, the realism and historical
accuracy of a model are critically important determinants of a
client's understanding of, acceptance of, and confidence in
model-based results. Moreover, Peterson[10] shows that a histo-
rically accurate model is much more likely to be correct.
However, in creating the "right tool for the job at hand" I
strive to take a broad, pragmatic view of adequacy. Although
quite important, historical accuracy is one of several key fac~
tors. Various degrees of perfection are possible. What is
really called for in a particular situation? And within time,
financial, and client relationship constraints, how should this
dimension of model adequacy be traded off against the others?
This perspective leads me to favor situation-dependent criteria.
B. Judging Whether a Model is "Good Enough"
The most important judges of model adequacy are high-level
decision makers. If they are not "sold", a model will not be
successful. The next most important judges are the people in-
volved in building a model. They have to anticipate the decision
makers’ criteria and: be approximately on-target; they usually
295
have the’ technical knowledge and opportunity to more fully evalu-
ate the model; and their understanding, acceptance, confidence
often is a key determinant of the decision makers! attitude
toward the model.
Third in importance is the judgements of the technical
"fraternity" of model builders. This may sound like heresy, but
I believe that the peer review process is a “mixed bag". On the
one hand, it is part of valuable scientific tradition which sti-
mulates high quality work, the exchange of ideas, and scholarly
debate. On the other hand, it encourages out-of-context evalua-
tions and reliance on absolute standards. We should never forget
that the client comes first. Pleasing ourselves and our profes-
sional colleagues is no substitute for client satisfaction.
What characteristics of a model determine whether it is the
“right tool for the job at hand"? I believe that the relevant
factors include the model's scope; the amount of detail in the
model; the amount of data collection underlying its development;
its historical accuracy; and the amount of model testing that was
done. The required scope, detail, data collection, accuracy, and
testing depend on the specifics of each situation:
1, the types of questions being addressed;
2, The attitudes, experience, and organizational posi-
tion of the client;
3. Who, beyond the people involved in developing and
using the model, must be "sold" on the results;
296
4, The institutional systems and processes through which
implementation of model-based decisions must take
place; and
5. Perceptions of “what's at stake".
The ideas of situation-dependent model adequacy are best illus-
trated through specific examples.
297
IV. A PAIR OF ILLUSTRATIVE EXAMPLES
How differences in circumstances can lead to different (but,
in each case, quite adequate) models is clearly illustrated by
contrasting two models developed five years apart for the same
organization. he first was built in 1974-75; the second, in
1980, In both case, the client was the Life Insurance Division
of a very large diversified financial services company.
A. The 1975 Model
This project focused on the Division's variable annuity
products group.[11] In the years preceding the project, the
variable annuities group had grown dramatically in sales, booked
business, capitalization, and personnel. However, by the end of
1973 management became concerned with certain aspects of this
phenomenal growth. For one thing, the group was not yet produc-
ing operating profits, despite (or, as some people seemed to
feel, because of) the growth. This condition was significant as
a deviation from earlier expectations, but took on real urgency
as a result of the large amount of corporate capital involved.
Furthermore, inflation and a prolonged stock market decline
were beginning to take their toll. It was becoming harder to
sell. “"Persistency" (the ability to keep customers, once they
are sold a policy) was worsening steadily. Costs were rising
even as service declined. Management was concerned about this
298
situation. When would the variable annuities group become pro-
fitable? When would it stop consuming corporate capital and
generate cash surpluses instead? Could the group absorb addi-
tional growth in the short-term? What if the stock market didn't
turn around soon? Discussion centered on the desirability of a
"pause" -- a 1-2 year slowdown during which the group could
"catch its breath". There seemed to be good arguments both pro
and con.
It was evident that the issues surrounding growth strategy
for variable annuities were highly complex. The objective of the
project was to develop an analytical tool for sorting through all
of those complexities and efficiently testing a broad array of
strategic options.
While the resulting model was quite comprehensive, it was
designed to focus on a particular set of strategic issues. It
was this problem focus that dictated what to include in the
model, and what level of detail was required in each portion.
The specific issues which this model addressed were:
1. Rate of growth;
2. Control of. costs;
3. Service adequacy;
4. Improvement of persistency;
5. Evolution of the marketing organization;
6. The effect of stock market performance; and
7. Achieving profits.
299
Exhibit 2 shows an overview of the 1975 Model. It repre
sents the variable annuity product group and the divisional
marketing organization (which sells all of the Division's product
lines). The key exogenous inputs describe performance targets
for the group (e.g., sales, costs, persistency, profits); the
availability of corporate capital; economic conditions (e.g.,
inflation, stock market performance); and the characteristics of
competitors’ products. This model has approximately 600 varia-
bles.
The principal sectors of the 1975 Model can be seen in
Exhibit 3, Many aspects of the variable annuities group are
simulated within the model. The Product Line Sector gives an
overview of the generation of new products, including their re-
quirements for computer systems and administrative support. The
Service Sector represents the servicing of new sales and business
on the books. It includes manual clerical operations, computer-
ized service functions, and the interplay between the two. The
hiring, turnover, and efficiency of service staff are explicitly
modeled, as is the impact of automation. The initiation of new
computer system developments and the maintenance of existing
computer systems also are represented in the Service Sector.
As its name suggests, the Sales Sector represents the deter-
minants of variable annuity product sales. It is a rich, de-
tailed formulation, combining the effects of: salesforce size,
composition, and experience; product characteristics; pricing;
service adequacy; commission levels; managerial concerns and
300
THE COMPANY
PERFORMANCE
TARGETS
‘THE LIFE INSURANCE DIVISION
4 DIVISIONAL,
PRODUCTS
CAPITAL,
ECONOMIC COMPETITORS’
CONDITIONS PRODUCT
CHARACTERISTICS.
EXHIBIT 2
THE 1975 MODEL
301
VARIABLE ANNUITY
PRODUCTS
PRODUCT LINE
SERVICE
SALES
PERSISTENCY
ACCOUNTING
MGT. CONTROL,
MARKETING
ORGANIZATION
© CAREER FORCE
@ BROKERS
EXHIBIT 3
PRINCIPAL SECTORS OF THE 1975 MODEL
302
priorities; and external factors (e.g., inflation, stock market
performance, competitors products). ‘The Persistency Sector
models the "quality" of sales and the factors which cause cus-
tomers to cease paying policy premiums. Many of the factors
which determine sales also affect persistenc’
e.g., sales force
experience, service adequacy, managerial concerns, and external
conditions.
There are two aspects to the Accounting Sector. One part
represents the entry of newly-sold policies onto the company's
books, and their subsequent aging, persistency, and payment of
benefits. This is done in'considerable detail, to give reliable
financial results from model simulations; it is central to the
calculation of variable annuity revenues and cash flows. The
other part of the Accounting Sector assembles various revenues
and costs, and produces income statements and balance sheets.
The Management ‘Control sector describes the allocation of
managerial attention and influence in response to various dimen-
sions of performance, e.g., sales, profits, costs, and persisten-
cy. This sector represents the existent management control
system, and is easily modified to test alternative structures.
It is quite conceptually rich, showing how changing managerial
priorities react to, and feedback to affect, group performance.
The Division's marketing organization was broken down into
two components. The Career Force Sector models the recruiting,
training, experience level, time allocation, and attrition of
salesman who work exlusively for the Division. The Brokerage
303
Sector parallels the Career Force Sector, but represents inde~
pendent insurance brokers who sell several companys’ products.
The 1975 Model was used both to diagnose the causes of
existent problems and to reveal fundamental "truths" about the
variable annuity business. In particular, much time was devoted
to explaining the inter-relationships between sales growth and
profitability. Opportunities for performance improvement were
examined in such areas as:
1. Adequacy of service and system:
2. Managerial priorities in controlling the various
(somewhat conflicting) dimensions of group perforn-
ance;
3. Mix of sales between career agents and brokers
4. Control of persistency;
5. Sales growth targets; and
6. Recruitment and allocation of personnel.
During this project, we worked with a small Task Force that
was led by the Vice President in charge of the variable annuities
group. The Vice President proved to be an extraordinarily astute
and motivated client. He maintained a very high level of person~
al involvement, immersing himself in the project down to the
smallest technical details. Moreover, he was by nature a strate
gic thinker. He was prepared to challenge "conventional wisdom”
and to champion new ideas.
Several members of the Task Force took the time to scruti-
nize the equations and simulation results. They wanted to satis-
fy themselves that the model was reasonable on a detailed level
304
and to understand “where the simulation results come from". Of
great importance, though perhaps surprising, the Vice President
and his comptroller were part of this group.
We devoted a considerable amount of effort to improving the
historical accuracy of this model. Simulated values for a large
number of variables were explicitly compared with historical data
over the period 1970-1974. The historical accuracy of the 1975
Model is illustrated by the two examples in Exhibit 4. The model
generally produced results, within 10¢ of historical values; in
some areas, the accuracy was consistently within 5%. We achieved
a consensus that the base simulation was historically valid, and
the best existing estimate of what the future held in store.
The general conceptual framework provided by the model, the
initial analysis results, and our best forecast for 1975 through
1980 became inputs to management's determination of near-term
growth targets. The major company decision was to slow down
growth in variable annuity sales in order to improve profitabil-
ity. The question then became: what is the best set of policies
for achieving this goal?
The model's credibility was significantly enhanced when,
late mm 1975, it became evident that model-generated forecasts of
sales, persistency, and profits were much more accurate than
estimates produced by "conventional methods". Especially in the
eyes of managers who had not participated in model development,
this was a critical test.
305
simulated
FIRST-YEAR
-—7 LAPSE RATE
actual
octual
4
1971 «197219731974 1975
EXHIBIT 4
HISTORICAL ACCURACY OF THE 1975 MODEL
306
At that point, the model was expanded in several sectors
where more detailed answers to the strategy implementation ques-
tion were reguired. Policies in the areas of product mix, pri-
cing, salesforce compensation, salesforce size, customer service
expenditures, and persistency control were analyzed with the
model. The results of these analyses significantly influenced
key managers’ perceptions of the issues, and shaped the policy
decisions which ultimately were made. We consider this project a
success,
B. The 1980 Model
Pour and a half years later, we undertook another project
for the same organization. We were invited back because of the
perceived success of the 1975 Model. As we were discussing the
new assignment, I was told: "The model was even more correct
than we were willing to accept at the time".
This project focussed on marketing strategy for the entire
Life Insurance Division.[12] It was the culmination of a process
of strategy development and evaluation which had been underway
throughout 1979 and reached a significant plateau with a draft
strategy paper in January 1980. In general terms, the objective
of this project was to clarify, test, and refine the 1/80 stra-
tegy. To be somewhat more specific, the goals were:
1. To more precisely define the elements of the 1/80
strategy, e.g., growth targets for both career agents
and brokers; the number of people involved in each
proposed distribution system change; the cost and
productivity impacts expected from each change; the
307
intended allocation of selling effort among products;
the timing of distribution system and product chan-
ges.
2. To develop and test a set of explicit assumptions
about the functioning of the Life Insurance Division
as a business "system", e.g., the impacts of infla-
tion on costs and on sales productivity; the sens:
tivity of field personnel (in terms of recruitment,
retention, and time allocation) to compensation,
product competitiveness, and their own morale; the
effect of distribuiton system growth and turnover on.
sales productivity.
3. To develop and test various assumptions about the
external environment, e.g., the competitive position
of each major divisional product in each market where
it might be sold (in terms of price, service, and
features); future economic trends (inflation rates
and business cycles); the maximum profit margin
sustainable on each major product (given competitive
and economic conditions); the future salability of
permanent life insurance.
4. To project the specific impacts of the 1/80 strategy,
and many other prospective changes, on divisional
financial performance (profits, sales, and costs) and
on the “health” of the marketing field organization
(as indicated by compensation per person, morale, and
turnover).
5, fo refine the 1/80 strategy wherever possible, taking
into account the risks posed by adverse conditions
e.g., lower than expected inflation; declining pro-
fitability and/or salability of permanent life insu-
rance.
6. To forge a consensus among senior divisional managers
about the marketing strategy and its implementation.
To all those involved in the project, the last objective was
Paramount. The organization ana conduct of the project were
carefully designed to bring people with different points of view
toward consensus.
Exhibit 5 presents an overview of the 1980 Model. Its
structure reflects the purposes enumerated above. Unlike the
308
COMPETITIVENESS
OF PRODUCTS
PERSISTENCY
EXHIBIT 5
THE 1980 MODEL
THE COMPANY
PERFORMANCE
TARGETS
THE LIFE INSURANCE DIVISION
INFLATION
1975 Model which focused on one product group, this model repre-
sents the full range of divisional products (aggregated into six
groups). Similar to the 1975 Model, it includes the Life Insu-
rance Division's marketing organization. In the 1980 Model,
accounting is for the entire division. The important external
inputs describe performance targets for the division (e.g.,
sales, costs), inflation, persistency, and the competitiveness of
each product. This model has approximately 1000 variables.
The principal sectors of the 1980 Model are indicated in
Exhibit 6. The Marketing Sector determines the size and composi-
tion of the division's sales organization in response to recruit~
ing and assignment policies, and the turnover of personnel. Four
alternative distribution channels are explicitly modeled: the
career agent force, life insurance brokers, casualty insurance
brokers, and direct selling from the home office. For each
distribution channel, the model represents the number of "produ~
cers", their experience, and the amount of staff support and
supervision they receive. Salesforce time allocation and attri-
tion depend on such factors as commission levels, product compe~
titiveness, support and supervision, and morale.
The Sales Sector determines sales for each of six product
groups in three different markets. Sales are calculated for each
combination of product, market, and distribution channel, as a
function of producer time allocation and productivity. The
productivity of time allocated to a product depends on product
competitiveness, price, inflation, and salesforce experience. AS
310
MARKETING ORGANIZATION
@ CAREER FORCE @ CASUALTY BROKERS
@ LIFE BROKERS © HOME OFFICE
( )
PERMANENT LIFE
PRODUCTS
© SALES
DIVISIONAL ACCOUNTING
@ REVENUES
@ costs
: EXHIBIT 6
PRINCIPAL SECTORS OF THE 1980 MODEL
341
noted above, the first three of these causal factors are external
inputs to the model. However, their impacts are guite complex
and differentiated by distribution channel, product, and market.
The Accounting Sector computes revenue and costs by product
group, to yield operating earnings. Two sources of revenues are
considered -- investment income and policy premiums. The latter
is calculated from the cumulation of new business sold, decreased
by lapses and maturities. A very detailed breakdown of expenses
occurs in the 1960 Model. Many categories of head office and
field costs are calculated, based on fixed and variable compo-
nents. This degree of financial detail was necessary for several
reasons:
1. To examine how shifts in product, market, and/or
distribution channel emphasis would affect divisional
profits;
2. To determine the existence of “economies of scale";
3. To accurately portray the timing of investments and
returns under various scenarios; and
4, To analyze the overall effect of inflation on divi-
sional profits.
The 1980 Model was used to structure an important debate
about assumptions and strategy alternatives. It forced people to
be very explicit. The need to specify, initially to flow dia~
grams and later in mathematical equations, a theory of how the
division functions and how its financial performance is deter-
mined caused managers to spell out; argue about, and ultimately
agree upon dozens of critical assumptions and hypotheses. A-
B12
chieving a consensus about assumptions was a major step toward a
consensus regarding a marketing strategy.
Moreover, the model facilitated the testing of a wide range
of strategy alternatives and economic/competitive scenarios.
Approximately five hundred tests of this kind were performed with
the 1980 Model to evaluate possible elements of a marketing
strategy (singly and in many combinations). Among the areas
examined were:
1, Mix of sales among alternative distribution channels;
2. Recruitment and allocation of personnel;
3. Emphasis of various combinations of products and
markets;
4, Establishment of new distribution channels;
5. Sales growth targets;
6. Alternative budgetary constraints;
7. Methods for establishing future manpower require-
ments;
8. Actions to increase productivity; and
9. Risks posed by adverse conditions.
This project was organized to engage many senior managers in
the process of defining assumptions, designing a model, evalua~
ting the model, specifying tests to be performed, interpreting
the results, and formulating a recommended marketing strategy.
The emphasis was on detailed strategy design and strategy imple-
mentation planning. We worked with a larger and more diverse
Task Force than in the earlier project. It was chaired by the
Vice President in charge of marketing -- an outstandingly suc-
AS
cessful life insurance salesman and sales manager; a very dynam-
ic, action-oriented person. The other twelve members of the Task
Force included five additional Vice Presidents from the home
office (one of whom was our client for the 1975 Model, by then in
a more senior position) and two regional marketing Vice Presi-
dents; the remaining members were planning managers. Overall,
this was not a group of abstract thinkers. They were deeply
interested in the details of products, markets, and distribution
channels.
As the project progressed, several Task Force meetings were
devoted to a very detailed review of the model's structure and
the assumptions which went into it. Many significant additions
and refinements came from those meetings. In particular, much
effort was invested in reviewing and refining the model's fin-
ancial structure. This was necessary to ensure that comparisons
of projected financial performance from one experiment to the
next were valid.
The historical accuracy of the 1980 Model is illustrated in
Exhibit 7. For the variables shown and most other model varia~
bles, simulated values for the period 1970-1980 were within 5-10%
of actual historical data. This was an important check on the
model. It meant that model relationships, which seemed reason-
able individually, collectively produced performance which was
consistent with actual history. It indicated that no important
relationship existent over the historical period was omitted.
314
PRODUCTIVITY
1970 1975 1980
— Simulated
Actual
EXHIBIT 7
HISTORICAL ACCURACY OF THE 1980 MODEL
315
This gave the Task Force confidence that the model was useful for
Projecting divisional performance into the future.
A detailed list of conclusions and recommendations resulted
from this project. Most were incorporated in a Life Insurance
Division marketing strategy document which was issued at the end
of 1980. The final product was greatly superior to the 1/80
Graft in terms of its specificity, actionability, organizational
commitment, and projected performance. In the interim, as a
direct result of the project, senior managements' thinking on
several key strategic issues had beeh reversed. Once again, we
consider this project a success.
316
Vv. CONCLUSIONS
The preceding chapter described the "right tool" for two
different jobs. Because the two models were developed for the
same organization, with many of the same people involved (both as
client representatives and model builders), the comparison is
reasonably controlled. The following table summarizes the key
differences between the two projects:
A. The Situation 1975 Model 1980 Model
1. Problem Focus overall growth strategy marketing
strategy
2. Issues problem diagnosis; strategy
fundamental "truths" refinement
and imple-
mentation
3. Client strategic thinking; action-
Attitudes analysis-oriented oriented
concern re.
details
4, Who Else corporate manage- corporate
Had to ment manage-
be Sold ment
5. Avenues of sales targets; per- sales tar~
Imple- sonnel actions, bud- gets; per-
mentation gets; system develop- sonnel
ment; mgt. controls actions;
budgets
6. What's at urgent problem; very urgent
Stake large financial risks problem;
very large
financial
risks
B. The Model
1. Model Struc- one product; rich many pro-
ture internal dynamics ducts, mar-
ket, and
distribution
channels;
more exogen-
ous inputs
2. Size 600 variables 1000
variables
3. Type of breadth re. determin- depth re.
Detail ants of sales and marketing
profits
4. Model Evalu- detailed review; detailed
ation historical accu~ review;
racy; correct historical
forecasts accuracy
5. Data interviews; some interviews
Collection numerical data large
amount of
numerical
data
6. Model Use @iagnosis; policy eval- policy
uation; forecasting evaluation;
risk
analysis
7. Principal detailed diagnostics financial
Outputs summaries
8. Significant 100 500
Simulations
These differences are instructive. The 1975 Model focused
on a fundamental question of system behavior; the principal
clients were analytically-oriented strategic thinkers; strategy
implementation involved changes to all important subsystems. The
1980 Model was created to guide refinement and implementation of
an existent strategy; the client's mindset was quite different
because the subject was “marketing strategy", not overall busi-
ness strategy, implementation involved a more restricted set of
variables (i.e., those over which marketing managers had con-
trol).
Little wonder that the resulting models reflected those
major differences. The 1975 Model had rich internal dynamics,
encompassing all subsystems which significantly affected sales
and profits; it was used for detailed problem diagnosis and,
because of its robustness, forecasting. Ther 1980 Model was
Narrower (it focused on marketing variables), but far more disag-
gregated; many factors which had been endogenous to the 1975
Model were external inputs’ in this case; because of its disag-
gregation, much more numerical data gathering was required for
the 1980 Model; a very large number of policy tests were per-
formed; because there were many exogenous inputs, forecasting was
less meaningful than "risk analysis" with respect to variation in
the external scenarios.
The different tradeoffs between breadth and depth produced
models of roughly comparable size. The two models achieved about
the same degree of historical accuracy, but under different
circumstances and, hence, with different implications. With its
richer endogenous dynamics, the 1975 Model was more self-con-
tained and more valid for forecasting. Thé 1980 Model produced
highly reliable indications of the relative financial performance
of policy variants.
The two examples show that successful models are persuasive,
not simply to modeling technicians, but to high-level decision
B19
makers. - These executives repeatedly ask themselves: Is this
model worth the costs? Is it believable? Does it tell me impor-
tant things I don't already know? Does it give me answers I can
use? Is it an effective weapon for getting what I need to be
successful? The two case examples demonstrate how technically
different products can be entirely adequate for their respective
situations. They illustrate why I concluded that successful
modeling results from applying situation-dependent criteria of
model adequacy.
320
ll.
12.
REFERENCES
See, for example, Theil, H., Principles of Econometrics,
John Wiley and Sons, New York, 1971. *
The basic reference is Pratt, J.W., H. Raiffa, and R, Schlai-
fer, Introduction to Statistical Decision Theory, McGraw-Hill
Book Co., New York, 1965.
A more complete discussion of this conceptual model can be
found in Weil, H.B., "The Dynamics of Strategy Implement-
ation", Dynamica, University of Bradford (U.K.), Vol. 9,
Pare I, Sumer 1883.
Roberts, E.B., "Strategies for Effective Implementation of
Complex Corporate Models", in Managerial Applications of
System Dynamics, E.B. Roberts, ed., M.1I.T. Press, Cambridge,
ie Tg 7e.
Weil, H.B., "Achieving Implemented Results from System
Dynamics Projects: The Evolution of an Approach", in Ble-
ments of the System Dynamics Method, J. Randers, ed., M.I.T.
Press, Cambridge, Ma., 1979; Weil, H.B., "Effecting Strategy
Change with System Dynamics", presented at the 1981 System
Dynamics Research Conference, Rensselaerville, N.¥., October
1981, copies available from the author.
Roberts, op cit.
Weil, op cit.
Frohman, A.L. and Kolb, D.A., “An Organization Development
Approach to Consulting", Sloan Management Review, Vol. 12,
No. 1 (Fall 1970).
The problems of indirectly estimating model relationships
are discussed in Peterson, D.W., "Statistical Tools for
System Dynamics", in Elements of the System Dynamics Method,
J. Randers, ed., M.1.T. Press, Cambridge, 1980.
Ibid
This model is described in more detail in Weil, H.B. et al.
"Growth Strategy in a New Business Area: A Simulation
Analysis" presented at the 1974 Summer Computer Simulation
Conference, Houston, Texas; copies available from the auth-
or.
The process followed in this project is discussed in Weil,
“Effecting Strategy Change ....", op cit.
Bat
