RISK MANAGEMENT IN COMPLEX PROJECTS
Henry Birdseye Weil
William J. Dalton
Pugh-Roberts Associates
PA Consulting Group
41 William Linskey Way
Cambridge MA USA 02142
ABSTRACT
This paper describes the use of System Dynamics models to manage the very substantial risks
associated with complex design, development and production projects. The authors present a systematic
approach to controlling the risks associated with a project's cost, schedule and technical performance.
The steps in this process include: (a) simulation of an appropriate "ancestor" project to illuminate sources
of performance uncertainties and variances; (b) establishment of'a baseline projection for the project of
current interest; (c) simulation analyses to identify the sensitivity of the project's performance to various
contingencies (e.g. design changes, resource shortages, changes in government regulations, various
technical problems, problems with subcontractors, delays in other interdependent projects); (d) analysis
of various "hedging" and "insurance" strategies for reducing the project's vulnerability; and (e) planning
responses to contingencies for which risk avoidance strategies are not practical.
A hypothetical case is used to illustrate the approach. It includes simulation analyses that
highlight the interconnectedness of various types of risks and the tradeoffs among: (a) project
performance objectives; (b) project management policies; and (c) project risks.
L INTRODUCTION
The development of complex "systems" in fields such as aerospace, nuclear power, information
technology, and transportation poses substantial management challenges. These development projects
span long time intervals, in some cases a decade or more. Large investments, at times billions of dollars,
are involved. The risks are very high. Cost escalation, delays, and technical problems can undermine the
financial feasibility of a project, jeopardize its completion, threaten the solvency of participants, provoke
bitter legal disputes, and lead to governmental inquiries. If a major project goes wrong, the result often is
a financial and political disaster, Consider, as examples, the U.S. nuclear power, space shuttle, and naval
shipbuilding programs. Much is at stake for corporations and countries. Their long-term competitive
position and security can hang in the balance,
Complex projects are inherently vulnerable to performance problems. Indeed, it is their
complexity that makes them so vulnerable. These projects are "complex" because they tend to have:
multiple -- stages of design, procurement, construction, and testing; interacting
technical disciplines; organizations involved (prime and sub-contractors,
vendors, design agents, customers, regulators); and possible sequences for
accomplishing the work,
- 39 -
changing -- customer requirements and performance priorities (schedule, cost,
technical); government regulations and standards; work scope; technologies;
resource availability; and contractor productivity and work quality.
delays -- in discovering rework; experiencing the full effects of events and conditions
that impact the project; in perceiving true project performance; and
implementing management responses.
The key words in explaining the complexity of such projects are "multiple, "changing," and
“delays.”
Why do these projects get into so much trouble? Because of their complexity, they are difficult
to control under the best of circumstances. Dysfunctional behavior by the parties involved -- for
example, ignoring major risks, "success oriented” planning, poor initial project definition, reluctance to
admit problems, excessive pressures, confrontation instead of cooperation -- often amplifies the
vulnerability of such projects to performance problems. And once initiated, there is a tendency for
problems to "snowball" out of control.
The challenge of risk management in complex projects involves early identification of major
tisks, systematic analysis of these risks, strategic control of the project's risk exposure, and continual
learning. Until recently, there has been insufficient systematic analysis of, and learning from, past
experience with the problems of complex projects. Managers lacked tools powerful enough to effectively
analyze and control such projects.
This paper describes a new approach to managing project risks that is based on computer
simulation modeling. It begins by discussing the nature of project risk and reviews the sources of risk
revealed by analyses of more than sixty complex development projects. The paper then outlines a
systematic process for controlling project risks. The next section presents a hypothetical case of project
risk analysis. This example uses a model of an aerospace design and production project, but is
representative of a wide range of complex projects.
0. THE NATURE OF PROJECT RISK
Risk arises from uncertainty, Outcomes that are entirely certain are riskless, while outcomes
that are difficult to attain or to predict are "risky." This is particularly true if there are significant gains or
losses contingent on the outcome. In a practical business sense, a “high risk" situation exists when: (a)
significant amounts of money, reputation, or other important assets are at stake contingent on a future
outcome, that (b) is difficult for management to achieve or forecast. Complex projects are a perfect
example.
Depending on the nature of the project, the "assets" at stake can include profit or loss on the
contract, the contractor's reputation and customer relationships, the customer's operating costs and
effectiveness, the market positions and profits of all involved, the safety and welfare of the public, and in
the longer term, national competitiveness and security. Thus, the stakes often are high, but a favorable
outcome can be very difficult to achieve.
The outcome of a complex project can be measured in terms such as cost, timeliness, technical
formance, quality, value for money (performance/cost), and social impacts, The major sources of risk
quality
= 40 -
are factors which both: (a) have a significant effect on project outcome; and (b) are surrounded by
significant uncertainty. Otherwise, as a practical matter, management needn't be concerned.
Analyses of more than sixty projects in the aerospace, electronics, computer software,
telecommunications, shipbuilding, electric power, and transportation industries have revealed a set of
recurring, major sources of performance risk. The most important are:
© inadequate initial project definition
* overly ambitious schedules
bd success-oriented management policies
* complex project organizations
* rapidly changing technologies
* resource shortages
bs interdependencies with other projects
* down-stream design changes
© problems with vendors or sub-contractors
¥ changes in number of units to be produced
® changes in government regulations
¥ forces majeures
These factors are significant for several reasons, Some -- for example, ambitious schedules and
success-oriented budgets -- are almost unavoidable by-products of highly competitive markets. A
contractor who is risk averse in bidding will not win many major projects, at least not until customers
become much wiser and more sophisticated about their own interests.
Other factors reflect unrecognized deficiencies in project management. For example, multi-
company project consortia are increasingly common. But project control systems rarely take explicit
account of the slower information flows, the more cumbersome decision processes, the extra time
required to implement actions, and the diminished control over resources that typically exist in complex
project organizations.
Still others on this list are largely beyond management's control. Major advances in
technology, a basic re-think by the customer of his requirements, or an incident that causes regulatory
authorities to "go back to the drawing board” may obsolete aspects of project definition or work already
accomplished. An inter-dependent, but separately contracted program (the engines for a new aircraft)
might experience delays or technical problems. A key sub-contractor may have its work interrupted by a
strike, a fire, or an act of terrorism. The occurrence of such factors, particularly their timing and
magnitude, is difficult to predict.
These risks can be, and often are, inter-connected. For example, consider the following, all-too-
familiar, scenario. A contractor wins a project where, initially, the end-product is not fully defined.
Soon it becomes apparent that changes and additions to the customer's specifications are reducing
productivity and causing extra work. The project falls behind its ambitious schedule. The delays expose
the project to unexpected technological and regulatory changes, and hence more rework. There is no
provision in the project schedule and budget for any of this. Further delays and cost escalation prompt
the customer to change the design and the number of units to be produced, in attempt to stay within
budget limits. The resulting disruption of the project aggravates vendor performance problems. Then the
a
contractor, in an attempt to salvage the project, agrees to an overly ambitious, success-oriented, and
potentially disastrous "rescue plan." Frequently, there is this pattern where the incidence of one risk
exposes the project and increases its vulnerability to risks from other sources.
The impacts of these risk factors are delayed, non-linear, indirect, and self-reinforcing. Hence,
before the fact or even after they have occurred their full significance is difficult to perceive. Worse still,
because their effects can be quite counter-intuitive, these factors often prompt dysfunctional management
actions. Typical examples include over-staffing in an attempt to achieve the schedule, forging ahead
with downstream work even though the prerequisites are not ready, turning testing into a public relations
event, and clinging stubbornly to infeasible performance targets.
Figure 1 is a diagram of some key determinants of project performance. The relationships
depicted in this figure are so general that they pertain to a very broad range of projects. The existence of
many circular paths of cause and effect -- involving combinations of risk factors, management decisions,
and performance variables -- should be clear. It is through these "feedback loops” that the self-
reinforcing dynamics build and the indirect effects propagate. Project performance catastrophes arise
from the combination of: (a) high levels of risk; (b) interconnected risks; (c) the non-linear impacts of
risk factors; and (d) self-reinforcing degenerative dynamics.
‘SeouENce
ohne
sea, yoiwon ates, snooucron mm
Gunes ——> betkeey —> hiusagury—> prosuetiny —> herent
Nomoone Te : secouronto
hia
BEB once | RE scource RBS
Figure 1. Key Determinants of Project Performance
Reducing a project's vulnerability to performance problems requires a multi-pronged attack on
the degenerative dynamics. It involves a carefully developed set of actions that reduce: (a) the key
uncertainties; (b) the project's sensitivity to disruption; (c) the magnitudes of disrupting factors; and (d)
the likelihood of dysfunctional management responses.
This approach is distinctly top-down and strategic in its philosophy. Strategic risk management
starts by recognizing the major sources of overall performance risk and the inter-connectedness of key
risks. Based on that understanding, one defines and evaluates various options for desensitizing a project;
in effect, partially immunizing it against certain risks. One also assesses the value of reducing the range
of uncertainty surrounding significant risk factors. For example, how much is it worth to have extra
- 42-
quantities of critical personnel and materials on hand? To have an agreed cutoff date for design changes?
To have advance warning of new government regulations?
Furthermore, this approach recognizes the typical dysfunctional management actions that occur
in complex projects as problems begin to "snowball." Where possible, it seeks to prevent or mitigate
such actions. With that objective in mind, one looks carefully at the important performance tradeoffs.
What is the tradeoff between project schedule and cost? How much can be saved by adopting the cost
minimizing schedule? What is the tradeoff between productivity (in the narrow sense) and work quality?
Is it better in terms of project completion time and costs to apply fewer, but more experienced personnel?
Also in this spirit, strategic risk management recognizes the value of thoughtful contingency
planning. Dysfunctional actions often occur under great pressure at times of crisis. It is not possible or
cost/effective to immunize a project against all significant risks. But the existence of good contingency
plans greatly increases the likelihood that management will act correctly if a seriously disruptive situation
were to arise. For example, what should management do if a key sub-contractor is hit by a strike? If
government safety standards suddenly were to change?
i. A SYSTEMATIC APPROACH TO CONTROLLING PROJECT RISKS
Over the past fifteen years, System Dynamics has emerged as a powerful methodology for
analyzing the performance of complex projects. Using the techniques of System Dynamics, computer
simulation models of a very wide range of projects both in North America and Europe, have been
developed. Unlike other methodologies, System Dynamics explicitly addresses the dimensions of
complexity listed above (remember "multiple," “changing,” “delays"). System Dynamics models can
answer many critical questions about project performance:
> why did certain problems occur?
* what would have happened without certain events or conditions?
® what will performance be under a specified set of circumstances?
* what if management took various actions?
These capabilities make the System Dynamics methodology a very appropriate and effective
tool for strategic risk management. A System Dynamics model can be the basis of a systematic process
for identifying and controlling the most significant project risks. The steps in this process include: (a)
simulating an "ancestor" project; (b) establishing a performance "baseline" for the project of interest; (c)
identifying performance sensitivities; (d) defining the major sources of risk; (e) analyzing options for risk
reduction; (f) evaluating risk/performance tradeoffs; and (g) preparing contingency plans. The next
section contains a hypothetical case example that illustrates the process.
Iv. PROJECT RISK ANALYSIS -- AN EXAMPLE
Imagine the following situation. Company X has won a very competitive negotiated bid for the
design and prototype production of wing sets for a commercial aircraft manufacturer. It is clear from the
beginning that this project will be a real challenge, given the low fixed price negotiated by the aircraft
manufacturer, the aggressive schedule required, the inherent technical difficulty of the work, and past
= 43 -
Company X experience with projects of this type. Since this project is so critical to Company X's overall
financial performance and reputation over the next several years, the wing project manager has decided
to take a very proactive approach to risk management.
The computer simulation model of the wing project expands significantly on the rudimentary
structure in Figure 1, forming a comprehensive representation of the key interrelated activities on the
project. These activities include design of the wing, procurement of components from outside vendors,
and the production and testing of the prototype wing sets, The wing project model is a descendant of
over sixty prior project models, so the structures used to represent these project activities have been
tested and validated over a wide range of project types and conditions.
A baseline projection for the wing project has been established by first calibrating the model,
using known historical performance on a similar prior project at Company X, and then carefully
modifying those factors (related to work content, project conditions and management style) expected to i
differ significantly between the wing project and this "ancestor". |
This baseline projection raises immediate concems for the wing project manager, since it shows
significantly higher costs than allowed for in the fixed price project bid. With the assumed project
structure, conditions and management policies, the simulation shows that incurred costs will total about
$139 million compared to the fixed price of about $100 million. The project also overruns its schedule
by 50 weeks, beyond the originally planned four year duration. This baseline shows lower than expected
productivity and higher levels of rework (relative to the assumptions in the bid) in both engineering and
prototype production. Unfortunately, the dynamics underlying this performance do not seem unrealistic
to the wing project manager. In engineering, the low productivity results from a combination of factors:
the original specification of the design, the experience levels of the design engineers, the availability of
necessary vendor design, and the amount of design work being performed out of sequence. In
production, the problems include late or poor quality design products, late material from vendors, and the
average experience level of the production staff. To some extent, these sorts of problems were
anticipated in the original bid, but not to the extent they occur in the baseline projection.
Given that this baseline projection assumes "business as usual" out to the end of the project (i.e.
No special initiatives), the project manager feels that this performance can probably be improved upon.
His primary concem now is that, as bad as this baseline looks, it doesn't take into consideration a whole
set of risks that he knows to be common on this type of project. For example, one major area of risk
involves the behavior of the customer, who might:
m fail to properly define the design task to be performed
* impact design progress by indecision on key design issues
* delay the release of drawings due to late approvals
* fail to make available promised customer-furnished equipment
~ add scope to the engineering or production efforts
make changes within the existing project scope
‘These customer-related risks are interrelated, in that a customer who is “difficult” will tend to
impact the project in many or all of these ways, rather than just one or two. To test the potential impact,
the baseline simulation is altered to reflect some degree of each of these additional risks, creating a
"Medium Risk" scenario. The result is an increase in overall project cost from $139 to $179 million, and
a further extension of the schedule by 26 weeks! Direct labor hour expenditures are driven up from the
baseline projection by reduced productivity and, more importantly, increased levels of rework in both
= 44 -
i
i
i
engineering and production. In engineering, this increased need for rework results primarily from more
design work being performed out of sequence, lowered design team morale, and more incorporation of
upstream errors or omissions in downstream design products. In production, labor hours are driven up by
reduced availability and quality of the design products supplied by engineering, and a lowered average
experience level among the production work force. Support labor hours on the project also increase,
primarily because the project lasts longer.
But wait, there's more,..the project manager ruefully admits to himself that there is another
entire category of risks common to this sort of project. These risks include:
*
lower than expected regional availability of labor
* higher workforce attrition rates
* slower vendor delivery (of both design products and material)
slower management perception of variations in actual productivity
slower discovery of rework in engineering and production
*
*
When the wing project simulation is re-run with the customer-related risks plus these resource-
and project control-related risks (creating a “High Risk" scenario), the picture is even gloomier than
before. Overall project cost increases from $179 to $265 million, and completion is delayed an
additional 45 weeks! Most of this cost increase comes in prototype production; engineering labor hours
increase by "only" 7%, whereas in production the increase is 120%. Engineering productivity is reduced
by delayed vendor design products needed for the primary design work, and the effects from slower
discovery of design rework. Increased production labor hours result from both reduced productivity and
increased rework levels. The drivers here include lowered production workforce experience levels (from
hiring more deeply into the available regional labor pool, in part due to the higher attrition rates) and late
vendor material deliveries. Figures 2 and 3 below show engineering staffing and productivity for the
baseline projection and these two risk scenarios (with time on the X-axis in decimal format).
te eee ook
TIME
Figure 2. Engineering Staffing
Having defined the two risk sets (the first customer-related, and the second resource- and project
control-related), the wing project manager believes he has now considered all those sources of risk which
are both high impact and of fairly high probability. Clearly, performance on the project is very sensitive
to these risks! The next logical question is; what might be done to minimize or reduce these risks, and
as YS ce
Engineenng Productity (Drawings/Work-Yean)
teak ceeetdied
snot Pee
ThE
Figure 3. Engineering Productivity
how much would the project benefit? The project manager can identify five potential areas for initiatives
aimed at risk reduction:
schedule -- What if the schedule is slightly extended, and the amount of schedule overlap
between the engineering and prototype production phases of the project is
reduced, to give the design and dependent procurement activities more lead
time over production?
workforce management - - What if hiring onto the project is carefully managed (and
constrained where necessary) to maintain higher average experience
levels in the engineering and production workforces?
test program -- What if a more aggressive test program is pursued, including an earlier
Start to testing and a stronger tolerance for test failures in order to maximize
engineering and production rework discovery?
customer relationship - - What if, through proactive cooperation and early relationship-
building with the customer, key concessions can be gained which include the
assurance of timely design reviews, an agreed design freeze at the end of the
first year of the project, limitations on scope increases and design changes, and
no excessive pressures exerted on the project manager by the customer?
improved technology - - What if a more powerful Computer-Aided Design (CAD)
system is implemented as both a basic tool in the wing design and for
visualization and optimization of the prototype production process?
The effect on project performance of each of these management responses, implemented
cumulatively on top of the "High Risk" scenario, is summarized in Figure 4 below. While the wing
project is very sensitive to the risks analyzed, the responses tested here are clearly quite effective in
neutralizing these risks and even significantly improving on the project performance relative to the
baseline projection. Individually, these responses have varying amounts of leverage on engineering and
prototype production cost, the delivery schedule achieved, and the delivered prototype quality (measured
inversely here by the percent of undiscovered rework still present in the delivered wing sets). For
example, the customer relationship and technology responses are particularly effective in containing
engineering costs, while the schedule and workforce responses are most leveraged with respect to
prototype production costs. The schedule, test program, customer relationship and technology responses
all help control the schedule overrun. The schedule and test program responses are most effective at
increasing delivered quality. An important note to these simple analyses is that there has been no
consideration of any direct costs of implementing these responses -- for example, the purchase price and
training costs associated with the introduction of a new CAD system at Company X.
Baseline Medium High Schedule Workforce Testing Customer Technology
Risk Risk Response Response Response Response Response
Total Project 139 179 265 230 «194 174 120 101
Cost ($M)
Engineering 40 59 63 57 55 54 32 25
Labor Cost ($M)
Production 44 59 = 130 104 75 61 38 28
Labor Cost ($M)
Schedule 50 76 121 94 89 53 19 4
Overrun (Weeks)
Undiscovered
Rework at 12 10 12 05 AL 03 AS 08
Completion (%)
Figure 4. Summary Simulation Results
Lastly, the wing project manager asks the obvious question: if these responses improve project
performance in the presence of the defined risks, should they be implemented no matter what the
circumstances? Implementing these responses in the absence of risks (using the baseline projection as
the starting point) shows the project being completed at about break-even cost (approximately $100
million), and with a small schedule overrun - essentially the same result achieved by these responses in
the presence of the defined risks. The difference in project cost, schedule and delivered quality made by
these responses therefore depends directly on the extent to which the defined risks actually occur. The
cost of implementing these responses can be appropriately viewed as an "insurance" investment, where
the return from this investment is determined by the conditions under which the project is performed.
v. CONCLUSIONS
Several fundamental conclusions can be drawn from our simulation-based analyses of this
hypothetical aerospace project. While the analyses reported here are limited in scope, we are reassured
by very similar findings from analyses of other complex programs in such diverse areas as shipbuilding,
nuclear power, information systems, and transportation.
we AF
1 Design changes drive up project cost. Particularly in an environment characterized by
open competition, new technologies, and evolving governmental regulations (e.g., regarding safety and
the environment), some changes in designs, specifications, and standards are inevitable. However, it
should be clearly recognized that the volume of changes has a highly non-linear, adverse impact on
project costs. Moreover, the later in a project the changes come, the greater their adverse impact.
Several important policy conclusions follow from this, for both the project manager and the
customer. Changes to the design, specifications, and standards of a complex project should be minimized
wherever possible, The perceived benefits from design "improvements," exploiting the latest technology,
meeting new user requirements, or complying with higher safety standards must be explicitly weighed
against the increase in project costs. This cost assessment should be comprehensive and precede
acceptance of the changes. A frequent mistake is to accept such changes as “obviously desirable" or "of
minor cost consequence” and simply forge ahead. Conflicts of interest often arise, and must be resolved
equitably. The simulation methodology described in this article is a powerful means of fast, accurate
quantification of the labor and schedule consequences of proposed changes.
In any event, there needs to be a cut-off to changes at some appropriate point. This cut-off
might be the product of contractual or less formal negotiations between the project manager and the
customer, or might require agreement with governmental regulators. Simulation analyses indicate that
the appropriate cut-off point is before the start of significant production or construction activities. For
example, a conspicuous feature of both the Japanese and French nuclear power programs -- both have
achieved cost and schedule performance far superior to the U. is an early cut-off to changes.
Dy Parallelism increases project risks. "Parallelism" means substantial overlap among
sequentially inter-dependent design and production phases of a project. Sometimes called “fast track," or
"design while building,” this strategy permits aggressive project scheduling. Unfortunately, simulation
analyses show quite clearly that significant parallelism greatly increases the vulnerability of a project to
cost escalation and, perhaps surprisingly, to schedule delays. Parallelism increases the amounts of design
and construction rework. It substantially amplifies the adverse cost impacts of changes. And it makes
the project less able to contend with any problems from subcontractors or vendors.
In an environment of continual or, worse, accelerating changes to a project's design,
specifications, and standards there is a logic to getting the project finished as quickly as possible. This is
the only way to stop the flow of changes! However, it is a strategy for making the best of a bad situation,
If changes are minimized, and if they are cut-off entirely in advance of the start of production, the cost
minimizing schedule has only moderate parallelism. One key element of "advanced" Japanese
shipbuilding techniques is holding off on production until design is essentially complete.
3. There is an optimal project schedule. As suggested above, each complex development
project involves a significant cost/schedule trade-off. In other words, there is a cost minimizing
schedule, and attempts to follow significantly faster or slower schedules will lead to higher project costs.
Faster schedules involve excessive rework and disruption from changes, vendor problems or resource
limitations. Slower schedules expose the project to more changes, workforce turnover, overhead costs,
and inflation,
The optimal schedule depends on the environment surrounding the project. Difficulty in
containing changes, rapidly evolving technology, substantial inflation, and high interest rates all suggest
a faster schedule. A complex project organization (e.g., an international consortium of many contractors
and sub-contractors), tight labor market conditions, incomplete customer requirements, competing
demands on key resources, dependencies on the performance of some other complex development project
|
|
CC BY-NC-SA 4.0