A System Dynamics Model of Government Engineering Support
During the Development Phase of a Military Acquisition Program
J. E. Bartolomei
United States Air Force
Wright-Patterson AFB OH, 45431
(937) 255-6522; (937) 255-3672
Due to the increase of system complexity and the existing draw down of manpower
allocations, today’s acquisitions environment desperately needs a systems approach to decision
making. Many studies have been performed to model the entire government acquisition
environment. Due to the high degree of aggregation, front line decision makers have had no
use for the information these models provide.
This research focuses on the Air Force’s largest functional support element in aircraft
systems development, engineering. | will only consider one phase of the government acquisition
cycle the Engineering, Manufacturing, and Development (EMD). This is the development cycle,
which begins with initial contract award (Milestone II), through the production approval
(Milestone III). This model is a building block to ultimately help leaders determine the required
skill-set and manpower to perform activities which can meet short term requirements while
minimizing the intrinsic cost, schedule, and performance risks associated system development.
The simulation presented will be used by the as an alternative decision making tool for
manpower allocations for government organic engineering workforce during an eight year
development effort.
Key Words: military, acquisition, simulation, manpower, engineering, system dynamics,
Functional Analysis System Technique (F.A.S.T), Management Causal Matrix (MCM)
Introduction
Since end of the Cold War, the U.S. Government has made concerted efforts to reduce
the costs of its acquisition processes. One strategy has been the elimination of activities that are
not necessary or cost effective. This has been a difficult transition, as the entire Defense industry
wrestles with conducting business within a reformed acquisition environment. No government
institution has felt the ramification of the new business practices as much as the System Program
Offices (SPOs). In the USAF, the SPO is responsible for managing the entire lifecycle of a
designated system. This includes the initial concept exploration and requirements generation
through the development, production, and retirement of the system.
For years, SPOs participated in many activities that overlapped contractor efforts. The
generation of a Government Statement of Work, Government specifications, and other standard
practices made goverment acquisitions very costly and inefficient. SPOs were filled with
military and government service engineers, financial managers, program managers, and contract
specialists. All were assigned to duplicate contractor effort in managing the cost, schedule, and
performance of the system.
In 1991, the Government A cquisition community began to rethink the manning and skills
mix for weapon system development efforts. AFSC (Air Force Systems Command) Commander
General Y ates and the Acquisition leadership were faced with a problem. General Y ates stated,
"AFSC does not have a credible, quantitative process to determine the right numbers and skills
of people to properly match acquisition workload, nor can it effectively defend the acquisition
workforce" (Y ates:1991).
A team embarked on a three-year project to determine a process for managing the
existing acquisition force. The strategy to attack the problem was first to utilize other
govemment manpower models in combination with "top-level intellectual involvement" to
develop USAF-specific manpower models. The commander believed that it was necessary to
determine, quantify, and defend the value of an organic acquisition workforce; otherwise, the
acquisition community would continue to dissolve. Although the effort received great visibility
and attention, the team was unable to determine manning models that could defend the value of
maintaining an organic workforce.
In 1995, the Secretary of Defense (SECDEF) mandated several changes to the
govemment acquisition process. The Air Force translated these mandates into several
"Lightning Bolt Initiatives" released by the Office of the Assistant Secretary of the Air Force for
Acquisitions (SA F/A Q) to facilitate a reform process due to the dismantling of the Defense
budgets. Lightning bolt number 3 was entitled, "System Program Office 'Slim-Fast' Plan." The
goal of the Lightning bolt was to "eliminate all unnecessary SPO activities thereby reducing
program costs" (Druyun:1995).
The methodology for attacking the problem was to investigate programs that had
demonstrated success in a reform-like environment. One arena was classified acquisitions, or
Special Access Required (SAR) programs. Historically, SAR programs had successfully
performed their mission with a substantially smaller workforce than unclassified SPOs
(Druyun:1995). A tiger team assembled to develop tenets applicable to unclassified programs
which could help SPO Directors (SPD) to achieve efficiencies in SPO operations and ultimate
reductions in manpower.
New SPO manning goals were established setting new thresholds for manning acquisition
programs. Large development programs that previously employed over 300 military and civil
servants were limited to 140 total personnel. Systems in the production phase that once had 150
employees were challenged to employ 50. The goals represented the total workforce manning,
including government and support contractor resources. These goals were quite aggressive, up
to a 90 percent reduction for some efforts. No apparent analysis was presented to verify or
validate these numbers.
After two years of interviews, assessments, and analysis of streamlined acquisition
practices, the Lightning Bolt team presented a list of "inherent government functions" to be
retained by aSPO. These functions include the following:
I. Contracting - The processing of the contractual documentation and obligation of the government to pay
for provided materiel and/or services.
II. Program Management - Evolves around understanding and managing risk through evaluation of
program cost, schedule, and performance. A key component of this area is technical assessment -
determining pass/fail of intermediate and final test criteria.
III. Requirements Determination - Setting the quality, quantity, and performance characteristics required
from a procurement.
IV. Budgeting and Financial Management - The programming for, obligation of, and accounting for
SPO funds.
It was determined that all other activities could be "source through the prime contractor,
support contractors, or eliminated completely" (Lightning Bolt #3:1995). The functions were
then reduced to several tenets described by the team as approaches which, if implemented
intelligently, would lead to streamlining improvements. These tenets are as follows:
A. Aggressive Risk Management - A move away from risk avoidance toward risk management
B. Insight vs. Oversight - Understanding contractor processes and managing the program via process
metrics
C. Clear Accountability in Design (CAID) - To the extent practical, the Air Force assumes no design
responsibility below the functional baseline (system specification) level until the end of EMD
D. Integrated Weapon System Management (IW SM) - Non adversarial team membership to include the
contractor and user
E. Reduce Contract Data Requirements List (C DRLs) - Use existing contractor systems for insight
F. Communication of Performance specifications - What we want the system to do, not how to build it
G. Flat SPO structure - accessibility to the Program Director
H. Maximum use of electronic media
I. Maximum use of commercial off the shelf (COTS) items— Only develop what needs to be developed.
Additional tenets were provided to the government system program director (SPD) to
consider. These embrace the following key elements:
. Maximize the teaming efforts to include other government agencies
. Establish long-term government-contractor relationships
. Contractor develops a logistics support plan focused on 4 critical parameters
A
B
C. Minimize the number of contracts/line items and make them milestone based
D
E. Minimize and refocus engineering staff
FE,
Borrow expertise rather than maintain within the SPO
Openly, the team admitted that the "report is not a ‘cookbook’ or a mathematical model
for SPO sizing.” They explained that their intent was to present a “tool box” for SPDs, which is
to be “applied thoughtfully, based on the careful judgment of the program office personnel"
(Lightning Bolt #3:1995).
The goverment acquisition community generally agreed that these steps could optimize
certain portions of the acquisition cycle. However, many of the tenets were difficult to integrate
in existing programs and some were unrealistic for some programs.
It was interesting that the tenets removed the engineering function. Government
engineering support was one of the largest resources previously utilized in the system programs
offices. Removing engineers would immediately reduce the manpower load by 50 percent for
development activities. This may be attractive for the acquisition reform leadership, but it was
unreasonable for the services and the contractors who rely on the engineering support to provide
independent assessment of a program’s technical performance. Many philosophical debates
persisted throughout the acquisition community on the usefulness of maintaining a large
engineering function within the SPOs. However, no analytical solutions could quantify the value
of organic engineering support.
System Dynamics Modeling of G overnment Acquisition Systems
In the Air Force, I located five efforts at modeling all or part of the DoD acquisition
system that were performed in the late 1970's and early 1980's. These include Elder and Nixon,
Lawson and Osterhaus, Kaffenberger and Martin, Whittenberg and Woodruff, and Gonzalez and
Sarama. The first 5 studies were performed during the Cold War ams race. Whispers of U.S.
Goverment acquisition reform were evident in the text, but there were irreconcilable differences
in fundamental assumptions, model boundaries, political, and environmental issues to further
their research.
Elder and Nixon developed a conceptual model of the USAF program management
activities being performed at A eronautical Systems Division. They did not produce a completed
model (Elder and Nixon:1976).
Lawson and Osterhaus developed a six-sector model depicting the entire DoD acquisition
process, circa 1980. The causal relationships described in the text were not well understood or
parameterized to be useful. The process was completed by a series of interviews with Senior
Defense acquisition leaders and executives (Lawson and Osterhaus:1978).
Kaffenberger and Martin added four more sectors to Lawson and Osterhaus’ model.
Their model captured and described several causal relationships and systemic behaviors of the
US- USSR ams race. Their contribution rested primary on the structure of the system and
provided no validation or analysis (Kaffenberger and Martin:1979).
Whittenberg and Woodruff provided a capstone thesis, which completed the work of the
previous studies as well as the work of Bechtel and Sweeney. Their comprehensive model of the
acquisitions was quite an undertaking. Like the others, the model was designed for, and
aggregated at, the highest level. Though the model was mun and the provided some useful
information to acquisition policies for the cold war environment, little was applicable to today’s
current world environment (Whittenberg and W oodruff:1982).
Gonzalez and Sarama developed a causal model that looked at the acquisition process
from a Washington DC policy perspective. They modeled sectors including the political,
economic, contractor, and multiple government agencies at a very high level of aggregation.
They were unable to develop a computerized model of the system, but the well-defined causal
structure provided insight to the complexity of the system (Gonzalez and Sarama:1999).
The very high level of aggregation and the enormous scope of these projects provided
little detail or information for frontline managers to make decisions.
Model Objectives
The general objective of this research was to develop a conceptual understanding of the
complex, dynamic nature of a government development program, and subsequently develop a
computerized model that reflects the structure of the engineering function within a system
program office (SPO). In the USAF, the SPO is responsible for the managing the lifecycle for a
designated system. The lifecycle includes the initial concept exploration and requirements
generation through the retirement of the system. The SPO consists of US government employed
program managers, engineers, financial managers, and contract specialists.
The specific objectives were as follows:
1. Identify the structure of the system using traditional systems engineering tools
such as Functional Analysis Systems Technique and Design Structure Matrix in concert
with a system dynamics approach.
2. Isolate the interactions and influence of the components and variables within the
system.
3. Describe the decision structure that determines the allocation of engineers to the
SPOs.
4, Construct a mathematical model that represents the components, relationships,
information flows, and decision policies of the system.
5. Develop a computerized model that can be used as a learning laboratory for
policy analysis and development to optimize engineering support of a high-risk USAF
development activity.
6. Identify areas of sensitivity or critical issues in engineering manpower allocation.
Methodology:
Information Gathering: A military development program is a complex and poorly understood
system of multiple interrelationships and interdependencies. The role of the engineering
function within this structure was extremely difficult to define initially. To gather information
about the system, I employed group and individual knowledge elicitation techniques. These
methods and tools are described in the following sections.
F.A.S.T.: Defining the model boundary and determining an adequate degree of aggregation was
the first task. The Functional Analysis System Technique (F.A.S.T.) was used as a group
knowledge elicitation method. (Bartolomei et al:2001) Several F-22 systems experts developed
to describe the existing organizational structure by function. Over 150 functions performed by
the program office were defined using the F.A.S.T. methodology. (Fowler:1990) Of the 150
functions, the team determined that engineering support was only required for certain functions.
Based on the F.A.S.T. exercise it was determined that the value of engineers to a program
was to identify risk, mitigate risk, and broker information. The goal of the model was to
determine the proper number of engineers required to achieve the government's cost, schedule,
and performance goals and ability to adequately inform the customers.
Once the functions were identified, the team identified all of the daily activities that are
performed to support the functions, the customers of the functions, and the required manpower to
support the functions. Measures of performance for the activities within the system were also
identified. As a result of this activity, all of the variables existing with in the system had been
defined. The next step was to determine the interrelationships, which existed within the system
and the predicted behaviors of the system.
Management Causal Matrix: A Management Causal Matrix was used to organize the variables
and to visually represent the interrelationships that exist in the system. The purpose of the
Management Causal Matrix was to provide a visual structure of the system. The matrix form is
very conducive for identifying interrelationships within the system. Once this was
accomplished, the boundary of the system was defined. The next task was to explored the
dynamic relationships and information feedback found with in the system. Attached is a diagram
of the management casual matrix of the engineering ina SPO. (Bartolomei:2001)
Individual Interviews: Several interviews were performed with experts in the govemment
engineering community. Interviewees included the following individuals: the USAF
Aeronautical Systems Engineering Director and former F-22 Program Chief Engineer, the USAF
Aeronautical Systems Chief System Engineer, the former F-22 SPO Chief Engineer, the Director
of Engineering of the USAF Aging Aircraft Program, and the current F-22 SPO Deputy Director.
Prior to the interview process, several key dynamic relationships were isolated to present
to the interviewees. Each interviewee was permitted to describe the behavior of the presented
interrelationship and provide further comments in other areas that they felt needed to be
addressed. The interviewing process was iterative, and each interviewee was allowed to see the
comments of the other individuals. As a result, the individuals reached a consensus on the
critical behaviors to be modeled. The method was similar to Ford and Sterman’s expert
knowledge elicitation method (Ford et al:1998).
Define System Behaviors: Behaviors identified during the interview sessions are identified in the
following sections.
User Requirements Over Time:
This variable looked at the change in user requirements over time. Many weapon system
development activities begin with a fuzzy set of user requirements proposed for the development.
For aircraft development, many requirements include technologies that are unproven and
extremely high risk. As a result, program managers and govemmental agencies often manipulate
the requirements during the development phase.
Figure 1 is a graphical representation of varying user requirements. Above 0 indicates a
net gain of requirements or a requirement increase. Below 0 indicates a decrease.
= = = -User Requirement
Recpirements
\
\
J
CDR GRD FLT
User Requirements Vs. Time con
Figure 1. User Requirements Behavior
CDR represents the critical design review. GRD, represents ground testing, and FLT
represents the beginning of flight test. The graphic shows that user requirements are generally
stable until CDR. CDR is the first time that the program clearly communicates to the user the
design proposed to meet the user requirements. Many times, there are discrepancies in
requirement definition, and requirements are often increased. The CDR timeframe is also one of
the last times in the process that the user can add work to the contract before it is too costly. This
is why a slight increase in requirements is illustrated.
Ground testing, GRD, is a significant milestone for many programs in that it signifies that
the first articles have been produced. Immature technologies and gold plating requirements are
addressed. The users often relieve requirements to an acceptable level of risk, due to cost and
schedule constraints. This is illustrated by the slight decrease of the overall system requirements
during the last half of the development program.
Technical Maturity vs. Time:
Technical maturity, for most programs, has a goal-seeking behavior as seen in Graph 2.
100% P------------=====
z
rf
2
3
5 = = = User Requirement
& Tech Maturity
CDR GRD FLT
Tech Maturity Vs. Time Time
Figure 2. Tech Maturity Behavior over Time
This graph describes the percent of the system meeting the user requirement. This is a
highly aggregate representation of the system. The inherent risks of multiple subsystems
interacting makes this metric almost impossible to quantify since many portions of the system
exceed user requirements and others will never approach the desired capability. As a whole, the
interviewees accepted this to be an accurate representation of technical maturity over time.
Program Risk vs. Time:
Risk performance versus time is another dynamic relationship that is difficult for which
to gather data and quantify. Program risk is defined as the unidentified, “unknown, unknown”
factors inherent to the system, which affect cost, schedule, and performance. The behavior of
this variable is entirely system dependent. The role of government engineers is to provide
expertise to identify these risks and to help the contractor mitigate them.
= = = Program Risk
Program Risk Vs. Time Tine
Figure 3. Program Risk Behavior over Time
Figure 3 illustrates the behavior over time. Initially, a development effort has identified a
certain amount of programmatic and technical issues that are labeled as risk. Depending on the
maturity of the technologies associated within the system, risk is identified through CDR until
GRD. Once the initial development articles are produced, fewer risks are identified. The
development team, in concert with the users, will look at requirement relief and the application
of management reserve to mitigate the risk. The result is marginal risk as the development
program begins to transition into production.
Number of Program Inquiries vs. Time:
For highly visible and politically charged military programs, information brokerage is
one of the most important services a SPO provides. Engineers who understand the critical details
of system performance and technical risks are called upon to answer other government agencies
and Congress. The interviews reveal that answering inquires produces an S-curve behavior. In
the beginning of a program, there is little external interference due to the immaturity of the
weapon system. CDR is when programs generally start seeing inquiries. These inquiries come
from various groups and generally deal with design critiques, new requirements, and new cost,
schedule, and performance estimates. Programs offices see an exponential growth and then a
tapering of inquiries as the program approaches Milestone III and the production contract
decision, the biggest decision for any acquisition effort.
4 = = = Program Inquiries
CDR GRD FLT
a Time
#of Program Inquiries Vs. Time
Figure 4. Program Inquiry Behavior over Time
Engineering Manpower vs. Time:
The next relationship presented in the initial interview was the allocation of engineering
manpower over time.
Figure 5 illustrates the common approach to engineering manpower that reflects the
contractors manpower. Initially, manpower is added to the program at the beginning of the
program and reaches a peak near the CDR. Once the design is set both the government and
contractors dissolve the engineering workforce by two/thirds at completion.
= = = Manpower
l j ; Tine
Engineering Manpower Vs. Time
Figure 5. Engineering Manpower Burndown vs. Time
Cost to Mitigate Risk vs. Time:
This relationship illustrates the cost associated with making changes and the mitigation of
risks during the development phase. Until CDR, the cost of mitigating risks is relatively low.
After CDR, the same risks become increasingly more expensive. Early identification and
mitigation of risk is essential in minimizing program costs.
= = = Cost to Mitigate Risk
Cost: to Mitigate Risk Factor
s
N
CDR GRD FLT
Ti
Cost to Mitigate Risk Factor Vs.Time
Figure 6. Cost to Mitigate Program Risk over Time
Risk Impact to Schedule vs. Time:
Similar to the cost factor previously mentioned, there is a varying impact of risk to a
program's schedule. The earlier risk is identified, the less the impact. Figure 7 was generated by
the interviews. The curve is identical to the Cost to Mitigate Risk Factor mentioned above.
— — = Schedule Risk Factor
‘Schdule Risk Factor
NS
tR----"
CDR GRD FLT
Schedule Risk Factor Vs. Time Time
Figure 7. Schedule Risk Factor over Time
Model Description
Once the information- gathering phase was completed, the I-think® computerized system
dynamic modeling tool was used to develop a stock and flow structure. The development of the
simulation model was an iterative process. An object-oriented approach was used in which
individual sectors were developed and validated and then incrementally integrated with other
sectors. There were nearly 30 iterations and several expert interviews performed during the I-
think development process. The following sections describe the system dynamics sectors that
were modeled.
Requirements Sector:
This sector addresses the user requirements over the life of the development program.
The New Requirements variable is an input to the system and is an independent factor.
Requirements are reduced when the User Willingness to Relieve Requirements threshold is
exceeded. The Business Performance input (earned value) is the variable that affects the users
willingness to relieve the requirements. If the Eamed Value performance is bad, then the User
Willingness to Reduce Requirements increases. The Performance Requirement Gap
represents the difference between Tech Maturity and the user Requirements goal.
E-}2) 6
Performance Regs Gap
Tech aetury ¢
a Bn
Users wategrets oo Rebewe Regurecrers
Figure 8: Requirements Sector
Identify Risk Sector:
This sector models the identification of risk within a government program. The premise
is this: If the engineering support is manned with the appropriate number of individuals, the SPO
will identify risk at an acceptable rate. If there are too few individuals, then risk identification is
slow. Based on the behaviors presented earlier, the earlier risk is identified, the smaller the
impact to program cost. The longer it takes to identify a risk, the more expensive it is, by a
factor of ten, to mitigate.
Discovery Profile is an input variable, which is a time-phased input to Units of Risk to
be Identified. The Performance Regts Gap input drives manpower planning. Most SPOs
determine the number of individuals required on a program by the maturity of the system. Fora
more mature system, fewer engineers are required to identify risks. Available Manpower to
Identify Risk is the actual engineering manpower allocated by the SPO. The delta between
actual and required yields the Confidence in Risk Assessment variable. This factor determines
the Rate of Risk Identification variable that is the output factor for the Units of Risk to be
Identified.
EJ)
Figure 9. Identify Risk Sector
Mitigate Risk Sector:
This is the most complex sector in the program. The sector describes the flow of risk in a
program. There are two ways for programs to address risk. First, management reserve can be
applied to risk, thereby eliminating cost increase. Second, the SPO and contractor can work to
mitigate risk or cost growth potential through “other means,” including technical problem
solving, business practices, etc.
There are two major stocks in this sector, Reported Program Risk and Management
Challenge. Reported Program Risk is the amount of risk that is reported external to the
program. Management Challenge represents risks that currently exist yet are under mitigation.
The inputs to the Reported Program Risk stock come from the different inputs of risk
into the system. These inputs of risk include the Identified Risk variable from the Identify Risk
sector; Risk Increase Due to New Requirements, which represents additional costs due to the
user increasing the weapons system requirements; Unmitigated Risk, which represents the risk
that was unable to be mitigated; and Mitigated Risk Out, which represents one-third of the
mitigated risk that reenters the system. The outflow is the Management Reserve that is applied
to the risks. Another pathway for Program Risk Out is the Requirements Relief Risk Out. If
the user relieves requirements, then risk is reduced from the program at a rate defined as Risk
Relief Due to Reduced Requirements.
The Management Challenge input is dependent on the Selected Management
Challenge factor. This is a time-phased factor in which, early in a program, the SPO accepts a
larger percentage of risk as management challenge. This is because there is time to mitigate the
risks. As the program matures and the design is well established, less management challenge is
applied due to the decreased likelihood to reduce the program costs. There are two outputs to the
Management Challenge stock: the Unmitigatable Risk and Mitigate Risk Out. The
Unmitigatable Risk Factor is a time-phased variable that controls risk mitigation. Early in the
program, there is a much higher probability to mitigate risk than later in the program. The
Mitigate Risk Out variable is govemed by manpower allocation.
The Management Challenge figure dictates the number of engineers required to mitigate
risk, defined as Manpower Required. The Available Manpower to Mitigate Risk dictates the
number of engineers allocated to mitigate risk. These two variables are computed to determine
the value of Confidence in Risk Mitigation. This variable governs the Mitigate Risk Factor,
which determines the Mitigate Risk Out.
CJ) tore aa
Figure 10. Mitigate Risk Sector
Other factors in the sector include Remaining Risk, which represents the sum of
Management Challenge and Program Risk. MR Applied to Risk represents the number of
units of management reserve to be applied to risk. Units of Risk Per Units of MR isa
conversion factor between risk and management reserve. In reality, both risk and management
are measured in dollars. This simulation represents the value of one unit of risk to be
substantially less than a unit of management reserve by a factor of six. Total Program Risk is a
measure of the total risk associated with the program.
Technical Maturity Sector:
Tech Maturity is a goal-chasing curve and is an input into the system. Initial Maturity
is the maturity already achieved prior to the beginning of the development program. The
Requirements stock comes from the requirements sector, and is the variable that changes when
new requirements are added and subtracted. Program Schedule represents the number of
quarters the program office is planning to complete development. Projected Capability is the
percentage of the requirements the program expects to deliver to the user. When a program
begins development, many times meeting the full set of user requirements is an impossible task.
The user is reluctant to reduce requirements and is willing to progress with the program to push
the envelope of technology despite knowing that a 100% solution is unreasonable. The user is
also very unlikely to reduce requirements due to the concern of losing more than an acceptable
amount of capability.
od a Ey]
Figure 11. Technical Maturity Sector
Inquiries Sector:
This sector models the inquiries coming into a program office. There are three sources of
inquiry inputs: Planned Inquiries, Inquiries Due to Program Performance, and Inquiries
Due to Customer Satisfaction. Planned Inquiries are formal reporting inquires that each
program office is required to perform. Reports such as the Monthly Acquisition Report (MAR),
Defense A cquisition Executive Summary (DAES), and quarterly program reviews are examples
of recurring reports that engineers support and are included in the Planned Inquiries. Inquiries
Due to Program Performance are non-recurring inquires that are generated when the program
is experiencing poor Business Performance. The Inquiries Due to Customer Satisfaction
variable represents the additional inquiries that result from insufficiently supporting the inquiry
workload. The Answer Inquiries Factor govems the workload and is calculated by the ratio of
Available Manpower to Answer Inquiries and Required Effort Inquires variables. The
Required Effort Inquiries is a predetermined value that is dependent on the number of inquires.
The more inquires that are received, the more manpower that is required to answer inquiries. If
the SPO does not allocate enough engineers to answer inquires, or Inquiries Out. Inquiries
Out represents the remaining factor that increases the number of Inquiries Due to Customer
Satisfaction and is multiplied by the Answer Inquiries Factor.
Eye) aur Secor Fy
SB nf) Ariswer Inquities Factor
Auaitable Manpowes to Asie Inguiies 4
a
Inquities Oue to Customer Satitaction a
FE Roques Etter nquties
c
0
Business Performance:
s Jit
8) ” Inquires
5 Ineiies 0 Inquies Out
inqucies Que 19 Program Performance
Pranrsd inquiies
Figure 12. Inquiries Sector
Manpower Sector:
This sector contains the user-controlled input of manpower into the system. The EN
Manpower variable is the sum of the allocated engineering manpower.
o Manpower Sector 8
Available Manpower to Mitigate Fisk
Avsileble Manpower lo Kentty Risk
ra <8
5
*
EN Manpower
Avaliable Manpower to Answer Inquiries
Figure 13. Manpower Sector
Cost/Schedule Performance Sector:
This sector models Business Performance, Program Schedule, and Management
Reserve. Business Performance is the ratio of budgeted at Completion (BAC) and
Management Reserve (MR) divided by Estimate at Completion (EAC) and Reported
Program Risk, respectively. This ratio determines if the program is executable at the budget
determined at the beginning of the program. If the value is less than 1, then the user must
determine whether he is going to reduce requirements or seek additional funding. Other
information feedback is determined by this variable. If the business performance is poor (<1)
then the program could expect an increase in non-recurring taskers.
Program Schedule has an input and an output variable. Schedule In is a function of
factors that increase a program’s schedule. Reported Risk In is a variable that increases the
schedule if risks are identified late in a program. Early identification of risk does not have a
significant impact to schedule, whereas late risk identification is very costly and affects schedule.
The other input to Program Schedule is the impact that new requirements have on schedule.
The Time Factor New Rqmt and Schedule 2, in concert with the New Requirement input,
yields an increase in schedule that is dependent on when a new requirement is added. The later a
requirement is added to a program, the greater the impact it has on Program Schedule.
Mitigated Risk Out and requirement reduction variables govern Schedule Out. The
Mitigated Risk Out compensates for the Reported Risk In. If the SPO allocates the proper
effort to mitigate risk, some of the schedule impacts of the risks will decrease. There is also
corresponding schedule relief associated with requirements; this is defined by the variables Time
Factor New Rqmt and Schedule and Reqt’s Out.
Figure 14. Cost and Schedule Performance Sector
The next key component of this sector is the Management Reserve region. Each
program allocated approximately 10% of the contract cost for Management Reserve.
Depending on the risk associated with the program, this variable can be greater or less than 10%
and units can be added throughout the life of the program. The variable, New MR, isa
controlled input.
MR Out represents the sum of the variables that drains the Management Reserve stock.
These variables include; MR Applied to Risk, MR Risk Out, Cost to Mitigate Risk, and
Mitigated Risk Out. Cost to Mitigate Risk is a time dependent variable and works in concert
with Mitigate Risk Out. The two variables together establish that the point in program
development at which risk is mitigated affects the cost of risk. The later in development a risk is
mitigated, the greater the cost. The variable, MR Applied to Risk, is a control that allows the
model user to reduce a significant amount of risk in a specific time increment. MR Risk Out is
same output variable to the Program Risk stock. The variable MR Reduction Due to
Overmanning penalizes the model user for overmanning the program. If the ratio of manpower
required versus manpower allocated is exceeded, the program MR is reduced.
Model Analysis:
Due to the lack of substantial data, the model was not fit for performing critical statistical
analysis. The main objective was to determine if the current structure and multiple assumptions
yielded a reasonable response. Many scenarios were tested and the model was validated as a
good representation of the system structure. I will examine one scenario with varying manning
levels.
Adequate Engineering Support:
Below is several graph outputs from the model compared to the predicted behaviors with
reasonable engineering manning levels.
As indicated in figures 15 and 16, the behavior of achieved technical maturity vs. time
behaved very similarly to the predicted behaviors. Figures 4.3 and 4.4 illustrate the behaviors of
program inquiries.
_ a
4 100% F --------------===
%
i
y g
z = = = User Requirement
& Tech Maturity
CDR GRD FLT
] + Tech Maturity Vs. Time Time
Quarters
Figure 15. Figure 16.
Simulated Technical Maturity vs Time Predicted Technical Maturity vs. Time
p.
Go
6 a
£ ,
E fs
1 30: 7
i (f= = = Program Inquiries
CDR GRD FLT
i i i i Time
1 oot : t 4 #of Program Inquiries Vs. Time
2 a7 165 aa ma
Quarters
Figure 17. Figure 18.
Simulated Inquiries vs. Time Predicted Inquiries vs. Time
Figure 19 shows Management Reserve versus Remaining Risk over time. The
behavior represents a successful program where the difference between the Management
Reserve and the Remaining Risk represents the amount of funds remaining during the
development program. The two peaks on the Remaining Risk curves represent risk discovery in
the system. According to the predetermined inputs, the engineering manpower identified risk in
a timely fashion.
Quarters
Figure 19. Management Reserve and Program Risk over Time
Too Few Engineers:
The next graph represents a program office that has no engineers supporting the
development activity. The program office is relying on the contractor to identify and mitigate
risk. With all the variables remaining unchanged from the previous example, with the exception
of reducing manpower, the results are seen in Figure 20.
Quarters
Figure 20 Management Reserve and Program Risk Performance with No Engineers
The results are as expected. The Management Reserve bumdown is much slower, due
to the very slow identification of risks represented by the Remaining Risk curve. The sudden
peak a time=16, represents the contractor’ s identification of risks. The model is parameterized
so that at the halfway point the contractor begins identifying risk. The program goes
unexecutable at approximately time (T) =17, when the management reserve fails to cover the
program risk.
Too Many Engineers:
The next graph represents the behaviors of Management Reserve and risk in a program
office with too many engineers. The behavior is the opposite of the previous example.
Management Reserve experiences a steep decrease due to the early identification and mitigation
of risks, and the extra costs of extra manpower. At T=16 there is only very slight increase to the
Remaining Risk due to fact that contractors has very little risk to identify. Interestingly, the
program becomes unexecutable at T-17 because the risk, though small, exceeds the remaining
management reserve. The system model behaved as predicted by demonstrating the adversarial
effects of overmanning.
Dv niiemitncien 2: Reming Rsk
momen
YY —0000
Management Reserve and Program Risk Performance with Too Many Engineers
Early Requirement Changes
The next scenario examined the effects of requirement changes early in the program. The
parameters were reset to the adequate manning level described above. For the scenario, a 50%
increase of requirements was added to the program in the first 8 quarters of the development
cycle. Figure 22 illustrates the system behavior. The three step-like jumps at T=2, T=4, and T=6
represent increases in the requirement. The early changes to the requirements has little effect on
the Technical Maturity which matures before the end of the program.
$v recierens 2:Tech Matty
rf 20000
%
y 10090
100 875 1650 2425 3200,
Quarters
Figure 22 Tech Maturity and Requirement Behaviors with Early Increase in Requirements
Figure 23 illustrates the Management Reserve and Risk behaviors. Even with a 50%
increase in the requirements, the program is executable.
JO voragement Reseve 2: Remain isk
y DOOD pstsssesesnascssnaseesen
$
3] 1000.00
Quarters
Figure 23 Management Reserve and Program Risk Behaviors with an Early Increase in
Requirements
Late Requirement Changes:
The next scenario investigates the effect of late requirement changes in the last quarter of
the development phase. The results are significantly different than early changes. Figure 24
illustrates increases in requirements at T=22, T=24, and T=26.
Bove
a
%
reso mis Tad
Quarters
Figure 24. Tech Maturity and Requirement Behaviors with Late Increase in Requirements
Figure 25 goes unexecutable at T=25, when the risk exceeds the management reserve
available. Notice that the program actually goes unexecutable after only a 25 % increase in the
requirements.
BD vvvssenertReseve
yy monon
2 Remaining Risk
4 100000 fessssssssseceresse
Too a 1650 ne Tao
Quarters
Figure 25 Management Reserve and Program Risk Behaviors with an Early Increase in
Requirements
As expected, late increases in requirements have a grave impact on the performance of a
program.
Sensitivity Analysis:
Once the extreme conditions were tested, we tested different policies and performed
sensitivity analysis. To perform sensitivity analysis for this model, variables were individually
manipulated while keeping all other variables constant. If the dynamic behavior of the model
changed significantly, it was determined that the variable was a critical variable. The results of
the sensitivity analysis determined that manpower selection for each of the three control
variables, Manpower to Mitigate Risk, Manpower to Identify Risk, and Manpower to Answer
Inquires, were significant factors in the model.
During the sensitivity analysis two manning scenarios were explored. The first looked at
the ramifications of adequately manning the engineers who identify and mitigate risk in the
beginning of the program, and then removing them at the midway point.
BB vena e
Quarters Quarters
Figure 26 Management Reserve and Risk Figure 27 Management Reserve and Risk
Performance with Adequate Manning Performance with Adequate Manning time T=16
and no engineering manning T>16
The results demonstrated that the system is not sensitive to the removal of engineers who
identify and mitigate risks after the midway point of a program development. This does not say
to remove the engineering workforce completely, because engineers are still required to broker
information to the stakeholders and to communicate with the contractor. However, the SPOs
might consider reallocating the engineering workforce to shift focus from risk identification and
mitigation, to other areas.
The next scenario looked at the behavior of the system if the program has too few
engineers at the beginning of the program, and then increases engineering staff at the midway
point. Increasing engineering manpower did not improve the system risk performance. In fact,
the program became unexecutable at T=20, see figure 29, versus the no engineering model which
remains executable until T=24.
Quarters Quarters
Figure 28 Management Reserve and Risk Figure 29 Management Reserve and Risk
Performance with no manning Performance with No Manning T=0 to T=16
and Adequate Manning T>16
This result is a bit counterintuitive. One might think that adding engineering support to a
program would improve performance. However, the late addition of government engineering
manpower would also increase the contractors engineering manpower. The added expense
would causes the management reserve spending to increase at a faster rate.
Also, as mentioned in the section above, the system is sensitive to the changes in
requirements. Early increases of requirements, does not affect the program. Late increases of
requirements have grave consequences to system performance.
Conclusion/Recommendations:
The exercise of generating the model has yielded a much greater understanding of the
system. The systems experts have agreed that the basic structure of the current model is sound,
and the boundary definition and the degree of aggregation provides insight into the system.
The system experts were very pleased with the results of the model. The largest USAF
organic engineering organization has adopted the model as a baseline. The entire process enabled
the system experts to see the system in a new light. The rigorous information gathering and the
formal modeling process led USAF leadership ask the right questions. Each expert has
expressed that there is value in pursuing a systems approach to solving the manpower problems
over traditional empirical manpower models. They plan to further refine the existing model to
better understand the critical dynamic relationships and will pursue better parameterization.
Since system dynamics modeling is an iterative process, the next step would be to revisit the
critical relationships and reach a consensus of the linear and nonlinear dynamic relationship
through interviews and data gathering.
BIBLIOGRAPHY
Bartolomei, J. 2001. Management Causal Matrix: A Tool for Organizing Model Variable and
Interrelationships, 19th International Conference of The System Dynamics Society, Atlanta GA,
July 23-27, 2001.
Bartolomei, J. and Miller, T. 2001. Functional Analysis Systems Technique as a Group
Knowledge Elicitation Method for Model Building, 19th Intemational Conference of
the System Dynamics Society, Atlanta GA, July 23-27, 2001.
Blanchard, B.S. and Fabrycky, W. J. Systems Engineering and Analysis. Prentice Hall: Upper
Saddle River NJ, 1998.
Elder, J.E. and Nixon, M.B. 1976. “ The Application of System Dynamics to a Management
Model of Aeronautical Systems Division.” Unpublished master’s thesis. LSSR 22-78A,
AFIT/LS, Wright-Patterson AFB OH, June 1976
Ford, D. and Sterman, J. 1998a. Expert knowledge elicitation for improving mental and formal
models, System Dynamics Review 14(4), 309-340,
Forrester, J. W. Industrial Dynamics. The MIT Press: Cambridge MA, 1961.
Fowler T. 1990. Value Analysis in Design. Van Nostrand Reinhold: New Y ork.
Druyun, D. Principal Deputy Assistant Secretary of the Air Force for Acquisition And
Management, Washington DC, Personal Correspondence and Report, 6 Nov 1995.
Gonzalez, J.F. and Sarama, TJ. “Lean Methods Above the Manufacturing Floor in a Aerospace
Military Program.” Unpublished master’s thesis. Massachusetts Institute of Technology,
Cambridge MA, 1999.
Hill, J. and Warfield, J. “Unified Program Planning,” IEEE Transactions of Systems, Man, and
Cybemetics 2(5), 610-623, 1972.
Kaffenberger, S.A. and Martin, D. P. “A Dynamic Policy Model of the Defense Systems
Acquisitions Process.” Unpublished master’s thesis. LSSR 1-79A, AFIT/LS, Wright-Patterson
AFB OH, 1979.
Lawson, D. and Osterhaus, D.L. “A Conceptual Model of the Department of Defense Major
System Acquisition Process.” Unpublished master’s thesis. LSSR 1-79A. AFIT/LS, Wright-
Patterson AFB OH, 1978.
Sterman, J. Business Dynamics. McGraw-Hill: Boston MA, 2000.
Yates, R. W. Commander, Air Force Systems Command, Andrews Air Force Base, MD,
Personal Correspondence, 24 June 1991.
i] [sarin bunesrenuuae aeeerqeornien ger A
>| spaney sony eh
x seruedrey som oun ve
a ‘vayanjazos ya <iomny 902 si
04 403 siPanL 18% ole
amy Ayu
x ‘cegpserng euey
y wwewsrezng any eimpeyes ye evejeueg weeseg| a
x
quewsromngangimg ye neiaungawerieg| Pi
| pouszoena in
mee ge:
jung aueriea|
>| uojauanedruniiny ye s}duiag woos elelelel |e
resmmey qoummngieg G1en2y|
Schedule Risk Peformance
a
Identity Pick
Mitiqute Rizk
Cart Rirk Per
Support for a Development Program
Ri Repertinaiecuresy
‘ouinbuy weiBorg
fuga Auswoanboy|
uonzernes [ented] fe
7 uonpenes uobausd fe
uonse¢nes 1927) ‘fe
1 S2URMIOHOg WAR EL alele
7 ‘SDuRMEBDG Hele OD) alele
1 SpuaMuogog 42t4 4204] of ele
1 ‘SDuBMIOBE Hely SINPSYDS alele
7 faunaeyy e>144>9 | .
ig Buipung °
quDwcercy Hong ul S>USpINOD ale
7 foaunaay Bumedsy yi .
aeemeapnanisazeseasy ssemmamg|
identity Fick.
ivigate Risk
sro.
i
Pent
User
=
=
E__]rtenaae Rik
Inform Curtamer
=
fineening
Management Causal Matrix of Government Eng:
