The Opportunity Cost of
Solving Problems:
The Role of Testing in Product
Development
Laura Black
International System Dynamics
Conference
Motivation
‘We don’t know what we don’t know
until it’s rea/ late.”
--Program Manager
Motivation
m= Emergent dynamics in multiple projects
difficult to manage
¥Y Repenning’s work:
¢ Scarce resources creates interdependence among
projects
¢ Transient increase in workload can lead to
permanent decline in quality
m= Example: Design and testing are iterative
yet often treated as sequential.
L. Black 2000 rf
Mult-project Environment
Model Year s Model Year s+1 Model Year s+2 Model Year s+3
m Scarce resources for two projects at different phases
m Fixed annual launch date
m Bias toward work nearest deadline
m Early phase helps: lower probability of error
L. Black 2000 4
Modeling Assumptions
engineers allocated to design engineers allocated to revisions
parts to be \ prototyped _————< — | aw i S
aes P tested parts revised parts
designing testing revising parts
sé
testing
aa
engineers allocated to testing
m Early and current phases structurally identical
m Initial priority for work
Current phase: design -->testing --> problem-solving
Early phase: design -->testing --> problem-solving
L. Black 2000 5
As)
Year -1
Completion Loop
resources doing work in -_
allocated to work to do in Year -1
_ =+_ completed in
cv * Yeard <—> Year -1
maximum possible
number of errors in
transition in model year Dx] tansition in work
work to do ' + i completed
Tilting Loop
An) | |
v Acceleration :
+ . work
resources <d work to do in completed in
Year O
aigeated doing work in Year 0
Year 0
Year O
Completion Loop
L. Black 2000
Quality
L. Black 2000
Base Case
“Textbook” Allocation
Fraction Prototyped During Year -1 for Model S
3 i i
10) 0.2 0.4 0.6 0.8 1
Fraction Prototyped During Year -1 for Model S-1
Policies Explored
= Dedicating resources to testing doesn’t
ensure high quality.
= Prototype build during last 12 months
Stabilizes system slightly.
m Altering prionties for resource allocation:
m “Textbook” Allocation a “Urgent First’ Allocation
Y Current design Y Current problem-solving
¥ Current testing ¥ Current design
Y Current problem-solving ¥ Early problem-solving
v Early design v Early design
v Early testing Y Current testing
¥ Early problem-solving v Early testing
L. Black 2000 8
Dedicating People to Testing
With Prototype Build
SCZ
LZ LLTTZ
LISS LLLP
Build 34 Weeks Before Launch Build 16 Weeks Before Launch
L. Black 2000 10
Varying Priorities
‘Urgent First’ Allocation
0.2
Fraction Prototyped During Year -1 for Model S
‘Sizp of Shock to) 0.2 0.4 0.6 08 1
Fraction Prototyped During Year -1 for Model S-1
L. Black 2000 11
Policies Explored
(continued)
m Fixing fraction of problems identified
stabilizes system more, with a higher
equilibrium after a dip in quality.
= Postponing incomplete work keeps quality
high but at the expense of a reduced
model year.
™ Canceling incomplete work yields
immediate performance hit but no long-
term deterioration.
L. Black 2000
Fixing a Fraction of the Errors
O05
Size of Shod«
Postponing Work
Performance Quality
Decision 26 Weeks Before Launch Decision 26 Weeks Before Launch
L. Black 2000 14
Performance
Canceling Work
Quality
Performance Quality
With Checkpoint 12 Months Before Launch With Checkpoint 12 Months Before Launch
L. Black 2000 15
Conclusions
m Balancing work-to-do with resources available
assures long-term productive capacity.
v Managers can't balance what they're unaware of:
Parts to Be Designed + Identified Problems
=Total Work-to-Do
= Additional cycles of testing and problem-
solving can overload the development
system as much as too many projects.
L. Black 2000 16
Recommendations
m Dynamically allocating resources can help--if early
work receives priority.
m Test plans and results can serve as cross-
functional communication about total work-to-do.
= Targets for "what we don't know' raise awareness
of the need to test early in the development cycle.
= Metrics for problem resolution can aid project
management:
Y Prioritizing problems identified
vY Average time to resolve problem
v Percent of component-level concems identified in build
L. Black 2000 17
CC BY-NC-SA 4.0