Designing Public Health Dissemination and Delivery Systems’
Peter S. Hovmand, PhD
Washington University in St. Louis
phovmand@wustl.edu
Matt Kreuter, PhD
Washington University in St. Louis
mkreuter@gwbmail.wustl.edu
James W. Deering, PhD
Kaiser Permanente
James.W.Dearing@kp.org
Abstract
There is increasing recognition in public health that achieving population level
impact from even the successful translation of basic research to effective
interventions is inherently challenging. Only a small percentage of initial
interventions ever get implemented and of those very few do so at scale. A number
of efforts have sought to reduce the research-practice gap in public health. This
paper adds a promising tool to this endeavor by introducing the use of system
dynamics for the design of public health dissemination and delivery (D&D)
systems. We do so through two case studies. The first shows the role that system
dynamics played in assessing the relative impact of different designs for a D&D
system, while the second shows how system dynamics was used to help develop a
conceptual framework of factors influencing the performance of a D&D system.
Together, both projects highlight the contributions that relatively simple models
can make in dissemination science and practice through the design, testing, and
evaluation of public health D&D systems.
Keywords: public health, implementation, scale-up
To achieve significant improvements in public health, interventions need to be effective,
disseminated at scale, adopted, implemented, and sustained over time. Over the last 10 years, a
growing realization has emerged that the problem of improving public health may have a lot
more to do with dissemination, adoption and implementation of interventions than with
developing new interventions. Fields such as implementation science and translational research
have emerged in response to this realization, which have led problem and calls for brining in new
methods from other fields (Institute of Medicine 2001; Reid et al. 2005; Woolf 2008; Proctor et
al. 2011).
' This research was partially supported by the Bill & Melinda Gates Foundation, and the Health Communications
Research Lab at Washington University in St. Louis.
Dissemination and Delivery Systems 2
Knowing how to successfully scale an innovation is not new. Companies successfully
scale products and services all the time. It is not easy, but it is part of the routine process of any
successful enterprise that attempts to achieve scale either through growth or replication. In public
health, however, knowledge on how to scale innovations is relatively limited, and new
approaches are needed for how to think about and solve the various issues that arise in scaling an
innovation to address public health problems. Where companies such as Proctor & Gamble,
Coca-Cola, General Motors, Microsoft, and Apple have all developed systems to make their
products ubiquitous, similar systems for scaling up public health innovations are largely absent.
To address this, public health researchers have started to look toward business to understand how
to design and build better dissemination and delivery (D&D) systems (Dearing and Kreuter
2010).
From its onset, system dynamics has been used to understand the role of marketing,
research and development, and supply chains (Forrester 1968; Sterman 2000; Roberts 1964), and
emphasized the role of feedback in the design and management of such systems (Richardson
2011). While there has been growing interest in applying system dynamics to public health
(Milstein, Homer, and Hirsch 2010; Homer et al. 2008; Milstein et al. 2007; Jones et al. 2006;
Homer 1993), there has been little work done on the design of D&D systems for actual
implementation and scaling up of public health interventions.
This paper highlights two applications of system dynamics to designing D&D systems.
The first project focuses on the use of system dynamics to evaluate the conceptual designs of a
public health dissemination support system; the second project focuses on the use of system
dynamics to help develop a conceptual framework for assessing the readiness for scaling up
interventions and achieving impact. Both projects highlight the unique role that system dynamics
can play at the conceptual design phase.
1. Using Models for Analysis versus Design
Most applications of system dynamics in public health have focused on using system
dynamics to analyze various policies and strategies with respect to specific outcomes, that is,
focusing on questions such as what is the best prevention strategy? What is the best set of
policies and programs to implement for improving clinical outcomes? And, what is the most cost
effective intervention, strategy, or policy? For example, previous system dynamics health studies
have looked at the dynamics of specific health conditions and risk factors (Homer et al. 2008;
Ghaffarzadegen, Lyneis, and Richardson 2011; Jones et al. 2006), service delivery system
dynamics (Levin and Roberts 1976; Lane and Husemann 2008), prevention strategies
(Hassmiller Lich, Osgood, and Mahmoud 2010; Thompson and Tebbens 2007), the cost
effectiveness of interventions (Tengs, Osgood, and Chen 2001; Tobias, Cavana, and Bloomfield
2010), and strategies for managing chronic disease (Homer et al. 2004).
Such efforts typically require significant time and money to develop models; their
primary purpose is analysis. However, models can have other uses such as helping people better
conceptualize a system (Richardson 2011), develop awareness of the important resource stocks
(Warren 2004), designing better systems, and developing innovations. In this paper we highlight
the use of system dynamics as a system design tool.
Although analysis of policies and strategies can also be thought of as policy design and
strategy design respectively, the primary emphasis of such activities is oriented toward
maintaining the current system as opposed to a more fundamental transformation of the system
Dissemination and Delivery Systems 3
(Lane 2001, 2001). However, system dynamics has a rich set of applications where models are
used to design new systems including (e.g., business models, supply chains). In such
applications, simulation models play an important role for early and quick prototyping, testing,
and revising of ideas. Heuristically, models serve as a boundary object (Black and Andersen
2012) or generative metaphor (Schén 1979) that help the designers change their interpretation of
a situation, reframe a problem, and innovate, or provide designers with a new pattern language
(Alexander et al. 1977).
The use of models for system design differs from the more prevalent practice of analysis
in that the emphasis of using models for design is on rapidly evolving the structure of a system.
It is the difference between assessing (analyzing) the influence of different policies through
parameter changes and additions/subtractions of feedback loops, and considering (designing) the
performance characteristics of different systems.
2. Designing a Dissemination Support System
The first project began as exploration to test via computer simulation how a conceptual
framework for a dissemination support system might work (see Figure 1). The developers were
concerned about a prevailing idea in public health that all effective interventions should be
implemented as is, while there is much research and practice evidence that adaptation, not
adoption, is the rule. The developers pointed out that few if any interventions were evaluated in
terms of demand for the intervention by end users. Moreover, those that were in demand were
generally not ready for mass distribution. Drawing from research and examples from industry,
the developers hypothesized that the addition of (1) a user demand review panel, (2) design and
marketing teams, and (3) dissemination field agents would significantly improve the rate that
effective interventions were adopted and implemented because these additional actors would, on
the strength of their market and practice knowledge, effectively adapt the research product into a
market-compatible innovation.
Figure 1. Conceptual framwork for a proposed dissemination support system
user review design and dissemination
e ee panel marketing team field agents
° e ° e ° e
ee e eee
sete *. mmm . e yinie
SD DBD > Pye es
ee . . y e ° yb
e e e
ee °
e e
eee
“menu of evidence-based,
high-demand, practice-ready
interventions
pfoven interventions
with user demand
imerventions expert empirically-supported
being tested review interventions
t
user demand informs, drives feedback from users continuously
practice-based research improves programs
While individual components of this system had been tested in empirical pilot studies,
what was not known is how the overall system might work, what the relative contribution of
each component might be, and where to invest the next set of resources for further development
Dissemination and Delivery Systems 4
and testing. It was in this context that the system dynamics team was approached with the goal of
developing a simulation model to test various designs of the proposed dissemination support
system.
Over the next 12 months, the modeling team worked with the developers through a series
of unstructured group model building sessions to conceptualize, formulate, and review the
analysis of different designs shown in Figures 2 through 5. Each design was represented as a
separate model with health innovations entering the system from the left and moving
progressively toward the right and eventually being adopted and implemented by end-users. A
co-flow structure was used to keep track of key two attributes: effectiveness of the innovation
and demand for the innovation.
Figure 2 illustrates the main stock-flow structure of business as usual case consisting of
expert review panels. In this model, the prevailing assumption is that reviewing the published
empirical studies by an expert panel and making recommendations on best practices is sufficient
for getting innovations adopted and implemented. This model serves as the basis for subsequent
comparisons. The model in Figure 3 adds to the “business as usual” case user review panels.
User review panels consider whether or not there is actual demand for the innovation. The
model Figure 4 adds design and marketing teams.
Figure 2. “Business as usual” case with expert review panels
S——=P] salutons * PD} Adopted
‘Adoption
y y
Figure 3. Addition of user demand review panels
>| souutions fective In demand Adopted
Developing Expert ‘User ‘Adoption
avalon reviews reviews
sons oe al Userreviow obs ei
rejected
Figure 4. User demand review panels with the addition of design and marketing teams
Adoption
Solutions Etectve In demand -———— Adopted
Developing Expert User
solutions reviews reviews
User
Jutions obs revew re Jo demand
Inettectve jf, Erectve (SU hice = Adopted obs
rejected ‘obs
toretinng
x Refining ton
. 5 x demand ¢
etooing to
in demand &
Effective Etective
solution being solutions being
Elfecivesoulens — Efacive solution“
being retooled obs boing refined abe
Dissemination and Delivery Systems 5
Figure 5. User demand review panels with design and marketing teams and delivery teams
ri Adoption \
o Solutions | Effective >| In demand BR ‘Adopted
Been ;
Developing Beet User reviews a
solutions
User revi /
olatons obs intecio lknecim ane aed cage yreamans —/
rejectod retnng jf
Refining to in Y
tating ton
o & demand Se ge
Effective Effective
solitons being Yoluions being
retooled Yefined
Effective Solutions Effective solutions
being retooled 008 ‘boing refined obs
Design and marketing teams take effective innovations that have no demand from potential users
and retool them to increase demand, as well as refine innovations that are in demand to enhance
their overall appeal to end users. The model in Figure 5 adds delivery teams that essentially
function as agents similar to marketing, pharmaceutical and real estate agents that help match
end users to innovations, but importantly, also provide feedback to intervention developers and
design and marketing teams.
As part of the modeling process, two metrics were found to be especially useful for
assessing the performance of this dissemination support system. First, the average time from
developing a solution to diffusion represented how long it would take from the time of
introducing an intervention to its spread throughout an adopting population. This represented the
delay often discussed in diffusion of innovation literature reflecting how long it takes to get
effective interventions into regular use. The second metric was the ratio of the number of
solutions that had to be developed for every effective intervention adopted, which reflected the
overall efficiency of the system. Typical ratios cited from industry are on the order of 1000:1, so
the overall goal was to find a design of a system that could significantly improve this overall
efficiency ratio.
Each model was then simulated to assess the steady-state characteristics for average time
from development to diffusion, and the ratio of solutions developed for every effective solution
adopted. As a part of the modeling process, it became apparent to the developers that part of the
effectiveness of design and marketing teams as well as delivery teams could be in their ability to
correct the errors introduced from expert review panels and user demand review panels. That is,
these panels may not be perfect in identifying effective solutions and solutions that are in
demand, and thus pass on solutions downstream that should ideally have been filtered out. To
address this, we considered both a “best case” scenario where panels did not make mistakes, and
a “worst case” scenario where panels were wrong 50% of the time and passed on solutions that
were not effective (in the case of expert review panels) or not in demand (in the case of user
demand reviews).
Dissemination and Delivery Systems 6
Table 1. Results from simulation analysis of different designs of dissemination support systems
“Business as usual” 21 110 33 132
+ user review panels 4 bie 145 ane
+ user review panels 5 14 16 27
+ design and marketing teams
+ user review panels
+ design and marketing teams 5 13 16 25
+ dissemination field agents
Table | shows the results from the simulation analysis. Overall, the full combination of
different components had the best result, but was only marginally better than having only user
review panels combined with marketing and design teams. One of the key insights from this
simulation was realizing that the addition of dissemination field agents, which was thought to
have an obvious benefit, needed more exploration.
Developing the simulation model helped the developers to not only become more precise in
their operational definitions of their conceptual model, but also discover more nuanced dynamics
about how different designs functioned. In addition to realizing that the role of dissemination
field agents may need further exploration, the modeling-as-design process also helped the team
draw a formal distinction in the roles of design and marketing teams, and realize that what may
be a transient benefit may not have much impact on overall steady-state performance of a
dissemination support system.
3. Assessing Readiness of a Distribution System for Scale-up
The second project began as a follow up to a year of an extensive literature review; key
informant interviews from industry, government, and major foundations; and review of
publically available applied tools that organizations have developed to assess organizational
readiness for global health distribution and scale-up. One of the major outcomes from this first
year of research was a concept model of factors associated with successful scale-up of health
innovations and the realization that the complexity of global systems involved could benefit from
formal modeling and simulation. This led to reshaping the goal for the second year to focus on
developing two separate prototype simulation models, an agent based model and a system
dynamics simulation model, with the general purpose exploring the potential of using simulation
models to assess different scale-up scenarios.
This second case focuses on the system dynamics simulation model. The primary goal of
the system dynamics simulation model was to test the logical consistency of the conceptual
framework and develop a better understanding of the potential leverage points for intervention
since the various stakeholders in global health intervention -- funders, large intermediary
Dissemination and Delivery Systems 7
organizations like the World Health Organization, and ministries of health in low-income
countries—all have choices that they can make to strengthen or modify a planned intervention so
that it works more effectively or more efficiently. The model was developed over the course of
five months in parallel to separate agent-based modeling effort, and then presented for review to
a group of experts including academics, program officers, and program directors from various
non-profit and governmental organizations. The format of the review consisted of a 2-day
meeting that providing some overall context for the models, and then two sets of parallel sessions
for the agent based model and system dynamics model where participants.
On the first day, half the participants were in the system dynamics session where they
were introduced to the model, raised questions about the model and method, and provided
additional structures through a facilitated structure elicitation exercise. These changes were
incorporated into a second version of the model, overnight, and shared with the second half of
the participants on the second day.
The initial model (see Figure 6) depicts four major factors influencing delivery of
innovations: resources, relationships, motivation, and environment. Environment is shown as a
box around the entire system to reflect the assumption that environment affects the entire system.
This is modeled by having the environment influence both the inflows and outflows of the main
stocks. However, while resources, relationships, and motivation are endogenous, environment is
by definition exogenous. In addition to considering the major feedback loops of the proposed
framework for assessing readiness, the model also introduced several interventions to change the
system.
The model was initially tested with a number of standard tests including extreme
conditions, boundary adequacy, behavior reproduction, construct validity against the key
informant data, and the expert review. After testing and revision, the model was subjected to a
number of analyses. It was quickly found that the system was generally biased toward
innovations not scaling up. Somewhat surprisingly, inter-organizational relationships tended to
be a relatively weak influence in this system, which depended on collaborative and coordinating
activity across a team of organizations. Conventional wisdom holds that network structure plays
an important role in the spread of ideas; certain structures facilitate spread; others do not, based
on characteristics such as density of connections and where in a system of units an innovation is
seeded (Hinz, Skiera, Barrot, & Becker 2011). However, in this case, it turned out that overall,
networks alone do not drive the system at the aggregate level. In comparison, organizational
capacity and organizational motivation played a much greater role in increasing the delivery rate
in addition to improving relationships.
Dissemination and Delivery Systems
Figure 6. Initial system dynamics model for assessing readiness to scale-up innvations based on
conceptual framework from literature reviews, review of assesment tools, and key informant
interviews.
Losing
Resource relationships
intervention < ry
O
. Relationships
Building
resources lk
u \ - Forming
4na) F relationships
. Resource:
Losing he)
resources
wot
+
Relationship \
a intervention
Motivation __, Increasing
intervention ‘motivation
hej
(&)
Motivation
‘ a Delivery
bs @} -} rate
Zp ¥
Losing
motivation Delivery
Percent delivered
4 to target
communities
Environment
The expert panel review also pointed out the importance of demand or “pull” for
successful health innovation scale-up (Dearing and Kreuter 2010). These structures were
included in the model (see Figure 7) and resulted in a different set of behaviors from the first
version. Whereas the previous version always produced the S-shaped diffusion pattern when
scale-up was successful, the revised structure led to a more dynamically complex system where
some factors played a more important role in scale-up (motivation and resources) than
relationships. Specifically, interventions in motivation and resources could both produce S-
shaped patterns whereas interventions to strengthen relationships could only produce goal-
seeking patterns.
The general conclusions from the expert panel review were favorable for this stage of
modeling. Comparisons between the system dynamics model and agent based model led to the
conclusions that: (1) system dynamics models were advantageous for identifying where to
intervene, while agent based models were more suited to understanding the details of specific
interventions; (2) system dynamics models helped people understand the aggregate system more
broadly than agent based models; (3) agent based models were better for modeling the specific
structures of social networks and actor rules; and (4) the inductive nature of system dynamics
modeling-as-design made assumptions more transparent and increased the ability of participants
to assess those assumptions and hence its “trustworthiness”, which was itself something that
contributed to trusting the model more (i.e., by understanding its limitations better). System
Dissemination and Delivery Systems 9
dynamics modeling, again due to its technical ability of enabling participants to modify a model
in real time, is more participative and hence engaging than actor based modeling.
Figure 7. Revised system dynamics model for assessing readiness to scale-up innvations based
on input from expert review
Losing
rollionshiie
Resource
intervention
Building
resources
eS eS
Losing Ag) Pesoyees
poairee ; (sna
Motivation __, Increasing
intervention peestaion
iS
| (ke)
| ‘Motivation
\ e ‘ Delivery
("@s} rate
v/s
Losing Delivery
motivation +
~
Environment
The experts also felt that the model was missing structures that influenced sustainability
of implementation after scale-up. This is a model boundary issue. In the initial scope of the
model, sustainability was explicitly excluded from consideration because it was seen as being
outside the scope of the research focus. However, the experts reviewing the model disagreed
with this decision and felt that it would add an important dimension to the overall understanding
of scale-up.
4. Conclusions
Both models introduced here are relatively simple, and yet both highlighted the important
role that system dynamics can play in conceptualizing and refining a design of dynamic and
complex systems. In both cases, the process of formulating a simulation model forced a level of
specification that the developers had not previously made, gave them a language for doing so,
and provided a way to rigorously test the logical implications of their thinking. Of particular note
in both projects was how much time was spent defining and redefining terms. This is not
uncommon in a relatively new field as terms are still in flux. The difference that modeling makes
Dissemination and Delivery Systems 10
is that one can quickly eliminate ways of operationalizing and measuring variables that are
irrelevant.
Ata more general level, both applications emphasize the role of modeling in the design
process. System dynamics models are frequently discussed in the context evaluating different
policies and strategies, and yet, the greatest role for system dynamics may be as a tool to help
people understand and design simulations of sophisticated systems for anticipating and managing
change. In this case, there were two different ways of representing design. In the first case, the
focus was on building multiple simulation models to assess specific designs of different systems
and evaluate their relative performance. In the second case, we developed a more abstract model
to test a conceptual framework with respect to its hypothesized structure-behavior relationship,
and assessed different designs of implementation scenarios.
Moreover, both projects highlighted the benefit of rapidly building and iterating a
simulation model for refining how we think about dissemination and distribution systems, and
for doing so with experts who had no prior familiarity with system dynamics. Somewhat
unexpected was the similarity in both projects for the extensive need to clarify and define terms
more precisely within the context of a formal simulation model. While this is quite common with
formal modeling, what distinguished this application experience from other research by the first
author was how frequently well-accepted definitions from the research literature were inadequate
or failed outright in their logical consistency. The interpretation offered here is that this is
symptomatic of a situation where the object of study (dissemination and distribution systems) are
simply too difficult to study and theorize adequately without the aid of formal models.
While there are many approaches to developing formal simulation models, the ability of
system dynamics models to be easily conveyed through the visual language of stocks-and-flows
and feedback loops gives system dynamics a unique role to play in helping people to think in
new and creative ways about such systems, and to develop a shared appreciation for their
complexity.
Equally important is the fact that the visual representations can be quickly translated into
running simulation models that help people learn. In the first example, this was evident in the
fact that not only were multiple models developed relatively rapidly, but that this strategy was
based on several previous models in early sessions that tried to represent all the designs in a
single more generalizable model (somewhat analogous to the second example). That is, multiple
strategies were attempted and scratched before settling on this particular approach. In the second
case, the modeling was seen as an inductive means for giving the research team something to
look at and critique early on, an aspect of system dynamics modeling-as-design that also
provided important in rapidly incorporating feedback from the expert review session in the
eventual day one meeting into the model during its second day.
In both cases, no one expected the models to be perfect or represent everything in the
system. But being able to rapidly incorporate feedback and sometimes very substantial changes
led to an intuitive understanding and appreciation of how the modeling could evolve into more
sophisticated and empirically tested models in the future. The fact that one could work rapidly
became a persuasive reason for people to engage further in the modeling, offer more and better
feedback, and identify ways that they could use the existing models to think better about
dissemination and distribution systems.
Building large and sophisticated simulation models will always be an essential and
important part of system dynamics practice for major policy questions on health and other
matters. However, building such models takes significant resources including time, money, and
Dissemination and Delivery Systems 11
expertise, and presupposes that the end users of such models have a clear set of expectations that
are stable enough to allow a completed model to be relevant when its completed.
In this paper, we have highlighted a different use of system dynamics modeling that
focuses on modeling-as-design; a tool that can serve as an important boundary object for people
to think, explore, and innovate. While much has been made in the past about the differences
between qualitative and quantitative models (or causal maps versus simulation models if one
likes), this focus has overshadowed the potential importance of small simulation models that can
be rapidly built and developed to help people think better about a system. We may ultimately
find that in the world of potential models that can be built and impact the world, many fall into
this category of models as design and learning tools.
References
Alexander, Christopher, Sara Ishikawa, Murray Silverstein, Max Jaconson, Ingrid Fiksdahl-
King, and Shlomo Angel. 1977. A pattern language: Towns, Buildings, Construction.
New York: Oxford University Press.
Alexander, Christopher, Sara Ishikawa, Murray Silverstein, Max Jaconson, Ingrid Fiksdahl-
King, and Shlomo Angel. 1977. A patter language: towns, buildings, construction. New
York: Oxford University Press.
Black, Laura J., and David F. Andersen. 2012. Using visual representations as boundary objects
to resolve conflict in collaborative model-building applications. Systems Research and
Behavioral Science 29:194-208.
Dearing, James W., and Matthew W. Kreuter. 2010. Designing for diffusion: How can we
increase uptake of cancer communication innovations? Patient Education and
Counseling 81S:100-110.
Forrester, Jay W. 1968. Market growth as influenced by capital investment. Jndustrial
Management Review 9:83-105.
Ghaffarzadegen, N., James M. Lyneis, and George P. Richardson. 2011. How small system
dynamics models can help the public policy process. System Dynamics Review 27 (1):22-
44.
Hassmiller Lich, Kristen, Nathaniel D. Osgood, and Amad Mahmoud. 2010. Using system
dynamics tools to gain insight into intervention options related to the interaction between
tobacco and tuberculosis. Global Health Promotion 14:7-20.
Homer, Jack B. 1993. A system dynamics model of national cocaine prevalence. System
Dynamics Review 9 (1):49-78.
Homer, Jack B., Gary B. Hirsch, Mary Minniti, and Marc Pierson. 2004. Models for
collaboration: how system dynamics helped a community organize cost-effective care for
chronic illness. System Dynamics Review 20:199-222.
Homer, Jack B., Bobby Milstein, Kristine Wile, Parakash Pratibhu, Rosanne Farris, and Diane
Orenstein. 2008. Modeling the local dynamics of cardiovascular health: risk factors,
context, and capacity. Preventing Chronic Disease 5 (2):1-6.
Institute of Medicine. 2001. Crossing the quality chasm: A new health system for the 21st
century. Washington, DC: National Academy Press.
Jones, Andrew P., Jack B. Homer, Dara L. Murphy, Joyce D. K. Essien, Bobby Milstein, and
Donald A. Seville. 2006. Understanding diabetes population dynamics through
Dissemination and Delivery Systems 12
simulation modeling and experimentation. American Journal of Public Health 96
(3):488-494.
Lane, David C. 2001. Rerum cognoscere causas: Part II--Opportunities generated by the
agency/structure debate and suggestions for clarifying the social theoretic position of
system dynamics. System Dynamics Review 17 (4):293-309.
Repeated Author. 2001. Rerum cognoscere causes: Part I-how do the ideas of system dynamics
relate to traditional soscial theories and the voluntarism/determinism debate? System
Dynamics Review 17:97-118.
Lane, David C., and E. Husemann. 2008. System dynamics mapping of acute patient flows.
Journal of Operational Research Society (59):2 13-224.
Levin, Gilbert, and Edward B. Roberts. 1976. The dynamics of human service delivery.
Cambridge, MA: Ballinger Publishing Company.
Milstein, Bobby, J. B. Homer, and Gary B. Hirsch. 2010. Analyzing national health reform
strategies with a dynamic simulation model. American Journal of Public Health 100
(5):811-819.
Milstein, Bobby, Andrew Jones, Jack B. Homer, Dara Murphy, Joyce Essien, and Don Seville.
2007. Charting plausible futures for diabetes prevalence in the United States: a role for
system dynamics simulation modeling. Preventing Chronic Disease 4 (3):1-8.
Proctor, E., H. Silmere, Raghavan, Peter S. Hovmand, G. Aarons, A. Bunger, R. Griffrey, and M.
Hensley. 2011. Outcomes for implementation research: conceptual distinctions,
measurement challenges, and research agenda. Administration in and Policy in Mental
Health 38:65-76.
Reid, Proctor P., W. Dale Compton, Jerome H. Grossman, and Gary Fanjiang, eds. 2005.
Building a better delivery system: a new engineering/health care partnership.
Washington, DC: The National Academies Press.
Richardson, George P. 2011. Reflections on the foundations of system dynamics. System
Dynamics Review 27 (3):219-243.
Roberts, Edward B. 1964. The dynamics of research and development. New York, NY: Harper
and Row, Publishers.
Schén, Donald. 1979, Generative metaphor: A perspective on problem-setting in social policy. In
Metaphor and society, edited by A. Ortony. New York.
Sterman, John D. 2000. Business dynamics: Systems thinking and modeling for a complex world:
Irwin McGraw-Hill.
Tengs, Tammy O., Nathaniel D. Osgood, and Laurie L. Chen. 2001. The cost-effectiveness of
intensive national school-based anti-tobacco education: results from the Tobacco Policy
Model. Preventive Medicine 33:558-570.
Thompson, Kimberly M., and Radboud J. Duintjer Tebbens. 2007. Eradication versus control for
poliomyelitis: an economic analysis. Lancet 369:1363-1371.
Tobias, Martin I., Robert Y. Cavana, and Ashley Bloomfield. 2010. Application of a system
dynamics model to inform investment in smoking cessation services in New Zealand.
American Journal of Public Health 100 (7):1274-1281.
Warren, Kim. 2004. Improving strategic management with the fundamental principles of system
dynamics. System Dynamics Review 21 (4):329-350.
Woolf, Steven H. 2008. The meaning of translational research and why it matters. JAMA 299
(2):211-213.
Dissemination and Delivery Systems
13
