179
REGRESSION AND CASE STUDIES OF PUBLIC PROGRAMS:
DISCREPANT FINDINGS AND A SUGGESTED BRIDGE*
David P. McCaffrey
David F. Andersen
Paul McCold
Doa Hoon Kim
Nelson A. Rockefeller College of
Public Affairs and Policy
State University of New York at Albany
omments of Fred Thompson, Irene Lurie, and Richard Nunez are
gratefully acknowledged. Thi
This was a fully collaborative effort.
ABSTRACT
Case studies of regulatory and social programs suggest that policy
systems are dynamic. In the systems described, outcomes depend on how
numerous variables interact over time, and feedback among variables
"simultaneous" causation over multiple time periods -- is more a rule
than an exception. However, the most influential evaluations of public
prograns are studies using multiple regression. A recognized limitation
of multiple regression is its relative insensitivity to multiperiod
strategies, feedback among variables, and other dynamics. Accordingly,
we maintain, the findings of regression-based and case studies commonly
conflict. Simulation modelling can serve as a methodological bridge
between case studies and regression-based studies of policy systems,
fuiproving) CheOEeEICAL wodery GF thelayatans aid preViaing @ way te
evaluate the of i
models. The results of
some early experiments along these lines are presented.
SECTION 1: INTRODUCTION
Case studies of regulatory and social programs suggest that policy
systems are dynamic. In the systems described, outcomes depend on how
e
numerous variables interact over time, and feedback among variables --
‘simultaneous causation over multiple time periods--is more a rule than
an exception. For example, weak
er poor
often prompt opposition leading to strong regulation and management
reform; "unlikely" coalitions form routinely; and so forth (McCaffrey,
1982: 402-403; Bacow, 1980: ch. 5; Policy Studies-Urban Systems
Vol. 11).
1981:
How
r, the most influential evaluations of public programs are not
case studies, but rather studies using miltiple regression. A recognized
limitation of multiple regression is its relative insensitivity to
multiperiod strategies, feedback among variables, and other dynamics.
Statistical and data problems tightly constrain the number of variables,
and feedbacks that can be i ly built into
regression-based analyses of public programs (Meadows, 1980: 40-46).
Also, the research i
with i i
cross- "chin" i
of large numbers of
individual cases, rather than longitudinal, "thick" description of fewer
cases. Such distant examination of even many cases will detect only the
most obvious dynamic relationships, and even then not provide data that
are sufficiently rich to explain them.
An assumption of regre:
ion analysis is that variable interactions
feedbacks, nonlinearities, and other dynamics are not so important as
180
oe
to prevent the necessarily simplified regression model from picking up
the strongest signals in the policy system. The aspiration of regression
analysis is to detect central factors that dominate other relationships,
however complex.
Arguably, the method works quite well, for it appears to uncover
strong, stable signals in numerous policy systems. Regression-based
studies find, fairly consistently, that Occupational Safety and Health
Administration inspections do not reduce industrial injury rates or
increase investment in safety-related capital equipment (Smith, 1979;
Viscusi, 1979; McCaffrey, 1983). They find, fairly consistently, that
state certificate of need (CON) regulation of hospital investment does
not reduce capital investment (Steinwald and Sloan, 1980; Sloan and
Steinwald, 1980; Joskow, 1981; Fuangchen and McCaffrey, 1983); that
variability in school expenditures or teaching methods are not associated
istically with variati
in student (Hanushek, 1981);
and that the organizational characteristics of medical care, or
or ethnic i of indivi
are iated in
only trivial statistical ways with use of physicians’ services (Mechanic,
1978). The list of stable findings from regression analyses could be
lengthened almost at will.
But what is striking is that these stable statistical findings
commonly conflict directly with analyses using case study or other
"thick" descriptive technique
As we will see below, intensive studies
of firms suggest that OSHA does improve safety performance and increase
y case studies suggest that
~3-
certificate of need regulation does affect the medical technology market
and hospital investment (E.g., see U.S. Office of Technology Assessment,
1981: 32-38; Howell, 1981); that school characterististics do affect
student performance (Rutter et al., 1979); and chat organizational and
Psychosocial factors do substantially influence physician utilization
(Mechanic, 1979). Analysts sometimes label the case studies as
"interesting" anomalies, but maintain that greater veight should be given
* regression analysis of carefully selected samples (E.g., see Sloan,
1982: 200). We question this argument because the discrepancies between
case studies and regression results are too common and systematic to
reflect only the anomalous qualities of the cases.
Instead, this paper maintains that regression-based studies, however
stable their results, are not definitive. Practically none of the
Tegression based studies of occupational safety regulation, and very few
of the studies in other areas, are grounded in credible theories of how
the policy systems work. Although the empirical results are consistent,
it ie therefore difficult to know what the results mean. The problem
exists because the data used to operationalize variables me: jure the
variables only grossly or ambiguously. Or, worse, the variables
themselves have no theoretical meaning, serving only as rough proxies for
variables or processes that are undefined or unknown, but Suspected to be
somehow important. the gaps between data and variables, or the
ambiguities of the variables themselves, mean that even stable empirical
results tell us little.
181
-4-
‘ rT
Section Il of this paper illustrates these theoretical and empirical
problems in detail, focussing on occupational safety regulation. Other
policy areas could have been easily used (F¢ of
licy areas could hat n y used (For extended discussions
: ‘ ; A
lar problems in the research on hospital regulation, education, an
physician utilization, see McCaffrey, Andersen, McCold, and “ 11984).
Section III outlines a research strategy, relying on dynamic
simlation modelling, that could ease some of these problems. The first
element of this strategy is to construct simulation models of a given
policy system using both quantitative and qualitative data. Simulation
modelling is a method designed specifically to examine and structure
dynamic causal relationships; its distinctive strengths are those
qualities in which regression methods are weak. The method is also a way
to structure in theoretically coherent ways what otherwise would be
disparate, noncomparable case studies. The second element of the
strategy uses the simulation model-~and variations on it--to generate
synthetic data on the simulated “reality.” Regression models can ven be
used to estimate properties of the simlation model, and the relative
success of alternative regression specifications in describing the known
model can be noted. The two elements of the research strategy are
related, but each has a distinctive advantage. The first element would
improve causal mapping of the policy system; the second element would
of i ion models
provide a way to test the
prior to data collection for an evaluation project.
ape 182
SECTION 11: RESEARCH ON OCCUPATIONAL SAFETY REGULATION AND OTHER POLICY
SESEARCH ON OCCUPATIONAL SAFETY REGULATION AND OTHER POLICY
SYSTEMS: THEORETICAL AND EMPIRICAL ISSUES
SESTEMS: THEORETICAL AND EMPIRICAL ISSUES
OCCUPATIONAL SAFETY REGULATION
Since 1971 the Occupational Safety and Health Administration (OSHA)
has enforced standards for working conditions using inspections and
fines. Most of the empirical studies evaluating the impact of the
Occupational Safety and Health Administration present a "theory of work
injuries" (Smith, 1979: 147-151); @ “conceptual analysis of occupational
health and safety regulation" (Viscusi, 1979: 118-122); or a “model of
injury rate determination" (Mendeloff, 1979: 98-102). These theoretical
sections serve as interpretations of the results, if the results are in
the predicted direction and statistically significant. The theories
focus.on the associations between occupational injury rates (or injury
rate changes) and OSHA inspections, industry, the size of firms, firms’
employment changes, and firm-specific factors. Some studies have also
examined the impact of workers’ compensation benefit increases on injury
rates, but for a variety of reasons these studies have not been
integrated into the study of OSHA's impact.
Empirical findings regarding these variables are quite stable, with a
mix of positive, negative, and negligible associations. But either the
data used to measure the variables are quite incomplete, or the variables
themselves have no clear theoretical relationship to injury rates.
. -6-
Therefore, when the studies try to estimate how a policy variable--
usually the OSHA inspection variable--is related to injury rates,.the
relationship is very ambiguous. Or, when analysts only try to "control"
for variables so that the independent effects of the policy variable can
be
sessed, they cannot in fact assume that control variables do not
interact in unmeasured ways with the policy variable because they have
only the vaguest idea as to the dynamics of either the control or policy
variables (See Footnote 1 above). Consider the results for each
variable.
Safety and Health Admini i i
Analysts are interested in the penalties that firms expect for
violating OSHA's standards for working conditions. OSHA could induce
firms to invest more heavily in safety and health capital by increasing
the probability of inspections and/or the penalties associated with
standards violations (Mendeloff, 1979: 94, 108; Viscu:
» 1979: 123-125;
mith, 1979: 149; McCaffrey, 1983: 132-133). ‘The studies examine how the
rates of i ions in i the of i ions in
firms, or penalty levels affect injury rates:
The obvious ways to assess the impacts of OSHA inspections, since
reliable pre-OSHA and post-OSHA data are unavailable, are to compare the
injury rates of industries with relatively high and low rates of
inspections, or the injury rates of inspected firms with those of
uninopected firms. Several studies have found that high inspection rates
in industries are not associated with relative injury rate reductions:
Several analysts are skeptical of the velue of results for an aggregate
-7-
category like "industry." Robert Smith, for example, suggested that it is
relatively futile to estimate the effects of OSHA from industrial data
because the potential effect of OSHA is relatively emall and "likely
confined" to inspected firms (1979: 147, 169); thui
analysts prefer to
use firm-specific data rather than industrial date.
One cannot meaningfully compare inspected and uninspected firms.
Comparing them would require that inspected and uninspected firms behave
in similar ways, but because inspections can be triggered by employees’
complaints, catastrophes, growing hazards, labor-management problems, and
so forth, inspected firms may be qualitatively different from uninspected
firms. Thu:
analysts have opted to compare rates of firms that have
been inspected at different times of a given yea
jeually March-April
inspectees with November-December i ¢
P
are used rather than
ry y 90 that the i i
vould not have been prompted by temporary increases in injuries reflected
in the prior year's injury rate). Although the late inspectees would
presumably be similar in other respects to the early inepectees, their
annual injury rate would not be affected significantly by the November or
December inspection. Thus, the impact of the inspection on the annual
injury rate can be Quite
ly, the i i
variable is not associated with relative declines in injury rates.
The inspection variable for firms comes from the Bureau of Labor
Statistics survey of injuries and illnesses, and represents the month of
the first inspection that a firm had in the year (Zero if there was no
inspection). This inspection variable does not articulate possibly
-8-
important variations in enforcement. The variable does not indicate if
the inspection was a routine “general schedule" inspection, or whether it
was prompted by a major accident, an employee complaint, or was a
follow-up visit to check for abatement of violations cited in previous
inspections. The variable does not record multiple inspections, nor does
it indicate if the inspection resulted in any penalties. .
Finally, even. though the inspection data are reported by month,
variables which one wants to correlate with the inspection data are
reported only by year, and so one cannot measure temporal sequences.
Studies test for associations between changes in injury rates and the
occurrence of inspections. However, the resulting associations do not
indicate whether (1) the inspection altered injury rates (e.g., by
reducing the injury rate, or by increasing the injury rate by increasing
the reporting of injuries); (2) whether the inspection was induced by a
prior increase in injuries; or (3) whether both (1) and (2) occurred in
the firm, Indeed, it is not at all unreasonable to suspect that all of
these tendencies are real. Interpreting non-significant statistical
effects is difficult when, for example, inspections might improve the
level of safety in the firm but simultaneously sensitize workers and
those who record injuries to safety problems. (In a related observation,
|. Kip Viscusi suggested that increases in safety-health investment may
not reduce injury rates as workers will respond to gafer conditions by
“diminish{ing] their level of safety-enhancing agtions...that affect
either the probability of an accident or the size of the loss...For
example, workers may get more careless if the company adds guards or
-9-
184
safety cables to the machine” [p. 118]. The point, however, does not
figure in his statistical estimates of the net impact of OSHA).
The ambiguities of the inspection data become troublesome when one
contrasts studies of OSHA's impacts which use multiple regression with
those using other methods. The conclusion commonly drawn from the null
regression results for the inspection variable is that the expected
penalties for violating OSHA's standards are too low to justify
safety i + W. Kip Viscusi, in fact,
analyzed a ion of i ies'
y ated ii and
reported that a greater frequency of inspections was not associated with
higher i in
y capital from 1972 to 1975
(1979). “Another conclusion commonly drawn is that OSHA's standards may
not address true safety problems of firms.
However, studies which directly ask management or workers about
OSHA's effects on safety conditions or opetations, or look at
disaggregated investment data, imply that OSHA does affect firns'
behavior and their level of safety. After studying plants unionized by
the International Association of Machinists, Kochan, Dyer, and Lipsky
wrote that since OSHA's establishment "management has assigned a higher
priority to plant safety, the ability of the union to influence
management decision making on safety issues has increased, and the role
of
fety
has been bolstered." They reported
that 69 percent of the workers, and 48 percent of the managers, said that
OSHA had a "strong" or "very strong" impact on safety consciousness and
responsiveness in the plants (1977: 5, 36, 76-77). A survey of chemical
= "10 -
workers by Cambridge Research Reports for the Shell Oi] Company reported
that 64 percent of the workers said that safety conditions had improved
in their plants in the 1972-1978 period (1978: 110), and that OSHA was a
contributing factor. (See also Freedman, 1981).
Also, an Arthur Andersen and Company survey of 48 large firms, which
is the most rigorous study of regulatory costs to date, found that the 48
firms spent $68 million in capital outlays directly related to OSHA, and
$184 million in OSHA-related expenditures overall, in 1977 (1979: Section
8, p. 5). The report added that the firms had higher OSH-related capital
costs in earlier years, which included the period studied by Viscusi
(1972-1975). The Arthur Andersen study, which involved a detailed
analysis of 48 firms’ investment decisions in 1977, and less detailed
analysis of earlier decisions, directly conflicts with Professor
Viscusi's, which involved pooled
of
industry data, with inspection frequency being the OSHA policy variable.
Thus, multivariate studies ini
industry i »
injury rate, and inspection data, or data on firms’ injury rates and the
occurrence of inspections, find that OSHA's activities are not associated
with injury rate changes. Those studies examining particular groups of
firms in great detail, focussing on a variety of practices and decisions
within firms, find just as consistently that OSHA's activities do affect
firms’ behavior and the level of safety in firms. (The “level of safety"
is not the same as “injury rates" because the reporting of injuries may
vary for reasons other than the level of safety. But this would mean
that focussing solely on injury rates would be an inappropriate way to
evaluate OSHA's impact on occupational safety).
-~a.b-
One could argue that these effects may be limited to specific classes
of firms--e.g., the firms unionized by the i i
of
Machinists, chemical workers, or the large firms studied by Arthur
Andersen and Company. But this would mean that multivariate studies are
mis
pecified, leaving out important variables that do magnify OSHA's
effects. The argument would also--for several other re:
jons--be weak.
(For example, the 48 firms studied by Arthur Andersen accounted for
nineteen percent of all capital investments made in 1977 in the United
States; this would be a very large “atypical” cla
of firms [Arthur
Andersen Report Executive Summary, p. 7]).
Another possibility is that regulation affects firms' operations and
behavior in more complex ways than can be realistically described by
wultivariate regression studies of national data, or of rather skeletal
firm-specific data. Regardless of whether one regarde OSHA's possible
effects as desirable or perverse, it appears that one will not understand
their dynamics except through alternative methods.
Examination of the other variables included in studies of OSHA
underscores this point. These are typically. entered as control variables;
however, as alluded to earlier (See footnote 1), their connections to
injury rates ‘are so ill-specified that we cannot assume that they are
independent of OSHA's effects. These other variables are now discussed.
Industry Variables
Occupational safety hazards vary by industry. Some technologies are
tore hazardous than others. The marginal costs and benefits of safety
controls also vary by industry. Time series data which indicate the
-12-
dominant type of technology, or safety controls’ costs and benefits, are
often not available at the industry level and hardly ever available at
the level of the firm.
Also, trade associations, insurance companies, or other
organizations, for any number of reasons, attend to safety probleme in
some industries more than in others. For example, the chemical industry
is reportedly criticized far more heavily for--and is more conscious
of--toxic waste disposal problems than the primary metals industry,
although the industries’ waste outputs are similar, because of the
chemical industry's attachment to Love Canal and s.
Street Journal, June 30, 1983: 58).
‘A study by Lawrence Bacow found that
1 safety are i more vi in the
auto industry than in the steel industry for several reasons, although
doth
industries have formally strong agreements (Bacow, 1980: ch. 5).
Detailed comparative studies of industry attention to safety matters
(such as Bacow's) are rare. Other industry effects, in addition to
industry technology or varying attention to safety matters, may be
consequential as well.
Analysts try to control for industry effects by entering dumy
variables for industry (usually the two-digit Standard Industrial
Classification level). Note that this procedure does not indicate what
“industry effects" are; it only lets us assess whether or not some
non-specified industriel factors are associated with injury rates. It
does not let us
sess whether OSHA contributes to industry effects
by--for example-~giving unions in certain industries a "club" to enforce
safety agreement:
or by affecting the cost and effectiveness of
fety
eq
equipment by stimulating
for certain i
or through several other plausible channels, none of which are adequately
described by an inspection variable.
Employment: Le
Injury rates vary by employment size in a curvilinear way. Small
firms (1-99 workers) and large firms (500 or more workers) tend to have
lower injury rates than firms of intermediate size. One explanation
commonly noted in the quantitative research is that small firms closely
monitor safety conditions, and that large firms are likely to have
injury-reducing safety programs. No one really knows if this explanation
is accurate. Alternatively, true injury rates could be inversely related
to firm size because of economies of scale in safety work, but small
firms might have distincti
injury di Data on
safety record keeping violations in the product safety area
re
consistent with this view (Linneman, 1981: 474).
The size category var:
bles do not indicate the size-related factors
that influence firms' injury rates. Rather, the size variables only
indicate whether or not some non- or ill-defined size-related factors
appear to influence injury rates. If, for example, OSHA reduced true
injury rates in distinctively hazardous small firms, but simultaneously
led the firms to tighten up distincti i
injury di
one could easily find no net change in injury rates. To conclude that
OSHA did not affect firms’ behavior would be an error.
are i
ly with
Changes in employment in firms are usually
-W-
represented by a ratio of employment in a year t+1 to employment in a
year t (EMP,,,/EMP,); changes in employment in industries are often
represented by new hire rates. Again, while these variables appear to
"control" for employment increase effects, there is very little research
on what these effects are. Employment increases may be associated with
an influx of accident-prone or inexperienced workers; with an influx of
young workers who are more willing to complain about job hazards, report
injuries, and request inspections; with an increase in the ratio of
workers to i worker:
¢ in firms (increasing
rates of injuries per 100 employees relative to other firms); and other
factors. While time and time again employment incre:
e effects appear
important, there are no studies that would give us a reasonable way to
choose among these or other interpretations of the employment increase
variable.
'Firm-specific" tors
Firms in certain industries or size categories, with certain
employment level changes, or inspected by OSHA, undoubtedly have other
characteristics associated with injury rates. Particularly outmoded
equipment, 1 di ies, or ieti
of the
plant's immediate location are possibilities. These factors should be
reflected in each year's injury rate. One way to control for such
factors in a year t is to enter the lagged injury rate as a dependent
variable (Smith, 1979: 149). Consistently, this variable accounts for
65-90 percent of the variance "explained" in studies of dccupational
injury rates, with the figure increasing with sample size (Smith, 1979;
= 15 -
McCaffrey, 1983. Data on the proportion of explained variance due to
lagged injury rates are available from McCaffrey). Such a variable does
not indicate which plant-specific conditions tend to influence injury
rates; it simply "controls" for such conditions. Thus, note that the
variable which the results of i etudies of
i afety 1 is a proxy variable with no
referent 1. Imagine any plausible interaction between unique firm
and OSHA's exi
a6 proximity to an OSHA area
office and the ion it distri iss xii
8 who
commonly threaten to "call in OSHA," managers who for reasons of personal
biography stress safety, etc.
and these interactions could contribute to
thie variable in ways not picked up by an OSHA inspection variable, or an
inspection-lagged injury rate variable.
A final factor thought to affect injury rates is the liberalization
of states’ workers' compensation la
although the effects are
ambiguous. Benefit liberalization may decrease injury rates as employers
strive to reduce injuries to avoid higher insurance premiums.
Conversely, benefit liberalization may increase injury rates as vorkers
become more willing to report injuries, claiming compensation. Studies
of the association between benefit liberalization ard injury rates
report, fairly consistently, that benefit liberalization is associated
with increased injury rates (Worrall and Appel, 1982; Butler and Worrall,
1983; McCaffrey, 1983). None of these studies have looked closely at the
dynamics ing the isti iatii
her the
187
See
first effect does not exist; whether the first effect exists but is
somewhat offset by the second; or whether other factors might explain the
benefit liberali injury rate
Nor have the studies
the workers’
variable with OSHA,
F ii, Be iabl
Thus, we really do not know how
workers’ compensation liberalization ought to enter into a theory of
occupational safety regulation.
Summary
The results of research on occupational safety regulation are quite
stable, showing a mix of positive, negative, and negligible
associations. However, the results say surprisingly little about the
dynamics of occupational injury regulation. The data used approximate
only ambiguously the variables of interest. In most cases the operative
variables themselves are not specified or are ambiguously specified, eo
it is extremely difficult to interpret their individual effects or know
ible interactions with other variables,
of their po:
Porter, Connolly, Heikes, and Park (1981) outlined four types of
demands that might be made of regression. We might ask regression to
describe the characteristics of a single data set; to predict the
characteristics of future behavior in the policy system; to causally
model (or "explain") the patterns of behavior in the policy system; or to
causally predict how future manipulated shifts in one or more variables
in the eyatem will affect other variables in the system. Where does the
research on occupational injury rates leave us in terms of these levels
of policy analysis?
-.7-
The research does not describe the characteristics of single data
sets, because it does not describe the specific factors underlying injury
rate patterns in single data sets. Consequently, although one can
predict how statistical associations will turn out in additional data
sets, one is not predicting the dynamics of injury rates at all. One is
predicting only that the current ambiguities will persist. A fortiori,
the area has no basis on which to develop
models or causal
models of
1 safety ion, or i
occupational safety regulation policy.
The problems of research in occupational safety and health regulation
appear in numerous other policy areas.” These common problems involve
the relative is of
dyn
ice in i (or
even longitudinal) regression studies; the extent to which regression
results are not linked uniquely to theories of how policy systems
operate, or do not: suggest what adequate theories might be; and
aggregation of variables in ways that may level important sources of
variation among variables.
Detailed case studies, which distinctively avoid these problems (and,
taken alone, have other problems) often directly conflict with the
regression-based research in other areas, just as the two types of
research on OSHA conflict. The case studies indicate that certain
variables are related strongly in ways that are somewhat complex and
unfold only over time. Regression analyses of cross-sectional or even of
thin longitudinal data detect few of these relationships (McCaffrey,
Andersen, McCold, and Kim, 1984), It is reasonable to suspect that the
18 -
discrepancies between the two sets of findings reflect regression's
relative insensitivity to dynamic or longitudinal relationships.
Certainly the evidence for this view is not definitive. However, the
evidence is strong enough to lead us to believe that the marginal payoffs
of shifting some effort to process-sensitive analytical methods may be
quite high. As will be discussed in the next section, the payoffs would
be both conceptual and technical. The conceptual payoff would be more
modelling. The technical payoff
would be a pi to
how well ion models
do describe the characteristics of policy systems.
SECTION III: RESPONSES TO THE PROBLEMS: SIMULATION MODELLING AND
SYNTHETIC DATA EXPERIMENTS
This section discusses two ways to cope with the methodological
problems discussed in this paper. The first activity involves extending
case-study analyses through the creation of dynamic simulation models of
policy systems. ‘The purpose of this activity is to describe more
adequately how events, constraints, and feedbacks emerge in policy
systems like occupational safety or hospital regulation, and to identity
underlying structural properties of the systems. The second activity
involves an attempt to estimate experimentally the conditions under which
dynanic feedback driven systems may present problems for regression
analysis. "
-19-
CONSTRUCTING DYNAMIC SIMULATION MODELS OF POLICY SYSTEMS
Case studies and other types of qualitative research suggest that
policy systems. are dynamic and ought to be. observed longitudinally.
However, regression based methods do not handle such complexity well.
Because the use of regression analysis requires large numbers of
observations, analysts are necessarily constrained in the time period and
richness of the data collected for any single case. Analysts tend to
restrict exploration of causal behavior in policy systems, and modelling
of the systems, to accomodate what can be roughly tested with these quite
skeletal data. Even where the theoretical models are more fully
developed, the data are not sufficiently rich to allow adequate tests of
the models. Thus, the theoretical and empirical work on the policy
systems are, at best, loosely coupled. 7
Dynamic simulation modelling is a method that can link
regression-based results and the large amounts of case material in these
areas in a theoretically disciplined way. Like case studies, dynamic
similation modelling uses diverse sources of information. The method
uses the numerical data available to regression analysis, but the method
also draws on i iews, ici rg and other
non-numerical data in constructing models of how the policy systems
function. The method goes beyond case studies, however, in providing
some clear principles for i
the i ion into a i
model of the system. These principles are derived from feedback loop
theory (Forrester, 1961, 1980; Randers, 1980), and force analysts to
specify clearly--in both graphic and mathematical form—the multiple
= 20-
of variables and
in the system, the time periods
in which these effects play out, and how the multiple connections in the
system either amplify or dampen certain behaviors. The behavior of the
model can be projected into the future through computer simulation.
There is a constant interaction between the implications of the model and
behavior in the "
eal" world; discrepancies force the analyst to refine
the theoretical structure of the model.
Once the evidence gathered through essentially case study methods has
been assembled into a formal similation model, several features not
available through simple case studies are available. First, most of the
theoretical richness of the case study can be retained.
This is because
the relatively flexible ical form of si i
models allows for
the inclusion of a large
number of "hard" and "soft" effects. Second,
the simulation model will contain a mathematically explicit causal
structure. Whereas case studies can be rich in descriptive detail,
simulation models force the analyst to posit explicit causal hypotheses
concerning how the policy system operates. These causal hypotheses can
then be tested further via regression or other forms of statistical or
empirical analysis. A third feature of the simulation model is that all
assumptions of the model are available readily for inspection since they
have been cast into an unambiguous mathematical form. The model (and
hence the causal theory being tested) must be absolutely consistent and
logically complete. These features, often neglected in simple case
studies, must appear in a formal simulation model or the model simply
will not compile and run on the computer.
- ae
A fourth feature of the simulation model is that the behavioral
consequences over time of the causal hypotheses can be known with
certainty. That is, by actually running the model and creating a
simulated scenario, the model itself will generate synthetic time series
data that arise solely from the caueal assumptions built into the model.
By running the model and observing if the simulated behavior is
reasonable, the analyst can gain some additional insight into the
and logical of the
built into the
model. A fifth feature of a case study recast as a simlation model is
that analysts can easily insert alternative causal hypotheses into the
model, and then examine the implications of such changes over time.?
Thus, the first benefit of simlation modelling is that it can use
diverse sources of i ion, including ipti
rich case
studies, to theoretically model policy systems in a rigorous way.
USING DATA EXPERIMENTS TO TEST A k MODEL'S ROBUSTNESS
The second benefit of simulation modelling is that synthetic data
experiments, using the simulation model, can explore in mathematically
precise ways the and itivity of models.
The basic ideas behind ic data are, in princi
simple. First, the sii ion model--or a plausi liminary model--
is run a large number of times under stochastically varying conditions to
create a large number of observations. These observations are called
“synthetic data." Because the structure and stochastic character of the
model is known, the characteristics of the data generated by the model
are known as well. The experimenter ought to be able to design a
190
-22-
ying on si i lagged iabl
and variable transformations if necessary--to recover the known structure
and parameters of the data generating model.
Tne nature of the synthetic "reality" is easily changed by altering
the simvlation model; similarly, the specification of the regression
model may be changed easily. These changes may include “corrupting” the
data generating model by inserting amounts of measurement error,
inserting odd lations within i iabl
planting
plausible misspecifications in the regression model, and other probl
that are suspected to exist in real data and analysis. Because the
structure of the data generating model is known, the experimenter can
monitor in a mathematically precise way how well alternative regression
models recover the si ion model's and
under
these imposed and ification problems. ion of one
such experiment usually takes from five minutes to one half hour to
complete.
Such synthetic data experimente bear a family resemblance to Monte
Carlo simulations and often prove interesting because the statistical
properties of feedback-driven systems differ from those assumed by
ordinary least squares regression and most of its derivatives (Andersen,
1981). For example, Peter Senge (1977) has used this experimental design
to inv
tigate how well regression models can recover parameters from a
dynamic eyatem when the observed independent variables have been
corrupted by relatively small amounts of measurement error. For feedback
driven systems, the estii
tion models performed well for no or minute
-23.
quantities of measurement error, but the estimation model's performance
degenerated rapidly under small to moderate amounts of measurement error
(See also Johnson, 1980).
Also, Mass and Senge have used a similar design to investigate
several puzzles that arise when tests of statistical significance are
used to include or exclude variables when estimati from a
dynamic feedback-driven system. They concluded that simple tests of
significance may be poor indicators of which variables are important and
unimportant in determining the behavior of policy systems. (Mass and
Senge, 1980) Similarly, Luecke and McGinn (1975) used a modified Markov
chain to investigate how well regression might estimate schooling
effectiveness. Luecke and McGinn discovered that even when expenditures
for educational inputs such as better teachers and schools are known to
have an effect on student achievement in a simulated reality, estimation
models used in major evaluations of education still produced null
results
We have begun to extend this method to the study of regulatory
problems; the first results, dealing with a very simple model of OSH
regulation, are now discussed.
‘An OSH Data Experiment: Early Results
The B:
ic Model. A simplified simulation model of accident
generation within firms was created. According to the model, accident
rates within @ firm were an additive combination of factors related to a
firm's size, industry type, and safety programs, and a firm-specific
constant. As discussed below, one version of the model assumed that OSHA
191
- 24 =
inspections influenced the impact of firm's safety programs on accident
rates; another version of the model assumed that OSHA inspections did not
influence safety programs and accident rates. We explicitly set the basic
model to resemble closely the model specified by most regression studies
of OSHA's impact.
The model generated monthly accident rates for 1000 firms, and
accumulated these monthly rates into yearly total rates. The generating
formula used was:
Acc =K+(5*S8) +24 (541) +E
where
‘Acc = monthly accident rate;
K = a firm specific constant (value 0 through 1);
$ = the cumulative accident factor determined by a firm's safety
program (value 0 - 2) (Henceforth; this particular factor is called
the "safety factor);
2 © the accident factor determined by the size of the firm (value
0-1)
1 = the accident factor determined by the firm's industry type (value
0- 2)5
E = a random error term (value 0 - 1).
Firms fell into one of five categories of increasing size. If size
was 1 or 5, Z = 0; if size was 2 or 4, 2 = .5; andsif size was 3, 2 = 1.
‘Thus, the size-related accident factors followed the curvilinear pattern
reported in the literature. Two versions of the model generated
' 25-
data sets. In one version, it we
specified that OSHA inspections would
reduce the actual safety factor to zero for five month:
; this is called
the "OSHA-effective" model. In the second version, OSHA inspections did
not affect the actual safety factor; this
8 called the "OSHA-
ineffective"
model. Initially, all values were selected randomly for the
firms. A firm's safety factor, size, and industry were allowed to vary
between months using a Markov process of varying degrees of stability.
In the model, OSHA inspected ali firms in the second year (months
13-24), half early and half late in the year (in March-April or in
November-December). The data output for each of the 1000 firms included
the yearly accident rate for both years, firm size and type at the end of
the second year, and the month of inspection. The inspection variable was
coded 1 if the firm w
inspected early, and 0 if the firm was inspected
late.
A generalized multiple linear regression model was used to evaluate
the effects of OSHA inspections on yearly accident rates. Following the
earlier studies, the regression equation attempted to fit the accident
rate for the second year to the previous year's accident rate, firm size
and type, and whether the firm was inspected early or late. The
regression coefficient and significance level for the inspection variable
were recorded for the data sets generated by the OSHA-effective and
OSHA-inef fective versions of the model.
If the ion model
the data generated by
the simlation model, then the regression coefficients would match the
difference between accident rates of the two years. The actual impact of
192
~ 26 -
OSHA inspections on accident rates could be determined by simply
annual accid within the simul
where OSHA had an
effect from annual accidents within the simulation where OSHA had no
effect. This is because it all other ways, including random sequences,
these two model variations were identical. Furthermore, the F value for
the inspection variable would be highly significant for the
OSHA-ef fective simulation and would be insignificant for the
OSHA-ineffective simulation. A more detailed description of both the
data generating models (programmed in BASIC) and the regression models
(using ordinary least squares in SPSS) is reported elsewhere (McCaffrey,
Andersen, McCold, and Kim: 1984).
Basic Model Test. The ion’ model
the
simulated reality under a wide variety of stochastic conditions. The
relative size of the random error term, E, and the “stability” of the
Markov process that changed safety program's effects, firm size, and
industry type were varied substantially (A perfectly "stable" Markov
process has all unit values on the diagonal of the state transition
matrix-~a value as low as .5 was tested). Under stochastic processes of
varying stability, the regression model detected that OSHA reduced
accidents in the OSHA-effective simulation, and that OSHA did not reduce
accidents in the OSHA-ineffective simlation. Bear in mind, however,
that none of the empirical and theoretical problems discussed in Section
TI above were yet built into the data generating model. Two of these
problems were then introduced into the simulation, one at a time.
- 27
Misspecification of the OSHA ion Variable. As di in
Section II above, numerous empirical and interpretive problems surround
the OSHA inspection variable. In the OSHA-effective simlation earlier,
OSHA inspections always increase the effectiveness of safety programs; a
value of 1 for the inspection variable is a perfect and unique predictor
of systematic improvement. We relaxed this
sumptions slightly by
introducing varying degrees of mismeasurement to the OSHA variable. The
involved "miscoding" the i
variable for certain
varying percentages of the firms, rendering the variable inaccurate for
those firms. The results of this experiment are presented in Figure 1.
The horizontal axis reports the percent of the inspection variables
that have been miscoded in the OSHA-effective simulation. The vertical
axis presents the F statistic for the OSHA inspection variable (Scale 1)
and the predicted effect (B value) of OSHA inspections (Scale 2). The
horizontal dashed line reads on scale 2, and shows the "real" effect of
OSHA derived not by statistical inference but by directly comparing the
and
For no measurement error (i.e., no misspecification of the OSHA
variable) the i
model predicts OSHA's by
roughly a factor of 2. This result is highly statistically significant.
When the i ion variable for in
30% of the i firms
has been mi ified the ion model's icti
the correct magnitude of effect for OSHA inspections.
Both the predicted effect of OSHA inspections and the associated F
statistic decline ically as the mi
At
+= 28 -
roughly 45%
specification, the predicted OSHA effect is not
statistically significant, although OSHA is known to be effective in the
model.
The results suggest that measurement error in the OSHA variable
reduced the predicted effect of OSHA on accident rates, and degraded the
F statistic. Also, an estimate under considerable measurement error
matched the actual simlated effect of OSHA inspections.
OSHA Interact with Reporting Behavior. In a second
experiment, we specified in the simulation that an OSHA inspection
increased the reporting of accidents in the firm. We specified that,
Prior to the inspection, only some fraction of the accidents were
recorded; an OSHA i ion led to ing of all acci In the
real world such an effect is plausible, because inspections could
increase awareness of safety issues, improve recordkeeping lest
violations of reporting be cited by and so
forth.
Figure 2 presents the results of this experiment. The Percent of
Accidents Reported Prior to OSHA Inspection is recorded on the horizontal
axis. The vertical Scale 1 and Scale 2 are analogous to those scales in
Figure 1. Again, the actual
effect of OSHA i ions in
reducing accidents is computed by comparing the OSHA-effective to the
OSHA-ineffective simlation, and is shown by a horizontal dashed line
labelled
‘actual effect." The line labeled “apparent effect" is derived
by comparing the ive to the i ive sii i
This
line represents the pure effect of increases in accident reporting and
- 29 -
194
actual accident reductions due to inspections. Of course, such a curve
could never be computed in a real data analysis situation since the true
effect of OSHA on safety, and its effect through changes in reporting ere
deeply intertwined.
If it is assumed that all accidents were reported prior to OSHA
inspection, the regression model predicts an OSHA effect that it roughly
twice the actual effect. When roughly 96 percent of accidents had been
reported prior to OSHA's inspections, the regression model's estimate
coincides with the actual effect of OSHA on accidents reduction. At
roughly 91% of accidents reported prior to inspection, the regression
model predicts no effect of OSHA in the simulated world where OSHA is
known to have an effect. The OSHA effect is not statistically
significant from roughly 88% to 93% of accidents reported prior to
inspection. For lower prior reporting rates, the model predicts that
OSHA inspections actually increase accitaeice (Again, this is in the
simulation where OSHA is known a priori to make safety programs perfectly
effective).
These results map the statistical implications of a plausible
situation in which OSHA inspections increase accident reporting, and
simultaneously reduce accidents. Of course, the exact ranges of
significance reported here are artifacts of the simulation model being
used. The important point’ is not that prior accident reporting rates of
between 88 and 93% will produce null results nor that a 96% prior
reporting rate produces unb:
ed estimates of OSHA effect. Rather, the
point is that these types of experiments can alert the analyst to a
- 30 -
possibly important effect before detailed and expensive data collection
begins. Also, such experiments can be used to probe the sensitivity of
regression-based results to complications suggested in case studies.
Thus, the simulation models provide a bridge between case studies and
regression studies via synthetic data experiments.
Next Steps. In the experiments reported above, the simulation model
was built so as to conform to the specification most commonly used in
regression studies of OSHA effectiveness. The structure of the model wi
relatively simple, containing only a few lags, non-linearities and
feedback effects. A more ambitious task would be to base the simulation
model directly on the richer and more detailed insights gained through
case studies. Plausible theoretical and empirical complications would be
initially built into the model, rather than tacked on as in the simple
illustrative experiments described above.
Furthermore, the simulation model, should be subjected to more
extensive analysis and sensitivity testing before being used in synthetic
data experiments. For example, the estimated parameters from the
synthetic world should closely resemble in size, magnitude and
significance those. observed from real data sets. Such a full range of
tests were not performed in the results discussed above.
Finally, insights into how such synthetic data experiments should be
performed need to be refined by replicating these experiments in several
other policy areas (McCaffrey, Andersen, McCold, and Kim, 1984). Such a
research ‘program would generate guidelines for using synthetic data
experiments to bridge the gap between case studies and regre:
on studies
2a 195
of the impact of government programs. These experiments, and the
theoretical simulation modelling that precede them, could begin to remedy
problems iated with ical and poorly measured
variables, and excessively simplified causal structures.
~32-
FIGURE 1
Simulated Effect
of Measurement Error
Scale Scale
1 2
100-20
90
80
45 Predicted
70 << effect (B value)
60 (change due to
w“ inspections)
50-104
40
30
F value of
= 25 Predicted effect ~~ *™ NN
10 crwea
‘i of F (p05) ————->
10 20 - 30 40 30
Percent of Inspections Misspecified
NOTE: Scale 1. F value of inspection vas
‘Scale 2-
~33-
FIGURE 2
Simulated Effect of
Reporting Error
Scale Scale
“uo 36
120 3.0)
100 25
F value of
80 20 — predicted effect
60 15
40 1.0
205
oo SE)
\ \ee Pscoes dn teporting rate)
40 woh 7 Grey tas Be apace
1S \e fu effect
-2.0 : :
80 82 84 86 88 90 92 94 96 98 100
Percent of Accidents Reported
Prior to 0.S.H.A. Inspections
NOTE: Scale 1-F value of inspection variable
Scale
2: mean change in accidents due to O.S.H.A. inspections
{accidents per year per firm)
196
yp
Footnotes
1. Often these variables are entered only as "controls" for
environmental effects, and so it might seem unreasonable to expect
specification of their i h
lati ip to the
dependent variable. We do not think it is unreasonable for two
reasons. First, analysts commonly do assert a relationship between a
control variable and the dependent variable; the problem is that
there is usually no basis for the assertion. Second, having only
vague (or no) sense of the meaning of control variables can lead us
Fe overlook how they interact with policy variables. Usually the
descriptive data on the possible associations among control, policy,
and dependent variables are poor. (For example, only sparse
descriptive information on how OSHA inspections filter into firms’
behavior, or how hospitals make investment decisions [Sloan and
Steinwald, 1980: 18) are available). If a alysts had a reasonable
understanding of the relationship of the control variables to the
Policy and dependent variables they could assert knowledgeably that
control variables are theoretically independent of policy variables.
When they have only the slightest, if any, idea of the control
variables’ dynamics they cannot know when control variables are
independent of policy variables. Nor can they just look at
correlations between control and policy var:
bles, for if they do not
have well founded ideas about theoretical connections of control and
le
i
197
-35-
(continued).
policy variables they are not able to interpret the correlations.
(Nonsignificant correlations could reflect offsetting dynamics or
unmeasured interactions as easily as null associations).
An extended discussion of how the theoretical and empirical problems
discussed in this paper apply to hospital certificate of need
regulation, and also hospital rate setting regulation, is found in
McCaffrey, Andersen, McCold and Kim (1984).
However, the i
of dynamic
modelling are bought at a price. Such models are typically beset by
complex
Parameter estimation (Graham, 1980) as
well as overall model validation (Forrester and Senge, 1980). A trade
off emerges between the relative strengths of regression and dynamic
simulation modelling. Take parameter estimation as a case in point.
Simulation modelling uses a of subjecti
data sources both in formulating a rich causal structure as well as
in estimating the i with the st On the
ether hand, because regression analyses are restricted to av:
lable
longitudinal or cross-sectional data sets, they usually have less
richness of causal structure, but many fewer questions concerning
par due to the
of formal tests of
The data
reported
below represent one of the several possible ways that case studie
dynamic simulation models, and regression modele can be combined to
capitalize on the relative strengths of all three approaches.
- 36>
REFERENCES
Andersen, David
1981 "Kalman filter estimates of system states compared to the general
regression probl. International Journal of Policy Analysis and
Information Systems, 5: 95-109 (No. 2).
Arthur Andersen and Company
1979 Cost of Government Regulation: Study for the Business Roundtable
(Three Volumes). New York: The Business Roundtable.
Bacow, Lawrence
1980 Bargaining for Job Safety and Health. Cambridge: MIT Press.
Butler, Richard, and John Worrall
1983 ‘Markers’ compensation: Benefit and injury clains rates in the
" Review of 65: 580-589.
Cambridge Research Reports, Inc.
1978 Public and Worker Attitudes Toward Carcinogens and Cancer Risk.
ridge: Cambridge Reports, Inc.
Forrester, Jay
MIT Press.
1961 Industrial Dynamics. Cambridge:
1980 "Information sources for modeling the national e
onomy." Journal
of the American Statistical Association, 75: 555-566 (September).
Forrester, J., and P. Senge F
1980 Steet for Confidence in System Dynamics Models." Pp. 209-228 in
A.A. Legasto, J.W. Forrester, and J.M. Lynch (Eds.), System
Dynamics, TIMS Studies in the Management Sciences, Vol. 14.
‘Austerdam: North Holland Publishing.
Freedman, Audrey
to Health Risk. New York: The Conference Board.
1981 Industry Respons:
Fuangchen, Somporn, and David McCaffrey
e impact of certificate of need and rate setting regulation on
Paper presented to Fifth Annual Researc!
lation for Public Policy Analysis and ianspenesit®s
1983"
—— technology."
mference of Associa
-37- . - 38+
198
Graham, Alan McCaffrey, David
1980 "Parameter Estimation in System Dynamics Modeling." Pp. 143-161. in l9s2" and 1 Toward explaini
J. Randers (Ed.), Elements of the System Dynamics Method. a discrepancy." Administrative Science Quarterly, 27: 398-419
Cambridge: MIT Press. (September).
Hanushek, Eric 1983 "An assessment of OSHA's recent effects on injury rates."
: Journal of Human Resources, 18: 131-146 (Winter).
1981 "Throwing money at school! Journal of Policy Analysis and
Management, 1: 19-41 (Fall). McCaffrey, D., D. Andersen, P. McCold, D.H. Kim
Howell, Julianne 1984 "Dynamic Systems, Ambiguous Variables, and Discrepant Findings.”
Working Paper, Rockefeller College of Public Affairs and Policy,
1981 Regulating Hospital Capital Investment: The Experience in SUNY-Albany
Massachusetts. NCHSR Research Summary Series, DDHS Publ. No.
(PHS) 81-3298 (March). Washington: U.S. Department of Health Meadows, Donella
and Human Services.
: 1980 "The Unavoidable A Priori." Pp. 23-57 in J. Randers (Ed.), Elements
Johnson, Curtis of the System Dynamics Method. Cambridge; MIT Press.
1980 "Some effects of data errors in econometric models." TIMS Mechanic, David
Studies in the Management Sciences, 14: 143-159.
1979 “Correlates of physician utilization: ny ae major pobetutriere
Joskow, Paul studies of physician utilization find tr: pschos and
organizational effects?” Journal of Health ia Social Behavior,
1981 Controlling Hospital Costs. Cambridge: MIT Press. 20: 387-396.
Kochan, T., L. Dyer, and D. Lipsky Mendeloff, John
1977 The i of Safety Committ! 1979 Regulating Safety. Cambridge: MIT Press.
W.E. Upjohn Institute for Employment Research. °
“Policy Analysis-Urban Systems, Inc.
Linneman, Peter
1981 Eraluation of the Effects of Certificate of Need Programs Three
1980 “The effects of consumer safety standards: The 1973 mattress mes). Washington: U.S. Department of Health and Human
flammability standard." Journal of Law and Economics, 23: setts,
461-479 (October).
Porter, A., T. Connolly, R. Heikes, and C. Park
Luecke, Daniel, and Noel McGinn
1981 "Misleading indicators: the limitations of multiple linear
1975 " ion analysis and i i Can ion in ion of policy Policy
they be trusted?" Harvard Educational Review, 44: 325-350. Sciences, 13: 397-418,
Mass, N. and P. Senge Randers, Jorgen (Ed.)
1980 “Alternative Tests for ing Model Variables." Pp. 203-223 in
J. Randers (Ed.), Elements of the System Dynamics Method.
Cambridge: MIT Pre:
1980 Elements of the System Dynamics Method. Cambridge: MIT Press.
Rutter, H. B. Maugham, P. Mortimére, and J. Ouston
1979 Fifteen Thousand Hours: Secondary Schools and Their Effects on
Children. Cambridge: Harvard University Press.
-39-
Senge, Peter
1977 "Statistical estimation of feedback models." Simulation (June),
pp. 177-184.
Sloan, Frank
1982 "Government and the regulation of hospital care." American
Economic Review Papers and Proceedings, 72: 196-201.
Sloan, Frank, and Bruce Steinwald
1980 Insurance, Regulation, and Hospital Costs. Lexington: Lexington
Books.
th, Robert
1979 "The impact of OSHA inspections on manufacturing injury rates."
The Journal of Human Resources, 14: 145-170 (Spring).
Steinwald, Bruce, and Frank Sloan
1980 "Regulatory approaches to hospital cost containment: A synthesis
of the empirical evidence." Pp. 274-308 in M. Olson (ed.), A New
Approach to the Economics of Health Care. Washington: American
Enterprise Institute.
U.S. Office of Technology Assessment
1981 Policy Implications of the Computed Tomography (CT) Scanner.
Washington: U.S. Office of Technology Assessment.
Viscusi, W. Kip
1979 “The impact of OSHA inspections on manufacturing injury rates.
Bell Journal of Economics, 10: 117-140 (Spring).
Worrall, John, and David Appel
e and benefit utilization in workers’
The Journal of Risk and Insurance,
1982 "The wage replacement
compensation insurance
September: 361-371.
199
