Using System Dynamics to Model Student Interest in Science,
Technology, Engineering, and Mathematics
H. Alex Sanchez, Brian Wells, J oanne M. Attridge
Copyright 2009 Raytheon Company, ALL RIGHTS RESERVED
50 Apple Hill Drive
Tewksbury, MA 01876
978.858.5875 Telephone, 978.858.5996 Fax
alex_sanchez@raytheon.com, brian_h_wells@raytheon.com, joanne_m_attridge@raytheon.com
Abstract
This paper presents preliminary results of a Raytheon project that uses systems dynamics to
understand the intricacies of the U.S. educational system and to assist in exploring the effects of
policies and programs, with a goal of doubling the numbers of science, technology, engineering
and mathematics (STEM) college graduates by 2015. Specifically, a system dynamics model has
been developed, the initial version of which targets increasing the number of students both
capable and interested in pursuing careers in STEM disciplines. A few scenarios have been
analyzed that examine changes for improving student capabilities. Initial results provide insight
into the value and viability of a few proposed policies and indicate that with continued research,
model development, and analysis it will be possible to further assess proposed improvements in
the U.S education system.
INTRODUCTION
The Business- Higher Education Forum (BHEF) was founded to “advance innovative solutions to
[the] nation’s education challenges in order to enhance U.S. competitiveness.” (See
www.bhef.com) In its Spring 2006 Forum Focus, the BHEF described a future in which, owing
to a shortage of trained workers in the fields of science, technology, engineering, and
mathematics (STEM), the United States is no longer a leading contributor in science and
technology developments [1]. Though there has been debate over the nature, scale, and to a
degree the existence, of this problem, most experts seem in agreement that the problem is real
and increasing with time [2; 3]. To remain competitive in the global economy, the American
education system must provide an ever expanding and highly talented pool of STEM workers.
The downward trend in U.S. science and engineering degree attainment could significantly affect
the size and composition of the workforce available to industry. From 1980 to 2000, growth in
U.S. science and engineering degree production lagged growth in science and engineering jobs.
The dearth of U.S. job candidates was mitigated by an influx of foreign-born workers and low
retirement rates for scientists and engineers [4]. In projecting forward from 2002 to 2012, the
Bureau of Labor Statistics (BLS) estimates the need for science and technology workers will
increase by 26% compared to 15% for all occupations. They predict the need for
Page 1
computer/mathematical scientists will increase by 39% and the need for post-secondary teachers
will increase by 37% [5]. Without qualified teachers, the U.S. will have a very difficult time
training future generations of American-born STEM workers.
To address this problem, the BHEF launched a multi-year initiative, “Securing America’s
Leadership in Science, Technology, Engineering, and Mathematics,” to develop a strategy to
double the number of U.S. STEM college graduates by the year 2015. This initiative investigates
a variety of problems that exist in today’s education system, such as low student participation,
declining achievement in STEM subjects relative to other countries, the shortage of qualified
STEM teachers, and the lack of participation by women and minorities in STEM disciplines.
Raytheon Company Chairman and CEO Bill Swanson, who is co-chair of BHEF’s STEM
initiative, conceived of the idea of applying systems engineering to the U.S. education system as
a way to organize the problem and help to determine the effectiveness of proposed solutions so
that priorities could be set and guidance could be provided to policy makers. This innovative
application of systems engineering skills is part of the company’s multi-pronged approach to
improving science and math education.
Though the complexity of the U.S. education system makes it very difficult to isolate problems
for independent analysis, experts in a number of fields, especially the social sciences, have
produced a vast supply of studies and publications addressing a variety of issues. Economists
have looked at the role of incentives, such as teacher pay, in producing both well-trained STEM
teachers, and in attracting students to STEM fields [6; 7; 8; 9]. The charged political debates in
the U.S. over merit pay and student testing demonstrate the controversies generated by certain
economic approaches to this problem. Educational theorists have debated just how to define and
assess teaching quality, a debate that ties very closely to economic arguments over teacher pay
[10; 11; 12]. Initiatives aimed at improving student capabilities, such as reducing class size,
have been proposed and implemented with little supporting research. Subsequent research has
indicated that class size changes had limited effectiveness and has even had unintended
consequences such as creating a shortage of qualified teachers [13; 14].
Previous work has been done that applies systems engineering principles to the examination of
the education system. Of particular value have been the critical path analysis studies produced
by the California Council on Science and Technology (CCST) [15; 16]. This method uses a
static approach to examine the structure and effectiveness of the California education system. In
[16] the CCST researchers conclude; “Perhaps eventually, a more truly dynamic interactive
model may be achieved, one that would enable policymakers to understand and respond to the
functioning of the overall system on an ongoing basis.” The CCST critical path analysis
determined that a shortage of mathematics and science teachers persists within Califomia
schools, especially low-performing schools. CCST recommended the first step should be
legislation to collect teacher workforce data necessary for fully understanding and analyzing the
current situation and trends. These data will support initiatives aimed at improving teacher
recruiting, professional development and retention.
The analyses in this paper have drawn on the work of these experts - economists, educators,
political scientists, and leaders of industry. The majority of the existing research and analyses
are limited by being static and too narrowly focused; they investigate one part of the over-
arching problem and attempt to expand their conclusions from there. For example, many
changes, when applied, have been successful only in limited applications and others have not
Page 2
produced the expected results. Often the experts and the analysis produce contradictory results,
due to the scope limitations of their particular research studies. None of the previous studies
found provide a complete description of the K-16 education system and the problems that
confront the U.S. when it comes to the production of STEM graduates. These approaches have
been unable to describe how effects, impacts, and changes in one part of the U.S. educational
system flow through and impact the other parts of the system or how changes propagate through
time.
A dynamic systems engineering based tool that provides a means of examining the intricacies of
the entire U.S. education system is necessary to overcome the limitations of past research and
analysis. Such a tool will assist in exploring the effectiveness of proposed solutions, so that
priorities can be set and guidance provided to policy makers. In addition, the process of creating
this systems engineering tool provides a means of identifying potential solutions and an
organized approach for assessing them.
The systems engineering activities that led to the modeling approach and the focus of the
modeling efforts on teachers is reported in [17]. Early results were provided in [18] with a high
level overview of the modeling activities and some of the preliminary results.
The next section presents identified problems that exist within the U.S. education system, and
indicates some potential areas for change. This discussion is followed by a brief introduction to
the system dynamics modeling approach used for this study. These introductions lead into a
presentation of the system dynamics model of the U.S. education system, and the results of three
policy analyses. Two of the cases examined were judged to be unacceptable in the current
political environment. The third alternative provided the desired increase in STEM students and
appears to be within the realm of political acceptability. The results are summarized and plans
for future work, including on-going research and analysis, and development of an open source
implementation of the model, are provided.
THE UNITED STATES EDUCATION SY STEM
STEM education within the United States constitutes a very complex system that includes public
and private institutions starting at pre-school levels and continuing through colleges and
universities that offer graduate degrees. For this examination the model was limited to public
elementary and secondary schools, and colleges and universities that offer bachelor’s degrees in
a STEM discipline or a related teaching discipline. Within the U.S., the public elementary and
secondary schools contain 90% of the student population.
The U.S. public education system teaches approximately 3.6 million students in each grade level,
from first to eighth grade (see Figure 1). After eighth grade, students begin to drop out of the
school system and many do not graduate with their 12" grade class. Of the students who do not
graduate from high school within four years, about half never get a degree, while the other half
eventually get a degree or a general education development (GED) certificate. About 2.5 million
students graduate high school each year and most attend college at either a two year or four year
institution. Only 23% of the students enrolled in college (15% of the total 3.6 million
population) choose to major in a STEM discipline in college and about 40% of those who elect
STEM majors freshman year receive a STEM degree within six years (about 6% of the total 3.6
million student population) [19].
Page 3
STEM-Interested Compared to All in School
F Gracuate
STUDENT 9 5
POPULATION <"~
mritos 2
13
1
5
0
8th 12th Freshman BS
CAADE
Figure 1. Attrition among students (total), and STEM proficient and interested students.
Students are assessed regularly throughout their education to determine progress in mathematics.
For the purpose of this study we selected student capability in math as the indicator of STEM
interest. While it is understood that some students are proficient in math but not interested in
STEM, there may also be some who are interested in STEM but who have marginal proficiency
in mathematics. Math assessments divide the students into the categories of below-basic, basic,
proficient and advanced. Proficient and advanced students were equated with STEM interested
students, and the assumption was made that this equation is viable from an analytical point of
view, even though many exceptions exist. This assumption was supported by the fact that the
numbers of students who are proficient or advanced in math at the 12” grade level are
approximately equal to the number who declare their intent to pursue a STEM major freshman
year in college [20].
The students who are proficient or advanced in mathematics represent 36% of their class in 4"
grade (see Figure 2). This percentage gradually decreases at about 1.5% per year in elementary
and middle school and by about 3% per year in high school as the students progress through the
education system. By 12" grade only 17% of the student population is proficient or advanced in
math [19]. This flow of students from interested to uninterested, as represented by their
capabilities in math and their stated degree major in their freshman year is represented in the
model.
Among the many factors within the education system that influence student achievement and
interest in mathematics and science, research indicates that the quality of a student's teacher is
the most important factor. Statistical analysis indicates that teachers account for about 8.5% of
student variation in performance during elementary and high school [12]. Moving a student
from an average teacher to one in the g5m percentile increases the student’s rank by 7% [12].
Additional data from Gordon, Kane and Staiger provide an analytical basis for modeling teacher
influence on student performance and interest as a normal distribution [21]. Some teachers will
advance student rank, while others will reduce student rank. For the purposes of the model,
teachers who improve the average student's rank were defined as “STEM-capable” and the
remainder of the teachers as “not-STEM capable” (see Figure 3).
Page 4
% (of 3.6M) Proficient or Advanced
40
6% ra .
30 Ss io 15%
_ Declared |
20 +
oN
10
0 NY 6%
4th 8th 12th Receive
Grade \_ Degree |,
Figure 2. STEM proficiency declines in middle school (grades 4-8) and high school (grades 9-12).
Distribution data from [21] show a slight skew such that there are fewer STEM-capable teachers
than not-STEM-capable. The general form of these data appears to correlate with the gradual
decline in student proficiency as students progress through the grades. This apparent correlation
led to a model of student change in capability (and interest) based on the relative size of the
STEM-capable and not-STEM-capable teacher populations. Shifting the distribution changes the
number of teachers who are STEM-capable and the proficiency of the students. A mean of zero,
corresponding to equal numbers of STEM-capable and not-STEM capable, for instance,
produces no change in the numbers of math proficient students from one grade to the next.
Teacher Capability Distribution
1c =7.5% Mean =0 (blue) and -3% (red)
Not-STEM-Capable «<—}—* STEM-Capable
9.35
Percent Change in Student Rank
93 =
VA moe SS
VLE 9.2 aS
a 9.15. SS
a 94 SS
s15 -12 39 -6 3 0 3 6 9 12
15
Figure 3. Teacher capabilities can be modeled by a normal distribution related to their ability to impact student
performance.
Page 5
Changes in policy related to the number of STEM-capable teachers can shift the distribution
mean or reduce/increase the numbers of teachers in any part of the distribution. As suggested by
Gordon, Kane and Staiger, one of the potential methods for improving student performance is
through identification of teachers in the lowest quartile of the ranking and then reducing their
numbers through denial of tenure or development programs that improve their capabilities [21].
This concept was examined using the model and it was found that its effect was dramatic and has
the potential for eliminating the decline in student math proficiency as students progress through
the education system from 4" to 12” grade.
In order to simplify the model, and because of a relative lack of knowledge about the exact
distribution of teacher capabilities, the current version of the model computes the change in
student capabilities based on the ratio of two populations: STEM-capable teachers and not-
STEM-capable teachers. This simplification is approximately correct for symmetrical
distributions with means near zero (i.e. a fraction of a standard deviation) and is adequate for
examining the trends in student capability as changes to the ratio are made.
ROLE OF SYSTEM DYNAMICS
System dynamics has been used to evaluate the implications of success in the education system
[22], to study knowledge management in engineering education [23], and to study the
performance of research and development in the South African higher education sector [24].
Alan Gaynor in his book, Analyzing Problems in Schools and School Systems, proposes using
system dynamics modeling as a method for analyzing school systems. Gaynor develops a
dynamic hypothesis using methods similar to those employed in this analysis, and described in
the Model Structure section below, to support what he calls “The Effective Schooling Project.”
The essence of the problem posed in The Effective Schooling Project, is that in ineffective
schools initial differences in children’s readiness to lean at school were systematically
magnified over the course of their educations [25]. Gaynor however, does not develop a
mathematical model that can be simulated.
MODEL STRUCTURE
Reference Mode
Figure 4 illustrates the potential reference modes for the U.S. education system being studied in
this paper. The historical curve shows that approximately 3.6 million students enter the U.S.
public education system each year and only 6% of this population attains a STEM bachelor’s
degree within six years of entering college. The feared future is that the historical behavior will
continue at, or degrade to less than 6% as represented by the curve labeled Fear. The desired
future is to grow the 6% number to 12% by 2015. The pattern of behavior for this growth may
have a range of shapes as shown by the curves A and B. Curve A, in Figure 4, is an example of
exponential growth behavior that we would like to occur. This has the advantage of high rates of
change, but the disadvantage of delayed initiation. Exponential growth is a result of positive
(self reinforcing) feedback loops. Positive feedback loops cause growth, they cause
amplifications, and throw systems out of equilibrium.
Page 6
% (of 3.6 M) Students Receiving STEM Degrees
12%
% Receiving
STEM Degree
exponential growth
. sam goal seeking
Figure 4: Reference mode dynamically describing problem.
A positive feedback loop identified that generates exponential growth is shown in Figure 5. An
increase in STEM Capable Teachers causes more students to become STEM Interested Students.
As the numbers of STEM Interested Students increase more students will graduate from high
school and select STEM teaching as a career path. More STEM Students Selecting STEM
Teaching as a major in college results in more STEM Capable Teacher Graduates. More STEM
Capable Teacher Graduates means more can be hired and become STEM Capable Teachers. An
increase in STEM Capable Teachers causes an increase in the number of STEM Interested
Students, completing the positive feedback loop.
Students
a
STEM Interested
Students
STEM Students
Selecting STEM
STEM Capable wD Teaching
Teshers cy
\
| Creating STEM \
\ Capable Teachers |
4
STEM Capable STEM Teacher
Teachers Hiring College Students
? a
STEM Capable
Teacher Graduates
Figure 5: Positive feedback loop that creates more STEM Interested Students by creating more STEM Capable
Teachers.
Page 7
However, real quantities cannot grow forever. There are limits to continued growth. One limit
to this growth is the total number of students. The number of STEM Interested Students is a
percentage of the total number of students. There cannot be more STEM Interested Students
than the total number of students. An additional limit was added by assuming that students who
had lost interest in STEM did not regain that interest (see section on Modeling Assumptions).
This assumption along with using the fourth grade levels of proficiency as the initial conditions
for interest, set the upper limit at the number of proficient fourth graders.
Curve B, in Figure 4, is an example of goal seeking behavior. Goal seeking behavior is a result
of negative feedback loops. Negative feedback loops drive systems towards equilibrium; the
system seeks a desired state or goal. Negative feedback loops counteract forces that take a
system out of equilibrium. A negative feedback loop identified that generates goal seeking
growth is shown in Figure 6. There is a goal to maintain the ratio of students to teacher. This
goal determines how many STEM Capable teachers are desired. In the loop below, the state of
the system, STEM Capable Teachers, is compared to the desired state of the system, Desired
STEM Capable Teachers. When there is a difference between the actual state and desired state,
Gap Desired vs. Actual STEM Capable Teachers, administrators take corrective action
increasing or decreasing the number of teachers. If the gap is positive, STEM Capable Teachers
Hiring increases. This increase in hiring increases the actual number of STEM Capable Teachers
which decreases the Gap Desired vs. Actual STEM Capable Teachers, completing the negative
feedback loop.
4 Closing STEM
Sr reaches Teacher Gap STEM Capable
| Teachers Hiring
| 2)
a Gap Desired vs Actual
SHED Interested STEM Capable Teachers
Students ef
i Desired STEM we
~» Capable Teachers _
Desired Student
Teacher Ratio
Figure 6: Negative feedback loop that seeks the desired number of STEM Capable Teachers.
Page 8
There is not enough space in this paper to individually address every feedback loop. The
complete dynamic hypothesis developed for this project is shown in Figure 7.
Students
“BTEM Students ——
Selecting STEM
Industry
ye ~ \
LZ > stent wn capable Ny STEM Dace iP STEN industy
STEM oon “Teacher Attrition \ STEM Teachers / College Students
Capable Teaches eM noncapable —/ a \
Ke Teacher Atwiting _/ . STEM Students | Ae
f ~ 7 “ 4. Selecting STEM | |
/ More STEM non Capable Teaching i +
/ | \ paenen Teachers= Less STEM Ke | 8 -STEM Industry
f { \ 7 i: Interest if 58 = 6
[SoemGemle Nth aan itera tapas | Biren sian,
| n | Teachers Hing recor Poa Draining, [pestis sos Lesinty fk moat |
if o \ aD \ STEM non Capable } lary {aise |
I \ Z \ Teacher Gradyates / \ |
\ STEM stem eS ee < ee: \ 4
|\ teatiers & Capable Teachers ~~ STEM Teacher STEM I
\.Attniting Capable STEM Capable College Students ‘STEM Industry
Teachers + Teachers Hiring a oe i \ Hiring
et, * A a STEM Industry Dy \
| STEM Teacher gE Capable /Staxting Salary iS) \
\ \. Pool Draining / e i 00a
VS Tesh Gates : / wos
wu er a J Gap Desi vs Actual gteM indestry
Se STEM Capable STEM Industry Professionals ~
\. More STEM Students, More Teachers, ‘Teacher Starting Salary Professionals =. \
\ More STEM Teachers LessTeachers =F q + = | ey X
Gap Desired vs Actual H \ = |
\ EM Capable Teaches» \ \ Fee
+ Besired STEM f g %, \ SESTEM Industry
gy Capable Teachers 4 we hol Desired STEM Retirees
a see Closing STEM Capabl Industry Professionals
Desired Student
Teacher Ratio
Gap with NotSTEM
Figure 7. Dynamic Hypothesis.
Stock and Flow Model
The stock and flow model for the U.S. education systems represents the flow of students through
the system from birth to retirement. Stocks define the state of the system. They represent
“things” that accumulate, for example numbers of students. Flows define the rate of change in
systems states, for example the rate at which students graduate from high school. Various flow
paths model students who are interested in STEM and those who are not. Additional flows are
created to model students who become teachers or apply their STEM skills in industry. The
modeling method allows for numerous altemative flows and provides a means of controlling the
flow into and out of each stock (group of people) using the dynamic hypothesis as the basis.
Figure 8 provides a summary view of the complete stock and flow model developed; the full
model is too complex to capture in this document, but it is available to those interested in
examining it or using it for further research.
Page 9
effect STEM teacher
quality on student STEM
interest
Pe * \
: Teacher
secondary \
sre \ \ \
uninterested / | \ |
acai / _-* desired STEM total STEM |
onl teachers |
Sa ce) [pee
hide entering” | terested Sem) unlit | / desedsudentto | \ /
uninterested Saents ‘uninterested remaining L_Students, { ‘eacher atio —— \ hs
uninterested f ‘gap desired vs actual
sn 1 ‘STEM teachers |
Children elementary SEM scone STEM | Isren capatie
ered ent ) senemotirmiad college STEM Teachers STEM capable
capable teacher teachers rein
g graduating
College STENT
: Teacher
Tene conta wae STEM
children enterint STEM STEM secondary STEM Students: college STEM nonCapable ‘SIEM Retirees
‘nterestad © | [mlempsted lementary STEM | '2ereed | sterestod selecting wncepableteacher| “Teachers | STEM
~ interested remaining ‘—S STEM teacher college ‘graduating eagle
\ interested / a
<qap desired /
\ E Mee
tofl STEM
interested students ey Cage STEM
anes Interested aoe
secondary STEM — | [attested coteg STEM | Professionals =e
interested selecting industry oe
STEM industy college pelea rots ng
Figure 8: Simplified Student Stock and Flow Model.
The complete model begins with a simple left-to-right structure. (see Figure 8) Students are bom
and enter the education system on the left side of the model and then progress from grade to
grade, graduate from high school, attend college, get a job, gain experience, and eventually retire
out the right side of the model. The flow represented by this chain of events is subdivided in the
Kindergarten-12" grade years as follows: one chain tracks STEM interested students and the
other tracks students who are not interested in STEM. Students who do not pursue a STEM
major in college are not tracked post high school graduation. STEM interested students who
graduate from high school and pursue a STEM major in college, or an education major related to
STEM, are tracked, and flow into the next portion of the model.
The model includes a flow in each grade between the STEM interested students (stock) and the
STEM uninterested students (also a stock). These flows represent the rates at which students
become uninterested in STEM. For the initial study, only teacher influence and its effect on
STEM interest was considered. Teachers have the potential for moving students up or down in
the rankings. STEM-capable teachers move students up relative to the average, while not-
STEM-capable teachers move students down. For the model it was assumed that students who
are proficient or advanced at math are interested in STEM. (This was the most reasonable
assumption at the time the model was created because very little were available on student
interest. Recent data indicate that interest and proficiency are independent variables. These new
data are being used to change and expand the model at account for interest and proficiency
separately.)
Post-college the model is divided into two major chains: STEM interested students who pursue a
career in teaching STEM, and STEM interested students who pursue a career in industry. These
chains each have two elements: the time spent in college, and the time spent employed in the
Page 10
chosen profession. The STEM interested students who pursue a career in STEM teaching are
further divided into four chains: 5-8" grade STEM-capable teachers, 5.8" grade not-STEM-
capable teachers, 9°12" grade STEM-capable teachers, and g.12 grade not-STEM-capable
teachers. These divisions allow examination of the dynamics of being taught by a STEM
capable teacher versus a not-STEM-capable teacher.
MODELING ASSUMPTIONS
Data related to the U.S. education system are limited and often contradictory. An essential step
in the modeling process is to examine the data and determine if they are adequate for the creation
of a valid model. Often the validity cannot be established from the data, and in these cases
modeling assumptions must be made. The assumptions allow the modeling activity to proceed,
but each assumption must be validated with further research before the model can be declared
validated. Table 1 lists the modeling assumptions required for this evaluation due to limited data
availability.
Each of these assumptions, if changed, will have a significant impact on the modeling results.
One of the advantages of modeling the U.S. education system is that it allows for examination of
many possible assumptions to see which will have a significant impact on the results. The
assumptions that dramatically change the simulation results are the ones that should receive
priority in future research activities.
Table 1. Modeling assumptions.
Modeling Assumption
Rationale
STEM interest is closely related to STEM
proficiency.
At the time the model was created very little were available on student interest
in STEM. Recent data are being used to expand the model to independently
address student interest and proficiency.
A STEM-capable teacher maintains STEM
proficiency and interest within the class.
For the model, a STEM-capable teacher is defined as one who increases
proficiency of the class on average. The model predicts average behavior.
Not-STEM-capable teachers reduce student
proficiency average over a year.
This follows from the modeling definition of not-STEM-capable teachers.
Once proficiency (interest) is lost itis not
recovered.
While itis possible that students can recover from a bad teacher, research
indicates the effects last for years afterwards. This assumption also prevents
positive runaway in the simulation that clearly would not be representative of
the real world.
Administrators cannot determine which college
graduates will become STEM-capable teachers
Research that examines all teachers as a group indicates that 97% of what
makes a good teacher is not quantifiable or well known. Data specific to
STEM teachers are limited. The model assumes that the teachers hired match
the characteristics of the pool of new candidates (i.e. no sorting occurs when
hiring inexperienced teachers)
Denial of tenure will result in attrition of teachers.
There are no data that correlate attrition with denial of tenure. Tenure is rarely
denied in the current system, so very little data exist.
RESULTS
During the study many factors were examined and considered for implementation in the model.
After researching, evaluating and reviewing several policies, three were selected for detailed
examination using the model (See Figure 8 and Table 2). Two of the policies produced very
little improvement in student interest in STEM due to the limited ability of administrators to
selectively hire STEM-capable new teachers from the candidate pool of college graduates.
While these policies (increased salary and class size) might have the potential for improving the
system if administrators can selectively hire capable teachers, research data indicated significant
issues with implementation such as rigid salary structures, union resistance, public opinion, and
limited school funding. The policy that showed the greatest potential for increasing student
Page 11
interest in STEM is the introduction of attrition through denial of tenure to teachers who have not
demonstrated their capabilities to teach STEM within their first three years teaching. This policy
can be enhanced by training, mentoring and other teacher development programs that improve
performance. This approach could be implemented within the highly constrained U.S. education
system, and that has significant potential for improving the system.
A baseline model was run that introduced no changes to the U.S. education system. This run
used constant population statistics to avoid dynamic changes that result from population
variations. Initial conditions were set to continue current education system policies, resulting in
little change during the decades modeled. The level of student interest and capability in each
grade remains nearly constant as expected.
The second run of the model examined the results of implementing a policy that introduces
attrition within the ranks of teachers having three years of experience who were rated in the
lowest 10% of their peer group. A third run examined an alternate case with attrition of all
teachers rated in the bottom 25% of their peer group. These runs show sensitivity to this
particular change in education policy.
Table 2. Summary of results.
Hypothesis
Model Results
Factors
Conclusion
Identifying the not-STEM-
capable teachers and
improving their capabilities
or increasing their attrition
after three years will improve
student capabilities.
A dramatic improvement in
student capabilities is
produced by implementing this
policy
Data show that denial of
tenure is rarely implemented
[26]. Increasing attrition may
result in teacher shortages.
This policy has potential for
introducing improvements in
student capabilities in grades
4-12.
Increasing STEM teacher
salaries will increase the
candidate pool and better
teachers will be hired.
Assuming that administrators
can differentiate capable from
not-capable and that industry
does not compete, then
dramatic improvements seen.
Research shows
administrators cannot tell
capable from not-capable
coming out of college.
Increasing salary will increase
the candidate pool, but more
capable teachers are not hired
due to the selection process
and industry competes raising
their salaries to off-set the
teacher salary increase.
Increasing class size will
reduce the demand for
teachers and allow
administrators to be more
selective in hiring.
Assuming administrators can
differentiate capable from not-
capable, improvements are
seen. Natural attrition was
modeled. Assumed equal
attrition and replacement from
capable and not-capable
Research shows
administrators cannot tell
capable from not-capable
coming out of college. Using
the change to lay-off less
capable teachers was not
modeled.
The change delayed all hiring
for a couple of years so no
input changes occurred in that
time. Improvements did not
occur due to limited
differentiation during hiring,
Figure 9 displays the comparison between the baseline and the new policies that create 10% and
25% attrition among the lowest-rated third year teachers.
Implementation of this policy provided a dramatic change in the numbers of STEM-capable
teachers, and in the numbers of students who are proficient or advanced in math and presumed to
be interested in STEM.
The next “what if” scenario investigated was what would happen if the student-teacher ratio was
increased, making each class larger. In the baseline run the desired student teacher ratio was
17:1, i.e. 17 students per class. In the “what if” scenario the student-teacher ratio was increased
to 25:1.
Page 12
STEM Interested High School Graduates
Declaring STEM Major
1,200,000
2 ae
i 800,000
H
3 400,000
2 2 ry 5 ry 4 iy ry 5 o aT ra «
~ Baseline Year
+ 100% Attrition Bottom 10% 3rd Year
+ 100% Attrition Bottom 25% 3rd Year
Percent STEM Capable Teachers
4 aos
é
2007
2008
20009
2010
zou
2o12
2013
zou
2015)
2016
2017
2018
2019
2020
2021
2022
2023
2028
2028
Year
Baseline
-+-100% Attrition Bottom 10% 3rd Year
“+ 100% Attrition Bottom 25% 3rd Year
Figure 9. Baseline case (red) compared to the improvement provided by implementing the dynamic
hypothesis with attrition of lowest 10% (green) and attrition of the lowest 25% (blue).
It is seen (Figure 10) that as classroom size increases, a delay in the need for new teachers is
hiring of STEM capable teachers. The delay in hiring slows down
capabilities because improvement depends on the hiring of new
introduced, which delays the
the improvement in student
teachers.
The next policy change investigated was a one time step increase in teacher salary. This change
did not result in additional students being interested in STEM during their tenure in school. The
surprising result is caused by the competition between industry and the teaching profession for
STEM capable students. Increasing teacher pay to an extent that dramatically increases the
number of STEM capable students not selecting industry results in a shortage that causes industry
to raise salaries. The rapid industry response (within a year or two) offsets the change in teacher
pay.
Page 13
Classroom Size
12
ian [
ec L
Boas
ic
5
o
~ Baseline Year
+ 100% Attrition Bottom 25% 3rd Year Larger Classroom
100% Attrition Bottom 25% 3rd Year
STEM Interested High School Graduates
Declaring STEM Major
1,200,000
1,000,000 Ses
on =
$ 600.000 —
2
200,000
e268 8 3 2228 8 8
-* Baseline Year
+ 100% Attrition Bottom 25% 3rd Year Larger Classroom
-+- 100% Attrition Bottom 25% 3rd Year
Figure 10. Baseline case (red) compared to the improvement provided by implementing attrition of lowest
25% (blue), and the larger classroom with attrition of the lowest 25% (green).
SUMMARY
Application of systems engineering methods and system dynamics modeling has achieved the
study’s primary objective, to provide an organized means of examining and analyzing the U. S.
education system. The model was built by examining the numerous factors that influence
student performance and interest and the effects of these influences. The process of creating the
model looked at these influences one by one in an organized and rational fashion. Modeling
activities have indicated where additional research and data are needed and have provided a
means of showing the intended and unintended consequences of policies and actions.
At this time, the very limited research data available for building the current model make it
impossible to state results with high confidence. However, the model still begins to provide a
highly effective tool for understanding trends and for organizing the research process.
Continuation of research efforts, especially in those areas needed to enhance the model,
combined with additional model development and validation, can eventually provide an effective
tool for predicting with some certainty the results of policy decisions on the U.S. education
system. Achieving this goal may take time but this effort can provide tremendous benefits.
Preliminary analysis and modeling of the U.S. education system indicates that reducing the
numbers of teachers who are not-STEM-capable provides an effective method for improving
student performance. Research shows that school administrators can determine, after three years
Page 14
of assessment, which teachers are STEM-capable and which are not. This knowledge can be
used to deny tenure to the least capable teachers, which should lead to attrition. The system
dynamics model shows a dramatic change in student performance as a result of reducing the
numbers of not-STEM-capable teachers through either attrition or training. To date, there is no
indication that school administrators have the ability to delay or eliminate tenure for less capable
teachers. There is even less likelihood that school administrators will eliminate the least capable
teachers through firing or lay-offs. Less than 1% of teachers leave the profession as a result of
school staffing actions of all types [26]. In addition to resistance from school administrators, a
change of this type will face substantial resistance from teachers unions. The results of the
system dynamics modeling effort, once validated, should help to persuade policy makers,
teachers and school administrators to take bolder actions to improve the quality of education.
RECENT RESULTS AND PLANS
This paper presents the initial version of a system dynamics model of the U.S. education system
developed in 2006 and 2007. Initial investigations focused on grades Kindergarten-12 and used
aggregate data for the total U.S. public education system population. The available data clearly
indicate the existence of distinct populations who behave differently from the average student
within the U.S. education system. Among these populations are women and disadvantaged
students, especially those attending inner city schools. During 2008 versions of the model that
address these populations separately were created. In addition, the model detail and fidelity in
the college years has been enhanced to provide year by year flow of the students so that attrition
in the early college years can be examined.
Raytheon is continuing the modeling effort and is working with the Ohio State University to
make the model available as an “open source” on the internet to anyone who is interested.
Publication of the model and support for the model in the future is being provided by the BHEF.
It is hoped that additional research will be performed that enhances the model, increases its
fidelity, and provides validation.
On-going modeling activities are enhancing the model to allow separate examination of student
interest and proficiency. Recent research indicates that interest and proficiency are independent.
This is apparent in populations of women where proficiency far exceeds their interest levels in
engineering and the physical sciences.
ACKNOWLEDGMENTS
The authors would like to thank the Raytheon Company and BHEF for supporting and
encouraging this research activity. The authors would also like to acknowledge the Ohio State
University for their efforts to improve the model and to make it available as an open source.
Page 15
REFERENCES
[1] Business Higher Education Forum, “Can America Globalize Itself?” Forum Focus, Spring
2006.
[2] National Science Board (NSB), “America’s Pressing Challenge - Building a Stronger
Foundation”, January 2006, http://www.nsf.gov/statistics/nsb0602/nsh0602.pdf .
[3] Committee on Prospering in the Global Economy of the 21% Century, National Academy of
Science, National Academy of Engineering, Institute of Medicine, “Rising Above the Gathering
Storm - Energizing Employing America for a Brighter Economic Future,” The National
Academies Press, www.nap.edu, 2006.
[4] National Science Board, “Science and Engineering Indicators 2006”, Arlington, VA:
National Science Foundation, 2006, (volume 1, NSB 06-01), pp. 013-014.
[5] National Science Board, “Science and Engineering Indicators 2006”, Arlington, VA:
National Science Foundation, 2006, (volume 1, NSB 06-01), Figure 3-5, p. 3-9.
[6] Murnane, Olsen, Randall, “The Effects of Salaries and Opportunity Costs on Length of Stay
in Teaching: Evidence from North Carolina,” Journal of Human Resources, 1990.
[7] Wilson, Suzanne, Edutopia online, “Higher Pay, Higher Standards: Balancing the Equation in
Connecticut,” www.glef.org. 4/29/2003.
[8] Milanowski, Anthony, “An exploration of the pay levels needed to attract students with
mathematics, science and technology skills to a career in K-12 teaching,” Education Policy
Analysis Archives, 11(50), December 27, 2003.
[9] Podgursky, Michael, “Is Teacher Pay “Adequate?” JFK School of Government Cambridge:,
Harvard University, October 2005.
[10] Goldhaber, Dan, “The Mystery of Good Teaching,” Education Next, Spring 2002.
[11] Hanushek, Eric, “Teacher Quality,” from Teacher Quality, edited by Lance Izumi and
Williamson Evers, Hoover Institute Press, 2002.
[12] Koppich, Julia, “All Teachers are Not the Same,” Education Next, Winter 2005.
[13] Hanushek, Eric, “Some Simple Analytics of School Quality,” NBER Working Paper 10229,
January 2004.
[14] Hanushek, Eric, “Can Governments Legislate Higher Teacher Quality?” LACEA paper,
October 2001, http://www.nip-lac.org/proqrams lacea/Hanushek.pdf.
[15] “Critical Path Analysis of Califomia’s Science and Technology Education System,”
California Council on Science and Technology, A pril 2002.
[16] “Critical Path Analysis of California’s Science and Mathematics Teacher Preparation
System,” California Council on Science and Technology, March 2007.
[17] Wells, Brian, Alex Sanchez, Joanne Attridge, “Systems Engineering the U.S. Education
System,” (paper presented at the INCOSE 2008 Symposium, Netherlands, June 2008).
Page 16
[18] Wells, Brian, Alex Sanchez, Joanne Attridge, “Modeling Student Interest in Science,
Technology, Engineering and Mathematics,” (paper presented at the IEEE Summit, “Meeting the
Growing Demand for Engineers and their Educators,” Munich Germany, November 2007).
[19] Snyder, Thomas D., Alexandra G. Tan, Charlene M. Hoffman, “Digest of Education
Statistics 2005,” Institute of Education Sciences, National Center for Education Statistics, U.S.
Department of Education, NCES 2006-030, 2006.
[20] University of California at Los Angeles, Higher Education Research Institute (HERI)
Survey, http://www.gseis.ucla.edu/heri/index.php,
[21] Gordon, Robert, Thomas J. Kane, Douglas O. Staiger, "Identifying Effective Teachers
Using Performance on the Job," The Brookings Institution Hamilton Project, Discussion Paper
2006-01, April 2006.
[22] Mehmood, Arif, “Modeling Framework for Understanding the Dynamics of Leaming
Performance in Education Systems”, (paper presented at The 23rd International Conference of
the System Dynamics Society, July 17-21, 2005 Boston).
[23] Rodrigues, Lewlyn L. R., Martis, Morvin Savio, “System Dynamics of Human Resource
And Knowledge Management In Engineering Education,” Journal of Knowledge Management
Practice, Vol. 5, October 2004.
[24] Gorbbelaar, Saartjie, Buys, Andre, “Research and Development in the South African
System of Innovation - Application of a System Dynamics Model to the Higher Education
System”, (presented at the IAMOT 2006 Conference, Beijing, P.R. China, May 22 - 26, 2006 ).
[25] Gaynor, Alan Kibbe, (1998) Analyzing Problems in Schools and School Systems, Mahwah,
NJ: Lawrence Embaum Associates.
[26] Ingersoll, Richard M., “Teacher Turnover and Teacher Shortages: An Organizational
Analysis,” American Educational Research Journal, Fall 2001 Volume 38 Number 3, Table 5,
pp. 520-522.
Page 17
BIOGRAPHIES
H. Alex Sanchez is a Senior Principal Systems Engineer on the Mission Innovation Cross
Business Team for Raytheon Integrated Defense Systems (IDS). This cross product team uses
mission analysis, synthetic capabilities, systems design and architecture, and human interfaces to
assist IDS in becoming the world’s leading Mission Systems Integrator. Alex leads analysis
activities that examine system performance using discrete-event and/or system dynamics
modeling and simulations. Prior to this assignment Alex served as Program Manager for
Collaborative Solutions. In this position he developed supply chain processes and decision
support systems. Prior to joining Raytheon Alex worked in the semiconductor and jet engine
industries. Alex holds a Bachelor of Science in Mechanical Engineering from Boston University
(1995), and a joint Master of Science in Engineering and Management in System Design and
Management from the Sloan School of Management and the School of Engineering at the
Massachusetts Institute of Technology (1999).
Brian Wells is the Raytheon Chief Systems Engineer and a Senior Principal Engineering Fellow
within the Raytheon Corporate engineering organization. Prior to this assignment, he was the
Technical Director of the Future Naval Capabilities organization, the Total Ship Systems
Engineering Lead for the Navy’s DDG 1000 Zumwalt program and the Raytheon Chief Engineer
for the next generation aircraft carrier (CV N-21) warfare system development program. He has
held a number of management positions including manager of the Systems Engineering Center,
PATRIOT System Engineering manager, and manager of the Missile Concept and Design
department. He holds a Bachelor’s degree in Electrical Engineering from Bucknell University
(1975) and a Master's degree in Electrical Engineering from the University of Illinois (1976).
He is a member of the Intemational Council on Systems Engineering (INCOSE), Institute of
Electrical and Electronics Engineers (IEEE), the National Defense Industrial Association
(NDIA), and the Surface Navy Association.
Dr. Joanne Attridge works for Raytheon Integrated Defense Systems (IDS) as a Systems
Engineering Manager for the PATRIOT Radar Surveillance group. Prior to joining Raytheon
three years ago, Joanne worked as a Research Scientist in Radio Astronomy at the Massachusetts
Institute of Technology’s Haystack Observatory. She was nominated to the Systems Engineering
Technical Development Program (SEtdp) and awarded Raytheon Technical Honors in 2006.
Joanne received a Bachelors degree in Astronomy from Wellesley College in 1989, a Master’s
degree in Astronomy from Wesleyan University in 1992, a Masters degree in Physics from
Brandeis University in 1994, and a Ph.D. in Physics from Brandeis University in 1998.
Page 18
