Building Slightly More Complex Models: Calculators vs. STELLA
Diana M. Fisher
Wilson High School
1151 SW Vermont St.
Portland, Oregon 97219
1-503-644-2719
dfisher25 @verizon.net
If students are to develop the potential to effectively manage ubiquitous complex systems
it is becoming increasing important to develop systems thinking concepts and model
building skills formally at the pre-college level. This paper describes an experiment
conducted in two secondary school classrooms in the Pacific northwestern United States
to determine the importance of access to a relatively new modeling tool for students to
enable them to successfully create and analyze simple models that are slight extensions
of traditional models, as compared with using graphing calculators to build an analyze
the same extended model scenarios. Does the modeling tool make a difference? If it
does, access to such tools must be addressed before a broad spectrum of formal
curriculum can incorporate the system dynamics method, at the pre-college level.
In their article "Thinking about systems: student and teacher conceptions of natural and social
systems," Booth Sweeney and Sterman (2007) remind us how poorly some very educated adults
are at systems thinking and emphasize the need for building skill in this area, particularly in the
K-12 environment. If students are to develop the potential to effectively manage ubiquitous
complex systems it is becoming increasing important to develop systems thinking concepts and
model building skills formally at the pre-college level.
Current research in both mathematics and science instruction, at the pre-college level, support
active, student-centered involvement when new concepts are to be learned. It has also been the
experience of this author that students learn effectively when they are actively engaged in
building skill when working with abstract concepts. Mathematics, at the secondary school level,
is about learning relationships, growth and decay patterns, and building models to study
applications of these patterns. Middle school and secondary school' teachers have noted that the
STELLA software seems to be easier for students to use when building models, even of
traditional problems, in the math and science curriculum. All of the information to this point, on
the ease of building models using STELLA, has been anecdotal. But if students are to build
models of more complicated problems, so that feedback and other systems thinking analysis is
part of the problem being studied, it is important to determine if the software does make a
difference. This may not be a point that the System Dynamics community would question, but it
is important for the traditional math and science communities, since the primary tool currently
supported by almost all secondary schools is the graphing calculator. In many secondary schools
neither math nor science departments have regular access to computer labs.
" Students ages 12 to 18 years.
An experiment was conducted in two second year algebra classes at a secondary school in the
Pacific northwestern United States, to study the issue of the ease of building and analyzing
growth relationships when using a graphing calculator versus the STELLA software.
Student Experience
The students in the two classes ranged from grades 8 (14 years) to grade 12 (17 years). All
students have used graphing calculators for at least two and a half years, beginning with first
year algebra (if not before). The students in these classes used the STELLA software in the first
semester of this year (2007-08) on the following occasions. When studying linear functions,
they spent one class period (50 to 60 minutes) in the computer lab, building simple linear models
for three problems. When studying exponential functions another class period was devoted to
building exponential growth and exponential decay models. Similarly, one class period was
spent building quadratic models for various problems when the class was studying quadratic
functions. Near the end of the semester (January 2008) the classs spent three days building and
analyzing very simple drug models (a combination of linear and exponential structures). At that
time they were introduced to the Pulse and Step functions, briefly. On the fourth day, at the end
of the first semester the students were given another problem to study.
Choosing a Problem to Stimulate and the Creation of a Model
The problem given on the fourth day was identified as the Malthus Problem. In the Worldwatch
Paper’ "Beyond Malthus: Sixteen Dimensions of the Population Problem" the authors describe
the situation identified in Thomas Malthus' paper "An Essay on the Principle of Population,"
regarding the issue of population growth (exponential) and food supply growth (linear). The
new problem packet asked the students to determine why Malthus would think there would be a
problem when population grows exponentially and food supply grows linearly and what might
be done to mitigate the problem.
There were some serious constraints on the type of lesson that could be used for this experiment.
The problem had to be kept very simple so students could actually build the model using the
graphing calculator. The model had to contain functions that the students had studied recently
enough to recall the mathematical equations necessary for defining the population and the food
supply. Unfortunately, feedback analysis was not practical in such a simple problem.
Additionally, any discussion of feedback had been done very briefly when using the STELLA
software, so discussion of feedback would have been a disadvantage for those using the
calculator for this experiment. Also, the students using the calculator had to be taught the
equivalent of implementing the Step function for changing the value of a growth parameter, in
mid execution of a simulation. (This was introduced briefly to the class before the experiment
was conducted.)
° "Beyond Malthus: Sixteen Dimensions of the Population Problem. Lester R. Brown, Gary
Gardner, Brian Halweil, Worldwatch Paper. September 1998.
In the previous week (before the experiment) students were asked whether they preferred to work
with the graphing calculator or with the STELLA software. The students in each class were then
divided into four groups.
Group 1: Those students who wanted to use the graphing calculator and were assigned (at
random) to use the graphing calculator.
Group 2: Those students who wanted to use STELLA but were assigned (at random) to use the
graphing calculator.
Group 3: Those students who wanted to use the graphing calculator but were assigned (at
random) to use STELLA.
Group 4: Those students who wanted to use STELLA and were assigned (at random) to use
STELLA.
Students were identified only by their student number, their class period, and their preference for
modeling tool.
Students were given their assignment and given a reference page’ appropriate for their modeling
tool. Students were also reminded about how to write the electronic equivalent of scientific
notation, since most of the values they were to use in the models were in the billions or millions.
Students were given one hour to complete the assignment.
The Assignment Description
The assigned problem (packet) had several parts. After a brief introduction to the potential
problem students were asked to sketch three graphs. The time sequence was 200 years. Students
were to focus only on general shape for a food supply (grain production) graph, a population
graph, and a food per person graph. They were then to explain why they drew the food per
person graph in the shape they had chosen. This was all done before they were to use any
modeling tool.
The second part required students to build a food supply model, a population model, and
determine some way to get their modeling tool to calculate the food per person each year (for
200 simulated years). They were given values to use (from the Worldwatch paper). They were
to identify their STELLA structure or their math equation and also indicate how they calculated
the food per person. They were to get their modeling tool to display a food supply graph, a
population graph, and a food per person graph from 1950 to 2150. They were then asked to
explain any discrepancies between their original predictions of the shape of the graphs and the
model produced graphs.
* The reference page for STELLA contained an identification of each type of component, how to
create a graph (and put graphs on the same scale), how to create a table, and how to use a Step
function. The reference page for the calculator identified an example of setting up a piecewise
defined function (the calculator equivalent of changing a parameter value in mid-simulation, as is
done with the Step function in STELLA), how to activate and deactivate graphs, how to use the
Y-vars feature of the calculator to simplify typing in combinations of equations). All of these
calculator features had been briefly shown to the entire class before the experiment began. The
reference page was to help the students remember the process.
The calculator solution The STELLA solution
Y1=2.5£9(1.02)' population
Y2=631£6 + 22.65E6(t)
2.5e9
Y3=¥2 fal sek growth food per person
rate
food supply
2 63126 |
22.65e6
growth
Figure 1: The solution to the first model building section of the Malthus packet.
The third part of the packet required them to change some parameters in mid-simulation. After
establishing a minimum (arbitrary) food needed per person per year, a base case was established
for an approximate year when the food per person dropped below the minimum needed to sustain
each person. (Ignoring, of course, that food is not distributed equitably in the world.) Then the
population growth fraction was reduced in 1998 and a new year for food per person dropping
below the minimum needed was determined.
The calculator solution The STELLA solution
Two functions must now replace Y1 The net growth rate for the population structure
Y1=2.5E9(1.02)' (x < 48) must be changed to:
0.02 + STEP(-0.006, 1998
Y5 = 6.467£9(1.014)"" (x > 48) + ( , 1998)
Figure 2: The equations needed to modify the population in 1998.
Finally the food production was increased in 1970 and the model re-executed.
The calculator solution The STELLA solution
Two functions must now replace Y2 The growth (flow) for the food structure must
Y¥2=631£6+22.65£6(t) (x<20) be changed to:
Y6 = 1.08469 + 23£6(t ~ 20) (x2 20) 22,6586: STER(O-3560; 1970)
Figure 3: The equations needed to modify the food production per year in 1970.
To re-simulate with both changes:
The calculator solution The STELLA solution
Y7= (Y2+Y6) Just re-run the simulation
~ (Y1+Y5)
Figure 4: The equations needed to incorporate both a change in population and a change in food
production per year, for the final observation (graphical) of the year food per person drops below
minimum needed food per person.
The final part of the paper asked the students to explain the issue with food per person over the
next 100 years. It then asked students to come up with some policies that might stave off the
problem of insufficient food. Finally, students were asked if there would be groups who might
be opposed to their policies, and how they might convince those groups that the policies were
actually necessary.
Although there is no feedback analysis, an attempt was made to include some components of the
system dynamics process in the lesson (surfacing a students mental model, building an actual
model to test the mental model, using the model to help clarify the original problem behavior,
attempts to identify potential policies that might mitigate the undesired behavior the model
displays, and some thought about how those policies might affect different stakeholders.)
The results of the experiment
The assignment was broken down into 14 parts. For each part a score of | (correct or mostly
correct) or 0 (no response, incorrect response, or mostly incorrect) was recorded for each
student. One student's response was separated from all the others since he did not choose a
modeling tool preference and received zeros on all 14 parts.
Prediction:
Most students were able to draw a linear graph for food production and an exponential graph for
population growth correctly (over 80% for all groups except group 4 -> 71%). Predicting a food
per person graph was more difficult for the students (Groups 1, 2, and 4, about 65%, group 3,
59%) Students were given full credit for the food per person graph if they drew any graph with a
negative slope, or at least a negative slope toward the right end of the graph.
Setting up the initial models:
The calculator group was less successful writing the equations for their models (Group 1: 65%,
Group 2: 58%) than the STELLA group (Group 3: 76%, Group 4: 79%) When describing how
to calculate the food per person values for their model (over a simulated 200 years) the calculator
group averaged 60% and the STELLA group averaged 55%.
Modifying the initial model:
This is the segment of the experiment where the groups show the most significant difference. To
set up a minimum food per person baseline and compare, on the same grid, the calculated food
per person from the model was very difficult for the calculator group (average 30%) but better
for the STELLA group (51%). Answers were given full credit if the estimate for when the food
per person would fall below the minimum level needed, within 10 years of the "correct" year.
To change the rates in mid-simulation was almost impossible for the calculator group, better for
the STELLA group. Writing the correct equation to reduce the population growth rate at 48
years into the simulation: 9% for calculator group, 30% for the STELLA group. Estimating the
"correct" year (within 10 years) that this change will now have the food per person drop below
the minimum food per person: 5% for the calculator group, 34% for the STELLA group (some
students must have submitted a guess of the answer). Writing the equation to increase the food
production rate in the 20th year of the simulation was impossible for the calculator group (0%).
The STELLA group was more successful (30%). Estimating the "correct" year (within 10 years)
was not possible for the calculator group (0%) and a little more difficult than the previous change
(23%) for the STELLA group.
Explaining the results:
Correctly explained why there is a problem when population grows exponentially and food
production grows linearly: calculator group (42%) STELLA group (58%)
Suggested at least one reasonable policy to help mitigate the food shortage: calculator
group(42%), STELLA group (55%). Identified at least one subgroup of the population who
might not like the suggested policies: calculator group (34%), STELLA group (42%).
Explained at least one method to try to convince the disgruntled subgroup that the policy was
needed: calculator group (28%), STELLA group (42%).
It was possible for students who had a sense of the problem of food shortage at the beginning of
this exercise to (potentially) answer these final questions, even if their model did not work. The
lower results from the calculator group in this last section could be attributed to frustration at not
getting their model to work as required, and losing mental energy to continue. (Some students in
this group quit when they could not get their models to work.) On the other hand, actually
getting a model to work, for the STELLA group did seem to help them understand the problem
better. Their success at getting their model to work may have given them more stimulus to finish
the packet. Although these (emotional energy issues) were not questions that were quantified,
they play a significant part in a successful lesson.
A two proportion Z-Test
Let P, = Modeling performance/analysis of the group that used calculators
Let P, = Modeling performance/analysis of the group that used STELLA
Pa By
x x, =173 x, =219
oe : ; : 7 a n=448 | n, = 434
P=.444 | B =386 | B =.5046
Z=-3.539
p= .00020
Assume the null hypothesis, H,, is that there is no difference between the two groups (P, - P, =
0). The alternative hypothesis, H,, is that the performance and analysis of the calculator group is
less than the performance and analysis of the STELLA group (P, - P, < 0). The x values were
determined by adding all the correct responses for each group (calculator or STELLA) out of all
the possible correct responses for the specified group (the n values). The results of the two
proportion z-test produce a p-value = 0.0002. So the probability that there could be no difference
between the performance of the two groups would happen randomly only 0.02% of the time.
Therefore the data indicate strong evidence that using STELLA to create and analyze the
Malthus problem is significantly better than using the calculator to complete the same exercises.
Additional data about the four groups:
Let CCL represent average comfort level with computers as self-assessed by students (on a scale
of 1 to 5, with 5 being very comfortable). Let GPA represent average grade point average in the
advanced algebra class at the end of the first semester of the current school year (2007-08).
Group 1: Wanted to use calculators and did use calculators: CCL = 2.6, GPA = 82.6
Group 2: Wanted to use STELLA but used calculators: CCL = 3.17, GPA = 81.4
Group 3: Wanted to use calculators but used STELLA: CCL = 2.76, GPA 87.4
Group 4: Wanted to use STELLA and used STELLA: CCL = 3.07, GPA = 75.9
Malthus Experiment Analysis
100%
90%
oon H —
oo HG [|
60% +}
oe.
mez
ne3
nea
50%
Percent Correct
40% +4
30%
20%
10% 1
ow H im
Prediction P&F Prediction F/P Initial Model Calculate F/P Modify Model Explain Prob Design Policy
Analysis Segments
Figure 5: Average scores for each of the four groups on each segment of the Malthus
experiment.
Conclusion
The scenario of exponential population growth and linear food supply growth is about as simple
a problem as could be designed, given the constraints needed to allow students to build the
models using a graphing calculator. The overall results are disappointing, both because these are
college bound students who are fairly successful in school, and because we need students to be
able to analyze problems of much greater complexity than the one used in this experiment.
While the students in the calculator groups may improve their results if given more opportunity
to practice defining piece-wise functions on their calculator, that is not the point. The types of
problems that need to be added to the curriculum are those that require much more sophisticated
analysis than the simple problem given in this modeling exercise. If students, who are
comfortable using graphing calculators for most mathematical tasks, cannot extend their skill
level to include two relatively simple calculator tasks, especially when given an example to
follow, then it will be very difficult to expand the complexity of problems for this level of
student using that same tool.
This experiment indicates that there is a significant difference in the ability of students to
correctly build and analyze a problem that is a slight extension of what they have learned in class
when using either the calculator or the STELLA software. Add to this, that the students
have far less experience with the STELLA software than the graphing calculator, and the results
are even more impressive. The students who were able to use the STELLA software performed
significantly better than those students using the calculator, for the same task. This is a more
global initial evaluation and more follow-up analysis is needed.
If we want students at the secondary school level to build and analyze models of complex
systems, it is necessary to provide the students with tools, such as STELLA, to allow them to be
successful. Computer technology is available in most secondary schools in the United States.
Regular access to computer technology, by math and science classes, is a barrier. Until teachers
can have regular access to labs and appropriate modeling tools, teachers will not be able to
provide the experiences students need to enhance their analytical skill with complex systems.
Note:
It is intended that this be the first in a series of experiments designed to study the advantages of
incorporating system dynamics modeling and the system dynamics method of analysis in the
study of problems of a more complex nature than those currently presented in the secondary
school curriculum. Another experiment has already been conducted, but not yet analyzed. It is
entitled "The Study of Change in Behavior Over Time." It is divided into three parts: the study
of cause and effect, predicting behavior from viewing diagram structures, and identifying
multiple scenarios that match a given structure (transferability). This (continuing) series of
experiments is designed to develop statistical evidence with regard to the use of system dynamics
modeling at the 9-12 grade levels. The initial experiment, explained in this paper, was needed to
set the stage for future experiments. It was needed to (hopefully) convince administrators that
support for software designed to work seamlessly in building system dynamics models, would be
an educational advantage for students. Additional, it was intended to suggest that administrators
become attentive to the need to provide math and science teachers more access to computer labs.
References
Booth Sweeney, Linda and John D. Sterman. 2007. Thinking about systems: student and teacher
conceptions of natural and social systems. System Dynamics Review 23(2-3): 285-311.
J.W.Wiley.
Brown, Lester R., Gary Gardner and Brian Halweil. 1998. Beyond Malthus: Sixteen Dimensions
of the Population Problem. Worldwatch Paper 143. Worldwatch Institute.
Fisher, Diana M. 2001. Lessons in Mathematics: A Dynamic Approach. isee systems, inc.
Randers, Jorgen, editor. 1980. Elements of the System Dynamics Method. Productivity Press.
Sterman, John D. 2000. Business Dynamics: Systems Thinking and Modeling for a Complex
World. Irwin McGraw-Hill. New York.
Yates, Daniel S., Daren S. Starnes, and David S. Moore. 2005. Statistics Through Applications,
W.H. Freeman and Company. New York.
Appendix 1: More data about the composition of the groups
Group 1: 11 males, 9 females: one 8th grader, five 9th graders, nine 10th graders, four 11th
graders, one 12th grader: one 13 year old, two 14 year olds, seven 15 year olds, eight 16 year
olds, two 17 year olds: eighteen who were white, one Hispanic, one not specified ethnicity.
Group 2: 3 males, 9 females: one 8th grader, three 9th graders, four 10th graders, four 11th
graders: one 13 year old, two 14 year olds, four 15 year olds, three 16 year olds, two 17 year
olds: ten who were white, two Hispanic.
Group 3: 7 males, 10 females: three 9th graders, ten 10th graders, three 11th graders, one 12th
grader: three 14 year olds, seven 15 year olds, five 16 year olds, two 17 year olds: twelve who
were white, four Asians, one Hispanic.
Group 4: 10 males, 4 females: one 8th grader, two 9th graders, five 10th graders, five 11th
graders, one 12th grader: two 14 year olds, four 15 year olds, five 16 year olds, three 17 year
olds.
Group 5: 1 male (indicated no preference), 12th grader, 17 years old, Hispanic.
Appendix 2: Breakdown of group scores on each section of the Malthus Packet
Topic Group 1 | Group 2 | Group3 | Group4 | No pref
#Stu=20 | #Stu=12 | #Stu=17 | #Stu=14 | #Stu=1
wanted Cal | wanted ST wanted Cal wanted ST no pref
used Cal used Cal used ST used ST used Cal
1“Val=#Stu | I“Val=#Stu | 1“Val=#Stu | 1“Val=#Stu | 1“Val=#Stu
2™Val=% 2™Val=% 2™Val=% 2"Val=% 2™Val=%
1 | Correctly drew linear food 17 10 15 10 0
production and exponential 85 83 88 71 0
pop growth pre-model graphs
2 | Correctly drew food per 13 8 10 9 0
person pre-model graph 65 67 359 i 0
3 | Correctly explained reason for 13 9 12 10 0
food per person pre-model 65 75 71 0
graph
4 | Correctly wrote basic 13 7 13 11 0
equations for population and 65 58 76 79 0
food production
Correctly drew correct
STELLA model for
population and food
production
5 | Correctly wrote equation to 14 6 10 % 0
calculate food per person ai sD. 59 5 0
Correctly drew/explained
STELLA model structure to
calculate food per person
Correctly calculated year food
per person drops below
minimum food per person.
(within +-10 years)
10
Correctly wrote new equation
to change pop growth mid-
stream
24
Correctly estimated when
food per person drops below
minimum food per person
with change in pop growth
change
(Should probably given a hint
for this for calculator people -
we may need to discount this
question for calculator grp.)
24
Correctly wrote new equation
to change food production
growth
°
i)
24
i)
10
Correctly estimated when
food per person drops below
minimum food per person
with change in food
production growth change
(same potential problem as for
#8 for calculator groups)
24
Correctly explained why there
is a problem when pop grows
exp and food production
grows linearly
10
59
=)
12
Came up with at least one
reasonable policy to help
mitigate the food shortage
problem
13.
Identified at least one group
who might not like the
policies.
Explained at least one way to
try to convince the group that
policy is needed
Average comfort level with
computers
Student self assessment on
scale 1-5 (1 = low)
2.76
Average grade in the advance
82.6
81.4
87.4
75.9
66.5
algebra course for the first
semester (out of 100%)
