Does learning to reflect make better modelers?
Jill Slinger, Jan Kwakkel and Maarten van der Niet
Delft University of Technology, J affalaan 5, 2628 Delft, Netherlands
ABSTRACT
In this paper we address the role of reflective skills in the development and training of
new System Dynamics modelers at the tertiary education level. Over the last two years
students at the Delft University of Technology have written a reflective essay on their
experiences with the conceptualization, formulation, validation and use of a System
Dynamics model in addressing the fictitious, but realistic, problems of a public policy
maker. The degree to which they apply the cursory reflective training that they receive
and the effect of this intentional reflection on their acquisition and application of
modeling skills is evaluated. While some students do attempt to address the added value
of a modeling approach to their client and their role in actualizing this, the majority of
students focus their attention on the strengths and weaknesses of their model and the
dilemmas they face in executing the modeling cycle. By presenting the metaphor used in
teaching these reflective skills and analyzing the questionnaires completed by the
students, we are able to gain further insights regarding the views held by the aspirant
modelers of the choices they face in building and using their models. This then feeds
back to improve our teaching.
Keywords: Intentional reflection, modeling skills acquisition, problem based learning,
public policy, System Dynamics
Introduction
The bachelor’s program of the Faculty of Technology, Policy and Management was introduced in 2000 at
the Delft University of Technology in response to changes in the Dutch higher education system. The
introduction of the program was accompanied by a change in the curriculum. The original four year
program was spit into a three year bachelors’ program and a two year masters’ program. The emphasis in
the bachelors’ program is on teaching students how to analyze complex socio-technical problems. The
program includes technical courses, courses on policy and decision sciences and mathematical courses. The
first fully fledged modeling project which the students’ undertake is the System Dynamics project early in
their second year. This is then followed by other modeling courses including discrete event modeling,
statistical modeling and several conceptual modeling techniques. The keystone position of the System
Dynamics modeling project in the curriculum provided the basis for the decision taken late in 2005 to
provide a reflective training module in association with this project course. The effects of this training in
reflection related to modeling at such an early stage in the curriculum is designed to improve both the
modeling skills and the learning acquisition strategies of the students. The extent to which this goal is met
within the project course itself form the subject of this paper.
The System Dynamics modeling project is taught according to the problem based learning approach of
Barrows (1985, 1992), following the instructional principles derived by Savery & Duffy (2001) based on
the constructivist values of Lebow (1993). These principles are:
1. Anchor all learning activities to a larger task or problem
2. Support the learner in developing ownership for the overall problem or task
3. Design an authentic task
4. Design the task and the leaning environment to reflect the complexity of the environment they
should be able to function in at the end of the learning
Give the learner ownership of the process used to develop a solution
Design the leaning environment to support and challenge the leamer’s thinking
Encourage testing ideas against alternative views and alternative contexts
SY St oF
Provide opportunity for and support reflection on both the content learned and the learning process
The design of the problem based learning approach to the System Dynamics modeling project will be
described in terms of the task that students are required to undertake, the character of the leaming
environment in which they do this, including the support offered to them, and the embedding of this
learning in a reflective activity. The manner in which each of the eight instructional principles is met by
this course will be explained. An analysis framework in which the model is embedded will also be
presented and finally conclusions will be drawn regarding the utility to the students and teachers of such an
approach.
The task
An authentic, yet fictitious complex policy problem is prepared for the students, as recommended in
instructional principle 3. The types of problems that students have been asked to address in the last few
years are exemplified by the following three cases:
° A fibre optic cable problem of a fictitious city in the Netherlands aiming to support knowledge-
intensive businesses, inspired by newspaper articles on the problems faced by Amersfoort, a Dutch
city. A city councilor has committed himself to a particular (well researched) option for glassing
the city ring and needs to address the concerns of fellow councilors regarding the feasibility of
laying such a ring within the desired completion times as well as provide answers to questions
regarding the financial arrangements that the municipality can make with the potential service
providers to ensure the success of such a venture. Students figure as consultants with a modeling
specialization and are given information on the supposed failure of a similar project in another
Dutch city.
A request by an interdepartmental commission for information regarding the knowledge-based
service industry of western Europe. “Consultants” in the Netherlands are asked to simulate the
quality and service standard issues associated with a typical, small knowledge-based industry and
based on their insights provide recommendations to the Minister of Economic Affairs of a
developing country on measures to address the problem of declining standards. This case was
inspired by the work of Oliva and Sterman (2001).
A water resource management problem, inspired by the situation in the Gaza strip (Hoekstra et al
2002), in which the newly appointed Minister of Water Affairs of an imaginary country called
Archapela, requests a modeling study as a springboard to her development of new policies and
discussions with a commission comprising a number of other ministerial delegates.(Box 1). The
problem is supported by historical data and many relevant newspaper articles on groundwater
shortages and the effects of pollution of the water sources on public health.
Box 1: Typical problem situation
With the recent appointment of a new Minister of Water Affairs (Sept 2007) came the request to the
Delft University of Technology to update the understanding of the water situation in Archapela,
using the existing information. The objective of the project therefore is to gain an understanding of
how the water system functions and of the development of Archapela from 1983 onwards in order
to be able to make recommendations to enhance water management in the future. Specific
attention should be paid to the management of the water supply over the medium term (2008 to
2035) with some attention given to the long term (till 2050). Important factors in this respect are the
anticipated trends in the demand for domestic water, water for industry and agriculture, and the
quality and quantity status of the groundwater. Background information and data on the water
supply in the area are presented in the articles on the following pages
Each of these problems includes the following elements:
Request by a client for a modeling study (method not specified)
° — The client operates a complex multi-actor context in which the specified goals for the project are a
narrower reflection of the actual problem situation in which they have to maintain their own
political position while depending heavily on the “consultant” to provide them with relevant and
reliable insights and recommendations.
The emphasis of the client in their search for solutions is strategic, rather than operational, and
they focus on enduring solutions rather than short term gains. They have longer term careers in
public policy in mind.
The information supplied to the students is presented in a number of ways and is not always fully
consistent. Numerous articles are provided, tables and graphs of data or interviews with people
who present different aspects of the problem. A coherent whole has to be distilled from the
information-rich, yet disparate parts.
The problem itself is of a technical nature and involves the implementation of technology-based
solutions. It cannot be solved simply by conceptual modeling or spreadsheet modeling owing to
the presence of strong interactions (and feedback) between the constituent components of the
problem. Accordingly, the problem lends itself to a System Dynamics modeling approach.
In essence, the problem provided to the students places them in an advisory role to a public policy maker
and attempts to mimic the complexity of the situation of their client in finding solutions to technically-
based problems in the public domain. This is in accordance with instructional principle 4.
The learning environment
Students are required to complete the task of supplying model-based advice to their client within a period
of seven weeks. They have previously received training in differential equations, control theory and
System Dynamics modeling and have undertaken tutorial tasks in the PowerSim studio software package.
These tutorial tasks grow in difficulty during the course until the students are able to build and run a small
model such as the basic epidemic model. However, they have never undertaken the task of building their
own more complex model based on unstructured information. Students undertake the project in groups of
two, so that the social negotiation of knowledge is supported and stimulated (von Glaserfeld 1989). This is
further supported in the weekly half hourly sessions that they have with project supervisors. The task of the
supervisors is to support the leaning of the students, primarily by asking directed questions or responding
to the questions of the students. Supervisors are specifically instructed to act as a soundingboard and allow
the students to test ideas with them and with each other. Although there is a clear standard for the eventual
end product, there is no “model answer”. Instead the process of arriving at a conceptually sound, validated
model that allows the testing of policy alternatives and the provision of model-based advice to the client is
facilitated. Students are assisted during this process in reflecting on their approach by the questions of the
supervisors regarding their progress both in model development and in writing the accompanying report for
the client. In short, the leaning environment adheres to the instructional principles 5 and 6 and 7. In
addition to this, specific short training in reflection is given to the students and they are required to write a
reflective essay.
The training in reflection
Late in the third week of the project when the students have wrestled their way through a conceptual model
and are busy with the specification of a System Dynamics model in PowerSim Studio, a short lecture is
given on the process of modeling. Students are required to read the article entitled: “The unavoidable
apriori” by Meadows (1980). This, in order to stimulate them to think about whether, and why, System
Dynamics is an appropriate modeling technique for the problem at hand. They are encouraged to
remember that this may not be obvious to their client and certainly is not obvious to their political
opponents. Reasoned justification of their choice of modeling method and an openness to the possibility of
using other methods is the desired outcome.
The process of modeling is then further addressed using the metaphor of an optician. A customer seeks the
help of an optician because they have problems with their vision. They require a solution to their problem
and can judge the validity of the solution offered by the optician in terms of whether it helps them to see
better or not. They cannot fully judge the quality of the process followed by the optician in coming to such
a recommendation, only the value of the outcome to them. We view the optician as the modeler and the
customer as the modeling client. Then, just as in the metaphor the modeling client judges the value of the
model to them primarily on the basis of the outcome and the insights provided in regard to the original
problem, while the expertise of the modeler, similarly to that of the optician, can only fully be assessed by
colleagues and not the client. Only by exercising his reflective capacity and asking questions of himself
regarding the process followed in completing the assigned task can the modeler ensure his own adherence
to quality standards. The types of (simple) questions he can ask include:
° Is the model fit for purpose?
° How well did I construct and test my model?
° Are the results useful /good enough?
° How could I do better next time?
In addition, one can view the situation from an external perspective and ask questions about the relationship
between modeler and client and the utility and level of aggregation of the model.
The purpose of the illustrative metaphor is to allow the student to realize that they can choose to view the
situation from one of three perspectives, namely: (i) the client’s, (ii) the modeler’s and (iii) an external
perspective and that each of these perspectives leads to different reflective questions and hence different
insights. They are required to choose a perspective that they feel comfortable with and reflect on their
experiences during and at the end of the modeling project.
In addition, to support this reflection further the modeling cycle is allied to choices that they make in
developing and using their model. In essence, the model artifact forms the pivot in an initially convergent
and later divergent analytical approach (Figure 1). The steps that they make in drawing their conclusions
and making recommendations regarding policy solutions reflect back on the choices made in the process of
building and specifying the model. Clarification of these choices and the insight that it is the process of
reflecting on these choices that will make them better modelers is the desired outcome.
So, in accordance with the last instructional principles allied to the problem-based learning approach of
Barrows (1985, 1992), the opportunity for reflection is provided to the students as is the support to conduct
this. However, the embedding of the model artifact in the analytical process as in Figure 1, strives to
address the need to anchor the learning within the larger task of mastering the bachelor curriculum and
enabling the individual students to own their own learning process and task (instructional principles 1 and
2). But, what use do they make of this reflective opportunity and does it make them better modelers?
Problem Description
Formulating
problem, research
questions, and
model questions
Describing the
system
Developing
the model
Modeller
insight
‘Answering model
questions,
Answering research
questions:
Figure 1: The model as the focal point embedded in a convergent and then divergent analytical process
allied to the modeling cycle (on the right hand side). In drawing conclusions on the validity of
the model and its use, students need to reflect on the choices made in moving from the problem
formulation through the conceptual system description to the model itself. Similarly, in
providing answers to the clients’ problems students need to reflect on the initial assumptions
they made. The arrows on the left hand side represent these reflective learning activities.
Grading
The products of the System Dynamics Modelling project include a validated model and (tested) policy
options, submitted on CD-Rom, together with a report to the fictitious public policy client. The students
receive a group grade for the model (25% of the total grade) as well as the joint report (the majority of the
mark). This means that students who are good modelers, but poor in argumentation can still score
relatively well in the course compared with students who are poor modelers, but write well. An excellent
mark can only be attained if both the model and the report to the client are good.
The reflective essay on the other hand is deemed of sufficient standard provided that the argumentation and
structure are sound. Their opinions in regard to modeling and its value from their chosen perspective of
modeler or client are not evaluated so that they are free to be honest and to ensure that the reflection is
helpful to their development as a young professional. The essay is placed in the skills portfolio which they
develop over the course of the bachelors’ program in Technology, Policy and Management.
Results and Conclusions
Sixty-eight per cent of the students focus their attention on the strengths and weaknesses of their model and
the dilemmas they face in executing the modeling cycle. In essence they address the rather simplistic
questions listed previously and so do this well. Thirty per cent of the students choose the role of the client
and address the added value of a modeling approach in quantifying the potential gains of policy options and
their role in actualizing this. Only two per cent of the students choose the extemal perspective. This did
not surprise us, as they have not yet had exposure to various modeling techniques and so have little insight
regarding the relative merits of different techniques, their underlying paradigms and their most appropriate
level of aggregation.
However, categories of students can be distinguished according to their modeling grade and the degree of
insight evinced in their reflective essays. We observe that the students with excellent modeling grades
produced good reflective essays. Some students who achieved average modeling grades also produced
good reflective insights. In addition some students who were weak modelers produced adequate reflective
essays. In short, the level of the reflective essay aligned more closely with the grade for the modeling
report than with the total modeling grade. Our preliminary interpretation of these results is that enhanced
reflective ability on the part of students enables them to improve their use of a model in a client-oriented
situation, but does not necessarily help them make a better model.
Table 1: Categorization of students according to their modeling grades and their reflective insights. Dark
grey indicates that the majority of students within a grade range fall within a certain reflective
category. Light grey indicates that only a few students within a grade range are assigned to the
reflective category, while the hatched grey is indicative of intermediate numbers.
Reflection | poor adequate good
Model grade
fail none none
weak
average
excellent none none
In the questionnaire filled in by students, more than 70 % of the students indicated that the training in
reflection complemented the System Dynamics modeling course well to adequately.
To enable us to draw more definitive conclusions about the role and efficacy of reflective training in
making better modelers in future, we will conduct a longitudinal survey of students who have taken the
System Dynamics modeling course over the last five years. It is our long term goal to establish the extent
to which the tools offered as an aid to reflection, support the act of leaning and how effective this proves
to be for young professional practitioners. A fter all, Sterman’s masterly analysis of leaning as a feedback
process (1994), teaches us that we need to be open to the unexpected effects of our teaching activities and
welcome feedback on these.
References
Barrows H S (1985). How to design a problem-based curriculum for the preclinical years. New Y ork,
Springer Publishing Co.
Barrows H S (1992). The Tutorial Process. Springfield, Illinois: Southern Illinois University School of
Medicine.
Hoekstra A Y et al (2002). Corsa...
Lebow D (1993). Constructivist values for system design: Five principles towards a new mindset.
Educational Technology Research and Development, 41. pp 4-16.
Meadows D M (1980). The unavoidable apriori. In Randers, J. (ed.), Elements of the System Dynamics
Method, MIT Press, Cambridge, Mass.
Oliva R. and J.D. Sterman (2001). Cutting corners and working overtime: quality erosion in the service
industry. Management Science 47(7), July 2001 pp. 894-914.
Von Glaserfled E (1989). Cognition, Construction of knowledge and Teaching. Synthese 80 pp121-140.
Savery J R & Duffy T M (2001). Problem based learning: An instruction model and its constructivist
framework. CRLT Technical report no. 16-01, Indiana University, Bloomington. 17p.
Sterman J D (1994). Learning in and about complex systems. System Dynamics Review 10 (2-3): 291-330.
© 8%:
