A Judgment Approach to Estimating Parameter in Group Model-Building : A Case
Study of Social Welfare Reform at Dutchess C ounty
Tsuey-Ping Lee, Aldo Zagonel, David F. Andersen,
John W. Rohrbaugh, George P. Richardson
Nelson A. Rockefeller College of Public A ffairs and Policy
University at Albany, State University of New Y ork
Abstract
Building models directly with client groups has become increasingly common in
the field of system dynamics (V ennix et al., 1992; Vennix, 1996). For the past ten years,
the modeling group at the University at Albany has been experimenting with techniques
for handling the complex modeling and facilitation processes involved in group work.
This article extends the previous work of scripted techniques by discussing parameter
estimation techniques in group model building. The discussion of the paper is divided into
four sections : The first section states the purpose of the study and reviews the literature
of the past efforts on estimating parameter in group model building. The second section is
an overview of the approach for group parameter estimation designed by the Albany
modeling team. This overview section includes the problems addressed by each elicitation
step and the solutions for solving them. The third section is the case study describing the
data calibration conference at Department of Social Service, Dutchess County. This case
study presents how the Albany modeling team applied the designed scripts supporting
parameterization in a conference. Strengths and weaknesses of the application are
discussed. The last section of this article contains suggestions for future work on
estimating parameters in group model building. Due to page limitation, this paper only
presents the script of parameter estimation.
I. Background of Dutchess County Group Model Building
Department of Social Services of Dutchess county and Albany modeling team have
been through several conferences for group model building regarding social welfare
reform issue. The purpose of the social welfare reform model building is for policy analysis
and policy suggestion. Several products of the project so far show its contributions on
welfare policy analysis and the field of group model building (Center for Policy Research,
University at Albany, 1997s, 1998; Rogers, et al, 1997; Allers, et al, 1998). This project
at this point includes two models, one focusing on TANF (Temporary Assistance for
Needed Families) and the other on the Safety-Net (families ineligible for federal TA NF).
Two models are planned to be jointed together eventually. This Dutchess County data
calibration exercise focused on the Safety-Net model.
The Dutchess County data calibration conference is considered as a part of
continuous line of social welfare reform group model building at Albany. Although the
modeling team wants to get as much useful information as possible from one conference,
eliciting all the parameters of the model at one conference is usually not feasible.
Therefore, an upcoming concem is the efficiency of this conference. Generally, there are
three sources of the numerical value of parameters: firsthand data, computed value from
firsthand data, and expert judgment. By now, some firsthand data have been gathered
from past conferences and through it some numerical values of parameters have been
directly or indirectly obtained. Thus, this conference will concentrate on those unknown
parameters. After parameters are elicited, they can be fed back to the model for testing
model sensitivity and tailoring the model more completely.
Ninety percent of the participants of the calibration conference have been going
through several prior Dutchess welfare group model building conferences with Albany
modeling team. In other words, most of the participants are familiar with the social
welfare reform model under development. The following section will briefly describe the
Safety-net model.
II. The Nature of the Model
Figure 1 shows the major structure of Safety-net model. In this model, people
who receive safety-net public assistance come from either economic mainstream due to
overall economic situation or loss of TANF eligibility due to TANF maximum five year
time limit. Safety-net assistance recipients could leave the system because of job finding,
being sanctioned or exiting with non-work reasons. For those people being sanctioned,
there are chances for them to retum to the safety-net system. Part of those people who
are employed after safety-net assistance may stay employed and finally go into economic
mainstream. However, part of those post safety-net employed people may lose their jobs
and recidivate the safety-net system. In this model, several resources have impacts on the
flows. Basic services have an impact on flows of “from mainstream”, “departing” and
“sanctioned leaving”. Child Support Enforcement influences the flows of “from
mainstream”, “departing”, “sanctioning” and “into mainstream”. Monitoring affect the
flow of “departing”, “sanctioning” and “job finding”. Employment services influence the
flow of “job finding”. Job Maintenance services have an impact on the flows of
“recidivism” and “into mainstream”. The overall economic situation (represented by
unemployment rate) affect all of the inflows and outflows of the safety-net system. The
data elicitation technology is designed based on this model.
Loss of TANF y
Child Support eligibility ram
Enforcement
Employment
Services
Monitoring
Figure 1. Safety-net Client Stock-and-flow Diagram With Resource Impacts and
Unemployment Scenario
III. Data Calibration C onferencing : An Overview of the approach designed by
Albany modeling team
A data calibration conference is one in a series of group model building sessions.
Typically it occurs after an earlier conference designed to elicit model structure. As with
other group model building sessions, a calibration conference is supported by a team of at
least three professionals who fill specific roles during the meeting: the process facilitator,
whose function is to ensure that all group members are able to participate fully in the
process and that the group session is not dominated by a minority of group members
(Keltner, 1989), modeler/reflector who takes care of all the computer support and
information feedback tasks (Andersen & Richardson, 1997), and a correspondent who
monitors the process and electronically records the important details of the group’s
discussion for good documentation and a printed report to client group (Reagan-
Cirincione & Rohrbaugh, 1992). A typical room layout for a data calibration conference
is similar to the one for group model building workshop. It should include large white
boards, flip charts, swivel chairs and an overhead projector with projection pad (or beam
projector) linked to a computer in the background (Andersen and Richardson, 1997).
Extensive discussion of the room layout and the roles in the room are contained in
Richardson and Andersen (1995) and Andersen and Richardson (1997). Since the
conference is on a continuous line of group model building, those people who attended
previous group model building meeting are expected and preferred for data calibration.
A Script for Calibrating A Complete Stock and Flow Chain
This calibration script is based on a completely structured model - safety-net
model. The purpose of this script is to elicit parameters and table functions after the
structure of the model has settle down. Some potential problems could hinder group
communication during the data calibration session as well as the accuracy of the
calibration results from individual or group judgment exercises. Such problems include
participants’ different recognitions or definitions of a specific variable in the model, the
presence of dominant individuals, participants’ unfamiliarity of the model structure due to
memory limitation, individual human judgment inconsistency, high disagreement among
group members, to name but a few. The following section will discuss the problems
addressed by each elicitation step and the Albany team’s approach designed to mitigate
these potential problems.
1, Learning major structure of the model
The conference should be opened by reviewing the model to help participants
refresh their memories of the model’s major structure. This presentation should not too
detailed. A high level view of system structure including major feedback loops, stocks and
flows should be shown. Computer support using a beam projection of model output are
recommended here to show runs that are designed to raise attention and make participants
better understand linkages between system structure and behavior.
2. Clarify the definition of important variables
Richardson and Pugh (1981) pointed out that care must be taken that the data
gathered have the same meaning as the model parameters. Participants’ different
definitions of a specific variable can be a source of disagreements on the numerical value
of a related parameter. Skewed understanding of a variable will induce skewed parameter
elicitation. Therefore, for those variables that may have been ambiguous during discussion
in the group model building sessions, it is important to clarify their definitions before
parameterization. Definition should be drawn from past meeting minutes or report in order
to ground participants in the groups’ prior understanding of what a specific parameter
meant.
3. Elicitation of key parameters
Based on a complete stock-and-flow structure, the task of eliciting key parameter
here is specifically for equilibrating the stock-and-flow chain. Therefore, this is not a
global way of parameter elicitation.
An “Estimate-Feedback-Talk” group process intervention is applied to elicit key
parameters. The process facilitator first helps group members to understand parameter
definitions and how they are used in the model. This discussion is followed by individual
anonymous judgment. After that, information feedback is provided by a equilibrium
stock-and-flow spreadsheet in which various individual estimates can be entered. The
structure of this stock-and-flow spreadsheet is exactly the same as the structure of safety-
net model except the spreadsheet is in a equilibrium status. When various individual
estimates are entered into the spreadsheet, some related numbers will change accordingly.
Group members then discuss these estimates and their effects on the whole system. The
detailed of the elicitation process is as following.
a. Clarifying parameter definition and location.
As mentioned above, different definitions of a parameter being elicited may cause
difficulties reaching agreement among group members or even meaningless value
estimation. Thus, by presenting a major stock and flow diagram with clear parameter
explanations of how parameters are used in the model, the process facilitator helps
participants to understand the definition of the parameters and the effect of the parameters
on flows.
b Estimating paraneters by individual judgment
In order to collect estimates from all members present, the data estimation process
starts from individual anonymous judgment without group discussion. In group judgment,
research has revealed that groups have been found to perform under the level of their best
member, i.e. the member with the most accurate judgment (Miner, 1984). However, when
the group is unsure about who their “best member” is, research indicates that judgment is
best done in interacting groups provided that individuals first make their own individual
estimates before a discussion and subsequent group decision (Miner, 1984; Sniezek and
Henry, 1989)
Here, for individual judgment, a well-designed, typed-up list of key parameters
being elicited will be handed out to each participant. Based on individual judgment, each
participant is asked to put a numerical value for each parameter on the handout.
c. Tuning numerical value of parameters by group judgment
This is a judgment aggregation step combining behavior approach and computer
facilitation. It allows full interaction among group members. Unstructured discussion
could cause process losses during this phase of calibration. Therefore, a device that helps
4
to direct the whole discussion to be more structured is used. A stock-and-flow
spreadsheet (figure 2) is built to connect participants’ judgments on real world data to
model structure and help people to simply pay attention on the influence of value change
of each parameter on the value change of other variables. In figure 2, the cells with black
shading are used to input numerical value estimated by participants. Changing specific
parameter input will influence multiple other values of the overall stock and flow chain.
From the exercise, participants can lear more about the impact of specific parameters on
the overall stock and flow equilibrium. In addition, using the flow spreadsheet could help
to identify some contradictory numerical values.
Recid. fr.:
Individuals
LOS on SN:
Time underempl.:
(5 ERGs)
LOS on sanction: (1-2)
MEER (nos)
(3-6) Fr. sanctioned: In job prep.:
Lass
(4.6)
Participation fr.:
Fr. sanct. leaving:
(.2-.5)
Figure 2: Spreadsheet Used to Calibrate a Linked Stock & Flow Chain
4. A Script for Eliciting Multiple Linked Table Functions Associate With Stock-and-
Flow Chain
One of the important natures of the safety-net model is that a given flow is usually
affected by multiple resources. For example, all of child support enforcement, basic
services and monitoring influence departing flow (please see figure 1). Based on the
safety-net model, the script for table function elicitation here is specifically for multiple
elasticities impacting on a given flow point.
Table functions (graphical functions) are used to express non-linear relations
between two variables. Again, clarifying the variables related to table functions being
elicited should be focused before starting elicitation process. Then, table function
elicitation starts from an individual anonymous rank-ordering of comparative strength of
competing table functions. The judgment from individuals are aggregated statistically to
obtain an overall rank. The overall rank will help us to focus on those table functions with
higher strength. The ranking task is followed by a more complete discussion of those
table functions of comparative higher strength being estimated first.
a. Rank-Ordering the effect by individual j
Individual human judgment inconsistency is a problem not only in the field of
group decision making but also in the field of group model building. In order to overcome
the possible human judgment inconsistency, we design a column-by-column total effect
analysis and a row-by-row marginal effect analysis sheets (figure 3 and 4) for each
participant to rank order. In column-by-column form, each column represents the effect
of several services on a single flow. A blank cell indicates the existence of a relationship
between the service and the flow while the shaded cells implies no relations. For each
column, participants rank-order the services from most to least powerful in terms of
individual beliefs conceming the total effect on the flow. That is for the rate associated
with a given column which resource could have the greatest overall impact on the flow.
The most powerful variable will be ranked as 1, the next less powerful one will be ranked
as 2, and so forth. No discussion is allowed at this phase.
In the row-by-row form, each row indicates the effect of the service on several
flows. A blank cell indicates the existence of relationship between the service and the flow
while a shaded cell implies no relations. For each row, each participant is asked to rank-
order the flows from most to least influenced in terms of the marginal effect of the service.
That is for the rate associated with a given row which flow could be marginally affected by
the service. Marginal effect means the effect of one unit change of the service on the flow.
The flow being influenced most will be ranked as 1, the next less powerful effect will be
ranked as 2, and so forth. A fter integrating the two sheets, the existence of any judgment
inconsistency could be raised for further discussion.
1. Column-By-Column Analysis:
Rank-Order of Magnitude of The "Total Effect" of Several Services Upon A Single Flow Rate.
‘Senvles ye: reer families Sanctioni |Sanctione[oancetone Job Recidivis had of
View [siainateal apace @raturnin sinsteen
Sites ae tern RAStERS, ng d leaving 7 finding m a Sherdens
paste sn |S 0 0 0
apodtes
Emergence s
yy Services
t 0 S s S 0 S
i Support
Services
neasens A s n
mes
Bcd
Empey ae o s
we
series
Maintenan 0 s
S S 0
Figure 3: Column-By-C olumn Total Effect Sheet
2. Row-By-Row Analysis:
Rank-Order of the Magnitude of The "Marginal Effect" of A Single Service Upon Several Flow
perviceiv: From families . Sanctioni |Sanctione Sanctione Job Recidivis Into of
Flow |mainstrea| “85 | peparting one returnin| 12! valustese
losing ing |.a.lsavling finging | nar jane
hatens | E Fi :s eran
gfeenesw S 7) 0 0
8] services
wlemergenc S
5 |, services
caTTa 10 S Ss S 0 Ss
suppare
sariicar
Taare
q( 5 5 S
U | Monitorin
employe 0 3
ies
satvicae
Tob 0 Ss
Maintenan
A
T 15 S 0 0 0 S 0
“unempl.
rate"
Figure 4: Row-By-Row Marginal Effect Sheet
b Selecting most important effect for estimation
Due to limited time, it is not possible to elicit all the table functions at one data
calibration conference. At the conference, we simply want to elicit those table functions
that are considered most important by the client group. The way to choose these
elicitation-worthy table functions is to sum up the total ranking score for each cell and
concentrate limited time on those with the highest ranks.
c. Estimating table function by group judgment
We used a “marker pen and cling sheet’ way with full discussion among group
members. Participants were asked to sketch the curve relation of two variables of a table.
This would be considered the hardest part of data elicitation exercise because this exercise
is asking people to sketch their professional knowledge into a non-linear curve. Thus, the
process facilitator will help participants start from an anchor point which is drawn directly
or indirectly from observed data. From this anchor point, participants are asked to think
about the relationship between the two variables while one of the two variables (typically
on the X-axis) is two times as much as the anchor value and zero. Then, process facilitator
sketches the curve according to participants’ discussion of the shape of the curve between
anchor point and the other two points. The shape of the curve will be finished when the
group reach consensus.
IV. Further Work
This article attempted to report a script specifically for data calibration conference
in Dutchess County. The overall issue of evaluation of these group data calibration
techniques is worthy to be studied in the future. For instances, table function elicitation
could be done by either “marker pen and cling sheet” method or “pencil and paper”
method or even a method with computer drawing facilitation. Some experiment could be
conducted to find out which way is better under what circumstances. Another interesting
issue is what and how people learmed during the data calibration session.
Since group data calibration is time consuming, it is important to clarify the
purpose of the model and have good preparation before group data calibration exercise. If
the purpose of the model is policy analysis and, its policy implications do not change when
parameters are varied plus or minus some percent, then the parameters do not need to be
elicited any more accurately (Richardson and Pugh, 1981). Therefore, we suggest that a
complete parameter sensitivity analysis should be done before a group data calibration
exercise.
References
Allers, Robert, Robert Johnson, David F. Andersen, Tsuey-Ping Lee, George P.
Richardson, John W. Rohrbaugh, Aldo Zagonel (1998) Group Model Building to Support
Welfare Reform Part II: Dutchess County. Paper presented at the 16” Intemational
System Dynamics Conference: July 20-23. Quebec City, Canada.
Andersen, D. F, George . P. Richardson. (1997). “Scripts for Group Model Building.”
System Dynamics Review 13(2): 107-129.
Center for Policy Research. Albany, NY: Nelson A. Rockefeller College of Public A ffairs
and Policy, University at Albany, State University of New Y ork
— May 1997. Welfare reform project: Report of the group model building conferences
in Cortland County
— June 1997. System Thinking: A case study on welfare reform in Chugwa County
— August 1997. Welfare reform project: Report of the group model building
conferences in Dutchess County.
— September 1997. Welfare reform project: User’s manual to the TANF flight
simulator, version 1.0.
— October 1997. Welfare reform project: Report of the model calibration meeting in
Dutchess County.
— Nobember 1997. Welfare reform project: User’s manual to the TANF flight
simulator, version 2.0.
— April 1998. Welfare reform project: Parameter booklet for the combined TANF &
Safety-net model
Kelner, J. (1989). “Facilitation: Catalyst for group problem solving.” Management
Communication Quarterly 3: 8-32.
Miner, F. C. (1984). “Group versus individual decision making: An investigation of
performance measures, decision strategies, and process losses/gains.” organizational
Behavior and Human Performance 33: 112-124.
Reagan-Cirincione, P. John W. Rohrbaugh. (1992). Decision conferencing : A unique
approach to the behavioral aggregation of expert judgment. Expertise and Decision
Support. G. W. F. Bolger. New Y ork, NY, Plenum Press: 181-201.
Richardson, George P. and David F. Andersen. (1995). Teamwork in group model
building. System Dynamics Review 11: 113-137.
Richardson, George P. and A. L. Pugh, III (1981). Introduction to System Dynamics
Modeling with DY NAMO. Cambridge MA, Productivity Press.
Rogers, J., Robert Johnson, Aldo Zagonel, John Roghbaugh, David F. Andersen, Geroge
P. Richardson, Tsuey-Ping Lee. Group model building to support welfare reform in
Cortland County. Paper presented at the 15" International System Dynamics Conference:
August 19-22. Istambul, Turkey.
Sniezek, J. A., $ Henry, R.A. (1989). “Accuracy and confidence in group judgment.”
Organizational Behavior and Human Decision Processes 43: 1-28.
Vennix, J.A.M. (1996). Group Model Building: Facilitating Team Learming Using System
Dynamics. Chichester, England: John Wiley & Sons.
Vennix, J.A.M., David F. Andersen, George P. Richardson, John W. Rohrbaugh. (1992).
Model building for group decision support: Issues and alternatives in knowledge
elicitation. European Journal of Operational Research 59: 28-41
CC BY-NC-SA 4.0