Table of Contents
Information Value Production
of Influence Diagrams versus Level-Rate Models:
The Perception of Decision-Makers *?
L. Martin Cloutier
Department of Management and Technology
School of Management
University of Quebec at Montreal
315 Ste. Catherine East
Montreal, Quebec, H2X 3X2 Canada
Tel. (514) 987-3000, ext. 3732#, Fax (514) 987-3343
cloutier.martin@ ugam.ca
Félix Gaudet-Dumais
IBM Canada Inc.
Tel. (800) 465-7748, ext 58647,
Fax: (514) 938-6615
felixg@ca.ibm.com
Abstract
This paper examines the debates on the use and contribution of influence diagrams, or the
display of more qualitative dimensions of feedback structures, versus the use of level-rate
models or quantitative models in system dynamics (SD). The question of interest in this paper
relates to the perception of decision-makers on the use value of influence diagrams and of
level-rate models. This paper aims to understand the perception of decision-makers familiar
and unfamiliar with SD methods, on these issues as raised in the SD literature by Coyle
(2000, 2001) and Homer and Oliva (2001). This paper reports on an exploratory research
that analyses the statistical significance, or lack thereof, in comparing the viewpoints of two
groups of decision-makers; one that had familiarity with SD and one that had not. The results
are mixed, and by an interesting twist, substantiate some dimensions of the opinions of both
Coyle (2000, 2001) and Homer and Oliva (2001).
Keywords: influence diagrams, level-rate models, decision-makers, computer models,
perceptions, information.
" Presented at the 22” Intemational System Dynamics Conference, Oxford University, United
Kingdom, July 25-29, 2004.
* The authors would like to thank all respondents for their time and input into the study; and criticisms
and comments from the conference participants are gratefully acknowledged. The research
omissions and limitations remain the sole responsibility of the authors.
Introduction
The system dynamics (SD) literature has questioned the genuine contribution and added value
of SD methods and models. In particular, there have been recent debates about the use and
contribution of influence diagrams, which display qualitative structures of feedback loops,
versus the use of level-rate structures or quantitative models. On the one hand, Coyle (2000,
2001) emphasizes that information risks may be greater for the model end-user if the
quantification of variables leads to erroneous results. For example, this may refer also to
models that include variables quantified with “soft” data and information. On the other hand,
Homer and Oliva (2001) state that quantification almost always adds value to the study of a
problem, including the careful use of “soft” data, and that influence diagrams (ID) are
insufficient for analysis.
This debate is not new and can be found under various forms in the management information
systems literature. For example, research on decision support systems has found that decision-
makers are more likely to lend credibility to “soft” information like anecdotes, rumors, and
informed opinions, than to results produced by computerized decision support systems
(Silince and Saeedi, 1999). However, the literature emphasizes that decision technology and
human intuition can be used as complements: both have inherent strengths and weaknesses
(Hoch, 2001). A corporate anthropology perspective would submit that a model in itself has
little or no value, but that relationships amongst model users are more important in generating
knowledge and “shared vision”. Individuals tend to “believe” in models they have developed
themselves, or in models developed by individuals they “believe in’; or who are, in their
view, credible model builders (Schrage, 2000). It is important to recognize, this debate takes
place amongst, and opinions are issued, by researchers, rather than by decision-makers or
expert users themselves.
To take the debate beyond discussion, this study examines the perceptions of decision-makers
regarding the use value of influence diagrams compared to the use value of level-rate models.
The objective of the paper is to compare the perceptions of decision-makers familiar and
unfamiliar with SD methods, and to analyze in part how their perception relate to the issues
raised in the exchange between Coyle (2000, 2001) and Homer and Oliva (2001). The
remainder of this paper reports on an exploratory study of the perceptions of two groups of
decision-makers: one that had familiarity with SD and one that had not. In the next section,
the design employed in the research process is described. A summary of empirical results is
presented, and a conclusion follows.
Research Design
The conceptual framework for this study follows from the hypothesis by Raghunathan (1999)
that raw data and expertise can be processed through information technology (IT) to generate
results that can be further processed by decision-makers to support decision-making (see
figure 1 below).
L-R Executives
—_——_— (Decision. |—————_
Raw Data (IT) Scenario makers) Decision
Expertise Results
(Information)
Figure 1. Production decision model (adapted from Raghunathan, 1999)
Five main steps were followed in the research design:
1) Development of an SD model (including an ID, a level-rate model, and simulation
results);
2) Presentation of models to participants familiar and not familiar with SD;
3) Development of a questionnaire;
4) Administration of a questionnaire to participants; and
5) Analysis of the results from the questionnaire using simple means test.
First, an SD model was designed according to the approach detailed in Roberts et al. (1983),
similar to the one by Sterman (2000). A context for a problem, the influence diagram, and a
level-rate model were specified and calibrated. The situation structured in the model is
analogous to the one presented in Goodman (1989) on the dynamics of a housing community.
This new model was calibrated to show the dynamics of the widely publicized rental housing
shortage (or “crisis”) for particular markets segments that has existed in the City of Montreal
area (Quebec, Canada) for a number of years. The model by Goodman (1989) was chosen
because of its intuitive appeal for describing this well-known housing situation, but mostly for
its generic character that could be adapted to a situation familiar to potential groups of
participants in the study. Moreover, it contained “soft” elements that could help a potential
participants appraise the “value” of having this type of information as part of a model. Finally,
the model can be displayed on a simple slide during a presentation, making it easier for
participants to see the “whole” picture and all feedback relationships involved. The model
was calibrated with historical secondary data sources (Statistics Canada; 2001a,b; Société de
habitation du Québec, 2000, 2002a, 2002b, 2003; Canada Mortgage and Housing
Corporation, 1992-2002, Institut de la statistique du Québec, 2002a, 2002b; newspaper and
magazine articles, etc).
The structure of the original Goodman (1989) model was slightly amended to account for the
situation in the new market to which it was applied. For example, Goodman's (1989)
feedback loop illustrating the availability land going into development is not an issue in the
Montreal market so was removed from the model. Because the model was mainly concerned
with the rental housing market, rather than the overall housing market, two more feedback
loops were added as displayed in figure 2. One feedback loop accounts for the outflow of the
rental population from the rental market to the property market. The second feedback loop
accounts for the incoming population from other regions in the province or from immigration.
These two changes were necessary to insure that the model would reproduce credible results
with respect to the times series available for the 1992-2001 period and to appraise results
generated by the model. The two “soft” elements in the original model, that is the
“attractiveness for migration multiplier’ and the “housing construction multiplier’, were
incorporated into this new model taking the same values as in the original model (Goodman,
1989). The model was subjected to a series of sensitivity analyses to examine of the
congruence between the historical data available and the results produced by a “business as
usual” simulation depicted by the movement of the level variables in the model. The modified
Goodman level-rate model is shown in figure 3.
foo Rental
fen Demolition
Jn Rate -
= —~s Rental Units |
a +
(
{ Average
\ Desired Lif etime
Rental Nd } of Rental
Units “— /, Units
a é eas KO
rental units
tae % | 2 A
wily
Sw —_—
Rental
= —> Population «—
Deaths | = a J} \ _
\") | + —» Ownership
sual
Figure 2. Influence diagram of rental housing dynamics
Second, a presentation was elaborated to describe all the elements of the model for
participants. The presentation included an overview of the SD modeling process, and
emphasized for the participants the intricacies of applying the SD approach. A particular
emphasis was placed on detailing the feedback loop structure using the influence diagram,
and for describing the level-rate model (including the use of soft variable relationships). The
historical congruence of the model was presented for the 1992-2001 period, and three
“scenario” results obtained from the level-rate model were presented. These scenarios amount
to a comparison of sensitivity analyses looking at exogenous parameter changes on three
topics: (1) the expectation of an increase in normal construction, (2) the expectation of an
increase in migration, and (3) the expectation of a life-style change affecting the number of
units per person.
Normal_rental_construction Initial_rental_units
O <)
Rental_uN
Rental_consétuction_rate R | demolition_rate Average_life_time_of_housing
we
Housing_construction_multiplier
Rental_ ratio Inog€upation_factor
Attractiverfess_for_Mmigration_multiplier
Normal_outmigration |
Units_per_person
Normal_migration
Death_factor
Or
Departure_migration_multiplier
Figure 3. Level-rate model of rental housing dynamics
Third, a questionnaire was designed with structured and semi-structured questions. Questions
addressed several topic areas: (1) participants’ previous training in decision technology in
general and their professional use of such software, (2) their perceptions of the influence
diagram, (3) their perceptions of the level-rate model, (4) comparisons of their perceptions
between the influence diagram and the level-rate model, (5) an open-ended question asked the
participants to make a choice between the ID and the level-rate models and to justify that
choice. The questions from (6) to (9) asked about socio-demographic information (years of
experience, nature of the position held (operational, tactical, strategic), etc.). Questions (2),
(3), and (4) were structured-type questions with responses on a Likert-type scale from 1 to 5
(ranging from “strongly agree” to “strongly disagree”). The formatting, general layout, and
ordering of questions followed recommendations found in Dillman (1999). The questionnaire
was pre-tested once with a group of 19 participants, all candidates in a master’s program in
MIS at the University of Quebec at Montreal. These pre-test participants were graduate
students enrolled in a research methods seminar who had received training on the design and
administration of questionnaires in the previous weeks. One advantage of that group, in
addition to the fact they were familiar with general principles of questionnaire design and
administration was that it possessed a group profile similar to the profile of potential
participants. Some of the participants had taken an SD course in other settings. Several
comments were made about the presentation and the questionnaire; and both were
subsequently refined in preparation for data collection.
Fourth, presentations were made and the questionnaires were administered during the fall
2003. This led to many challenges. First, it was difficult to find a group of participants
familiar enough with SD to be able to lend judgement on these methods, a requirement of the
study. Second, because of scheduling and other difficulties it was not possible to have all
participants attend the presentation and participate in the data collection session at the same
time. The presentation and the questionnaire had to be repeated, and thus, a written script was
developed to follow each time. The participants identified were mostly executive MBA
candidates attending a degree program. The presentation was shown to a group of executive
MBA candidates at the University of Quebec at Montreal that had taken a 3 credit SD course
(in which they were trained with a dedicated software and completed a major case study of a
technology enterprise in which they elaborated an influence diagram, built and calibrated a
quantitative model). The SD group included 27 participants, and the RE group had a total of
20 participants. The RE group consisted of experts on the subject matter of the model: real
estate. The questionnaire was repeated twice to insure a sufficient number of participants with
this expertise. These groups included executive MBA candidates specialized in real estate
(RE) management and a group of senior analysts and directors at the Canada Mortgage and
Housing Corporation. The presentation and data collection process were executed three times,
excluding the pre-test.
Fifth, the data collected was summarized using descriptive statistics and simple tests on the
homogeneity of the variance, prior to conducting tests on the mean (Student and Aspen-
Welch’ t tests), to compare results between the two groups of participants.
Research Results
This section presents the results of the research in three parts. The next section examines
participants’ perceptions of the information produced by influence diagrams. Then results
about perceptions of level-rate models to support decision-making are detailed. Perceptions of
soft variables used in level-rate models are examined. The last section concludes with the
' The Aspen-Welch t-test was used to calculate the t ratio when the difference in the variance between the two
groups of participants for a particular question, as determined by Levene’s F-test, was shown to be statistically
significant.
participants’ perceptions of the information produced by the influence diagram and the level-
rate model.
Perceptions of influence diagrams
Results for the three questions addressing participants’ perceptions of the information
produced by influence diagrams are presented in table 1. In all cases, the variance was found
to be non-homogeneous (p = 0.05). For all three questions, the mean difference between the
two groups was statistically significant.
The SD participants were more likely to agree with the statement that an influence diagram
helps understand a system, while RE participants were neutral, showing a statistically
significant difference between the groups (X,) = 2.21, X,, = 3.00, p = 0.004). The results
obtained for the statement that an influence diagram would help make a decision were almost
the same as for the one on helping understand a system (X,) =2.23, X,, =3.15, 9 =0.003).
The RE participants were less likely to perceive a more intuitive prospective inference from
influence diagrams than SD participants, showing a statistically significant difference between
the groups (X,) =2.00, Xp, =2.74, 9 =0.019).
Table 1 : Perception of the information produced by influence diagrams’
Levene's F test - t test between two
Statement Mean homogeneity of variance means
Xgp Xpe | (9 =0.05)| Interpretation | (9 =0.05) | Interpretation
The information
produced by an ID helps} 2.21 3.00 0.000 Reject Ho 0.004 Reject Ho
understand a system
The ID would help you
make a decision
The ID allows for a more
intuitive prospective 2.00 2.74 0.004 Reject Ho 0.019 Reject Ho
inference
T The results are calculated by the mean on the Likert scale defined as follows: 1: strongly agree, 2: agree, 3:
neutral, 4: disagree, and 5: strongly disagree.
2.23 3.15 0.047 Reject Ho 0.003 Reject Ho
The results of analysis showed that participants with an SD background tend to perceive
influence diagrams positively, but only mildly so. However, SD participants rated the
influence diagram more positively than the RE participants on this set of questions. The SD
group perceived influence diagrams as reliable information sources to describe the structure
of a system, while the RE group remained neutral.
Perceptions of level-rate models
The second set of statements, reported in table 2, examined perceptions about the use of level-
rate models and results by both SD and RE participants. The goal of this set of questions is to
appraise whether decision-makers have confidence in the information produced by computer-
based level-rate models.
In response to the statement about the information produced by the level-rate model helps
understand a system, the results showed no statistically significant differences between the
means for the two groups of participants. Both SD and RE participants are leaning towards
agreement (X.) =2.18, Xx: =2.67, 9 =0.09).
On whether level-rate models add a useful value to decision-making, the difference between
RE participants and SD participants were found statistically significant different (x,) = 2.00,
Xpz = 2.50, @ = 0.044). The SD participants clearly agreed with the statement, while RE
participants were at mid-point between agree and neutral.
In response to the statement on whether level-rate models are a rational technological mean to
conduct convincing analyses, similar results were obtained than for the previous question. In
this case, SD participants were near agreement and RE participants near neutral, showing
even stronger statistically significant differences between both groups (X,) = 1.96, Xx =
2.85, P = 0.003). The SD group perceived level-rates models add useful value for decision-
making and that computer technology helped support convincing analyses.
One question addressed the issue of multiple data sources used in model calibration and the
potential for misleading results. The result to this question showed no statistically significant
differences, but SD participants were at mid-point between disagree and neutral, while RE
participants were neutral (X,) =3.50, X,, =3.05, p =0.09).
The two statements about “soft” variables in level-rate models generated a set of mixed
results. In response to the statement that soft variables in the level-rate model make scenario
results improbable, SD participants were was at mid-point between neutral and disagree,
while RE participants were neutral (X,, = 3.50, X,, = 2.90, p = 0.032). However, this
difference between the two groups of participants was sufficient to obtain a statistically
significant difference. But, both groups of participants were neutral on whether the soft
variables included in the level-rate model would make scenario results more realistic (X,) =
2.81, Xp. = 3.00, @ = 0.45). It can be concluded that SD participants rejected the notion that
“soft” variables made scenario results improbable, but both groups did not perceive that
results were more realistic due to soft variables. These results challenge the arguments by
Coyle (2000), that the quantification of soft variables might be misleading, and by Homer and
Oliva (2001) that SD can handle soft information.
Table 2: Perception of the information produced by level-rate models’
Statements
Mean
Levene's F test -
homogeneity of variance
ttest between two
means
(@ =0.05)/ Interpretation
(p =0.05)
Interpretation
The information
produced by the L-R
model helps understand
a system
0.018 Reject Ho
0.090
Accept Ho
The results from the L-R
model add a useful
value to decision-making
0.157 Accept Ho
0.044
Reject Ho
The L-R model is a
rational technological
mean to conduct
convincing analyses
0.001 Reject Ho
0.003
Reject Ho
The L-R model is
calibrated with multiple
data sources and results
can be misleading
3.50
3.05
0.968 Accept Ho
0.090
Accept Ho
Soft variables in the L-R
model make scenario
results improbable
3.50
0.786 Accept Ho
0.032
Reject Ho
Soft variables in the L-R
model make scenario
results more realistic
2.81
3.00
0.451 Accept Ho
0.453
Accept Ho
T The results are calculated by the mean on the Likert scale defined as follows: 1: strongly agree, 2: agree, 3:
neutral, 4: disagree, and 5: strongly disagree.
Perceptions of influence diagrams and level rate models compared
The set of questions in table 3 aimed at studying the perceptions of both groups regarding
influence diagrams compare to level-rate models.
The first statement asked participants to rate whether “The ID and the results from the L-R
models are complementary”. The SD participants responded near agree, but RE participants
were at mid-point between agree and neutral. This resulted in a statistically significant
difference of opinion between the two groups of participants groups (Xj) = 1.73, Xp, = 2.45,
p=0.003).
The influence diagram and the level-rate model were neutrally perceived to be reliable
information sources for understanding the real estate rental market amongst RE participants,
while SD participants were more in agreement with the statement, resulting in a statistically
significant difference (X,) =2.35, X,, =3.15, p =0.013).
Table 3: Perception of the information produced by influence diagrams and level-rate models'
Levene's F test - ttest between two
Statements Mean homogeneity of variance means
Xgp Xe | (9 =0.05)| Interpretation |(@ =0.05)| Interpretation
The ID and the results
from L-R models are 1.73 2.45 0.351 Accept Ho 0.003 Reject Ho
complementary
The ID and the results
from the L-R model are
reliable information
sources to understand
the real estate rental
market
The results from the L-R
models have more
information value than
the ID
T The results are calculated by the mean on the Likert scale defined as follows: 1: strongly agree, 2: agree, 3:
neutral, 4: disagree, and 5: strongly disagree.
2.35 3.15 0.043 Reject Ho 0.013 Reject Ho
17 2.25 0.646 Accept Ho 0.074 Accept Ho
Finally, both groups were rather in agreement in response to the statement about whether the
results from the level-rate models have more information value than influence diagrams, for
which SD participants and RE participants have shown no statistically significant difference
(X, = 1.17, Xy, = 2.25, p = 0.074). But the means showed a trend towards an agreement
between the two groups of participants, which would support the arguments of Homer and
Oliva (2001), leaning towards quantification.
Conclusion
The goal of the research was to take a closer look at the debate about the use of influence
diagrams, which are qualitative representations of feedback structures, and of level-rate
models and results, which are quantitative output from quantitative simulation, to help with
decision-making.
The issues raised by the debate in the SD literature are found also in other fields. The goal of
the research presented here is to help improve our understanding of the information needs of
decision-makers. Most, studies have provided comments on the debate using the experience
and opinion of academic researchers. As was seen, taking the issues to decision-makers
introduced a new perspective to the debate. It also highlighted possible difficulties involved in
the process of assessing the relative contribution of both the influence diagram and the level
models independently or in conjunction.
In particular, the results showed that across all three sets of questions examined, the SD group
tend to be more enthusiastic about SD-related methods than the RE group. For example,
10
familiarity about the method created less doubt about the contribution of influence diagrams
to describe a system amongst SD participants, while the lack of prior exposure to influence
diagrams by participants in the RE group did not help convince them of they had much
informational value. In general, quantitative information seemed to fare better with both
groups.
Further research on this topics will help understand how the fit between the SD method,
including, influence diagrams and formal level-rate models and the information needs of
decision-makers can be improved or better satisfied.
References
Canada Mortgage and Housing Corporation (CMHC). 1992 - 2002. Rapport sur les
logements locatifs (Montréal). Montréal, QC : CMHC.
Coyle, G. 2000. Qualitative and Quantitative Modeling in System Dynamics: Some Research
Questions. System dynamics review 16: 225-243.
Coyle, G. 2001. Rejoinder to Homer and Oliva. System dynamics review 17: 357-363.
Dillman, D.A. 1999. Mail and Internet Surveys: The Tailored Design Method, Second
Edition. Toronto, ON: Wiley.
Goodman, M.R. 1989. Study Notes in System Dynamics. Waltham, MA: Pegasus
Communications.
Institut de la statistique du Québec. 2002a. Naissances et taux de natalité. Québec 1951-2001.
Institut de la statistique du Québec. 2002b. Décés et taux de mortalité. Québec 1951-2001.
Hoch, SJ. 2001. Combining Models with Intuition to Improve Decisions. The Wharton
School. New Y ork: Wiley.
Homer, J. and R. Oliva. 2001. Maps and Models in System Dynamics: A Response to Coyle.
System dynamics review 17: 347-354.
Raghunathan, S. 1999. Impact of Information Quality and Decision-Maker Quality on
Decision Quality: A Theoretical Model and Simulation Analysis. Decision Support Systems
26: 275-286.
Roberts, N., D. Anderson, R. Deal, M. Garet and W. Shaffer. 1983. Introduction to Computer
Simulation. Portland, OR: Productivity Press.
11
Schrage, M. 2000. Serious Play: How the World’s Best Companies Simulate to Innovate.
Boston, MA: HBSP.
Sillince, A.A.J. and H.M. Saeedi. 1999. Computer-Mediated Communication: Problems and
Potentials of Argumentation Support Systems. Decision Support Systems 26: 287-306.
Société d’ habitation du Québec (SHQ). 2000. Le marché de l’habitation durant les deux
derniéres décennies. Montréal, QC : SHQ.
Société d'habitation du Québec (SHQ). 2002a. Pénurie de logement-la situation. Available
at : www.shq.qc.ca.
Société d’ habitation du Québec (SHQ). 2002b. L’ habitation au Québec. Bulletin trimestriel de
conjoncture 6 (2) : table 11.
Société d'habitation du Québec (SHQ). 2003. La pénurie de logements locatifs. Available at:
www.shq.gouv.qc.ca/fr/penurie_logements/situation.html.
Statistique Canada. 2001a. Estimations de la population. Ottawa, ON : Statistics Canada.
Statistique Canada. 2001b. Census. Ottawa, ON : Statistics Canada.
Sterman, J.D. 2000. Business Dynamics. New Y ork: McGraw-Hill/Irwin.
12
Back to the Top
CC BY-NC-SA 4.0