Integrating System Dynamics and Multicriteria A nalysis:
Towards Organisational Learning for Performance Improvement
Sérgio P. Santos’, Valerie Belton and Susan Howick
University of Strathclyde, Department of Management Science
40 George Street, Glasgow - G1 1QE, United Kingdom.
Phone/Fax: +44 141 5484361/+44 141 5526686
E-mail address: sergio@ mansci.strath.ac.uk
Abstract
The design, implementation and use of adequate performance measurement and
management frameworks can play an important role if organisations are to succeed in
an increasingly complex, interdependent and changing world. Yet, despite widespread
recognition of the importance of performance assessment, there are some issues which
require further study if measurement systems are to be effective in supporting the
decision making process. This article argues that the integration between System
Dynamics and Multicriteria Decision Analysis can address some of these issues, and
ultimately contribute to improve organisational performance. To support this claim,
several problems that make performance measurement systems fall short of their
potential are outlined and, a discussion about how the integration of system dynamics
and multicriteria analysis can help organisations overcome these problems is
presented.
Keywords: Performance Measurement and Management, System Dynamics,
Multicriteria Decision Analysis, Integrating methods.
1 - Introduction
The environment within which most organisations operate is changing rapidly.
Organisations failing to adapt and respond to the complexity of the new environment
tend to experience, sooner or later, survival problems. In this climate of change, the
development, implementation and use of adequate performance measurement and
management frameworks is one of the major challenges confronting organisations and
can play an important role in their success.
Any review of the academic and practitioner literatures illustrates that organisational
performance measurement is a subject receiving considerable attention. Indeed, some
authors argue that a performance measurement revolution is taking place (see, for
example, Eccles (1991) and Neely (1998, 1999)). Several factors justify this current
interest; of particular importance are the growing dissatisfaction with traditional
performance measurement systems and the new environment faced by most
organisations.
In an attempt to address some of the criticisms of traditional systems presented by
several authors (see, for example, Kaplan (1983), Lynch and Cross (1995), Banks and
Wheelwright (1979), Fitzgerald and Moon (1996), Tumey and Andersen (1989)), and to
deal with a rapidly changing environment, several performance measurement systems
(PMS) have been proposed in the last decade. The Balanced Scorecard (Kaplan and
Norton, 1992), the Performance Pyramid (Lynch and Cross, 1995) and the Results and
Determinants Framework (Fitzgerald et al. 1991) are only a few examples.
However, in spite of the availability of various approaches to develop PMS, it is
recognised that some issues deserve further research. The identification of key
performance factors (or performance drivers), a better understanding of the inter-
relationships and the consideration of trade-offs between performance measures, the
dynamism of organisations and the dynamism of measurement systems are only a few
examples of topics where further research is required. The key argument presented in
this paper is that the integration between System Dynamics (SD) and Multicriteria
Decision Analysis (MCDA) can provide a useful framework in which to explore some
of these issues, and consequently, to better understand and ultimately improve
organisational performance.
However, we believe that if this support is to be effective, there are some technical and
conceptual problems dealing with integration which have to be solved.
The paper is structured as follows. In Section 2 we discuss why PMS frequently fail in
supporting the decision making process, focusing particularly on issues where the
integration of SD and MCDA can be helpful. In section 3 we outline the strengths of
these two approaches to show how their integration can bring new insights to inform
and support performance measurement and management. An illustrative example from
the health care sector is presented. Finally, in Section 4 we conclude with some closing
remarks.
2 - Performance Measurement vs. Performance Management
It is widely recognised that there is no perfect PMS applicable in all contexts. The
appropriateness of a specific measurement framework may depend on several factors.
The organisational context and the purpose of performance measurement are only a few
examples. In reality, these and other factors have led to the development of a wide
variety of frameworks of different characteristics and of varying complexity. Also, there
appears to be a general consensus that measurement systems should be designed,
implemented and used so that they enable continuous performance improvement rather
than simply control or monitor (Fitzgerald and Moon (1996), Lebas (1995), Neely et al.
(1995)).
However, improving the performance of an organisation is not a straightforward task. In
most of the cases, PMS do not provide decision makers with the information they most
need to effectively manage performance. In reality, diverse reasons may be pointed out
to justify why so many efforts to improve performance have not met with great success.
We believe that one of the major reasons why PMS fail is related with the large and
complex amount of information they provide, conjointly with the absence of approaches
to assist decision makers understand, organise and use such information to manage
organisational performance. Neely et al. (1995), for example, argue that most PMS
provide “too much data and too little analysis’. Due to the limited information
processing capabilities of the human brain, we believe that the use of approaches such
as SD and MCDA can be very valuable to assist decision makers to understand and
organise this information (usually of a conflicting nature and involving delays,
feedback, non-linearities, and so on) in order to develop and implement more coherent
and better informed action plans.
However, we believe that the support of these approaches can be extended beyond a
better analysis and use of the information resulting from measurement. Namely, it is our
belief that the integration between SD and MCDA can also provide very valuable
insights to address some weaknesses in the design of performance measures and in the
implementation of measurement systems.
For example, it is recognised that the identification of factors affecting performance and
their relationships is one important step in PMS design. However, it is also recognised
that much more has to be done in this topic (Neely (1999), Flapper et al. (1996), Bititci
and Tumer (2000)). From one side, the practice shows that several organisations create
performance measures on an ad hoc basis (Flapper et al., 1996). From another side, with
a few exceptions (for example, Kaplan and Norton acknowledge the inter-relationships),
little consideration is given in the literature to the relationships between performance
measures. Trying to identify factors affecting performance and their relationships,
Suwignjo et al. (2000) used cognitive maps. However, we believe that cognitive maps
alone do not allow participants to fully understand the interconnections between these
factors due to the existence of non-linear interactions between them, delays and
feedback loops. To deal with the dynamic complexity inherent in social systems and to
infer dynamic behaviour, quantitative simulation is required (Senge (1990), Sterman
(1989a, 1989b)). Consequently, we believe that the translation of qualitative diagrams
into a simulation model using the SD approach can enrich the analysis and provide very
useful insights for the design of measurement systems.
It is also recognised that there are frequently conflicting performance measures, and
therefore trade-offs among these measures are inevitable. Frequently, actions taken to
improve one measure may lead to a decline in other performance measures. Or, due to
time delays in feedback, some actions may produce long-run effects in performance
different from their short-run effects. Given that the relative importance of various
policy impacts is seen differently by different stakeholders, and usually it is not possible
to meet all stakeholders’ aspirations, trade-offs must be made (although they are often
not made explicit). Some of the most well known performance measurement
frameworks (for example, the Balanced Scorecard, Results and Determinants
Framework and Performance Pyramid) emphasise the need of measurement systems to
make explicit the trade-offs between the various performance measures, but are vague
in how to deal with these trade-offs. In reality, we can say that in the existing PMS the
consideration of trade-offs between measures and the policies’ evaluation process are
omitted or not explicitly addressed. We believe that the use of an appropriate MCDA
procedure can be helpful in this context. The need of additional research dealing with
the trade-offs among performance measures is, indeed, recognised in the literature (see,
for example, Ittner and Larcker, 1998).
Another area where we believe integration can be very valuable is related with the use
of performance measures to evaluate the effectiveness of different policy alternatives or
courses of actions to improve performance. This is, indeed, one important purpose for
which performance measures are often developed (Stiefel et al. 1997). However, to
judge the effectiveness of policy alternatives without the support of appropriate tools is
frequently a very difficult if not impossible task. From one side, stakeholders have
different points of view of what represents good and bad performance of an action, and
consequently it is likely that these stakeholders adopt a range of different measures for
judging the performance of policy alternatives. From another side, the long time delays
and systemic effects of actions, the fact that cause and effect are often distant in time
and space and, that obvious interventions do not always produce expected
consequences, also make this judgement very difficult (Senge, 1990). In addition, even
in those situations where measurement systems provide information to judge actions,
this information refers frequently to decisions that already have been made.
Measurement systems do not explain how performance will evolve in the future. It is
therefore our belief that the integration between SD and MCDA can play an important
role to achieve this objective in the most effective way. From one side, SD modelling
allows decision-makers to verify, through the use of ‘microworlds’, and consequently,
in a risk-free environment, the effect of different actions on a system’s performance
measures over time. That is, SD enables a greater understanding of the effects of actions
already implemented but also of the effects of alternative actions to be considered for
implementation. From another side, the use of a MCDA approach enables the decision
makers to develop an explicit evaluation process of these actions, helping decision
makers to leam about the problem they face and about their own and other values and
judgements (Belton, 1990).
An additional issue on which little research has been conducted is related with the
dynamism of measurement systems. Although it is widely recognised that
organisational performance measurement systems should be dynamic, evolving over
time, most organisations have only static performance measurement systems, and much
of the work that is currently ongoing in the field of performance measurement is static
in orientation (Bititci and Turmer (2000), Neely (1999), Suwignjo et al. (2000),
Waggoner et al. (1999)). However, given that performance measures change over time
as well as their importance to the stakeholders, measurement systems need to be
sensitive to these changes. We believe that the SD and MCDA approaches allow
decision makers to review and update systematically the measurement system, taking
into consideration these changes. From one side, SD models can help decision makers
gain insights of system’s behaviour over time which may reveal very valuable to review
and update PMS. From another side, the use of a MCDA procedure allows decision
makers to review and reprioritise the weights for each performance measure, reflecting
how important a performance measure is to the decision maker at a given moment in
time.
Finally, it is our belief that integration may also assist the implementation of
measurement systems. That is, once the measurement system has been developed it has
to be implemented. There is, however, some evidence that this is not a straightforward
task due to “fear, politics and subversion” (Neely et al. 2000). Given that SD and
MCDA have individually proved their potential to inform and support decision making,
working as a vehicle to reach consensus, ownership and commitment among decision
makers, we believe that their effective use in the context of performance measurement
can facilitate the implementation phase.
These are some of the issues dealing with the design, implementation and use of
measurement systems which require more attention if performance assessment wants to
succeed in supporting the decision making process and in improving organisational
performance. In reality, taking an holistic view of the field of performance
measurement, we can conclude that much of the literature and practice has focused on
the design of performance measures and measurement systems, with less concem for
managing performance. Certainly, the design and control of isolated measures is a
valuable exercise in the sense that it informs decision makers about how the
organisation is performing against goals, and it assists in identifying an organisation’s
strengths and weaknesses. However, it does not explain why an organisation is
performing in such a way, nor what corrective actions to adopt when needed. To be
efficient, measurement frameworks should enable organisations to manage, rather than
simply measure their performance. That is, if performance measurement wants to lead
to enduring and continuous performance improvement, then the design of measurement
systems, their implementation, analysis and use should form a continuous loop (Figure
1).
Fig. 1 - Life cycle of the performance measurement and management process
DESIGN
(what, when and
how to measure?)
IMPLEMENT MEASURE
(what happened?)
PLAN J
ANALYSE
(what... which Com] .
actions to adopt?) (why did it happen?)
That is, a robust framework for performance measurement and management should
assist the: design of the measurement system (this process enables decision makers to
develop an integrated set of performance measures, to set targets and to define the
procedures for periodic data gathering); measurement of key metrics (this process
informs decision makers about how the organisation is performing); analysis of metric
results (this process enables decision makers to understand why the organisation is
performing in such a way and to identify the need for corrective actions); planning of
corrective actions (this process enables decision makers to test and evaluate the effects
of alternative corrective actions in the organisation's performance, leading to better
informed decisions); and implementation of corrective actions (once appropriate actions
have been identified, this process should support their implementation). Notice,
however, that the process of performance measurement and management should be
iterative and not a linear sequence of steps. This is indicated by the arrows in the centre
of the diagram.
In the following section we attempt to demonstrate, in more detail, how the integration
between SD and MCDA can assist the different stages of the performance measurement
and management process, and consequently, how the integration can help decision
makers to close this loop.
3 - Integrating SD and MCDA: Towards a more effective management of
organisational performance
From the previous sections we can conclude that much of the work in the field of
performance measurement has focused on measuring rather than on managing
performance. That is, the existing PMS tend to provide information about the
performance of the organisation and whether corrective actions are required or not.
However, these systems neither provide participants with tools to identify the causes of
poor performance nor provide participants with tools to help them in evaluating and
selecting appropriate corrective actions. Given that, in most of the cases, understanding
the causes of poor performance and determining the proper action plan for performance
improvement require detailed analysis of the structure of the problem under study and
the consideration of trade-offs, we believe that SD and MCDA can play a major role in
these phases. However, the support of these approaches can be extended to the design
and implementation of measurement systems.
In this section we give a brief overview of SD and MCDA and we build on the strengths
of these approaches to show how their integration can bring new insights to inform and
support performance measurement and management.
3.1 - System Dynamics
System dynamics was conceived and developed in the late 1950s and early 1960s at the
Massachusetts Institute of Technology by Jay Forrester. Indeed, the advent of SD is
generally considered to be the publication of Forrester’s pioneering book, Industrial
Dynamics in 1961. Since then, significant advances have been made, and a cursory
examination of the literature indicates that the number of organisations using SD
models for the development of both strategic and operational policies is growing
rapidly. An overview of SD can be found, for example, in Forrester (1961), Richardson
and Pugh (1981) and Sterman (2000).
SD models are frequently developed and used to represent, analyse, and explain the
dynamics of complex systems. The dynamics or behaviour of a system is defined by its
structure and the interactions of its parts. The main goal of SD is to understand how this
behaviour is produced, and use this understanding to predict the consequences over time
of policy changes on the system.
Although SD models can help decision makers in enhancing understanding of system
behaviour over time, SD models do not concern themselves with the explicit evaluation
of this behaviour. That is, a pattem of behaviour is frequently presented as preferable to
another, based only on the modeller’ s intuition (Gardiner and Ford, 1980). The rationale
behind the decision reached is omitted or not clearly documented. This is, indeed, a
criticism to SD models (see, for example, Legasto et al. (1980) and Kadanoff quoted in
Gardiner and Ford (1980)). As pointed out by Gardiner and Ford (1980, pp. 242), “the
emphasis and focus [of SD models] is on developing models that show consequences,
not on formally evaluating these consequences” (text in brackets added).
However, evaluating, and choosing between alternative courses of action is not a
straightforward task. In the context of SD, as in many others, the decision maker is
confronted with a large and complex amount of information, usually of a conflicting
nature and reflecting multiple interests. Consequently, the use of an appropriate MCDA
approach can be very valuable to assist decision makers organise such information in
order to identify a preferred course of action (see for example, Belton 1985).
3.2 - Multiple Criteria Decision Aid
MCDA is now 30 years old, and it is an important area of Operations
Research/M anagement Science (OR/MS). Since the first session devoted to multicriteria
analysis in a scientific congress, organised by Roy during the 7th Mathematical
Programming Symposium, which was held in The Hague in 1969, the field of MCDA
has seen remarkable growth.’ On one hand, important theoretical results have been
achieved leading to the development of several multicriteria methods. On the other
hand, the number of real world applications documented in the literature is increasing
considerably. A synthesis of the main streams of thought in this field can be found in
Belton (1990), Stewart (1992) or Mollaghasemi and Edwards (1997).
MCDA is designed to take explicitly into account multiple and usually conflicting
objectives in supporting the decision process. In this way, MCDA methodologies can
help decision makers to learn about the problems they face, and consequently to make
better informed and justifiable choices. This is a view shared by many prominent
researchers in the field (see for example, Belton (1990), French (1988), Goodwin and
Wright (1998) and von Winterfeldt and Edwards (1986)).
That is, in the same way that one of the principal benefits arising from the use of a SD
model is to enable the decision maker to gain a greater understanding of the system of
interest, one of the main advantages from the use of a MCDA approach is the learning
which occurs about the problem faced and the alternative courses of action.
Furthermore, the use of a MCDA approach enables the decision maker to develop an
explicit evaluation process, which might be used to justify and explain to others why a
particular option was selected (Belton (1990) and Goodwin and Wright (1998)).
3.3 - Integrating SD and MCDA to support the performance measurement and
management process
The integration of MCDA and SD is not new. Y et it has attracted only the attention of a
few researchers over the last two decades, despite the recognition that integration can
give rise to a new synergistic approach. To the best of our knowledge, there are no
studies striving to fit the synthesised approach into the setting of performance
measurement and management.
Although we can find in the earlier 70’s the first attempts to integrate simulation models
with multicriteria approaches, the earliest attempt to use MCDA and SD in an integrated
way that we are aware of was by Gardiner and Ford in 1976. The methodology used and
the results achieved are reported in Gardiner and Ford (1980). Other examples
providing evidence that integration has potential to be applied in a wide variety of
settings can be found in Hammond et al. (1977, 1978), Mumpower et al. (1979), Dennis
et al. (1983), Gruver et al. (1984), Reagan-Cirincione, et al. (1991), Andersen and
Rohrbaugh (1992), Hsiao (1998), Brans et al. (1998), Kunsch et al. (1999) and Rosas
Flunger (1999).
The purpose of this section is to show that the integration of SD and MCDA can also be
applied and provide very useful insights when supporting the performance measurement
and management process. Particularly, we intend to discuss how this integration can
support the design, implementation and use of measurement systems in health care
organisations. Although this integration has potential to be applied in a wide variety of
organisations, four main reasons make us believe that it can provide very valuable
insights in NHS Hospital Trusts.
First, Hospital Trusts are complex systems in which several parts interact over time.
Because of the breadth of activities within hospitals and the interconnectedness between
them and the influences arising from the extemal environment, measuring and
managing hospitals’ performance is especially challenging. Unless some attempt is
made to understand these interactions and to co-ordinate the actions of the different
parts, the system will under-perform considerably. For example, within a hospital the
accident and emergency (A &E) department does not exist in isolation. It cannot operate
independently of the surgery department, nor of the management of elective patients,
nor of the management and staffing of beds on wards. Actions in one of these parts of
the system affect the others (sometimes counter-intuitively), and consequently affect the
dynamics of the whole system. Moreover, the performance of the A&E department is
also influenced by elements outside the boundary of hospitals including GP referrals,
capacity of community care services, technology, as well as economic and social
conditions. The achievement of high levels of performance by Hospital Trusts and their
continuous improvement demands that these interactions be understood and effectively
managed. Consequently, performance evaluation frameworks designed for hospital
performance should provide decision-makers with this understanding, which is of vital
importance to appropriately inform and assist decision-making.
Second, the design of any system to measure and manage performance in Hospital
Trusts needs to take into account the interests of multiple stakeholders including
patients, health care professionals (clinical and nursing staff), administrative and
managerial staff, purchasers (GPs fundholders and Health Authorities), as well as
central government (NHS Executive). Usually, these stakeholders have different points
of view of what represents good performance of Hospital Trusts. It is therefore likely
that they adopt a range of different measures for judging the performance of Hospital
Trusts. The large variety of points of view - either conflicting or overlapping - between
stakeholders justify, by itself, the need for an integrated and coherent set of performance
measures to assess the performance of NHS Hospital Trusts, as well as the need to make
trade-offs between objectives when assessing the performance of Hospital Trusts.
Indeed, the health care reforms in place in the U.K. have emphasised the need for trade-
offs in the design of performance measurement systems, reflecting the need to take into
account the interests of different stakeholders.
Third, although Hospital Trusts in U.K. have in use performance measurement systems,
most of them present several weaknesses. As it has been argued (Ballantine et al. 1998),
in these systems: performance measures (financial and non-financial) are usually poorly
integrated; the interconnections between performance measures across performance
dimensions are very difficult to establish; cause and effect relationships are very
difficult to identify and consequently are not explored; specific targets or standards
against which goals attainment can be measured are frequently absent; and, as a
consequence of all these weaknesses, strategic decision making is deficiently supported.
Fourth, performance assessment has been stressed as central in all areas of the NHS,
including Hospital Trusts (see for example, Department of Health (1997) and NHS
Executive (1999)). Indeed, in the White Paper - The New NHS: Modern, Dependable,
the Department of Health argues that the new NHS in the U.K. will be performance
driven. Therefore, we believe that attempts should be made to develop and provide NHS
Hospital Trusts with performance measurement and management systems which assist
them in conforming with this objective, leading to continuous improvements in the
health care sector.
These are some of the reasons that make us believe that the proposed methodological
framework can support Hospital Trusts in measuring and managing performance. In
what follows and using a simple illustrative example, we intend to discuss how can this
support be concretised. Naturally, other organisations where similar conditions apply
can also benefit considerably from the insights provided by the framework under
consideration.
It is widely accepted that effective performance measurement systems should provide
decision-makers with information about the degree to which organisational objectives
are achieved and how well the organisation is performing its tasks. To get this
information, an appropriate set of performance measures is required. However, the issue
of which performance measures a given organisation should adopt is not a
straightforward one. Although the design of performance measures has been widely
discussed in the literature (for an exhaustive review of the literature see Neely et al.
1997), there is no consensus concerning the best way to develop performance measures.
It is however recognised by several authors that PMS should align performance
measures with the strategic objectives of the organisation (see for example, Globerson
(1985), Kaplan and Norton (1992) and Lynch and Cross (1991)). In this way, it is
assured that the system will provide information on whether these strategic objectives
are being successfully implemented or not, and additionally, it is assured that if
corrective actions are required, measures consistent with these objectives will be
adopted. Yet, despite this, it is recognised that several organisations develop
performance measures on an ad hoc basis and without taking into consideration the
relationships between measures. It is also recognised that even when more “structured”
frameworks for performance measurement are adopted, little guidance on how the
appropriate measures can be identified (Neely et al. 2000) and how to capture an
holistic view of the system being assessed (Sloper et al. 1999), is provided.
The first stage in developing a performance measurement and management system
should aim, therefore, to identify the strategic objectives or orientation of the
organisation and the factors that are critical to its success. Strategy formulation and
performance measurement design should, however, work in an interactive way. In the
case of NHS Hospital Trusts let us suppose that this orientation is ‘to promote effective
delivery of high quality services’. We must recognise, however, that such overall
objective is too broad for managers to evaluate how well it is being achieved by
Hospital Trusts. It does however provide a starting point for specifying the lower-level
objectives, or performance dimensions as we will refer to them, and ultimately the
performance measures, that are critical to evaluate its achievement.
Performance measures are defined and used to inform decision makers about to what
extent the organisational objectives are realised. This means that we need to arrive at a
set of performance measures which can be assessed, preferably, on a numeric scale.
When the initial performance measures elicited from the decision maker are ambiguous
or ill-defined they may need to be broken down into more specific measures before
measurement can take place.
For example, in the White Paper, The New NHS: Modern and Dependable, it is
highlighted that performance measurement and management in the health care sector
needs to support the following objectives: fairer provision of services, higher quality,
improved value for money, greater responsiveness and thereby better health. In order to
evaluate the achievement of these objectives, six dimensions of performance were
defined in the Performance Assessment Framework (NHS Executive, 1999): ‘effective
delivery of appropriate health care’, ‘health improvement’, ‘fair access to services’,
‘efficiency’, ‘patient and carer experience of the NHS’ and ‘health outcomes of NHS
health care’. That is, these are the factors which directly (or indirectly) contribute to the
achievement of the overall objective stated previously.
Notice, however, that although the identification of these factors or dimensions
constitutes an important step in the performance assessment process, they do not
immediately present a workable framework for detailed evaluation of the performance
of Hospital Trusts. To effectively and thoroughly assess, for example, if a Hospital
Trust is improving the health of the population, if it is providing clinically effective
health care or if it is ensuring that people’s ability to obtain health care is related to their
needs, these dimensions should break down further. In general, the objectives or
dimensions should be decomposed to a level where they can be easily assessed. That is,
they should be ultimately decomposed into performance measures.
Performance measures provide very valuable information to evaluate how well Hospital
Trusts are performing against the stated objectives. For example, to assess the
‘efficiency’ of a Trust, that is, the way in which a Trust uses its resources to achieve
value for money, several performance measures might be defined. Day surgery rates,
length of stay in hospital, unit costs and labour productivity, are only a few examples.
Several tools or facilitative processes can be used to foster creative thinking in order to
identify these performance measures. We can anticipate, for example, that the use of
Post-Its complemented with qualitative maps (structuring the ideas generated) may be
very valuable at this stage.’
Given that in most of the cases a wide range of performance measures is generated, it
can be necessary and worthwhile to bring some structure to this list of measures. The
use of a hierarchy, or performance measures’ tree as we will refer to it, can help in
structuring these measures and can be equally useful in forming the basis of a multi-
attribute value function analysis. Figure 2 is based on the High Level Performance
Indicators (NHS Executive, 1999), and gives an example of a performance measures’
tree that can be used to assess performance in Hospital Trusts.
|___ Overall Performance _| Performance Dimensions | Performance Measures _|
r Level 1 r Level 2 r Level 3 1
eallh
Improvement
Surgery Rates
Siz of Inpatient Waiting
Day Case Rate
engt of Stay in Hospital
Eifective Delivery of
Appropriate Health Care
Eifectve Delivery of
High Quality Services
Patients who wales than 2
Tous fr admission
‘Outpatients sen within
13 weeks of GP referral
Pation/Carer
Experience of
the NHS:
Patons with operations cancled|
for non medial reasons
1% of tose on wating ist
‘ating 12 months or more
Health Outcomes of
‘NHS Health Care
‘Adverse evens!
complications of treatment
Notice that the process of identifying and structuring performance measures in this way
offers several advantages. First, it assures that performance measures are designed in
line with the strategic orientation of Hospital Trusts. As a result, these measures,
directly or indirectly, provide information on whether the strategy defined in stage one
is being successfully implemented and, additionally, encourage behaviours consistent
with this strategy (Neely, 1999). Second, going through the process helps to clarify
people’s thinking on the subject and on their objectives. This leads to a clearer
understanding of what should be measured, why and how, and provides insights for
better decisions. Third, it provides the basis of a multi-attribute value function analysis
that may be carried out not only to assess how well an hospital trust is performing but
also to support the decision making process if policy options need to be analysed and
evaluated.
It is important to emphasise, however, that the generation of a proper set of performance
measures encompassing all areas of concem is not always an easy task. This is, indeed,
an iterative process which should not finish with the design of a ‘first’ performance
measure’ s tree. Notice that, while this tree provides information about the links between
performance measures and performance dimensions, and between these dimensions and
overall system performance, it neither shows how performance measures interact with
one another nor provides significant insights about possible intervention or leverage
points. The use of both qualitative and quantitative SD modelling can play a very
valuable role to foster this understanding.
Qualitative modelling, for example in the form of a causal loop diagram (see Figure 3),
can play a fundamental role in this phase for several reasons.
First, it gives a clear picture of the different elements of the problem and the
interconnectedness between them (cause and effect, feedback loops, delays and so on).
This facilitates the understanding of systems behaviour and results, ultimately, in a
deeper understanding of what one should care about in measuring the organisation’s
performance. For example, the performance measure ‘waiting time’ is considered a
proxy measure of the Patient/Carer Experience with Hospital Trusts. Lower values
reflect a better experience of patients and consequently a higher satisfaction level.
Figure 3 shows how this measure interacts with other ones. For example, an increase in
the admission rate will tend to lead to a decrease in the time that patients have to wait
until they are seen by a doctor. Conversely, an increase in the size of the waiting list
should result in an increase in the waiting time, other factors remaining constant. Notice
that the use of causal loop diagrams (CLD) allows to identify feedback loops, and it is
the interaction between these loops that determines the dynamics of the system. As
argued by Wolstenholme (1990) “it is the analysis of such loops which facilitates
understanding of how the processes, organisational boundaries, delays, information and
strategies of systems interact to create system behaviour.” Moreover, important delays
can be represented using causal loop diagrams. Delays may be critical in the health care
sector given that often trade-offs between the short- and long-run effects of policies on
performance have to be considered.
Second, it allows the identification of intervention points or policy levers that can be
used to control the performance of Hospital Trusts. For example, to increase the
likelihood of achieving the desired level for the performance measure stated previously,
a possible course of action is to reduce the duration of treatment. As can be seen in the
CLD (Figure 3), reducing the length of stay of patients in hospital would lead to an
increase in the discharge rate and to a decrease in the number of patients in hospital,
allowing higher admission rates and, ultimately, shorter average waiting times.
However, Figure 3 also shows that while shortening the duration of treatment might
lead to shorter waiting times, it also increases the likelihood of inappropriate discharges.
It is, therefore, likely that some of these patients will return for further treatment either
as elective patients or as emergency cases, generating in the future an increase in the
waiting list. Furthermore, if it is reasonable to suppose that NHS may in part control the
performance of certain variables, there are others which overall control is beyond
Hospital Trusts. The CLD shows that many factors, some of which are outside the direct
control of hospitals, such as GP referrals or the capacity in community care, have an
important bearing on the performance achieved by a particular hospital. That is, the
CLD reveals, on its own, that the measurement and management of performance in
NHS Hospital Trusts is not a straightforward task. To improve performance in the
health care sector a great understanding of the problem under study is required. As
argued by Ballantine (1998), “control and co-ordination of a variety of activities, carried
out by various organisational groups, is necessary”. This section also intends to show
that this control and co-ordination can be much easier if participants benefit from the
insights provided by SD simulation modelling.
Elective
Patients" Hospital
A wating Tim Capacity
GP ( \
referrals. ~ 5 aaa" . j
Elective Elecive A y
Patients in Admission rca
__® Waiting List Rate CA
wm = Capacity
* nade N Utilisation
at ® patients in
Readmissions La
Ward
Discharge
Rate +
ee an: :
Treatment \ a
+ a +
oY
community AGE AGE
Care Patients in ‘Admission
Waiting List Rate
* Capacity 19 we \
Accident and eieyee +
Emergency + .
Patients AGE y
Patients’
Waiting Time
Figure 3 - Causal loop diagram
We can conclude, therefore, that the increased understanding about the feedback
structure of systems provided by the CLD can be particularly helpful to the decision-
makers in Hospital Trusts. From one side, in defining an integrated set of performance
measures and in clarifying their interrelationships. From another side, in identifying
potential intervention points where effective change can take place.
However, while CLDs (and qualitative diagrams in general) may assist users in the
understanding of some relationships between measures, suggest performance measures
and leverage points, they present some limitations. Therefore, they should be used as an
intermediate step in the process of performance measurement and management. For
example, when used alone, CLDs do not allow participants to understand the dynamic
behaviour of factors affecting performance and do not allow a full assessment of the
interconnections between these factors. Consequently, the translation of these diagrams
into a simulation model using the SD approach may be necessary. In the context of
performance measurement, this is a view also shared by Linard (1995) and Sloper et al.
(1999).
Building and running a SD simulation model it is much easier to understand the process
underlying performance generation and consequently, to identify the factors that are
susceptible to lead changes, also called ‘performance drivers’. In addition, building a
simulation model of the system under consideration and defining relevant variables and
the relationships between variables, might be useful for identifying performance
measures that might have been omitted in the previous stages. Furthermore, some
performance measures originally regarded as unimportant or overlooked might seem
worthy of remark after running the simulation model. For example, waiting time of
A&E patients is a performance measure often used to evaluate the effectiveness of
Hospital Trusts. However, when taken alone, this measure can provide misleading
information given that reductions in the delay of A&E admissions might cause delays in
elective inpatient admissions and an increase in the rate of cancellations of pre-
scheduled non-emergency admissions. Therefore, although the importance of a rapid
response time to A&E patients cannot be underestimated, this measure must be jointly
assessed with other measures. Using a SD simulation model it is possible to present
evidence that waiting time of AGE patients and elective cancellations are linked and
compensating measures, and as such, they should be jointly assessed (see for example
Lane et al. 2000). Ignoring this kind of relationship between measures leads often to a
situation where the proposed actions produce results opposite to those intended. For
instance, in the previous example, due to the delay in inpatient admissions and to the
higher number of patients with operations cancelled for non-medical reasons, it is likely
that some of these patients would become emergency cases, generating in the future an
increase in the AGE patients, making things even worse.
The development and use of a simulation model constitute, therefore, an important step
in getting a thorough understanding of the relationships between measures and in
getting a consistent and integrated set of performance measures. However, a simulation
model may also play a vital role in testing and comparing alternative actions to improve
system’s performance. As exemplified previously, in some cases, if participants do not
have access to a simulation model to test and design policy actions there is the danger
that the selected policies will worsen the problem instead of amending it. In other cases,
even if a given situation improves, as a result of adopting a given course of action, it can
be only temporarily, and consequently, other policies could have been more effective.
For example, Wolstenholme (1999) demonstrates through the use of SD modelling that
an increase in hospital bed capacity is not the most effective solution to reduce total
patient waiting times. He illustrates that, when bed capacity is increased although more
patients are admitted to hospital, the effect is temporary. As soon as the new capacity is
full, the number of patients in hospital wards stabilises and the pre-hospital waiting time
increases again. Furthermore, a counter-intuitive behaviour may occur as a consequence
of this type of policy. For example, the additional bed capacity introduced can
stimulate more demand for hospital treatment or, at least, encourage more GPs referrals
to hospital.
To understand the ‘dynamic complexity’ inherent in these situations, a SD simulation
model is required. By developing and running this model participants can understand
the stock and flow structure of systems and observe the changes that occur over time in
the variables of interest.
Stock and flow diagrams are usually used as a basis for developing this simulation
model. As with CLDs, stock and flow diagrams show relationships among variables.
However, unlike CLDs, and as can be seen in Figure 4, stock and flow diagrams
distinguish between different types of variables: stocks, flows and information.
Inappropriate_Disharges
sei.
write 40.) 1
O
Waitng._List
Tdmision_rate ae
GP _referals
ies Ase Community. Care. Okcharge_rate
Figure 4 - Stock and flow diagram
The stock and flow diagram in Figure 4 constitutes a simplified representation of the
way in which patients flow from the community into the NHS Hospital Trusts and back
into the community. From the stock and flow diagram we can also see how different
performance measures interact. However, one of the major advantages of this diagram is
that it can be used as a basis for developing a SD simulation model which allows to
represent explicitly the system’s internal structure and this structure is often the
underlying source of the problem. Thus, by finding and modifying this system structure
we are able to improve organisational performance in the most effective way.
According to Lebas (1995, pp. 28), “understanding the processes underlying
performance is the only way to define the measures that lead to actions. If we
understand which of the steps in the process is defective, appropriate corrective action
can be identified”.
Once the performance dimensions and an integrated and consistent list of performance
measures have been defined, targets should be set in order to evaluate goals attainment.
That is, having identified the performance dimensions and measures which the decision
makers consider to be relevant to evaluate organisational performance, the next step is
to set targets and to find out how well the organisation is achieving them. These targets
can be set in different ways. One possibility which we believe is useful in the case of
Hospital Trusts consists in establishing targets based on a range of ‘acceptable
performance’. The upper limits of this range may include, for example, industry
benchmarks and the lower limits may represent the worst tolerable performance for each
measure (see Table 1).
Table 1 - Performance Targets
Performance Measures
Surgery Sizeof DayCase Length of Adverse
Rates Inpatient Rate Stay in 22 events
Waiting List Hospital
Best attainable performance
Achieved performance
Worst tolerable performance
It is important to emphasise that both the publication of ‘league tables’ for some
hospital performance measures and the increased understanding obtained with the
development and use of SD models can assist decision-makers in setting these
boundaries. It is important to observe that although providing valuable information to
the decision-makers, this table focuses on individual measures. As a supplement to this
information, an overall view of the performance of the hospital under consideration can
be obtained by aggregating these measures into a single indicator of overall
performance. Notice, however, that an overall view of the performance of an Hospital
Trust cannot be obtained if these data are not normalised into a common scale.
That is, even when a set of measures and corresponding targets are defined, other issues
have to be addressed. From one side, it is very likely that this set will be composed by
multiple and heterogeneous measures of performance, which cannot easily be reduced
to a single dimension. From another side, it is very likely that performance measures
will conflict with each other in that the improved achievement with one measure can
only be accomplished at the expense of another. MCDA approaches have proved their
potential in dealing explicitly with these trade-offs and in integrating multiple
heterogeneous measures into a single or a few key indicators of overall performance.
Notice that, while the focus of most performance measurement systems is on individual
measures, with the use of an appropriate MCDA approach decision-makers can gain an
overall view of the performance of an hospital by aggregating performance measures.
Because the Multi-attribute Value Function (MAVF) approach is one of the best suited
approaches to assist decisions involving a short-list of altematives and many criteria
(Belton, 1986), it is well suited in the present context.
The procedure we propose to carry out this analysis (that is, to quantify the factors in
performance and to arrive to an indicator of overall performance) makes use of a
hierarchical, weighted additive value-function® and is supported by the use of the
software V-I-S-A°, Notice that at this stage we have identified already the performance
measures and corresponding targets which the decision makers consider to be relevant
to evaluate the performance of the Hospital Trust. Therefore, the next step is to find out
how well the Hospital Trust performs on each of the lowest-level measures in the tree
(Figure 2).
There are many possible ways to evaluate or score the performance of the Hospital with
respect to these performance measures. A possible procedure is to compare the actual
performance of the hospital trust against the targets defined in table 1 and score it using
a normalised 0-100 global scale on which the 0 and 100 points are defined by the worst
tolerable and best attainable possibility for each performance measure. The scoring
process can be realised through direct rating or by using value functions. As we can see
in Figure 5, scores are directly related to the nature of each measure, and when
appropriate they reflect non-linearities of values scale which may exist for some
measures. Figure 5 also shows how well the hospital trust performs regarding the
measures ‘Day case rate’ and ‘Size of Inpatient Waiting List’.
Br
etal
| a
sh Setingy
ama) re |
‘4 SA Wort 0 Wort
ure: ¥
Figure 5 - Part of the V-I-S-A analysis
Having scored the performance of the hospital trust with respect to all the measures at
level 3 of the performance measures tree, the next stage is to weight those measures to
reflect their relative importance to the performance dimensions at level 2. As for scoring
there are many possible ways of weighting performance measures. Independently of the
procedure to be adopted, it is important that people do take the range over which the
measure is assessed into account when assigning importance weights. The weights for
the higher-level measures (performance dimensions) in the value tree are found by
summing the appropriate lower-level weights. These weights can be assessed either by
direct comparison of the performance dimensions at level 2 or by selective comparisons
of performance measures at level 3.
Once these weights are defined, we are in a position to find out how well the hospital
trust performs in each performance dimension and how it performs overall (Figure 6).
This is done by using a hierarchical weighted value function.
Ee ES Bendion Speer Graph Satie Hee
oC Bee ec
Sk ae le El =m “E |
Figure 6 - Determining an indicator of overall performance using V-I-S-A
A thorough sensitivity analysis can be carried out to analyse or explore how robust the
overall score obtained by the Hospital Trust is to changes in the inputs to the model,
particularly to changes on priorities and values.
Notice that the procedures carried out so far using V‘1-S-A, allow us to look individually
at the scores on each of the performance measures (Figure 5, Figure 6 - A and B) but
also to evaluate how the hospital scores in each of the performance dimensions (Figure
6 - C) and over all the performance measures (Figure 6 - D). This information is very
valuable to assess how well the hospital is performing. By scoring and reporting results,
decision makers can identify where performance has been strong, and where
improvement is required. But, as stressed in the previous sections, this information is of
little or no help in driving hospital trusts if it is seen as an end in itself. That is, although
these scores allow decision makers to know how the hospital compares with the defined
targets on a range of measures, and therefore to know what is working well and what is
not, this does not provide a strong basis from which to manage effectively for
improvement. To be effective, a performance measurement and management system
should support decision making, informing decision makers, in between other things,
about which are the causes of poor performance and which actions to implement to
obtain effective and appropriate change. That is, about how can performance be
improved.
We believe the integration can greatly assist decision makers in making a better analysis
and a more effective use of the information resulting from measurement. That is, we
anticipate that the support of integration can be extended beyond the design of
performance measures and measurement systems and assist decision makers in testing
and design policies for effective performance improvement.
Very often, the information that performance measures and measurement systems
provide is not enough to assist decision makers in improving performance. Performance
measures and measurement systems are, in general, very efficient in diagnosing
symptoms of problems. However, to effectively manage performance, leading to
continuous and lasting performance improvements, decision makers need to identify the
causes of symptoms (not the symptoms themselves) and the proper corrective actions.
Many PMS have fallen short of their potential because of failure to provide the decision
makers with the understanding and support necessary to do this. However, identifying
the causes of problems and developing appropriate solutions is frequently a difficult
process for the unaided decision maker. As pointed out by Sterman (2000, pp. 27)
“people generally adopt an event-based, open-loop view of causality, ignore feedback
processes, fail to appreciate time delays between action and response and in the
reporting of information, do not understand stocks and flows and are insensitive to
nonlinearities that may alter the strengths of different feedback loops as a system
evolves”.
The focus on the causal structure of problems and the search for leverage points in the
system are some of the strengths which make SD to be an appropriate approach to foster
understanding of the process underlying performance generation and to identify the
factors that are susceptible to lead changes. The use of SD can be, therefore, a very
valuable exercise to assist decision makers gain a greater understanding about how the
organisation is performing and why. And notice that SD modelling allows not only to
understand what happened but can also provide very valuable insights about what might
be about to happen next. SD simulation modelling allows to compute and plot the
performance measures over time, and to predict the need for corrective actions. If
warming indicators of potential problems are identified (for example, when deviations
from targets are detected or undesirable behaviour is expected to occur) corrective
actions can be quickly tested and implemented, averting problems. In some cases, this
understanding leads by itself to the solution of a problem. In other cases, to find a
solution is not so obvious and the use of a multicriteria procedure may reveal very
valuable.
Using SD modelling several alternative actions can be simulated and their impact on the
performance measures of the system tested. However, as stressed previously, the
selection of the ‘best’ action plan is not straightforward. Stakeholders have different and
often conflicting objectives and, as a result, trade-offs must be made. Given that the
decision maker is confronted with a large and complex amount of information and given
that it is likely that none of the alternative courses of action optimise all performance
measures, we believe that the use of an appropriate MCDA approach can be very
valuable to assist the decision process.
Suppose, for instance, that the results of measurement indicate that the hospital trust is
performing poorly in which concems the size of waiting lists for elective surgery. How
can the hospital reduce waiting lists? This issue clearly involve multiple, conflicting
objectives and it is very likely that there will be alternative courses of action or
strategies to be considered. For example, to reduce the size of inpatient waiting lists, a
possible action to implement is to reduce the length of treatment in hospital. However,
this procedure would increase the likelihood of inappropriate discharges and,
consequently, would increase the number of emergency re-admissions. That is, to
improve one performance measure (for example, the size of waiting lists) we have to
sacrifice the performance of another measure (for example, the effectiveness of
treatment). Figure 7 shows two policy alternatives and some of their impacts that are to
be evaluated.
30.
25.
20.
waiting_list_AL
— 7 waiting_lst_A2
15
0 10 20 30 40 50
Time
504
—-Emerg_readm_A1
—z-Emerg_readm_A2
0 10 20 30 40 50
Time
Figure 7 - Graph showing the ‘size of inpatient waiting list’ and ’28 day emergency
readmission rates’ under two policy alternatives
As we can see, although both alternatives might produce an improvement in the size of
the waiting list, they might increase the number of emergency re-admissions. In
addition, Figure 7 provides evidence that the consequences of both alternatives are
dynamic. How then, to establish criteria on which to base the evaluation of alternatives
with outcomes that vary dynamically over time? Gardiner and Ford (1980), and
Andersen and Rohrbaugh (1992), for example, suggest some ‘curve characteristics’ that
can be taken into account to capture the most important aspects of the system’ patterns
of behaviour through time. But, do decision makers evaluate system performance
through time by consistently using a fixed set of curve characteristics? Or, do decision
makers prefer to evaluate alternatives based on an overall evaluation of the system’s
patterns of behaviour? Work on these and other issues is being accomplished by the
authors.
The integration between SD and MCDA can play, therefore, a fundamental role at this
stage of the performance management process - analysis of results and planning of
performance improvement strategies - for several reasons. First, SD modelling can
provide insights which may be very valuable to understand why hospital trusts are
performing badly. Second, using a SD simulation model several alternative corrective
actions can be simulated and their impact on the performance measures of the system
tested. For example, alternative plans or schemes for the allocation of resources between
the different groups of patients (elective, emergency and day case patients) or between
the many stages of the patient flow process, can be tested using this simulation model.
Third, it is likely that none of these alternative courses of action will be able to optimise
all performance measures given that some of them are conflicting. When this happens,
the information resulting from running the simulation model can be passed to the
multicriteria model and explicitly evaluated. That is, the alternatives to be considered
for analysis and evaluation by the MCDA approach will be the different plans of action
suggested by those with expert knowledge in the area and the criteria for the evaluation
of these plans may initially taken to be the performance measures presented in Table 1.
Finally, it is important to emphasise that the development of the proposed framework
should be a recursive process. Moreover, since the organisation’s environmental
conditions are constantly changing, and new strategies need to be developed to cope
with these changes, the system proposed must be regularly monitored and updated.
4 - Closing remarks
Any review of the literature illustrates that performance measurement is a field
attracting considerable attention, but where some issues require further study if
measurement systems want to succeed in supporting the decision making process and in
improving organisational performance.
Improving the performance of an organisation is not a straightforward task. Instead, it is
frequently a complex and poorly defined problem. A problem which solution often
requires a process of organisational leaming enabling decision makers to change the
way they think and act, and consequently, enabling a more effective use of the available
information.
It is therefore our belief that approaches which allow decision makers to identify and
understand the causes of poor performance, which allow decision makers to understand
the implications of alternative courses of action before they become operational and
which help them in evaluating and eventually selecting appropriate corrective actions,
can provide very valuable insights when supporting the process of measurement and
management.
SD and MCDA are two approaches to modelling which have individually proved their
potential to inform and support decision making. We believe that the integration of
these approaches, bringing together their complementary strengths, can provide a
valuable tool for understanding and informing decisions about organisational
performance.
However, we also believe that if this integration is to be successfully implemented some
technical and conceptual problems dealing with it have to be addressed (see, for
example, Andersen and Rohrbaugh 1992). To investigate theoretically and empirically
the effects of integrating SD and MCDA in the context of performance measurement
and how to integrate these two approaches in the most efficient and effective ways is,
indeed, the main goal of the research that is being carried out by the authors at the
University of Strathclyde.
Some issues which prevent organisations getting the most from their PMS were
discussed in this paper. While integration may not be the solution to all these issues, we
believe it brings new insights to inform and support the different stages of the
performance measurement process. Namely, we believe it is worth consideration for
several reasons: First, it allows the design of a measurement system aligned with the
strategic objectives of the organisation. Second, the factors affecting performance and
their interrelationships can be explicitly identified. Third, it provides a way of creating a
consistent and integrated set of performance measures. Fourth, it offers a powerful
frame in analysing the ways by which changes in system’s performance occur. Fifth,
trade-offs between the different performance measures and dimensions are explicitly
addressed. Sixth, it empowers and involves individuals. Finally, and a consequence of
all the previous reasons, this integration provides a powerful tool for organisational
learning.
REFERENCES
Andersen, D.F. and J. Rohrbaugh. 1992. Some Conceptual and Technical Problems in
Integrating Models of Judgement with Simulation Models, IEEE Transactions on
Systems, Man, and Cybernetics 22(1): 21-34.
Ballantine, J., S. Brignall and S. Modell. 1998. Performance measurement and
management in public health services: a comparison of U.K. and Swedish practice.
Management Accounting Research 9: 71-94.
Banks, R.L. and S.C. Wheelwright. 1979. Operations versus strategy - trading
tomorrow for today. Harvard Business Review May-June: 112-120.
Belton, V. 1985. The Use of a Simple Multiple-Criteria Model to Assist in Selection
from a Shortlist, Journal of the Operational Research Society 36: 265-274.
Belton, V. 1986. A comparison of the analytic hierarchy process and a simple multi-
attribute value function. European Journal of Operational Research 26: 7-21.
Belton, V. 1990. Multiple criteria decision analysis: Practically the only way to choose.
In Hendry, L. and R. Eglese (Eds.). Operational Research Tutorial Papers.
Operational Research Society: Birmingham; 53-101.
Belton, V. and S. Vickers. 1990, Use of a Simple Multi-Attribute Value Function
Incorporating Visual Interactive Sensitivity Analysis for Multiple Criteria Decision
Making. In Bana e Costa, C.A. (Ed.). Readings in Multiple Criteria Decision Aid.
Springer-Verlag: Berlin; 319-334.
Bititci, U.S. and T. Turmer. 2000. Dynamics of performance measurement systems.
International Journal of Operations & Production Management 20(6): 692-704.
Brans, J.P., C. Macharis, P.L. Kunsch, A. Chevalier and M. Schwaninger. 1998.
Combining multicriteria decision aid and system dynamics for the control of socio-
economic processes. An iterative real-time procedure. European Journal of
Operational Research 109: 428-441.
Dennis, R.L., T.R. Stewart, P. Middleton, M. Downton, D. Ely and M. C. Keeling.
1983. Integration of technical and value issues in air quality policy formation: A
Case Study. Socio-Economic, Planning Sciences 17(3): 95-108.
Department of Health. 1997. The New NHS: Modem. Dependable. London: DoH.
(Available on http://www.open.gov.uk/doh/newnhs/newnhs. htm).
Eccles, R.G. 1991. The Performance Measurement Manifesto. Harvard Business
Review Jan/Feb: 131-137.
Fitzgerald, L. and P. Moon. 1996. Performance Measurement in Service Industries:
Making it Work. CIMA.
Fitzgerald, L., R. Johnston, S. Brignall, R. Silvestro and C. Voss. 1991. Performance
Measurement in Service Business. CIMA: London.
Flapper, S.D., L. Fortuin and P.P. Stoop. 1996. Towards consistent performance
management systems. International Journal of Operations & Production
Management 16(7): 27-37.
Forrester, J. W. 1961. Industrial Dynamics, MIT Press, Cambridge, Massachusetts.
French, S. 1988. Decision Theory: An Introduction to the Mathematics of Rationality.
Ellis Horwood: Chichester.
Gardiner, P.C. and A. Ford. 1980. Which Policy Run is Best, and Who Says So?. In
System Dynamics: TIMS Studies in the Management Sciences, Legasto, A. A., J. W.
Forrester and J. M. Lyneis (Eds.). North-Holland: Amsterdam 14; 241-257.
Goodwin and Wright. 1998. Decision Analysis for Management Judgement. John
Wiley: Chapter 2.
Globerson, S. 1985. Issues in developing a performance criteria system for an
organization. International Journal of Production Research 23(4): 639-646.
Gruver, W.A., A. Ford and P.C. Gardiner. 1984. Public Policy Analysis Using Three
Systems Science Techniques. IEEE Transactions on Systems, Man, and Cybernetics
SMC-14(2): 355-361.
Hammond, K.R., J.K. Klitz and R.L. Cook. 1978. How systems analysts can provide
more effective assistance to the policy maker. Journal of Applied Systems Analysis
5(2): 111-136.
Hammond, K.R., J.L. Mumpower and T.H. Smith. 1977. Linking Environmental
Models with Models of Human Judgment: A Symmetrical Decision Aid. IEEE
Transactions on Systems, Man, and Cybernetics SMC-7(5): 358-367.
Hsiao, N. 1998. Conflict Analysis of Public Policy Stakeholders - Combining Judgment
Analysis and System Dynamics Modeling. Research Paper. University at Albany: 1-
44,
Ittner, C.D. and D.F. Larcker, 1998. Innovations in Performance Measurement: Trends
and Research Implications. Journal of Management Accounting Research 10: 205-
238.
Kaplan, R.S. 1983. Measuring Manufacturing Performance: A New Challenge for
Managerial Accounting Research. The Accounting Review LVIII(4): 686-705.
Kaplan, R.S. and D.P. Norton. 1992. The Balanced Scorecard - Measures That Drive
Performance. Harvard Business Review Jan/Feb: 71-79.
Keeney, R.L. and H. Raiffa. 1976. Decisions with Multiple Objectives: Preferences and
Value Tradeoffs. Cambridge University Press: United Kingdom.
Kunsch, P.L., J. Springael and J.P. Brans. 1999. An adaptive multicriteria control
methodology in sustainable development - case study: a CO2 ecotax. Submitted for
publication to JORBEL the Belgian OR Journal.
Lane, D.C., C. Monefeldt and J.V. Rosenhead. 2000. Looking in the wrong place for
health care improvements: A system dynamics study of an accident and emergency
department. J ournal of the Operational Research Society 51(5): 518-531.
Lebas, MJ. 1995. Performance measurement and performance management.
International Journal of Production Economics 41: 23-35.
Legasto, A.A. Jr. and J.A. Maciariello. 1980. System Dynamics: A Critical Review. In
System Dynamics: TIMS Studies in the Management Sciences, Legasto, A. A., J. W.
Forrester and J. M. Lyneis (Eds.). North-Holland: Amsterdam; 14, 23-43.
Linard, K.T. 1995. Dancing Towards Disaster The Danger of Using Intuitive Indicators
of Performance. Proc. 1995 International Conference of the Australasian
Evaluation Society.
Lynch, R.L. and K.F. Cross. 1995. Measure Up! How to Measure Corporate
Performance. 2nd Edition. Blackwell Publishers: Oxford.
Mollaghasemi, M. and J.P. Edwards. 1997. Making Multiple-Objective Decisions -
Technical Briefing. IEEE Computer Society Press: California.
Mumpower, J., V. Veirs and K.R. Hammond. 1979. Scientific Information, Social
Values, and Policy Formation: The A pplication of Simulation Models and Judgment
Analysis to the Denver Regional Air Pollution Problem. IEEE Transactions on
Systems, Man, and Cybernetics SMC-9(9): 464-476.
Neely, A. 1998. Measuring Business Performance - Why, what and how. The
Economist Books: London.
Neely, A. 1999. The performance measurement revolution: why now and what next?.
International Journal of Operations & Production Management 19(2): 205-228.
Neely, A., H. Richards, J. Mills, K. Platts and M. Bourne. 1997. Designing performance
measures: a structured approach. International Journal of Operations & Production
Management 17(11): 1131-1152.
Neely, A., J. Mills, K. Platts, H. Richards, M. Gregory, M. Bourne and M. Kennerly.
2000. Performance measurement system design: developing and testing a process-
based approach. International Journal of Operations & Production Management
20(10): 1119-1145.
Neely, A., M. Gregory and K. Platts. 1995. Performance measurement system design -
A literature review and research agenda. International Journal of Operations &
Production Management 15(4): 80-116.
NHS Executive. 1999. Quality and Performance in the NHS: High Level Performance
Indicators. London: NHS Executive. (Available on
http://www.doh.gov.uk/indicat/nhslpi.htm).
Perny, P. and A. Tsoukias (Eds.). 1996. Theoretical foundations of MCDA. Journal of
Multi-Criteria Decision Analysis 5 (special issue): 79-159.
Reagan-Cirincione, P., S. Schuman, G.P. Richardson and S.A. Dorf. 1991. Decision
Modeling: Tools for Strategic Thinking. Interfaces 21(6): 52-65.
Richardson, G.P. and A.L. Pugh III. 1981. Introduction to System Dynamics Modeling
with DYNAMO. Productivity Press: Cambridge, Massachusetts.
Rosas Flunger, R. 1999. L’approche de la dynamique des systémes et l'aide a la
décision multicitere comme outils d’apprentissage organisationnel. Ph.D.
Dissertation, Universite Paris IX - Dauphine.
Senge, P.M. 1990. The Fifth Discipline: The Art & Practice of the Learning
Organization. New Y ork: Doubleday Currency.
Sloper, P., K.T. Linard and D. Paterson. 1999. Towards a Dynamic Feedback
Framework for Public Sector Performance Management. International System
Dynamics & ANZSYS Conference.
Sterman, J.D. 1989a. Misperceptions of Feedback in Dynamic Decision Making.
Organizational Behavior and Human Decision Processes 43(3): 301-335.
Sterman, J.D. 1989b. Modeling Managerial Behavior: Misperceptions of Feedback in a
Dynamic Decision Making Experiment. Management Science 35(3): 321-339.
Sterman, J.D. 2000. Business Dynamics Systems Thinking and Modeling for a Complex
World, London: McGraw-Hill.
Steuer, R.E., L.R. Gardiner and J. Gray. 1996. A Bibliographic Survey of the Activities
and International Nature of Multiple Criteria Decision Making. Journal of Multi-
Criteria Decision Analysis 5: 195-217.
Stewart, T. J. 1992. A Critical Survey on the Status of Multiple Criteria Decision
Making - Theory and Practice. Omega 20(5/6): 569-586.
Stiefel, L., R. Rubenstein and A.E. Schwartz. 1997. Using adjusted performance
measures for evaluating resource use. Annual Research Conference of the
Association for Public Policy Analysis and Management, Washington. (Available
on: http://urban.nyu.edu/research/stieffel-et-al/)
Suwignjo, P., U.S. Bititci and A.S. Carrie. 2000. Quantitative models for performance
measurement system. International Journal of Production Economics 64, 231-241.
Turney, P.B. and B. Anderson. 1989. Accounting for Continuous Improvement. Sloan
Management Review Winter, 37-47.
Von Winterfeldt, D. and W. Edwards. 1986. Decision Analysis and Behavioral
Research. Cambridge University Press.
Waggoner, D.B., A.D. Neely and M.P. Kennerley. 1999. The forces that shape
organisational performance measurement systems: An interdisciplinary review.
International Journal of Production Economics 60/61, 53-60.
Wolstenholme, E.F. 1990. System Enquiry - A System Dynamics Approach. John Wiley
& Sons: Chichester.
Wolstenholme, E.F. 1999. A patient flow perspective of U.K. Health Services:
Exploring the case for new “intermediate care" initiatives. System Dynamics Review
15(3): 253-271.
| Financial support for the research being carried out by the author has been gratefully received from the
Fundacao para a Ciéncia e a Tecnologia, under grant SFRH/BD/823/2000.
? Pemy and Tsoukias (1996) present the ‘Theoretical foundations of MCDA’. A bibliographic survey of
the activities and international nature of Multicriteria Analysis is provided by Steuer et al. (1996).
3 Cause Maps and Causal Loop Diagrams seem to be effective tools in helping to structure in a more
formal way the ideas or performance measures which have emerged with the use of Post-Its.
“ Notice that, the need for various dimensions of performance and for different measures across these
dimensions is justified by the multidimensional nature of factors that affect performance, and by the
existence of stakeholders with different, and usually conflicting interests and points of view.
5 Notice that, for an additive model to be valid it has to satisfy the independence assumptions. If this is
not the case, a more appropriate model should be used.
® V-IS-A (Visual Interactive Sensitivity Analysis) is a multicriteria decision support system based on a
multi-attribute value function. Belton and Vickers (1990) provide and overview of the use of a simple
multi-attribute value function incorporating V-I'S-A. Keeney and Raiffa (1976) and Winterfeldt and
Edwards (1986) explain in detail the multi-attribute value function.
