Table of Contents
Go Back
Towards a Taxonomy of System Dynamics Models of the
Information Systems Investment A ppraisal Process
Michael Kennedy
Information Management and Modelling Group
School of Computing, Information Systems and Mathematics
South Bank University
Borough Road, LONDON, SE1 OAA, UK
Tel: (+44) 020 7815 7416; Fax: (+44) 020 7815 7499;
E-mail: mike.kennedy@ sbu.ac.uk
Key words: System Dynamics, Information Systems Investment A ppraisal Process
Abstract
A number of papers have been published on various issues relevant to System Dynamics (SD)
Models of the Information Systems Investment Appraisal Process from several academic and
professional viewpoints.
The paper contends that there is a need to catalogue and classify this work in order to
highlight potential areas of research in this field of study and to identify system archetypes at
different hierarchical levels and discover new ones.
The initial taxonomy is based on a limited literature survey. The contribution and relevance
of each of the sources examined is briefly described. The taxonomy classifies the completed
investigations into five specific areas of concern and five viewpoints.
1, Introduction
The objective of this paper is to facilitate and structure debate on the use of system dynamics (SD)
(Forrester, 1961) for the Information Systems Investment Appraisal [ISIA] Process. It builds on
previous papers by the author (Kennedy, 1996, 1999, 2001).
To guide management decision making in the ISIA process a number of “traditional” investment
appraisal techniques, based on financial management techniques are normally employed. As
commonly used these “traditional” investment appraisal techniques such as Payback, Accounting
Rate of Retum [ARR], Net Present Value [NPV] and Intemal Rate of Retum [IRR], are not able to
measure many of the benefits offered by IS investments that are intended to gain tactical or strategic
business advantages. This is a particular problem with those projects designed to achieve a
‘transformation’ of the business processes (Kennedy, 1999).
It is contended that these essentially static modelling approaches are inadequate for this application
domain because many IS investments are designed to improve the operation of business activities
that are dynamic, complex, non-linear systems. Such a system can be characterised by interactions
of closed chains (or feedback loops) that, when combined, define the structure of the system and
hence how it behaves over time. I therefore believe SD to be an appropriate modelling technique for
the ISIA Process. A number of authors have addressed some of the problems of the ISIA Process
and the findings from their investigations are outlined in section 3 below.
2, An initial Taxonomy of System Dynamics Models relevant to the Information Systems
Investment A ppraisal Process
The initial Taxonomy, presented in Table 1, is based on a limited survey of completed investigations
relevant to the Information Systems Investment Appraisal Process. The findings from these
investigations are briefly described in section 3 below. In light of this limited survey, I believe there is
a need to catalogue and classify this work. I present an initial attempt at this below in order to
highlight future areas of research in this field of study and to identify system archetypes developed
from different viewpoints. The completed investigations are classified into five specific areas of
concem (Business Value of Proposed Process Changes, Models to Contribute to our
Understanding of the IS Development Lifecycle & Project Management, Process Flight Simulation,
Strategic Decision Making Process and the problems of current ISIA practice & the desirable
attributes of an ISIA system) and five viewpoints (System Dynamicists [SD], Software Engineers
[SE], Investment Analysts [IA] Simulation Modellers [SM] and Domain Specialists [DS]). Some
work spans more than one category.
AREAS OF CONCERN
BusinessValue | IS Dev. LifeCycle& | Process Flight Strategic Decision | Problems of
of Process Project Mgt. Simulation Making ISIA
Change & Desirable
Attributes,
y | Sens Coyle (1996) ‘Abdel-Hamid &
Dynamics Wolsthenholme Madnick (1991)
Professionals | et al (1993)
1 J sotware Lehman (1994) Rubin etal (1994)
Engineers Lehman (1996)
Lehman (1997)
E Lehman (1998)
Jensen & Tonies (1979)
Kahen etal (2000)
WwW Wemick & Lehman
£1999)
Domain Area | Anderson (2000) Dyson (1990) Hares &
Specialist, O'Regan B & Moles | Royle (1994)
P R (2001) Willcocks
(1992)
O'Regan B &
Moles R
0) (2001)
Investment Remenyi etal
| | Analysts (1996)
Serafeimidis
& Smithson
(1995)
N Doherty &
King (2000)
T | Simulation | De Jong and Savolainen (1995)
Modellers Looijen(1999)
S
Table 1; Preliminary classification of SD related work in The ISIA Process Management
3. Published Work relevant to the Information Systems Investment A ppraisal Process
A number of authors have examined some of the problems with the Information Systems Investment
Appraisal Process management domain and suggested ways in which SD may add to the Analyst's
armoury. I shall briefly describe a selection of completed investigations and key findings.
Business Value of Proposed Process C hanges
Firstly, we may develop models of a business showing business processes before and after a
proposed process change. The anticipated value of the benefits derived, (in terms of greater
revenues, resources saved or perceived improvements in quality or reputation), can be compared to
the estimated costs. This would he of considerable value in evaluating “Process Transformation” or
“BPR” type projects.
A classic example of how constructing a SD model may lead to a better decision being taken and, in
this instance, avoid an expensive emor being made, is the “Domestic Manufacturing Company
(DMC)” case study as described and analysed in Coyle (1996). This case (disguised & simplified)
is based on a consultancy assignment. It is described in Kennedy (2001).
Coyle summarises the implications for the Information Systems (IS) investment appraisal domain as:
“Testing the effects of putative information systems in the inexpensive world of the model is a most
fruitful area of System Dynamics application”
This concept can be extended into a structured method. Wolstenholme et al (1993) describes such
a System Dynamics based methodology for MIS evaluation, which he and his co-workers termed
“BISEM” (Bradford Information System Evaluation Methodology). They state that it is intended to
be used as a “complement to existing methodologies for the structured development of MIS”. They
envisage that it “is to operate in parallel with the evolution of the MIS life cycle, but to remain at an
objective, strategic level in contrast to, but supporting, the detailed development of the MIS”. It is
described in Kennedy (2001).
As in the example drawn from Coyle (1996), above, Wolstenholme et al’s work takes a strategic
view of MIS and is aimed at providing a systemic and dynamic evaluation of the effect of a MIS on
its host organisation. This contrasts with the (isolated) project bias of most Information System
Evaluation Methodologies (Hares & Royle, 1994).
A further example of the application in business value to process change can also be noted in health
care. Anderson (2000) highlights the benefits of using SD modelling in a health care environment. An
example of SD modelling is then applied to a health care situation, particularly the effectiveness of
several IT applications to help minimise medication errors that result in adverse drug events (ADE).
A SD model was constructed of the situation which (when used in practise), resulted in a reduction
of ADE’s. Importantly, errors from ADE cost additional monies to correcting mistakes, and the
modelled reduction in ADE’s helped to reduce costs in the worked example.
De Jong and Looijen (1999) seek to develop a simulation model for evaluation of IT service
processes. The authors identify the characteristics of the underlying IT service process and treat
these as inputs to the model. IT tasks are then linked together to form IT service processes. The
resulting tasks structures are implemented in an ARENA simulation modelling tool. Problems,
however, were encountered defining uniform IT tasks, as some tasks could not be placed into a
process. A change of approach was then used focusing on Goals Question Metrics. They conclude
that simulation proved to be a good approach for evaluating IT management but found it difficult to
process the inputs to form the simulation model.
Models to Contribute to our Understanding of the IS Development Lifecycle & Project
Management
Lehman and his co-workers examine the various ways in which SD may contribute to our
understanding of the IS development lifecycle, (especially in respect of evolutionary software), from
a Software Engineering perspective.
In a sustained series of experiments (Lehman, 1994, 1996, 1997, 1998; Kahen et al, 2000;
Wemick and Lehman 1999) termed the “FEAST” study, they have constructed and compared the
results from ‘black box’, ‘white box’ (SD) and other classes of process models of selected
collaborator systems. These systems are selected so as to take advantage of the available process
evolution metrics and other global data about process components and structures that other
members of the Software Engineering community have collected over many years. These papers are
described in Kennedy (2001).
Their major contribution to this area is in providing the evidence (from a Software Engineering
perspective) that the IS development lifecycle may be viewed as a “complex multi-level multi-loop
feedback system, the long term behavioural pattems and trends of software processes are largely
determined by its intemal dynamics which, in tum is feedback generated and controlled”. In later
work they have described the evolution of their own approach towards simulating the effects of the
decisions made by the managers of these processes and generally placing greater emphasis on
‘human factors’, so moving closer to those authors approaching the issues from a traditional SD
approach. The key findings of a selection of their papers are described below.
In a counter-point to Lehman and his co-workers, Abdel-Hamid & Madnick (1991), examine the
various ways in which SD may contribute to our understanding of the IS development lifecycle (and
project management in particular) from a SD perspective. Despite the different starting points there
are considerable overlaps in the two teams’ conclusions.
They summarise some of the problems of Software Engineering as “the record shows that the
software industry continues to be plagued by cost overruns, late deliveries, poor reliability, and
users’ dissatisfaction.”
They conclude that the root cause is that although there have been major technical advances, “A
comparable evolution in management methodologies, however, has not occurred”. Their work is
described more fully in Kennedy (2001).
In a similar vein to Lehman's line of reasoning they suggest that this “micro-oriented” work is a
useful beginning in helping us obtain a better understanding of software development. They add,
“However, before we can say that we have a complete understanding, it is necessary to show that
our knowledge of the individual components can be put together in a total system.”
They quote Jensen and Tonies (1979): “There is much attention on individual phases and functions
of the software development sequence, but little on the whole life cycle as an integral, continuous
process - a process that can and should be optimised”.
Savolainen (1995) focuses on the need for dynamics to be incorporated in IT evaluation, “the
evaluation of IS is a dynamic process as the requirements for evaluation and the forms of evaluation
vary with the changing environment.” The paper seeks to highlight the dynamic evaluation with
specific regard for the IS life cycle. Firstly, evaluation is taken from the IS developers viewpoints
specifically with regard to intemal efficiency. Next the focus of different interest groups is studied
with specific focus of the change of interests in these groups across the evaluation process. Lastly
the change in evaluation criteria over the IS lifecycle is studied. The author concludes, “the
performance evaluation framework must always be customised according to the contingencies in the
information systems environment.”
Process Flight Simulation
Thirdly, we may develop ‘Process Flight Simulators’. The concept is that a dynamic model is built of
an organisation that allows managers to simulate and study situations before encountering them in
reality and so deepen their understanding of the organisation and the likely impact of policies and
decisions.
Rubin et al (1994) describe the use of “Process Flight Simulation”, using SD techniques in this
domain that helps assess the impact of improvements in process maturity. They describe the
construction of a dynamic model of an organisation. This model may consist of processes, events,
pattems of behaviour, structures and information feedback flows. Once managers are confident that
they have developed a satisfactory model of their organisation, they can simulate a wide variety of
business circumstances and scenarios. They state “...We have provided a simulation model that
helps a manager ask “what-if?” questions about different management scenarios. It’s our strong
believe that organisations should simulate their software processes before embarking upon
expensive and potentially disruptive changes to their existing organisational culture.”
Strategic Decision Making Process
Fourthly SD can play a significant part in the strategic decision making process. This clearly has
implications for the Information Systems (IS) investment appraisal domain but this aspect is not dealt
with in detail in this paper. As an example, Dyson (1990), states, “In order to evaluate possible
future states of the organisation, which is the result of adopted strategies and the impact on the
organisation of uncontrolled inputs, some kind of corporate system model of the organisation is
required.”
Dyson (1990) states that currently, in most organisations, this will entail the use of a corporate
financial model to provide financial projections. Such models typically involve accounting
relationships and what Dyson terms “rudimentary attempts” at modelling the behaviour of the
organisation. Due to the inadequacy of these “rudimentary attempts”, he states that there has been a
“growing interest in the development of behavioural simulation models using ideas of system
dynamics”.
O’ Regan and Moles (2001), utilise the benefits of SD modelling in a case study on Intemational
Minerals Investment. Two examples are used by the authors to demonstrate the benefits of this.
They conclude, “the greatest advantage in adopting system dynamics as an analytical tool for
investment analysis is that it exposes the many interrlationships which structure and influence the
behaviour of a complex system.” However they also note SD modelling “does not by itself provide
objective answers. Instead it is a leaming device, an aid to understanding. It is not a replacement for
analytical thinking, but rather complementary to it.”
Problems of current ISIA practice & the desirable attributes of an ISIA system
This section of the paper briefly reviews some issues in the Information Systems (IS) investment
appraisal (IA) domain. These are examined in greater depth in previous papers by the author
(Kennedy, 1996, 1999, and 2001). An exploration and classification of these issues will form the
basis of establishing criteria to determine how SD modelling may assist in addressing some of the
problems of current ISIA.
Most organisations are still using ‘traditional’ financial management investment appraisal techniques
(Hutchinson, 1995; Weston & Copeland, 1988), such as Payback, Accounting Rate of Retum
[ARR], Net Present Value [NPV] and Intemal Rate of Retum [IRR] for evaluating all IT
investments (Hares & Royle, 1994; Ballantine et al, 1995; Remenyi et al, 1991). It is argued that
although these “traditional” investment appraisal techniques are suitable for evaluating IT investments
that automate the organisation’s operations, where the prime motive of the project is cost
displacement, they are not suitable for evaluating IT investment that are intended to gain tactical or
strategic business advantages (Kennedy, 1999).
Hares & Royle (1994) state that there is much to be gained by ensuring that the IA is conducted against
a clear strategy plan of projects that are judged to be worthy of investment
It is argued that because one of the major benefits offered by a MIS is gained by an increase in the
quality of the decision making within the organisation it cannot be evaluated effectively by techniques
that only consider quantitative and financial data. For IT investments that are designed to gain
strategic advantage the benefits are likely to be even more difficult to measure than those of an MIS
because of the increased number of extemal and intemal factors which are involved Kennedy,
1999).
A fundamental reason why IA approaches have had difficulty dealing with more “modem” IS
investments is that they generally emanated from a traditional view of corporate structures and
decision making based on division and analysis, which attempt to break every aspect of the
organisation into small, discreet chunks. Since the 1980's the emergence of cross-boundary and
integrative models of management such as TQM (Total Quality Management), BPR (Business
Process Re- engineering), supply-chain management, enterprise resource planning (ERP) and
electronic commerce have lead to a growing interdependence between differing aspects of the
organisation (Maani & Cavana, 2000). This change requires an integrated ISIA method that is able
to deal with interconnections and feedback.
Research has shown that as a result of the past problems of IT and investments, with various authors
suggesting desirable attributes for an investment appraisal tool/method.
Serafeimidis and Smithson (1995) quote figures on the increasingly high spend of IT projects and
concem of senior management regarding the IT payback. IT is entering the “transformation” stage
and therefore needs more rigorous approaches to IT investment. The authors quote the following
requirements for an investment appraisal framework: a conceptual framework; business driven
evaluation; increase understanding and leaming of the business and of the evaluation process;
provide complete life cycle coverage; provide adequate and relevant information for the evaluation
process; trigger the identification of stakeholders and improve communication between them. The
authors then suggest a framework, which include various modules: CSF’s; Objectives;
Requirements; Decision; Direction; Delivery. Concluding the authors say “evaluation needs to be
more wide-ranging, to incorporate the richness of the intangible and unexpected costs, benefits and
risks of modem IT projects.”
A similar theme is continued by Remenyi et al (1996). These authors advocate modelling as vital
part of ISIA. The issues of the importance of modelling are raised specifically with regard to
management and planning of information systems. The authors focus in three levels of modelling:
macro (high level), meso (intermediate levels) and the micro (detailed level). Macro models are
defined as “the situation which they represent, in general terms.” The meso model adds greater
detail, especially with regards to results. Lastly the micro model takes the meso model and tries to
quantify the situation. Finally, the merits of modelling are put forward, “Unless some detailed
modelling is done, management will not have much of an idea of what lies ahead.” They also
mention, however, care needs to me taken needs to be used carefully “particularly to the
assumptions underlying the models.”
Doherty and King (2000) focus on the important role of human, organisational and economic issues
in systems development projects. Research is highlighted which points to inadequacies in dealing
with organisational issues leading to systems failure. Primary research focused on a questionnaire
completed by 593 senior IS executives. Results showed an increasing number of organisations
addressing a wide range of non-technical impacts throughout the systems development process. But
the authors suggest the evaluation process should be more continuous as opposed to pre and post
implementation evaluation.
Willcocks (1992) suggests the following evaluation guidelines:
¢ Link evaluation across stages & time
¢ Involve key stakeholders in evaluation at all stages
« Assess the actual against the planned impact of IT
* Evaluate & re-evaluate at all stages of the project
* The concept of leaming should be central to the evaluation process. The clamour for adequate
techniques may reveal a ‘quick-fix’ orientation; in the long min getting it right may prove more
difficult but add greater value
4. Conclusion
SD has a direct, but yet to be realised in widespread commercial application, potential for evaluating
IT investments. Several of the approaches examined in this paper utilise a system dynamics
modelling tool to simulate the likely effect on the organisation of a proposed IT investment so that
the likely benefit can be evaluated by management in advance and without disturbing the actual
system. Additionally, an organisation could have an enhanced understanding of cost structures, time
dependencies and human resources that would enable the model to estimate intangible costs and
benefits that traditional cost/benefit analysis (CBA) cannot measure.
A fundamental reason why current IA approaches have had difficulty dealing with more “modem”
IS investments is that they generally emanated from a traditional view of corporate structures and
decision making based on division and analysis, which attempt to break every aspect of the
organisation into small, discreet chunks. The emergence of cross-boundary and integrative models
of management requires an integrated ISIA method that is able to deal with interconnections and
feedback. SD can satisfy this need. (Kennedy, 2000; Maani & Cavana, 2000)
This initial taxonomy brings together work derived from several viewpoints. From a SD viewpoint,
Coyle (1996) summarises the implications for the ISIA domain as “Testing the effects of putative
information systems in the inexpensive world of the model is a most fruitful area of system dynamics
application’. While from a SE viewpoint, Lehman (1996) briefly reviews the difficulties encountered
in achieving further major improvement in the software evolution process. He suggests that this may
be due, in part, to the fact that the “global process” is a “complex, multiloop multilevel feedback
In the author's opinion the case for using SD in the ISIA process is considerably strengthened by
the congmence of arguments stemming from several academic and professional disciplines. In many
cases authors working from different traditions appear unaware of the parallel work being done in
other domains. One of the purposes of this paper is to show the links between such disparate work
and to attempt to establish an inter-disciplinary consensus.
Kennedy (2001) has suggested some of the possible causes of why there has not been a
widespread commercial adoption despite the impressive range of potential advantages for utilising
SD in the ISIA process.
Acknowledgements
The assistance of James Pearson-Kirk and Joseph Adjei in preparing this paper is gratefully
acknowledged.
References
Abdel-Hamid, T. &. Madnick, S. E. (1991) Software Project Dynamics, Prentice Hall,
Englewood Cliffs, New Jersey, USA.
Anderson J. (2000) Evaluation of Information Technology in the Delivery of Health Care Using
Computer Simulation, Proceedings of the Seventh European Conference on I.T. Evaluation,
Dublin, Ireland.
Ballantine J., Galliers R.D. & Powell P.L. (1995) Daring to be Different: Capital appraisal and
Technology Investments, Proceedings of the Third European Conference on I.T. Evaluation,
Bath, UK.
Coyle R.G., (1977) Management System Dynamics, John Wiley & Sons.
Coyle R.G. (1996) System Dynamics Modelling, Chapman & Hall.
DeJong. W & Looijen M. (1999) Simulation as Evaluation Tool in IT Service Management: W. de
Jong, M Looijen, Proceedings of the Sixth European Conference on I.T. Evaluation, Uxbridge,
UK.
Doherty N. & King M. (2000) The Treatment of Organisational Issues in Systems Development
Projects: The Implications for the Evaluation of Information Technology Investments, Proceedings
of the Seventh European Conference on I.T. Evaluation, Dublin, Ireland.
Dyson R. G. (1990) Strategic Planning in Dyson R. G. (Ed.) Strategic Planning: Models and
Analytic Techniques, Wiley.
Forrester J W., (1961) Industrial Dynamics, MIT Press.
Hares J. & Royle D. (1994) Measuring the Value of Information Technology, John Wiley &
Sons.
Hayes R.H., Garvin D.A. (1982) Managing as if Tomorrow Mattered, Harvard Business Review,
May-June.
Hutchinson R. (1995) Corporate Finance. Stanley Thomes
Jensen, R. W. and Tonies, C. C. (1979) Software Engineering. Englewood Cliffs, NJ, Prentice-
Hall.
Kahen, G., Lehman, M. M., Ramil, J. F. and Wemick, P. D. (2000) Dynamic Modelling in the
Investigation of Policies for E-type Software Evolution, ProSim 2000, International Workshop on
Software Process Simulation and Modelling, London, UK, 12 - 14 Jul. 2000.
Kennedy, M. (1996) Investment Appraisal of information Systems Projects: Altemative
Methodologies Explored, Proceedings of Financial Information Systems Conference, Sheffield
Hallam University, UK.
Kennedy, M. (1999). Some Issues in Building System Dynamics Models designed to improve the
Information Systems Investment Appraisal Process, Proceedings of Seventeenth International
System Dynamics Conference, Wellington, New Zealand
Kennedy, M. (2001). The role of System Dynamics Models in improving the Information Systems
Investment Appraisal in respect of Process Improvement Projects, Proceedings of Nineteenth
International System Dynamics Conference, Atlanta, Georgia, USA
Lehman, M. M. (1988) Feedback, Evolution and Software Technology - The Human Dimension,
ICSE 20 Workshop on Human Dimension in Successful Software Development, Kyoto, Japan,
20 Apr. 98
Lehman, M. M. (1994) Feedback in the Software Evolution Process, Keynote Address, CSR
Eleventh Annual Workshop on Software Evolution: Models and Metrics. Dublin, 7 - 9 Sept.
1994, Workshop Proceedings, Information and Software Technology, special Issue on Software
Maintenance, v. 38, n. 11, 1996, Elsevier, 1996, pp. 681 - 686
Lehman, M. M. (1996) Process Improvement - The Way Forward, Invited Keynote Address,
Proceedings of the Brazilian Software Engineering Conference, 14 - 18 Oct. 1996, pp. 23 -
35
Lehman, M. M. (1997) Process Modelling - Where Next? Most Influential Paper of ICSE 9
Award, Proceedings of the of ICSE 19, Boston, MA, 20-22 May 1997,pp. 549 - 552
Maani, K. & Cavana R. (2000) Systems Thinking and Modelling - Understanding Change and
Complexity, Prentice Hall.
O’Regan B. and Moles R. (2001) Modelling to Leam - A case Study in Intemational Minerals
Investment, Corporate Environmental Strategy, Vol. 8, Number 4, 2001.
Remenyi D.S., Money J. and Twite A. (1991) A Guide to Measuring and Managing IT Benefits.
NCC Blackwell.
Remenyi D., Sherwood-Smith M. and White T. (1996) Out comes and Benefit Modelling for
Information Systems Investment, Proceedings of the Third European Conference on LT.
Evaluation, Bath, UK.
Rubin H. A., Johnson M. and Y ourdon E. (1994) With the SEI as My Copilot: Using Software
Process ‘Flight Simulation’ to Predict the Impact of Improvements in Process Maturity, American
Programmer, Vol. 7, No. 9, September 1994, pp. 50-57.
Savolainen V. (1995) Analysis of the dynamic nature of information systems performance
evaluation. Proceedings of the European Conference on Information Systems, Athens, Greece.
Serafeimidis V. and Smithson S. (1995) Requirements for an IT Investment A ppraisal Framework
for the 1990's: Towards a more rigorous solution. Proceedings of the Second European
Conference on I.T. Evaluation, Henley-on-Thames, UK.
Wemick, P. D. and Lehman, M. M. (1999) Software Process White Box Modelling for FEAST/1,
Journal of Systems and Software, Vol. 46, Numbers 2/3, 15 Apr. 1999
Weston J.F., Copeland T.E. (1988) Managerial Finance Cassell.
Willcocks L. (1992) IT Evaluation: Managing the Catch 22. European Management Journal vol.
10 No 2 June.
Wolstenholme E. F., Henderson S. and Gavine A. (1993) The Evaluation of Management
Information Systems -A Dynamic and Holistic Approach, John Wiley & Sons
Back to the Top
