Table of Contents
SYSTEM DYNAMICS AS THE LINK BETWEEN CORPORATE VISION
AND KEY PERFORMANCE INDICATORS
Keith Linard Cheryine Fleming Lubomir Dvorsky
Director, University of New South —_ Assistant Director Strategic Planning Consultant - CSC(Aust)
Wales - Centre for Business Coordination, Science Policy Division, UNSW Postgraduate Student
Dynamics & Knowledge Defence Science & Technology Idvorsky@ csc.com
Management Organisation, Department of Defence
k-linard@ adfa.edu.au Cherylne.Fleming@ cbr.defence.gov.au
ABSTRACT:
The BSC has gained corporate support because of its potential to link vision, strategy and
deliverables via a coherent set of KPIs. Unfortunately, the BSC has largely failed to deliver this
ideal because of the lack of a rigorous methodology for selecting the metrics and for establishing
the relationship between the metrics and the corporate strategy. Rather, metrics are typically based
on group consensus or individual intuition, influenced strongly by data availability and the effort of
collection and maintenance.
This paper proposes an approach whereby possible metrics are plotted against the high level
activities of the organisation. Cognitive mapping and hierarchical cluster analysis are used to
derive a rational basis for identifying interrelationships between the metrics and their linkages to
corporate strategy and vision, as a precursor to the development of a balanced scorecard framework.
Keywords: SYSTEM DYNAMICS; COGNITIVE MAPPING; CLUSTER ANALYSIS; BALANCED SCORECARD
ANY FOOL CAN DEVELOP PERFORMANCE INDICATORS
Performance information is a critical tool in the overall management of programs,
organisations or work units. It is also crucial to public sector accountability. Put
simply, it is the main means through which assurance is provided transparently to
the Parliament and public that the Government's objectives are being met.!
Despite strictures regarding the critical importance of performance indicators, a review of hundreds
of evaluation and performance management documents on the Australian Federal Government web
sites and in the Australian Defence Managers’ and Goverment Managers’ Toolboxes shows no
evidence of the development of any rigorous methodology for the choice, specification or testing of
performance indicators. Stripped of their platitudes, the only guidance is consensus and common
sense. As indicated, however, in Linard’s papers the 1995 and 1997 Australasian Evaluation
Conferences’, there is significant evidence that ‘common sense’ indicators can produce
counterintuitive results. Linard (1995) also highlighted the limitations of the ‘logic framework’, as
commonly applied, in the development of performance indicators.
This paper develops a rigorous framework for developing performance indicators and describes its
application in the Defence Science and Technology Organisation. It highlights the significant
improvement in understanding of performance indicators arising from this approach.
1 ANAO. Better Practice Guide Performance Information Principles. 01/11/1996
4 Linard, KT. Dancing Towards Disaster . .. The Danger of Using Intuitive Indicators of Performance. Intemational
Conference of the Australasian Evaluation Society 1995
Linard, KT & Paterson D. Building a Learning Organisation - Evaluating Performance Criteria through Simulation Games.
Intemational Conference of the Australasian Evaluation Society 1995
LOGIC ANALYSIS - FOUNDATION OF PERFORMANCE MEASUREMENT
Program logic analysis may be defined as the systematic study of the presumed relationships
between political & regulatory environment, program resource inputs, ongoing program processes
/ activities, short term outputs, longer term results, and program objectives.’
The 1984, Financial Management Improvement Program (FMIP) Handbook, Evaluating
Government Programs‘, proposed a structured logic analysis framework for planning and
implementing program evaluations. The program logic framework was further developed by the
various State Governments and private consultants over the subsequent decade.
A program logic framework remains significant in various federal departmental evaluation
manuals” and in the reports and guidelines « of the Australian National Audit Office’ and the Federal
Department of Finance and Administration’.
Figure 1 depicts the most recent Department of Finance and Administration presentation of such a
program logic framework.
Basic Outcome and Output Structure
Figure 1: Causal Logic Model for Government Program (DOFA 2000)
The logic model “... provides for the use of output groups that can strengthen the strategic and
causal connections between each level.”® “The process of developing and analysing the underlying
logic of programs ... is a powerful mechanism for identifying the key areas and issues within a
3 Linard (1995).
a Department of Finance. Evaluating Government Programs - A Handbook. AGPS, Canberra, 1987. pp.13-15.
4 E.g., Royal Australian Navy, Directorate of Corporate Management, ABR 2010, The Navy Quality Manager. Defence
Centre, Canberra. Second Edition, 1996, Chapter 11. That document in tum draws heavily on Linard, KT, Evaluating
Govemment Programs - A Handbook. AGPS, Canberra, 1987.
& ANAO. Better Practice Guide Performance Information Principles. 01/11/1996
% Department of Finance & Administration. The Outcomes & Outputs Framework Guidance Document. Nov 2000
8 ANAO (1996). Op. cit.
program, particularly in relation to outcomes, and hence enables the development of useful
performance information”.
Why Logic Analysis ?
Conceptually, the DOFA 2000 model is the same as the 1984 FMIP model, with minor terminology
changes. The purpose in undertaking a logic analysis is to describe clearly the intended or assumed
processes by which a program is expected to accomplish its objectives. This presumes that the
program can be considered, and represented, as a causal system, as illustrated in its simplest form in
Figure 2.
Political Cultural &
Legal Environment
“_.. systematic study of the presumed
re presumed to] . . “
Resource Inputs bring about | relationship between...
Ongoing re presum:
Program 0 bring about
Activities
Short Are presum’
Term to bring about
Outputs
Achievement
of Objectives
Figure 2: Essence of Program Logic - Causal Chain Leading from Inputs to Outcomes
Figure 1 and Figure 2 presume that the allocation of resources (including staff, managerial skills,
staff skills, purchased services, equipment, accommodation, finance etc) will be adequate to
implement the planned products and services. These products and services in turn are presumed to
be adequate to achieve the planned sub-outputs, which in tum, result in achievement of the planned
outputs. Finally, the aggregate achievement of these outputs, together with the administered items,
are presumed to bring about the planned program outcomes. Such a model points to a host of
assumptions, and raises many questions for the planner, in particular the relationship between the
different program elements, and the effect on overall objectives achievement of inadequate
performance in particular elements.
Most government program activities are, of course, very complex: there are often many interactions
with other programs and with the external social and political environment; the ‘causal chain’ from
program activities to expected results is often unclear; and the mapping from outputs to outcomes
or objectives is often problematic. Logic analysis inevitably involves simplification of the real
world. Nevertheless, it imparts a salutary rigour to program design, implementation and evaluation.
Analysis of the program logic aims to identify the risk of breaks or dispersion in the logic chain
(Figure 3) by addressing:
* what are the key assumptions underlying the program and how will program success be
affected if they are not valid;
* what tests or measures might be appropriate to check whether these assumptions are sound;
* what are the most sensitive or significant variables and what monitoring and control
measures can we implement to rectify problems as they occur;
¢ what changes to program design, program management or program operations would
improve the likelihood of achieving the desired objectives;
¢ how do various program element activities relate to each other, and how does performance
in one affect the other activities;
¢ what aspects of the program are likely to be affected significantly by other programs or by
factors outside of the program manager's control; and
¢ what are the significant unintended impacts of the program.
BENEFICIAL
OR
DETRIMENTAL
CAUSAL DESIRED SUCCESSFUL
Croonly sn PROCESS C43 OUTCOME PROGRAM SG
which
c=» THEORY
PROGRAM did not
IES 2 ED xx ©
did not es
ES > IMPLEMENTATION
CE QED “me ©
Figure 3: Logic Analysis Helps Identify Breaks & Dispersion in the Causal Assumptions
Steps in Doing the Program Logic Analysis
Assuming that the objectives or outcomes of the program have been defined at least in general
terms, and that the broad strategy for achieving them has been decided in principle, the analysis of
program logic involves the following steps:
(1) identify, and hierarchically order, the full range of outcomes expected or objectives to be
achieved from this program;
a. sub-dividing program operation into a small number of major outputs, activities or
discrete phases (between 5 and 10 elements is desirable, at least initially);
(2) identify the inputs, processes, extemal influences, intended outputs and any significant side-
effects (whether desirable or undesirable) of each of these phases;
(3) specify for each phase and the program as a whole the perceived logical relationships (that
is, the hypothesised cause-effect relationship), between the inputs, operating processes,
external influences, the outputs or outcomes and the objectives;
a. confirming with the program managers / subject-area experts that the logic model is
a fair representation of what is expected /supposed to happen;
(4) qualitatively or quantitatively model the assumed logic (e.g., using system dynamics
modelling), to identify:
a. which are the key assumptions or most sensitive variables underlying the logic of the
model and what are the implications if they are not valid;
b. identify the evaluation questions which are relevant to the testing of these
assumptions; and
c. identify the performance indicators or measures which are warranted in each phase
to monitor the program; and
d. identify appropriate policy levers and the associated business rules for responding to
off-trend performance.
Step 1, hierarchical ordering of outputs and outcomes, has been extensively addressed at successive
AES Conferences over the past decade, and will not be canvassed here.
Step 4, program modelling is addressed in a companion paper at this conference.
Our focus is on steps 2 and 3, identifying the components and the assumed cause-and-effect logic of
the program. The UNSW Centre for Business Dynamics, through its consulting and its PhD
program, have evolved a structured methodology for addressing this area.? Two invaluable tools
applied in this are cognitive mapping and statistical cluster analysis.
TOOLS FOR INITIAL CAUSE-AND-EFFECT MODELLING
Cognitive Mapping
Cognitive mapping derives from the Psychology of Personal Constructs developed by cognitive
psychologist, George Kelly’®. An individual’s or a group’s perception and understanding of a
problem is captured in a ‘cognitive map’ which consists of interconnected sets of elements
(words/ideas) representing relationships that are perceived to exist among the attributes and/or
concepts of a given environment.
Cognitive maps are used primarily to assist decision-makers gain a clearer understanding of the way
different stakeholders view a problem. They provide a powerful and effective way to capture and
analyse stakeholder concepts about strategic issues. Experience in many environments, including
the Australian Defence Organisation (ADO), demonstrate that it can result in high levels of shared
understanding among stakeholders, analysts and decision makers.
A ‘cognitive map’, ‘concept map’ or ‘causal map’ consists of nodes, known as concepts linked by
arrows. The arrows may carry a sign, though the absence of a sign is usually taken to indicate a
positive link and a negative sign indicates a negative link. Causality feedback and dominant
mechanisms may be derived from these maps. McLucas, in his PhD program at the Centre for
Business Dynamics, has developed a robust methodology, within an Australia context, for the
application of this technique.!!
Cluster Analysis
Cluster analysis is suggested as an initial step in structuring complex multifaceted problems. It is
appropriate to any scale or any type of problem and is particularly suited to unstructured problems
A detailed methodological framework was developed in 1997 for the Department of Defence for the ‘Defence Preparedness
Resourcing Model - Scoping Study’. See also: Linard KT, Quality Assurance in System Dynamics Modelling. School of
Civil Engineering, ADFA, 1998.
x Kelly, G. A. (1955). “The Psychology of Personal Constructs: A Theory of Personality.” Norton, New Y ork
McLucas, A. Qualitative and quantitative techniques for addressing systemic complexity in the context of organisational
strategic decision-making. PhD Thesis. UNSW. Aug 2001
-5- 5
and non-quantitative data. In essence, cluster analysis provides a rigorous basis for linking or
grouping ‘objects’ (e.g., policies, objectives, KPI's) into groups or clusters, such that the objects in
a given cluster tend to be similar to each other in some sense, while objects in different clusters tend
to be dissimilar. Activities (or strategies or KPI's), which are similar to each other in so far as they
serve similar objectives, are grouped into the same cluster, whilst different clusters separate
activities that are dissimilar in so far as they are oriented to different objectives.
Cluster analysis has a theoretically sound basis and has been widely applied across many disciplines
for decades. All major statistical software packages include this tool.
Cluster analysis helps achieve a better understanding of the problem being tackled, its basic
structural components and their interrelationships, because it forces a simple yet systematic review
of complex issues, and leaves the software to do the analysis. Careful attention at the problem
definition stage can avoid ambiguities of meaning and intent; help eliminate unsuitable "package"
solutions; and help ensure that the problem solution is based upon an understanding of the causal
factors and not simply data related to the problem symptoms .
The first step in doing a cluster analyais is to identify those attributes or key characteristics of the
program most relevant to the problem at hand, for example, the initial expression of KPI's and the
organisation’ s strategies. The attributes are arrayed against each other in a matrix (see table 1), and
compared pair wise (e.g., Activity_A.1 against Strategy_1.1).
The next step is to classify all attributes with respect to each other. Proceed by asking the question
of each attribute pair: to what degree does an attribute impact on or support the other? Ideally, this
step is done with key stakeholders so that they own the end result.
There are diverse ways that the multidimensional effects may be coded. The simplest approach is:
(a) If an attribute enhances or supports the other: this is termed a positive interaction, and is
coded with a value of ‘+1’.
(b) If an attribute detracts or militates against the other: this is termed a negative interaction and
is coded with a value of ‘-1’.
(c) If there is no interaction, either (+) or (-), or if the relationship is uncertain or could be
either (+) or (-) depending on circumstances, it is coded with a zero value.
If there is good understanding of the interrelationships, a finer discrimination in the cluster analysis
would be achieved by coding on a scale, for example, of (+10) to (-10).
Strategy Number
11 u2 | 22 | 21 | 23 etc
Activity (Rank from -10 to +10)
Activity A.1
Activity A.2
Activity B.1
etc
Table 1: Example of a Strategy / Activity matrix
Clearly there is subjectivity in such assessments, and the judgements in this step may be a source of
disagreement. But since the process forces the judgements to be made explicit there is at least a
focus for discussion of differences of opinion. In the case of strong disagreement, separate analyses
may be undertaken to see if the implications are significant in the end result.
Output of the Cluster Analysis
A variety of measures of similarity (indexes of cluster "goodness") are available with most cluster
analysis programs. Probably the most useful output, however, is the dendogram, or "tree plot",
Figure 4, which graphically represents the results of hierarchical clustering.
Troe Cragram the 22 Cazen
Complete Linkage
Euchdoan distances
Figure 4: Dendogram Showing Relationship between KPIs & Strategy
This cluster tree plot allows one to gain rapid insights into the clustering. One may look for sub-
sets which are clearly defined by the clustering. These are indicated by clusters which join together
at a relatively low level (left hand side of Figure 4). In the example, indicators and strategies which
join close to the origin are a more homogeneous (or tighter) cluster than are activities which form
further from the origin.
CASE STUDY IN DEFENCE SCIENCE AND TECHNOLOGY ORGANISATION (DSTO)
The DSTO is part of the Australian Department of Defence. DSTO’s objective is to give advice
that is professional, impartial and informed on the application of science and technology that is
best suited to Australia’s defence and security needs.
To achieve the necessary high level of confidence that the scientific advice given to Defence is
right, DSTO conducts scientific investigations and undertakes research projects. Its understanding
of developments in defence technology assists Defence in making informed decisions on the
purchase of new equipment. DSTO also supports existing capabilities by enhancing operational
performance and by reducing the costs of ownership of major defence equipment, including through
life-extension.
DSTO Balanced Scorecard Project
The Balanced Scorecard (BSC) for DSTO was introduced to senior management against a
background of changes as a model to assess the organisation’s performance. The changes
articulated by the DSTO senior management team involved the environment that DSTO needs to
operate in externally, and within defence. In the external environment there is a worldwide trend to
compete R&D in both private and public organisations, with the recent Government's Innovation
Action Statement clearly articulating the necessity for all R&D funds to be competed. This places
-7- 7
pressure on DSTO from other government agencies to demonstrate our contribution to National
wealth. With the current Defence environment there is an ongoing management reform agenda
focused on making Defence more responsive to goverment, improving accountability and an
emphasis on stronger leadership throughout the portfolio.
This changing environment leads the senior managers of DSTO to articulate where they currently
see DSTO and where they would like to see DSTO in the future. Their key messages for the future
success of DSTO include:
* Scientific excellence
* Close to our clients: the ADF is our Partner
* Source of defence science knowledge
¢ Independence of Advice
* Creative / innovative solutions
* Responsive / accountable
To demonstrate that DSTO is continuing to achieve against these areas, strategies were developed,
with objectives and milestones. If DSTO’s organisational performance is to continue to be
successful then these strategies must relate to the BSC. This paper discusses the method used to
ensure DSTO’s change strategies and BSC are aligned.
For DSTO the BSC has a hierarchy of concepts. The highest level is the Critical Success Factor
(CSF); these are core activities that DSTO must get right if it is to be successful as an organisation.
Each CSF has between one and three Key Performance Indicators (KPI); these are activities or
objectives for the organisation and often form the basis of an implementation strategy. The
measures or metrics of the BSC relate to the KPI and more specifically the intent behind that K PI.
BSC - a Driver of Strategy
From Kaplan and Norton, and from the prolific marketing of BSC solutions by management
consultants, we know that a “good” BSC is not simply a limited list of measures gathered into four
categories. Rather, a “good” BSC “... should tell the story of your strategy” '*, communicating and
promoting adherence to the strategy to all levels of the organisation.
A good BSC “... tells everyone in your organization, in a single page, the story of your entire
strategy: Every measure is part of a chain of cause-and-effect linkages. All measures eventually
link to organizational outcomes. A balance exists between outcome measures (financial and
customer) and performance drivers (value proposition, internal processes, learning & growth)” .!°
A good BSC will reflect the vertical cause-and-effect relationships for any given objective and
summary measure, in the same way that the causal relationships are reflected horizontally across
the business value chain by the four perspectives. In other words, having delineated the causal
relationship between the Leaming/Innovative Perspective, the Intemal Business Process
Perspective, the Customer Perspective and the Financial Perspective we take each summary
objective/measure and disaggregate them to determine causation."
B Renaissance Worldwide Strategy Group. “The Balanced Scorecard -- An Overview”
http://www.rens.com/viewpoint/papers/scorecard. html
a Kaplan RS, "The Balanced Scorecard", July 13, 1999. http://www.mastersforum.com/kaplan/kaplan.txt
Be Jamesford Consulting, “The Balanced Scorecard - A Strategic Management System”, http://www jamesford.co.uk/the.htm
=B 8
“Every measure selected for a Balanced Scorecard should be part of a chain of cause-and-effect
» 15
relationships that represent the strategy” .
These perspectives have been adapted in the ADO and are known as People Matter (Learning /
Innovation) Enabling Business Processes (Internal) Government as Customer (Customer) and
Government as Owner (Financial). To align with the ADO, DSTO has adopted the same structure.
Understanding C ause-and-Effect Relationships
Virtually every reference to cause-and-effect in the BSC literature and the implementations in major
corporate BSC software systems are flawed:
* cause-and-effect necessarily involves a time lag between cause and effect, however the BSC
measures and presents both indicators at the same time with no analytical basis to present
the implications of the time lag;
¢ the presentation of a cause-and-effect chain is a uni-directional causality which totally
ignores feedback, and especially delayed feedback effects, whereas there are invariably
inter-relationships both within and between the sectors of the BSC.
This paper discusses an approach to developing a BSC which includes these basic relationships to
provide greater understanding on the impact strategies have on organisations.
Interrelationship amongst performance drivers - unidirectional causality
In most of their writings, Kaplan and Norton ambiguously describe the relationship between KPI on
the BSC. Every example presented and every diagram describing cause-and-effect presents or
implies a unidirectional impact, as illustrated in Figure 5.
The Revenue Growth Strategy The Productivity Strategy
“Reduce the volatility of earnings by broadening the “ Improve operating efficiency by shifting customers
sources of revenue from current customers" to more cost-effective channels of distribution”
FINANCIAL
PERSPECTIVE
Improve Retums
Broaden Revenue
Mix
increase
Customer
Confidence in
our Financial
‘improve Operating
Efficienc
‘CUSTOMER
Tncrease PERSPECTIVE
Customer
Satisfaction
through Superior
we xecution
IWTERNAL
PERSPECTIVE
inderstand Develop. Cross-Sell Shift to Provide
‘Customer New the Product Appropriate Rapid
Products, Line Channel Response
Segments,
A
LEARNING
PERSPECTIVE
Tacrease Employes
Satisfaction
‘Recess to
Strategic
formation
Figure 5: Unidirectional Cause & Effect Chain Supporting Strategy
(Kaplan and Norton, 1996).
Ibid.
Figure 5 assumes that ‘Employee Satisfaction’ will not be affected by changes in work processes
(internal perspective), changes that might involve staff reductions, reskilling, higher intensity work,
etc. It assumes that there is no feedback from success in the customer sector to ‘Employee
Satisfaction’. Arguably, the feedbacks from the ‘Internal’ and ‘Customer’ perspectives would have
greater impact on employee satisfaction than the ‘causes’ suggested in the Learning perspective.
Also, Figure 5 ignores the obvious feedback link from financial returns, which provides the
capacity to invest in leaning and growth or reengineering of internal processes.
In developing the draft Defence strategy map (Figure 6), the same unidirectional approach has been
adopted. The detail of the strategic objectives (the concept bubbles) is not important. However the
nature of the relationships and especially their uni-directional causality are.
The relationship between the strategic objectives and between the themes (strategies) reinforces in
the unidirectional model introduces difficulties, with concepts only appearing once in the strategy
map - and without reference to any of the other strategies. The concepts in the strategy map are
often relevant for more than one theme For example, the concept of risk is only identified once,
however the Defence White Paper (Oct 2000) discusses many types of risk, for example: technical
risk, business risk and operational risk. This complexity could be included in the later drafts of the
Defence BSC and strategy map by applying the method outlined in this paper. Indeed the current
draft Defence strategy map would be one of the key inputs to that process.
MISSION: The defence of Australia and its national interests
VISION: A force for good, a force to be reckoned wth, a force to win
[Vl breakable rules: Never Mislead the Minister Never Abuse Authority/Power Never Leak Information Never Condone Poor Performa
DRAFT as 12 jue 02
Figure 6: Draft Defence Strategy Map
-10- 10
BUILDING THE DSTO STRATEGY MAP
Cluster Analysis to Clarify the Complexity
The starting point for building the DSTO Strategy Maps was the use of Cluster Analysis. Figure 7
illustrates part of the matrix charting Change Strategies (CS) and sub-objectives as columns against
the CSFs and KPI of the DSTO BSC as rows. A score of relevance of the KPI was placed at the
intersecting cell of the KPI. This basic input for a cluster analysis can be used at various stages in
developing a BSC. For example, in relating KPI to strategy, the management team (individually or
collectively) assigns to each KPI a score (e.g., on a scale of -3 to +3) based on judgement of the
degree to which a given KPI supports (1 to 3), is neutral (0) or detracts from (-1 to -3) a given
strategy.
In a strategy-based organisation, the idea is that your strategies will transect the BSC perspectives
(Kaplan and Norton 2001). When system dynamics theory was introduced into the DSTO BSC,
this recognised that feedback loops will also transect the perspectives linking CSFs and KPI of each
section of the business. As such, a successful strategy will have CSFs from each perspective from
the BSC identified on the strategy map or cognitive map. Such a map will demonstrate the
relationship between the various concepts on the BSC.
The simplicity of this matrix-based approach allows the senior managers to provide their
perspective. The following analysis brings together the various perspectives with the cognitive
mapping to provide a framework for discussion and debate.
ry 5
DSTO CHANGE STRATEGIES
BALANCED
SCORECARD
KPI's
|
|
Epes
|
|
T
[—o
11
Figure 7: Matrix of DSTO change strategies by DSTO BSC KPI
The results of such an exercise, e.g. on an Excel Spreadsheet, can then he fed directly to a cluster
analysis program (standard with most statistical software packages). A variety of output formats is
possible, including the dendogram tree in Figure 4. When analysed results drawn from these charts
are used to determine how relevant specific KPI were to each CS and sub-objective. Questions can
then be asked where no relationships between the CSF and CS were found for example is this
measure adding value?
The BSC literature emphasises the importance of focusing on a small number of KPI, but again
gives no suggestions on how this selection might be done. Statistical cluster analysis provides a
-11- 11
simple and robust technique. In essence, cluster analysis groups like factors with like, and separates
unlike from unlike, allowing for informed decisions on which KPI add value.
The dendogram (refer Figure 4) identifies ‘clusters’ of (in this case) KPI, providing a rational basis
for the development of the performance indicator hierarchy in the BSC and for the analysis of inter-
telationships. The results showed relevance between the KPI and their associated DSTO CS. In
addition, the visual representation showed whether the KPI was working to achieve the CS or
working against it.
Cognitive Mapping to Comprehend Presumed Causality & Feedback
The process for developing DSTO’s cognitive map was built on the cluster analysis previously
described. After it was determined that all CS and sub-objectives were being measured, cognitive
maps were designed for each CS. A cognitive map was completed for each CS (Figure 8) with a
table describing the relationships (Figure 9).
> Validation of causal linkages.
“¢— _ |2.2Provision of independent
lexcellent S&T service Positive
= 11.1 Delivery of Defence Outcomes Positive
4.2 Ensure Defence |4.1 Maintain an excellent science
y e [Confidence in DSTO. nowietine base: Positive
1 3a Alingment with Defence Partners
. * — _|strategic Plans Positive
|2.3 Positioning for future S&T
4 ~ __|capatitity Positive
“ 2.3 Positioning for future S&T
x 12.2 Provision of a
-) ~~) independent excellent capably. Positive
S&T service —* [ensure Defence Confidence in DSTO
Negative
Figure 8: Change Strategy Figure 9: Description of relationships for each of
Cognitive Map the change strategies
An overall cognitive map (Figure 10) was then drawn using the intersections collected, and
described in the associated tables from each of the CS maps. The overall map shows the strength of
the connections between concepts
Figure 10 illustrates the end product of a cognitive mapping process focusing on elements of the
draft DSTO BSC. The draft BSC exhibited the traditional uni-directional cause-and-effect chain.
The cognitive mapping process, combined with a cluster analysis, not only revealed profound inter-
telationships, but highlighted several negative relationships (dotted lines) which had not previously
been appreciated. A negative relationship suggests that a particular strategy designed to promote
achievement of one strategy has a side affect of countering another strategy ... a possibility ignored
in the BSC literature.
The group dynamics involved in developing the DSTO BSC has evolved significantly since the
cognitive mapping concept, and the proposal to take a more analytical approach to developing the
DSTO BSC, was first initiated. The start point was an audience who appreciated the benefits of a
systems approach to various topics, but had no exposure to using system dynamic approaches in
planning, evaluation or implementing management initiatives.
The first approach was presented to a deeply sceptical audience. Once they had participated in the
cognitive mapping and viewed the impact of the cluster analysis a shared understanding was
-12- 12
developed, most importantly in the areas where the group disagreed. This process provided a
framework to facilitate discussion, a common reference point and a way of improving the DSTO
BSC to reflect the change strategies the senior leaders felt intuitively the organisation needed to
progress.
fa
| De of |
CS ————_
yr \ Goteomes /
vaith Defence | /
Reriners Stetowie! / |
Peat |
4
| Pos
\ Fumeset
\ capability /
me
i a
{ =
\ d Freearatces | \
eee aces )
—* == RY
Fiqure 10: Overall Top Level Cognitive Map
eae 13
