A pilot System Dynamics model to Capture and Monitor Quality Issues in
Higher Education Institutions: Experiences Gained
Michael Kennedy
Information Management and Modelling Group
School of Computing, Information Systems and Mathematics
South Bank University
Borough Road, LONDON SE1 OAA
Tel: (+44) 171 815 7416 Fax: (+44) 815 7499 e-mail: kennedms@sbu.ac.uk
Key words: Higher Education, Quality Issues, Quality Model Dynamic Simulation
Abstract
This paper describes and reports on a system dynamics (SD) model, developed as a pilot study in order to assess
the feasibility of modelling the complex, interdependent set of variables concerned with the various aspects of
managing quality in higher education.
The quality of higher education delivered is a major concern for students, institutions and government
departments, particularly as the "unit of resource" continues to decline. Quality issues impinge on all aspects of
an institution's planning, students' and staff performance, administration and finance.
The initial set of influences were based on relationships identified by researchers in the field of quality standards
in higher education. This initial structure was validated and calibrated for this School by incorporating academic
and research staff perceptions. These were captured through a survey, subjected to statistical analysis, and were
incorporated in the model. Currently, the model contains over 100 variables. A number of runs were made at an
average of 25 seconds per run on a Pentium 266 MMX with 126 MB RAM ina Windows NT environment. The
model is simulated over an eight-year period.
From the results of this pilot study, it would appear that Higher Education departments may obtain useful
insights into the likely impact of educational policies on the attainment of quality related objectives, through the
use of such a model. It is emphasised that these results are reported as the first stage of a long-term project. The
author would welcome comments from these with an interest in the field, particularly those interested in some
form of continuing dialogue or collaboration.
Introduction
This paper reports on a pilot study in developing a system dynamics (SD) model, designed to assist in policy
analysis in respect of quality issues by the School of Computing, Information Systems and Mathematics
(SCISM) at South Bank University. The project acted as a feasibility study in identifying those factors that
SCISM should consider when developing a quality management system within the Department. Under the
supervision of the Author, the questionnaire design, administration, interpretation and the modelling simulation
and analysis, was carried out by A. Mania with assistance from D. Williams.
As reported in a companion paper (Kennedy, 1998), Quality has been an issue of concern for many
organisations in their efforts to achieve their corporate objectives. For higher education institutions, the aim is
attaining high quality standards of education and scholarly activities. The main objective of this empirical study
was to gauge the impact of educational and managerial policies on quality through the use of a SD simulation
model. This initial model focuses on assessing various aspects of quality in the University: administration, staff
development, organisation effectiveness, funding, student performance, staff morale and motivation and
research.
The model was also used to assess the usefulness of SD in exploring quality issues and funding linked problems
in higher education departments and to explain how best to deal with those problems. The initial findings
suggest that a SD model may provide insights and support exploration of the process of achieving high quality
standards. Such a model may accommodate a non-linear and iterative view, define new boundaries to the
development process so as to include hard and soft issues, consider the University’s strategic objectives, and
acknowledge changes in the education environment.
Reviews of courseware and plans, assessment methods, course structure, quality of units and staff commitment
to teaching influence the quality of teaching, which in turn influence student and staff performance.
With the use of influence diagrams, these factors were further broken down into the key variables, which
influence quality management in higher education departments. These key variables were further refined by
adding performance tables. This process enabled the determination of negative and positive influences. When
the model was simulated, fuzzy sets came up with scores of overall quality of the process. Integrating fuzzy
concepts into SD models was another achievement of this project, although the main objective was to develop a
quality management model.
Quality is a fuzzy concept. It could be graded as High, Medium or Low. The system then uses this information
in assessing the overall quality performance. A typical rule conceptually takes the form of IF-THEN-ELSE.
Such a model is efficient in determining the level of quality over time. At run time, the block of rules interacts to
determine the mean average score of a particular attribute over time. Statistical control methods can be used to
control upper and lower limits in a quality performance problem. The third step was used to collect and analyse
data to test the model. Simulation runs were performed to test the limits of the model to determine whether it can
provide an insight and understanding of the quality management problems.
The analysis of the Quality Management process permits experiments with different scenarios while reflecting
the behaviour of expected performance. This will help both management and staff to investigate the impact of
specific policies before actually implementing them in the real system. For example, the management would be
able to gain insights on the effect of increasing class sizes on research, student performance and staff
performance. This study shows a potential role of SD in explaining some of the current phenomena in ever-
reducing resources and demand for high quality standards in higher education in the UK.
The model was calibrated using data captured from a survey of staff members within SCISM. Initial results of
the analysis indicate that contemporary university departments can explore quality control procedures and gain
an understanding into the likely impact of such policies. Many simulation runs were made at an average of 25
seconds on a Pentium 266 MMX with 126 MB RAM ina Windows NT environment. The ability of SD to deal
with compression of time and space provides the opportunity to help managers understand their problems and
find solutions for them in the shortest possible time. The model contains over 100 variables and further work to
expand the scope and model calibration is under way.
The strength of the SD is its ability to handle as well as the ‘harder’ quantitative factors some of the ‘soft’
quality issues. Although the development of this first model was relatively straightforward, we had some
difficulty with incorporating academic and research staff perceptions about quality. A survey was conducted of
all staff teaching and researching in the school and their perceptions and management's desired performance
over time was captured and incorporated in the model. An area where the model needs extending, is in the
student perception of quality issues in the university.
However, the current model at this stage provides some insights and is helpful in explaining some of the
dynamic structure identified. The model is simulated over eight years of the school’s management information
system’s life cycle. Current plans are to employ a full-time research staff to extend and calibrate further the
existing model. This experience shows yet another potential of the initial SD model, both as training and policy
analysis in the context of quality control in higher education in the UK.
Higher Education Quality Management Factors
Quality Management in Higher Education can be seen to consist of the various processes or activities within the
institution. Some processes are interrelated and their occurrence depends on other activities. Based on the work
of Cortada (1995), Ashworth and Harvey (1994) and O'Neil (1994), the author suggests the figure (below) to
represent the main issues addressed in Quality Management in Higher Education.
Administration
Departments
Effectiveness
Staff Performance | |
Departments
(faculties)
Student
Performance
Figure 1: Higher Education Quality Management Factors.
The above diagram gives a schematic view of the main activities affecting quality management of Higher
Education institutions from a high level perspective. These factors are examined in great detail in the
companion paper (Kennedy, 1998).
In order to measure the quality of institutions of higher education, it is important to assess the various factors
that contribute to its overall performance. These include (Ashworth and Harvey, 1994):
* — organisation and resources;
* — students and their support;
* teaching and leaming;
* curriculum ;
* funding;
* research;
* management and quality control policies.
Stages in the Development of the Model
There are three main steps involved in the development of the model:
The first step is to have a good understanding of the main attributes affecting the University as a whole and
formulating their relationships with one another. Influence diagrams were used to explain the main influences
showing whether they are a positive or a negative influence.
The second step is to build a prototype model using the information obtained from the influence diagram.
The third step will include the collection of data required to finish the model. This data was obtained from the
questionnaire of SCISM staff members. Simulation runs will be performed to validate the results throughout the
whole model-building process.
Step 1: Influence Diagram
The six key performance indicators of the level of quality identified are quality of Research, Teaching,
Professional activities, Administration support, Units and Student performance. The relationship of the variables
was captured using system qualitative analysis technique. Influence diagrams can be used to explain main
influences and the direction of such influences.
aie structure ‘actual publication
‘
“AI — socceal poe previo pubintion
oo -
:
soos funding pei tan es
aed tambt grant —> actual grant and fees
information reached student perception grant and fees A+
+
it methods + quality ——_ See re
quality of mnemal fundsing
‘commitment to teaching s
Le of units oy
x
student peFformance
external ‘ung sere oe
* et oan a
cs Niet
son >. cost
numberof graduates vy
iA \ chest \ eet
course, a planned publication
»
4, staff Performance lecture howrg student motivation
involment in plannig + + clgbs size
mesa st Ye og i NS Ir dents
sear Mubhos TS yreaniae policy student gumbers
+ +
staff training professional activities \ SS
* FIT paynil
i ia ‘s my + XN
training period
bug Fangs lyon dal ecrmet sion \.
fr \
A texchig mane
8 “ sini pti
A, at
allocated staff budget. //
“y time
fulltime
Figure 2: Influence diagram of Key Performance Indicators
From the figure 2 above, it may explain that funding (both internally and externally), previous publications and
the research budget influences Quality of Research. This then influences the number of research projects,
increase in both research staff and students, funding for more projects, and an increase in publications.
Student performance is influenced by student motivation, quality of facilities, student perception of employment
opportunities, quality of teaching, staff performance, student contact time, class size and staff motivation.
Student performance then influences the number of graduates completing courses and in tum increases both staff
motivation and performance.
The number of specialist staff, staff motivation, staff involvement in professional activity influences staff
performance, quality of teaching and student performance. This in retum influences student performance.
Staff motivation is influenced by communication overhead (communication lines with management),
remuneration (fringe benefits and salaries) and student performance. This increases staff involvement in
professional activities though this is also dependent on management policy on staff recruitment.
Constant review of courseware and plans, assessment methods, course structure, quality of units and staff
commitment to teaching influence quality of teaching. This in retum influences student performance and staff
performance.
Step 2: A Prototype Model
The second step was to build a prototype model using information obtained from the influence diagram.
The model building process took a top-down approach. There was a need to identify broad quality areas and
their feedback loops.
The model has 7 sectors:
* — Staff performance and productivity
* Budget
+ Funding
+ Student Performance
* Quality of Research
* Quality of Administration Support
« Equipment
Such a model above is efficient in determining the level of quality over time. At run time the block of mules
interact to determine the mean average score of a particular attribute over time. Statistical control methods can
be used to control upper and lower limits in a quality performance problem.
This results in the sectors of the Simulation model as shown in Figure 3 (below)
auoet 7
Budget Manageme <7
Management
Exim
salt per 7 Sudeep 7
pee staff performance Student
and productivity performance
|e
Research SZ
Funding <7
“ Funding
Figure 3: Simulation Model Sectors
Step 3: Data Analysis
The third step was used to collect and analyse data to test the model. Simulation runs were performed to test the
limits of the model to see whether it can provide an insight and understanding of the quality management
problems.
Many simulation runs were made at an average of 5 seconds on a Pentium 90 with 32 MB RAM. The ability of
System Dynamics to deal with complex time and space provides its ability to help managers understand their
problems and find solutions for them.
J 1 student performance 2: quality of teaching 3: staffing costs 4: total cost
lL 80.00%
5500000.00
40.00
15.00 |
0.00
1500000.00
A
_ | 74
“
1997.00 1998.60 2000.20 2001.80 2003.40 2005.00
N l= Pal Graph 1: p2 (Dynamic Behaviour) Years 12:31PM 5/5/98
0.00
1
2: 0.00
3: -2500000.00
4: -2500000.00
Figure 4: Simulation Results
Figure 4 above shows the simulation results of: quality of teaching, students’ performance, staff costs and total
costs, based on the model assumptions, over the simulation period.
Summary
The analysis of the ‘Quality Management Process’ at South Bank University’s School of Computing (SCISM)
gave the team a starting point to the model building process. The team was able to obtain some, but not all, the
data needed to use in the model through the use of a questionnaire to SCISM staff and figures obtained from the
Dearing report (1997). One main drawback was that there were no past data records to help gain an insight on
the progress made earlier to solve this problem using other methods. Through the use of the questionnaire
results, the team was able to identify the main problem areas facing the SCISM in achieving high quality
standards of education. The main key issues that needed to be addressed were staff performance, student
performance, research, teaching and learning, administration support and funding. The team therefore
concentrated on modelling these issues. Due to the fact that this pilot project was carried out during the summer
vacation, the team was not able to capture data on student perceptions on quality issues, which would have an
impact on the model.
The use of System Dynamics approach would enable the construction of a model that allows interactive ‘Process
Flight Simulation’. It permits experimentation with different scenarios, and while reflecting the behaviour of the
real system, there is no risk of disrupting it. This will help both management and staff to investigate changes to
the system before actually implementing them. For example, the School management would be able to gain
insights into the effect of increasing class sizes on research, student performance and staff performance.
References
Ashworth A., Harvey R., (1994), Assessing Further and Higher Education, Jessica Kingsley publishers,
London.
Cortada J W, (1995), TQM: Information Systems Management, McGraw-Hill, inc.
Dearing R., (1997), National Committee of Inquiry into Higher Education. (Dearing Report), HMSO or
http:/|www leeds.ac.uk/educol/ncihe/ see also http://d3e.open.ac.uk/D earing/ and http://www.sap.com/uk/
Kennedy M., (1998), Some Issues in System Dynamics Model Buiilding to Support Quality Monitoring in
Higher Education: Experiences Gained. Proceedings of 16" System Dynamics conference, Quebec City,
Canada.
O'Neil M., Nightingale P., (1994), Achieving Quality Learning in Higher Education, Biddles Ltd.
CC BY-NC-SA 4.0