Assessing the System-Wide Impacts of Automated
Voice Response Customer Service Technologies
Mohammad Mojtahedzadeh
David Andersen
Center for Technology in Government
University at Albany
State University of New York
Introduction and Problem Background
Between 1994 and 1995 the Office of Regulatory and Management Assistance (ORMA) of New
York State became interested in reengineering the processes whereby it interacted with the public.
Specifically, the agency was having difficulty responding to phone inquiries for business permits
in a timely way and was proposing that business permit information be integrated into an
automated voice response customer service system. ORMA approached the Center for Technology
in Government (CTG), a research and development unit of New York State Government,
requesting that such a system be developed on a prototype basis and evaluated for feasibility.
CTG, working with Precision Systems, Inc., developed a functioning prototype for ORMA’s
proposed solution and proceeded to evaluate its effectiveness, using a multi-method evaluation
typical of CTG’s approach to prototype evaluation (Andersen, Avery, Hyde, and Kelly, 1995). A
team of managers was convened in a group modeling session to estimate a cost-benefit model for a
fully functioning customer service system (Andersen and Rohrbaugh, 1994). A formal experiment
was conducted in which graduate students were asked to research a business permit problem by
one of three methods -- (1) calling into the present ORMA operator assisted system, (2) calling into
the new prototype system, or (3) doing the research without knowledge of ORMA (Andersen,
Avery, Hyde, Kelly, and Kim, 1995). Finally, a formal system dynamics simulation model was
constructed to assess what would be the overall impact of a fully functional system on the overall
business processes of ORMA if such a system were to be purchased and installed (Mojtahedzadeh
and Andersen, 1995). Using an approach first proposed by Wolstenholme (Wolstenholme,
Henderson, and Gavine, 1993) this simulation evaluated the potential impacts of business process
redesign on the overall organizational sub-system in relatively early design and testing stages of the
project.
Research Questions for This Paper
This paper has two purposes. First it presents some of the technical details of the system dynamics
simulation model created for the ORMA project. This model was designed to answer detailed
questions about customer response dynamics within ORMA. What are the impacts of various
types of equipment capacity constraints on the number of customer hang-ups? What is the
appropriate size of various system components (e.g., phone lines or computer ports) needed for
various customer volume scenarios? How do system components interact to drive overall system
performance? How should managers “grow” the customer service processes through various
modes of system behavior?
The second part of the paper looks into the role of simulation as one of several evaluative and
design tools being used in a business process redesign exercise. What types of questions are best
answered by system simulation in the evaluation of technology-intensive business process
redesigns? How does simulation best fit in with other types of evaluation and design technologies
such as experiments, customer surveys, interviews, and decision conferences?
311
A Simulation of ORMA’s Business Permit’s Assistance System
This section of the paper briefly describes the structure and behavior of a simulation model that
examines the technical and organizational consequences of installing a prototype automated
business permit system into the business processes of the agency. Figure 1 is an overall block
diagram of the agency’s system of processes for responding to customer inquiries. The call
processing system indicates how ORMA Bees : & :
handles calls into the system from call
placement to call completion. The system
consists of three sectors: (1) callers, (2)
ORMA's phone system, and (3) system
performance. The caller sector reflects
citizen demand for information delivery.
The phone system sector drives system
performance based on the balance
Being Served
by Voice Service
LoS
oa
between calls and call-processing capacity
(i.e., phone line, port, and operator). Lo
The system performance sector, on the E | 2
one hand, provides a basis for making Ce Being Served |
capacity expansion decisions. On the - by Operators —_
other hand, it provides information to the
reputational dynamics which drives long
term growth in ORMAs call volume. The
reputational effect is not formulated in the
model; instead, three scenarios of base
caller volume are introduced to capture
the potential effect of system performance
on customer satisfaction and, therefore,
on citizen demand. Figure 1: ORMA's Call-Processing System
Evidence analyzed in the simulation experiment indicates that under the current system, ORMA's
information dissemination performance by phone is significantly lower than it could be. As Figure
2 depicts, waiting time for an operator remains at its minimum until 11:00 AM, and then increases
rapidly to six minutes. Port utilization reaches one around 10:30 AM, when operator utilization
and port utilization are around 50 to 60 percent indicating the number of ports is much smaller than
needed to support the number of telephone lines and human operators which causes the voice
service to become ineffective. By the end of the day, the numbers of total calls, completed calls,
and attempts per completed call each attain their steady-state value. For 150 completed calls, 700
calls were placed. Each caller, on average has made almost five attempts to get through the system.
‘Completed Cals
a3)
Ovt-Input
Rat
-oo
i i H t _ i
5:00 1085 1230 145 rr 3:00 1085 1230 Tas 16:08
Figure 2: Base Run - Simulation of a Typical Day
Figure 4 presents a matrix of results from multiple runs of the simulation model examining the
impact of different strategy in call-processing capacity expansion on the system performance under
three scenarios. Under the base volume (150 callers per day), implementing the new system
316
hardware is equivalent to hiring one Ste ch
operator. When the call volume is ne tlers| Phone {OPE automate ARETE pS Wally
twice the base volume (300 callers per “toy [Syoteon|tt0S Business | peroomp | time for an
day ), installing the new system per day [System Permits _| leted calls Operator
y), , Base Run{ 150 | No ; 3 No High High
hardware becomes unavoidable, and iNT BOOT NS TT NE ton ae
hiring additional operators does not Rind 150 | Not 3 Yes] High Hien
compensate for the lack of new system = ["Rin'3"| "150 | Yes 13 1 Ves Low Tow
hardware. However, hiring one Rind PBC ves TS No Low Low
operator is equivalent to automating Run 1 300. | No 15 No. High ‘airly Low|
business permit information, when the " [Run 2” [300 [Yes 75 7"No Low High
the new system hardware is installed. Run3 | 300 | Yes {5 | Yes Low __ Fairly Low
Under the third scenario (450 callersa ["Run4~[~ 300 [Yes | 6 | No Low Low
day), improving system performance Run1 | 450 [Yes | 6} No High High
requires an even greater number of Run2 | 450 | Yes {8 No Low {Less High
telephone lines and ports than what the [Run3~ {450 “Yes 17” 7 Yes Low” +Less High
new system hardware offers. Run 4 450 [Yes 16 | Yes fairly Low: High
Figure 3: Summary of Results of Simulation Model
The Role of Simulation in Assessing Automated Customer Service Technologies
Evaluating a redesign of business processes supported by a new technology involves at least three
types of questions --technical questions about the technology itself, efficiency questions about the
overall cost and performance of the system, and effectiveness questions that get at quality of
service issues. As shown in Figure 5, the various evaluation methods used in this study all
address these three types of questions. However, each evaluation method addresses different
aspects of the overall problem. For example, with respect to technical issues, the prototype
answers directly the question, can complex permitting information be automated and what are the
technical problems involved with this implementation? On the other hand, the experiment can
systematically evaluate how customers react to specific technical components of the system (such
as the number of levels in the response hierarchy or the quality of the recorded voice). Finally, a
system dynamics simulation model can assess the size and scale of needed components when a
full-scale system is installed into existing or modified organizational processes. Figure 5 suggests
that process simulation models are best used as one of a suite of analytic tools in addressing this
type of business process redesign problem.
Type of Pratolype Cost & Performance Experiment Process
Question Group Conference and Survey Simulation model
Technology [ones Wecmation What are the anticipated How do customers react |What size components
Questions hipat are the technical customer volume by type or| to specific technical jaresriceded forvarious
. transaction? components? volume scenarios?
roblems and their solutions?|
What are the unit costs for |How do system
types of transactions? components interact to
Efficiency | What are hardware and |How much staff time per type] How long did customers —_|derive performance?
Questions software costs? lof transaction? What are have to wait? [Long term reputational
|simple cost-to volume leffect?
trade-offs? IManaging mode shifts?|
[Do customers like using the
lestectiveness| What are the best “look and Whats customer willingness | AC! 2f e907
fs SPSS | costraints on customer
Questions —_| feel” for customers? 0 pay for services? waiting time?
What are extended products
ind benefits from the system?
Figure 4: Multi-Method Approach to Prototype Evaluation
BA
Whereas Figure 4 emphasizes the differences in Redefin
purpose and outcome of various evaluative ‘Business
approaches, Figure 5 suggests that at another level Problem
of abstraction, all of the evaluation technologies Current
share a number of common features. All four Business Te
evaluation approaches discussed in Figure 4 -- Processes Business the Probelm
prototypes, cost and performance group Problem
conferences, experiments with customer reaction,
and process simulation models -- strive to create an Proposed
object or artifact (a prototype, a group model, or a
system simulation model) from which system Redesign
managers can learn lessons about their system.
Each approach involves steps to evaluate the object
of learning as well as steps to design a next stage in
the overall process.
Figure 5 makes the point that this learning loop of
evaluation and design can occur at several levels.
ra IT y
Solpition
Operationalize
The Solution
+AModel _ Evaluate
«A Prototype Solution
sete.
Figure 5: The learning loop
The innermost learning loop in Figure 5 involves proposing a technology-based solution (as a
prototype or as an overall process simulation model) and evaluating that technical solution. The
evaluation of the solution can lead to a redesigned solution which at the next stage of the project
yields a new proposed solution. However, as shown in Figure 5, the evaluation of the solution
can often yield a redefinition of the problem being solved. For example, in the ORMA example,
the initial problem centered on whether or not complex permit information could be automated.
Evaluation of the prototype concluded that it could, but that a key question was at what volume of
customer calls would such automation be cost justified? Redefining the problem often leads to an
examination of the business problem being solved and a redesign of the business processes for the
system as a whole. In the ORMA case, evaluation of the process simulation model revealed the
potential positive loop between customer satisfaction and the volume of incoming customer
inquiries. This loop implied that a very successful automated system would generate sufficient
customer satisfaction and hence repeat business (as a free public service) that the very success of
the system would swamp it in the future. The solution to this problem involved a redesign of the
types of services and the prices for those services being offered by ORMA.
References
Wolstenholme, Eric, Simon Henderson, Allan Gavine, 1993. Evaluation of Management
Information Systems: A Dynamic and Holistic Approach, John Wiley & Sons Ltd, England.
Andersen, David, Peter Avery, Stephen Hyde, and Kristine Kelly, 1995. "Voice Information
Response System for Business Permit Assistance", Office for Regulatory & Management
Assistance and Center for Technology in Government, CTG Project Repot 95-1.
Mojtahedzadeh, Mohammad and David Andersen, 1995, "A System Simulation of ORMA's
Business Permits and Phone-Based Public Assistance Program" (CTGORMA-007).
Andersen, David and John Rohrbaugh, 1994. "Report on ORMA's Cost and Performance
Modeling Conference of October 24, 1994" (CTGORMA-005).
Andersen, David, Peter Avery, Stephen Hyde, Kristine Kelly, and Soonhee Kim, 1995.
"Description of and results from the Experimental Evaluation of the ORMA Prototype"
(CTGORMA-009).
3Go
