|
Contents:
Introduction
The BSC method
Evaluation of corporate information technology
with the BSC
Applying the IT balanced scorecard
IT balanced scorecard practice
Conclusion: efficiency and effectiveness
References
Biographies
1. Introduction
The evaluation of the IT function, as with the evaluation of IT investments,
remains the subject of many academic and business discussions. In IT issues-studies
- where managers are asked what they find important in corporate information
technology - we can always find this subject under the name "Measurement
of IT effectiveness and productivity". In a British study (Galliers et
al., 1994), this issue was ranked 9th by IT managers, and in a very recent
US publication, (Brancheau et al., 1996) it was ranked 11th. The many publications
in scientific journals and the much visited seminars and conferences, also
suggest a continued and actual interest.
There are reasons for this extensive public interest: IT is increasingly
becoming crucial to achieve organisational and strategic goals. Investments
in IT are also never ceasing to grow, and business managers worry about
the fact that the benefits of these investments might not be as high as
initially expected. The industry likes to call this phenomenon the "IT
investment-paradox", or the "IT Black Hole". Large sums are invested in
IT, and seem to be swallowed by a large black hole without rendering many
returns. (Brynjolfsson, 1993 and Peppard and Rowland, 1995).
In order to be able to evaluate IT/IS-investments, many methods and
techniques have been suggested over the years. More traditional methods
focus on financial measures that have long been known: the "return on investment"
(ROI), the "net present value" (NPV), the "internal rate of return" (IRR)
and the simple and popular "payback time" (PB). These methods suggested
the paradox that we mentioned above, and urged researchers to search for
alternative ways of evaluating IT related investments.
Another approach to the problem is called "information economics" (IE)
(Parker et al., 1988 and 1989). This method allows to account for more
intangible benefits like a better customer service or a higher degree of
competitiveness. It also separates the benefits and risks into two domains
(a business domain and a technological domain) and evaluates these domains
jointly.
And now, the BSC found its way to evaluating IT and its investments.
Kaplan and Norton (1992, 1993 and 1996a,b) propose this method in order
to evaluate a company’s progress from four different perspectives: the
financial perspective, the perspective of internal processes, the client’s
perspective and the innovative perspective. This model can also be applied
to IT investments and to the IT function, as Gold (1992) and Willcocks
(1994) have already indicated in a conceptual manner.
In this paper a framework is developed for evaluating IT/IS based on
the BSC-technique. This evaluation is confronted with two kinds of tasks.
One task lies in trying to assess the contribution of a specific information
system or project. The other focuses on assessing the general IT function.
It deals with crucial questions like: how good is our corporate information
technology, how can we measure this function and how can we improve it?
In this article, we will try to take a closer look at these questions.
2. The BSC method
The inventors, Robert Kaplan and David Norton,
developed the method in three articles published in the Harvard Business
Review (1992, 1993 and 1996a). Their idea was that traditional financial
measures (like the ROI, for example) should be supplemented with operational
measures concerning customer satisfaction, internal processes and the ability
to innovate. These three measures would assure future financial results,
and drive the organisation towards its strategic goals while keeping all
four perspectives in balance. Kaplan and Norton undoubtedly knew the theories
behind business (process) reengineering (Hammer and Champy, 1993 and Davenport,
1993), where the authors also stress the importance of quantitative goals
and measures to drive the strategy. All these measurements (evaluations)
are framed in a strategic management system that drives improvement and
that allows to prepare for the future. To do this, the method uses a three-layered
structure:
-
the mission: management first states a mission (e.g. "to become our customers’
most preferred supplier")
-
the objectives: the mission is translated into objectives (e.g. "to provide
our customers with new products")
-
the indicators: the objectives can be measured through well-chosen indicators
(e.g. "percentage of turnover generated by new products")
Table 1 gives an example of the BSC-technique
applied to a company as a whole.
3. Evaluation of corporate information
technology with the BSC
This general BSC-framework found in Table 1 can
easily be translated to the more specific needs of the evaluation of an
IT function (Table 2). The proposed perspectives
(user-orientation, corporate contribution, operational excellence and future
orientation) differ from the general ones, mostly because the IT department
is an internal service supplier. The users are its clients, and the contribution
is to be considered from management’s point of view.
This BSC-evaluation for IT can be compared to the more general management
evaluation as Dickson and Wetherbe (1985) discuss it. They discuss the
critical success factors of the IT function and indicate that a measure
like "system availability and downtime" can be considered to be such a
factor. Yet the BSC-approach is more comprehensive: Dickson and Wetherbe’s
measures only regard the financial assessment and the efficiency of internal
processes.
Each of these perspectives have to be translated into corresponding
metrics and measures that assess the current situation. These assessments
have to be repeated periodically, and have to be confronted with the goals
that had been set beforehand. Hereafter, an overview of some generic IT
measures will be presented. These measures are generic, because each corporate
mission and each corporate set of objectives requires its own specific
measures. The measures presented here are largely extracted from traditional
IT management literature (Hamilton and Chervany, 1981; Dickson and Wetherbe,
1985), the Information Economics publications (Parker et al., 1988 and
1989) and the IT evaluation literature (Gold, 1992; Willcocks, 1994; Parker,
1996). The presented framework integrates these different approaches and
adds an important dimension: the evaluation becomes more dynamic and strategic
since measures are tracked and traced over time, and explicitly integrated
in the strategic management of the IT department. In this way, added value
can be created for the company.
It makes sense to distinguish two kinds of IT evaluation: the short
term financial evaluation and the long term oriented evaluation of IT projects
and the IT function itself. In Table 3 you will recognise
these categories: "Control of IT expenses" and "Sell to third parties"
are definitely focused on short-term evaluations. "Business value of new
IT projects" and "Business value of the IT function" are measures that
require a more prolonged time frame.
The traditional financial perspective is worried about the control
of the IT budget and the benefits that are possibly coming from the sales
of IT products and -services to third parties. Despite publications like
the Butler Cox Foundation’s (1990) where commercial activities of the IT
department are encouraged, these activities remain to be exceptional today.
A popular financial metric undoubtedly is the IT budget expressed
as a percentage of turnover. The comparison to other companies in the industry
may give useful indications. But these hints have to be interpreted with
care: higher or lower IT expenses may be caused by company-specific reasons.
A critical attitude towards these figures is absolutely necessary even
if the percentage is at the same level as the industry average. In addition,
variations from 1% to 8% of turnover have been known to occur, depending
on the IT intensity of the industry (Robson, 1994). The Butler Cox Foundation
(1990) has published global figures for the IT expenses per staff member:
for the financial sector this would come to more than 4000 British pounds
per employee.
IT projects must generate value for the company. Value is a much broader
concept than benefits (Willcocks, 1994). When implementing a new marketing
database for example, the substantially lower amount of programmer-intervention
necessary to execute an ad-hoc-query, will certainly generate (a maybe
modest amount of) direct benefits. But the real value of such a project
lies with the marketing-department: will the salespeople integrate the
database into their approaches and consequently achieve a higher turnover?
Value therefore implicates risk.
IT benefits have traditionally been measured by quite simple (at least
in theory) financial measures like the return on investment and/or the
payback period. The ROI is the ratio of average annual net benefits of
a project and the invested amount of money. The payback period is even
simpler to calculate: it results in a period of time that indicates how
long an investor will have to wait for the project to repay its initial
investment. These types of financial measures limit themselves to the financial
benefits, and do not incorporate values. The method of information economics
(Parker et al., 1988 and 1989) fills exactly this leap hole. Table
4 displays the Information Economics as recently adapted by Parker
(1996). You will find some elements of the business reengineering theories
in this outlook.
In essence, the information economics method is a scoring technique
whereby value and risk categories are attributed a score between 0 and
5. For a value category (marked with a ‘+’-sign), 0 would signify ‘no positive
contribution’, and a 5 would refer to a ‘large positive contribution’.
For a risk category (marked with a ‘-‘-sign) 0 would mean ‘no risk’ and
a 5 would signal a ‘large risk’. Each of these categories is assigned a
weight. By adding the weighed scores of the value categories and subtracting
the weighed scores of the risk categories, one can calculate the total
score of each project. Originally, the information economics framework
included 6 value and 4 risk categories, but in Table
4 we added the categories that Parker used in her 1996 work. We also
have to mention that these categories have an indicative, not an exhaustive
meaning. Each and every company should adapt these categories to its own
needs and specifications. The value of this method lies with the fact that
the scores are assigned by all parties involved: the users only score risks
and values in the corporate domain, and the IT specialists only score the
IT related categories. This way, the business contribution of the project
can be assessed jointly, and consensus can be reached on the evaluation
of a specific project.
Most value and risk categories of Table 4 speak
for themselves. A few of them might require a short explanation (see also
Parker, 1996). ‘Value linking’: incorporates the benefits and costs
in other (functional) areas. ‘Value acceleration’: a typical example
are the interest savings due to an accelerated cashing of invoices. ‘Value
restructuring’: refers to the efficiency and effectiveness of the
employees: does the new system free up more time to execute their own job?
‘Strategic IT architecture’ assesses the degree to which the project fits
into the IT plan. ‘Business strategy risk’ and ‘IT strategy risk’ respectively
refer to the degree of risk in terms of how well the company/IT department
succeeds in achieving its strategic objectives. ‘Definitional uncertainty’
indicates the degree of risk in terms of how clearly the functional requirements
and specifications have been agreed upon. ‘Technical uncertainty’ provides
a measure for the risk associated with dependence on new, immature technologies.
‘Business organisation risk/IT service delivery risk’ scores the degree
of risk in terms of how well the company/the IT department will be able
to adapt to the changes invoked by the project.
When we refer to the user, we have set our thoughts primarily on the
end user, the internal customer of the IT department. Secondarily, this
user could also be the company’s customer in the case of an interorganisational
system. The user orientation and the measurement of the customer satisfaction
were also heavily focused by the BPR-change methodologies that we referred
to earlier. The balanced score card now hands the techniques to measure
this dimension and manage accordingly.
The metrics regarding user orientation have three items to focus on:
to be the preferred supplier for applications and operations, the partnership
with the users and the user satisfaction (Table 5).
The percentages of the applications that are managed and delivered by
the IT department are heavily dependent on the company-specific situation.
When a company sets the ratio of internal versus external development,
it makes a strategic choice. In making such a choice, it will take into
account other factors like wanting to keep part of the development capacity
in house for strategic, highly competitive projects (Van Grembergen en
Vander Borght, 1997). This remark also goes for outsourcing computer operations.
Surveying clients (users) should play an important role in the evaluation
of the IT function as a whole. Especially important customers need to be
involved in such surveys. If the department would lose an important customer,
detailed research into the reasons behind this loss would certainly be
required. The indexes resulting from the involvement surveys that you find
in Table 5, are most important, but must be treated
with care. Hamilton and Chervany (1981) distinguish objective and subjective
measures. The indexes resulting from surveys are evidently subjective measures,
as opposed to most other measures that you will find here. Subjective measures
can be completed with a compliance audit, evaluating the user involvement.
It concerns primarily the measurement
and improvement of the two basic processes of the IT department: the development
of new information systems and the computer operations. We also focus on
other processes like PC supply, problem management, user education, management
of IT staff and their usage of efficient communication channels (Table
6).
IT should deliver high-quality service to its users and do this at the
lowest possible cost. This can only be achieved by optimally managing the
process and can be improved by following up the operational measures displayed
in the above Table. The measures in Table 6 should
not only be followed through time, but should also be compared to industry
standards and averages. Therefore it is important to use standardised measures
like e.g. function points. Function point analysis is a widely used output
metric that measures the number of inputs, outputs, inquiries and files
used in an application. The results of this analysis can be used to calculate
the number of function points written by a programmer in a specific unit
of time.
This way, function points will have to be used when benchmarking programming
productivity, because they are an accepted, standardised way of measuring
programmer productivity. In Figure 1 you can see how the evolution of a
programmer’s productivity can be followed. Of course, a graphical representation
can be equally well applied to the other measures.
3.4 Measurement future orientation
As an addition to measuring the performances of today, we also need
to measure the performances of the future. The measurement of the IT department
‘s future opportunities has to do with preparing the staff for the future,
preparing the applications portfolio for the future and putting effort
into researching new emerging technologies (Table 7).
The ability to deliver high quality IT services within 3 to 5 years
has to be prepared today. IT has to assess future trends and anticipate
them. The fact that unanticipated evaluations can probably be dealt with
through extensive external (often high priced) support, can be of some
comfort. Of course, the better solution is that internal people are well
educated for the future so that the right expertise can be found in-house.
4. Applying the IT balanced scorecard
In building a company-specific IT balanced scorecard, we propose the
following steps:
-
presentation of the concept of the IT balanced scorecard technique to top
management and IT management;
-
data-gathering phase where information is collected on the following items:
corporate and IT strategy, (traditional) IT metrics already in use for
performance measurement;
-
developing the company-specific IT balanced scorecard inspired on a "standardised"
model as presented in this paper and based on the Kaplan and Norton (1996b)
principles.
Following Kaplan and Norton (1996b) three principles have to be complied
with in order to develop an IT balanced scorecard that is more than a group
of isolated and eventually conflicting strategies and measures:
-
build in cause-and-effect relationships
-
include sufficient performance drivers
-
linkage to financial measures.
A strategy is a set of assumptions about cause and effect. If cause-and-effect
relationships are not adequately built in the balanced scorecard,
it will not translate and communicate the company's vision and strategy.
Cause-and-effect relationships can be illustrated as follows: if we guarantee
"zero" defects (operational excellence perspective), then we will meet
user expectations better (user orientation perceptive), and then we will
enhance the support of business processes (business contribution perspective).
A well built balanced scorecard needs a good mix of outcome measures
and performance drivers. Outcome measures like programmers' productivity
(number of function points per person per month) without performance
drivers like IT staff education (number of educational days per person)
do not communicate how the outcomes are to be achieved. And performance
drivers without outcome measures may enable to achieve short term operational
improvements, but will fail to reveal whether the operational improvements
have been translated in enhanced financial performance. An IT department
may invest significantly in employee training in order to improve employee
productivity. If, however, there is no outcome measure for employee productivity
(e.g. function points), IT management cannot measure whether its strategy
is effective. Table 8 shows some examples of outcome
measures with corresponding performance drivers. Outcome measures are more
or less generic (user satisfaction, productivity, employee satisfaction),
whereas performance drivers are more company-specific and are revealing
company strategy.
The IT balanced scorecard must retain a strong emphasis on financial
outcomes. "A failure to convert improved operational performance into
improved financial performance should send executives back to the drawing
board to rethink the company's strategy or its implementation plans" (Kaplan
and Norton, 1996b).
Further, we must keep in mind continuously that measurements are not
enough and that they must be used and acted upon by management: the balanced
scorecard is not only an operational but in essence a strategic management
system. The following steps to implement effectively the IT balanced scorecard
as a strategic management system are (Kaplan and Norton, 1996b):
-
Clarifying and translating vision and strategy, and attention to both the
cause-and-effect relationships and the performance drivers;
-
linking strategy to team and individual goals, and eventually linking employee
compensation to the balanced scorecard measures;
-
linking strategy to resource allocation, and determining stretch targets
and priority setting;
-
strategic feedback, and collecting and reviewing performance data about
the strategy and defining new strategic initiatives or adjusting existing
strategy.
5. IT balanced scorecard
practice
Recently the balanced scorecard is more and more applied to information
systems. One of the authors has been the witness of the start-up of an
IT BSC-implementation in two leading Belgian companies. It was demonstrated
in these two cases that there are many pitfalls when implementing the IT
BSC. Major (generic) shortcomings were:
-
The inspected IT balanced scorecards were operational management
systems but not strategic management systems due to the absence of long
term targets. Most of the system performance objectives were already met.
This means that either the performance objectives were sub-optimal and
should be further increased or that no strategic initiatives were needed
to further improve system performance.
-
It was identified that there were insufficient cause-and-effect relationships
and insufficient performance drivers. System availability, responsiveness
on development requests, and timely delivery of new applications were identified
as performance drivers for user satisfaction. However, there was no indication
of how the above will realised: by increasing employee skills and/or implementation
of new development tools, and/or by new project management methods?
-
There was insufficient communication: the distribution of the IT balanced
scorecard was restricted to senior management and IT management. Ideally,
it is communicated throughout the IT department in order to maximise employee
commitment. In addition individual objectives of the IT employees were
not linked to the IT balanced scorecard.
The study of the two cases also revealed that although each IT BSC is different
for each individual company, there is a need for a "standardised" scorecard
as presented in this paper. It was e.g. noticed that in one of the cases
the user view contained some typical internal operational measures, that
the business contribution view was poorly developed, and that the internal
operational perspective needed additional measures for PC acquisition,
problem management, user education, and management of IT staff.
Conclusion: efficiency and effectiveness
In this paper we proposed an evaluation framework
for the IT function based on the balanced score card technique, completed
with elements of information economics and business reengineering. Four
evaluation domains were identified and supplied with adequate measures:
business contribution, user orientation, operational excellence and future
orientation of IT. The suggested framework is a strategic management tool
that enables management to follow up the measures and to drive performance
based on the goals that were set and agreed upon in advance. Measurement
is a prerequisite to management.
The authors are convinced that such a tool can be of meaningful help
for both general management and IT professionals, and that this tool can
be implemented as such. Their experience however tells them that the installation
and maintenance of such a tool is difficult and requires substantial means.
But in many cases, the total cost of implementing such a tool can be lower
than expected, since many of the needed operational measures may already
be available.
Most of the presented measures are not new at all, but are used and
combined in a new way in this approach. The model we presented resembles
Hamilton and Chervany (1981), who distinguish the evaluation of the IT
efficiency and the evaluation of IT effectiveness. They define the primary
goal of the IT function to be the development and maintenance of information
systems that support corporate goals. This can be evaluated in two distinct
manners:
-
evaluating the efficiency of the execution of development and operations;
-
evaluating the effectiveness of the users that use information systems
to attain corporate goals.
The presented balanced score card evaluation of IT integrates these two
evaluations. Efficiency is typically dealt with in the domain of operational
excellence, while effectiveness is treated in the domains of business contribution
and user orientation.
References
- Brancheau, J., Janz, B. and Wetherbe, J., "Key issues in information
systems management: 1994-95 SIM delphi results", MIS Quarterly, June 1996,
pp. 225-242
- Brynjolfsson, E., "The productivity paradox of information technology",
Communications of the ACM, December 1993, pp. 67-77
- Butler Cox Foundation, Getting value from information technology, Research
report 75, June 1990
- Davenport, T., Process innovation: reengineering work through information
technology, Harvard Business School Press, Boston, 1993
- Dickson, G. and Wetherbe, J., The management of information systems,
McGraw-Hill, New York, 1985
- Galliers, R., Merali, Y. and Spearing, L., "Coping with information
technology? How British executives perceive the key information systems
management issues in the mid-1990s", Journal of Information Technology,
9, 1994, pp. 223-238
- Gold, C., Total quality management in information services - IS measures:
a balancing act, Research note, Ernst & Young Center for Information
Technology and Strategy, Boston, 1992
- Hamilton, S. and Chervany, N., "Evaluating information systems effectiveness
- Part I: comparing evaluation approaches", MIS Quarterly, september 1981,
pp. 55-69
- Hammer, M. and Champy, J., Reengineering the corporation, Harper Business,
New York, 1993
- Kaplan, R. and Norton, D., "The balanced scorecard - measures that drive
performance", Harvard Business Review, January-February 1992, pp. 71-79
- Kaplan, R. and Norton, D., "Putting the balanced scorecard to work",
Harvard Business Review, September-October 1993, pp. 134-142
- Kaplan, R. and Norton, D., "Using the balanced scorecard as a strategic
management system", Harvard Business Review, January-February 1996a, pp.
75-85
- Kaplan, R. and Norton, D., The balanced scorecard: translating a strategy
into action, Harvard Business School press, Boston, 1996b
- Parker, M., Benson, R., and Trainor, H., Information economics: linking
business performance to information technology, Prentice Hall, Englewood
Cliffs (NJ), 1988
- Parker, M., Trainor, H., and Benson, R., Information strategy and economics:
linking business performance to nformation technology, Prentice Hall, Englewood
Cliffs (NJ), 1989
- Parker, M., Strategic transformation and information technology, Prentice
Hall, Upper Saddle River (NJ), 1996
- Peppard, J. and Rowland, P., The essence of business process reengineering,
Prentice Hall, London, 1995
- Robson, W., Strategic management and information systems, Pitman Publishing,
London, 1994
- Van Grembergen, W. and Vander Borght, D., "Audit guidelines for IT outsourcing",
EDP Auditing, Auerbach 72-30-35, June 1997, pp. 1-8.
- Willcocks, L., Information management. The evaluation of information
systems investments, Chapman & Hall, London, 1994
Biographies
Wim Van Grembergen is Professor of Information Systems at the Business
Faculty of UFSIA (University of Antwerp) and the Executive School of UFSIA.
His teaching and research interests are business transformations through
IT and audit of information systems. He is a member of the Standing Committee
of the European Conference on Information Systems (ECIS). Until recently
he was Director of the MBA Programme and presently he is coordinator of
a professional two-year program on IT audit.
At the time of writing, Rik Van Bruggen was the co-ordinator of the
Information Systems Programmes at UFSIA’s Executive School (IPO - UFSIA
Management School). He recently accepted a new challenge as a project manager
for e-COM Interactive Expertise, a company specialised in implementing
company-wide intranets in large organisation.
|
