1.        Introduction

In an article written in 1991 Robert Eccles heralded a performance measurement revolution and predicted that “within the next five years, every company will have to redesign how it measures its business performance”.  This prediction has proved correct and “the shift from treating financial figures as the foundation for performance measurement to treating them as one among a broader set of measures” (Eccles, 1991, p131) has been embraced, to a greater or lesser extent, by both the private and public sectors. In this paper I will explore how “a broader set of [performance] measures” might be developed and implemented for an IT department within the context of the UK Health Service.

2.        Context

The foundations of the current National Programme for IT (NPfIT) in the National Health Service (NHS) were originally set out in the Department of Health’s strategy Information for Health (1998). This committed the NHS to:

§       Lifelong electronic health records for every person in the country;

§       Round-the-clock online access to patient records and information about best clinical practice, for all NHS clinicians;

§       Genuinely seamless care for patients through GP’s, hospitals and community services sharing information across the NHS information highway;

§       Fast convenient public access to information and care through online information services and telemedicine; and

§       The effective use of NHS resources by providing health planners and managers with the information they need.

Following the development of the Government’s ten-year plan for the NHS (2000), a supporting document Implementing the NHS Plan – Building the Information Core was published in January 2001. This developed the vision set out in the NHS Plan for a “service designed around the patient”. The delivery of a modernised health service was inextricably linked to the delivery of modern information technology as a key enabler of bringing about change in the way care is accessed and delivered. The Wanless Report (2002) included several key recommendations for IT in the NHS:

§         A doubling of IT spending that is protected to ensure that it is not diverted for other purposes;

§         Stringent national standards for data and IT set by the centre; and

§         Better management of IT implementation, including a national programme.

The report underlined the fact that the NHS Plan’s targets and modernisation objectives require a different, new approach to the development and delivery of IT:

“Without a major advance in the effective use of ICT, the Health Service will find it increasingly difficult to deliver the efficient, high quality service which the public will demand. This is a major priority which will have a crucial impact on the health service over future years.”

The NPfIT commenced in October 2002 and Delivering 21^st century IT support to the NHS started the process of connecting the delivery of the NHS Plan and the modernisation of services to the information strategy. IT will now be designed and delivered around the needs of the patient and service users, not institutions; shifting from systems running along institutional lines, dealing with only a portion of patient interactions, to whole health and social care community systems that track and record a whole user/patient journey. The key component of NPfIT is the building and availability of a 24/7 live patient record that all health professionals in whatever setting (hospital, primary care, community services) and patients can access.

Thus the importance of ICT in supporting the core business of an improved healthcare system and realising the targets and vision of the NHS Plan are beginning to be increasingly recognised. The priority which attaches to the contribution of ICT to the work of the NHS raises issues of how best to link ICT targets with business objectives and in turn determining how well the IT function is performing. One potential solution is the use of the Balanced Scorecard.

3.        Performance Measurement

Neely (1998, p5-6) offers a definition of a performance management system as one which “enables informed decisions to be made and actions to be taken because it quantifies the efficiency and effectiveness of past actions through the acquisition, collation, sorting, analysis, interpretation and dissemination of appropriate data”. In this context “effectiveness refers to the extent to which customer requirements are met, and efficiency is a measure of how economically the organisation’s resources are utilised when providing a given level of customer satisfaction”. The litmus test of a performance management system might thus be regarded as the use that is made of it to inform decisions and bring about improvements.

Performance frameworks fall into two broad categories: those which are designed for assessing business excellence e.g. the European Excellence Model (EEM), the Deming Prize of Japan, and the US Malcolm Baldrige National Quality Award; and those which are designed to help organisations develop performance management systems. The EEM is a diagnostic and self-evaluation tool whereas the focus of the Balanced Scorecard (BSC) is on providing a methodology to allow an organisation to turn its strategy into actual achievements.

The Balanced Scorecard (BSC) will be used in this paper as a conceptual and pragmatic model for the development of performance measures for an IT Department. The Balanced Scorecard (BSC) is a technique developed by Kaplan and Norton (1992) that helps organisational decision-makers to navigate the organisation towards success. It enables organisations to translate their mission and strategy into a comprehensive set of performance measures that provide the framework for a strategic measurement and management system.

The Balanced Scorecard measures organisational performance, with emphasis on financial objectives. However, it also includes the performance drivers of these financial objectives, and measures organisational performance across four balanced perspectives:

§         Financial;

§         Customer;

§         Internal Business Processes; and

§         Learning and Growth.

This is illustrated in Figure 1 below:

Figure 1: Kaplan and Norton’s Balanced Scorecard (Kaplan and Norton (1996a, 1996b))

Kaplan and Norton (1992) argue that traditional financial measures are backward looking. They try to address this inadequacy by complementing past performance measures (financial measures) with drivers of future performance indicators (customers, suppliers, employees, processes, technologies and innovation). The fundamental concept of the BSC is to derive the objectives and measures from the overall corporate vision and strategy and to use four perspectives as a “balanced” framework to monitor and achieve these objectives.

The Balanced Scorecard, with its four perspectives, has been criticised (e.g. Neely, 1998) for excluding employees inside the organisation, suppliers and other external partners, competitors, and regulators. For example, the supplier perspective would be extremely important in the IT field where services may be outsourced or where a supplier acts as a partner within a system procurement or application development project.

4.        Identification of performance measures for IT

The BSC framework will be used to develop financial and non-financial measures for an IT department. Two areas of IT activity have been selected to illustrate how this approach might be applied in practice i.e.:

§         Project Management; and

§         IT Help Desk.

Critical success factors will be developed for each area together with performance measures within each of the 4 perspectives of the BSC.

The performance management process for IT which underpins the development of these performance measures involves linking IT objectives to the corporate strategy, following a BSC approach to identify target measures, implementing and monitoring these measures in order to improve decision making and bring about improvements in IT processes and performance.

4.1         Project management

A formal project management methodology such as PRINCE II (Projects in Controlled Environments) may be used to manage ICT projects. One of the key objectives of an IT department is to identify and prioritise projects in line with the ICT strategy and corporate plan, and to adopt and apply sound project management techniques for each project undertaken. Projects may relate to a wide range of areas e.g. internet/intranet development, system development and implementation, infrastructure development etc. etc.

Critical success factors (CSF’s) for project management include:

§       Experienced and skilled project managers are available;

§       There is senior management sponsorship of projects;

§       Stakeholders and IT staff share in the definition, implementation and management of projects;

§       A project organisation is in place with documented roles and responsibilities;

§       There is an understanding of the abilities and limitations of the organisation and the IT function in managing large, complex projects;

§       All projects have a Project Initiation Document which includes project background and justification, Project Definition, Project Plan, Communication Plan, Project Quality Plan, Project Controls and Risk Log; and

§       The transition from the implementation team to the operational team is a well-managed process.

It is critical that the effectiveness and efficiency of projects are monitored utilising key performance measures. The measures developed for project management are in 4 linked areas - Financial, Internal Business Process Learning & Growth, and Customer – and this is illustrated in Table 1 below:

Table 1: Performance measures for project management

The ultimate aim in this area is that a proven, full life-cycle project methodology is implemented and enforced, and is integrated into the culture of the entire organisation.

Kaplan and Norton (1996b, p30) argue that “a strategy is a set of hypotheses about cause and effect. The measurement system should make the relationships (hypotheses) among objectives (and measures) in the various perspectives explicit so that they can be managed and validated. The chain of cause and effect should pervade all four perspectives”. The measures that have been identified for Project Management have been selected with this “chain of cause and effect relationships” in mind. Thus the measures act as indicators of the department’s progress towards meeting the overall objective and CSF’s for the Project Management process and are causally linked to one another. For example, the increased use of a formal project management methodology will drive the move towards more project team members being fully trained in project management methodology which will lead to an increasing number of projects being delivered on time and to budget and a higher level of organisational satisfaction with project delivered services.

4.2         IT help desk

The IT Help Desk provides first-line support and advice to users of IT systems in an organisation with the aim of ensuring that any problem experienced by the user is appropriately resolved.

Critical success factors for the operation of the IT Help Desk include:

§         Knowledgeable and customer-orientated support staff resolve problems in close co-operation with senior IT staff;

§         All user enquiries are consistently and thoroughly registered by the Help Desk;

§         User enquiries that cannot be resolved in a timely manner are appropriately escalated;

§         The clearance of user enquiries is monitored;

§         User questions are resolved in a timely manner;

§         Those user enquiries that cannot be resolved in a timely manner are investigated and acted upon;

§         Management monitors trends to identify root causes in a proactive manner and follows up with analysis and the development of solutions;

§         Organisational policies and programmes are defined for training users in technology and security practices; and

§         There is management awareness of support costs and these are charged back to the business.

The measures developed for the IT Help Desk are in 4 linked areas - Financial, Internal Business Process Learning & Growth, and Customer – and this is illustrated in Table 2 below:

Table 2: Performance measures for the it help desk

The ultimate aim in this area is that the Help Desk function is established, well organised and takes on a customer service orientation, by being knowledgeable, customer focused and helpful.

As previously noted the performance measures for the IT Help Desk have been selected with Kaplan and Norton’s (1996b, p30) “chain of cause and effect relationships” in mind. The measures act as indicators of the department’s progress towards meeting the overall objective and CSF’s for the IT Help Desk function and are causally linked to one another. For example, customer (and organisational) satisfaction is likely to increase if the time to resolve IT problems is minimised and the Help Desk service is delivered in a cost effective way. Furthermore, the need to effectively utilise the Help Desk resource will drive the provision of IT training programmes for staff in order to reduce the inappropriate use of the IT Help Desk. A more effective service could potentially be provided at higher cost, but judgements would need to be made as to the balance between the required level of service versus the resource available to provide this.

5.        How the measures will work

In considering how the IT measures which have been identified will work in practice it is important to bear in mind the behaviour they will encourage and whether or not this behaviour is desirable. Neely et al (1996, 1997), Neely (1998), and Bourne (2000) have developed a performance measurement record sheet which forces a series of questions to be answered in order to ensure that the measure is of practical value. There are 10 dimensions within this framework:

§         Title of the measure;

§         The purpose of the measure;

§         What corporate objective/s the measure relates to;

§         What performance target should be set;

§         How is the performance measure to be calculated;

§         Frequency of measurement and review;

§         Identification of who is responsible for measuring performance;

§         Source of the data;

§         Allocation of responsibility for taking action on the measure; and

§         Specification in outline of the types of action that can be taken to improve performance in

§         this area.

The performance measurement record sheet provides a formal framework which explicitly links measures to objectives, further defines the measures and how they will be evaluated, assigns responsibilities, and ensures that performance improvement is integrated as part of the overall process.

In Table 3 (at the end of the paper) a Performance Measurement Record Sheet has been completed for the six of the IT measures previously identified as a way of demonstrating how they would work in practice:

6.        Implementation

Olve and Sjostrand (2002, p106) emphasise the importance of what they call “the virtuous circle formed by strategy, control, measurement, learning, and back to strategy”. If, as part of the implementation of a performance measurement system, this continuous process is maintained then its full benefits can be realised.

The approach to the implementation of the performance measures for an IT department falls into three stages:

§         Stage 1 – Initial Development Process

§         Stage 2 – Continuous Use of the Performance Measures

§         Stage 3 – Refreshing and Updating the Performance Measures

Olve et al (1999) and Olve and Sjostrand (2002) propose a number of key steps in the implementation process which may be related to the three stages identified above.

6.1         Stage 1 – Initial development process

In this stage it will be necessary to:

§         Obtain senior management commitment and support for the project. It is important that the rationale for the development of a performance measurement system is communicated to all stakeholders in order to overcome potential barriers to implementation such as fears about the perceived adverse effects of performance measurement and process improvement;

§         Provide information and training in performance measurement and process improvement;

§         Form a small project team with appropriate membership in order to take the implementation process forward effectively and obtain ownership from the team and the organisation;

§         Decide on a department-wide implementation or pilot the implementation within specific areas of IT. The pilot would have the advantages of allowing the department to become familiar with the performance measurement process, learn from mistakes and gain the confidence and commitment of staff before rolling the process out to the department as a whole;

§         Clearly establish links between the development of measures with the corporate and ICT strategies; and

§         Integrate reward and recognition schemes with performance improvement as measured by the performance measurement system or BSC.

6.2         Stage 2 – Continuous use of the performance measures

In this stage it will be necessary to:

§         Set short-term and long-term goals for each measure which are consistent with the corporate strategy. Equally they must be realistic and attainable, but also challenging;

§         Develop flexible and effective systems and procedures to collect information necessary to monitor the performance measurement system. This should draw on readily accessible information and allow the automation of measurement; and

§         Focus on a balanced set of measures and explore the causal links between financial and non-financial measures to bring about process improvements. The emphasis here is on the department or organisation becoming a learning organisation.

6.3         Stage 3 – Refreshing and updating the performance measures

The development of a performance measurement system cannot be regarded as static and must therefore be kept under review as the corporate and ICT strategies evolve and change in response to internal and external drivers.

In summary the implementation process requires initial impetus, commitment, training and co-ordination in order to overcome cultural and organisational barriers; the performance measurement system must then be institutionalised as a process within the organisation so that the organisation’s focus is on what it is doing well, what it is not doing so well and what can be improved; and finally the performance measurement system must be kept under review and alive.

7.        Monitoring

A number of approaches to monitoring the IT performance measures are considered in this section, but the focus is on putting in place a monitoring process which will lead to improvements in the performance of the IT department.

7.1         Using performance data

It is important to avoid information overload, ensure that information is accurate, up-to-date and credible, use accessible information, present information in an attractive, easy to understand way and to select an appropriate frequency to monitor and act on performance information.

Performance data can be used to:

§         Predict trends in order to allow proactive decision-making and actions to be taken. For example, the analysis of the number of enquiries received by the IT Help Desk will in turn act as a predictor of future resources needed to support this function and the most effective deployment of staff resources;

§         Correlate results in order to concentrate actions that will give best results. For example, the correlation of information on budgets, timescales or project issues in relation to projects where a formal project management methodology was not used versus projects in which formal project management methodology was used, may give rise to more informed decisions about the appropriate use of project management methodologies; and

§         Access to historical performance information will allow year-on-year comparisons and therefore a sounder basis for making informed judgements.

In order to ensure that performance information is actively reviewed and used as a basis for implementing improvements, the use of a more formal monitoring framework could be considered. This would allow the IT Department to ask key questions about its performance:

§         What is our current performance?

§         How does actual performance compare with the target set?

§         What are the main reasons why the target isn’t being met?

§         What is the plan for corrective action?

§         Has the action been taken?

§         Does the action have the desired impact on the results of the measures?

Neely et al (1996), Neely (1998), Olve et al (1999) and Bourne (2000) highlight two frameworks – Ford QOS (Quality Operating system) and Xerox follow-up framework – that could be adopted for use by the IT Department to monitor and continuously improve its performance.

The Ford framework is shown in Figure 2 below and consists of 4 panels:

§         Panel 1 – the graph of actual performance against target;

§         Panel 2 – a breakdown of that result by the main factors contributing to the result achieving or missing the target;

§         Panel 3 – the action planned to improve the performance; and

§         Panel 4 – the record of the impact of the action taken.

Figure 2: The Ford QOS Measure Visualisation (Adapted from Neely et al, 1996)

The Xerox framework is shown in Figure 3 below and includes:

§         An owner, that is an individual responsible for preparing quarterly analysis of the trends, causes, strengths and areas for improvement as well as the action plan;

§         A sponsor: in Xerox’s case, a main Board Director;

§         A desired state, including the results, approach and pervasiveness and a 7 point rating where 7 is “world class performance”;

§         Performance;

§         Causal analysis;

§         Strengths;

§         Areas for improvement; and

§         Detailed action plan.

Figure 3: Follow-up of Measures at Xerox (Adapted from Olve et al, 1999)

The benefits of these frameworks are that they ensure that action is taken as an outcome of a performance monitoring process and that there is shared ownership of the drive towards improvement.

7.2         Benchmarking performance

Two approaches to benchmarking of the IT performance measures are possible:

§         Benchmarking of performance against comparative organisations e.g. using information derived from Gartner (information systems research specialists); and

§         Using a “maturity model” which allows an organisation to grade its IT processes in absolute terms from non-existent to optimised (from 0 to 5). This approach is derived from the Maturity Model that the Software Engineering Institute defined for maturity of the software development capability (Paulk et al, 1993). Against these levels an organisation can map:

§         The current status of the organisation – where the organisation is today;

§         The current status of the industry (best-in-class)  – the comparison;

§         The current status of international standards – additional comparison; and

§         The organisation’s strategy for improvement – where the organisation wants to be.

The maturity model is shown in Figure 4 below:

Figure 4: IT maturity model

In this model the scale from “non-existent” to “optimised” can be interpreted as follows:

7.3         Communication

In order to ensure that key staff are aware of the monitoring of the IT performance measures and that this is instilled in the culture of the department a number of actions may be taken:

§         Display of monitoring information on departmental notice boards and on the intranet;

§         Briefings on the results of measures become an integral part of staff meetings and one-to-one reviews of individual objectives. These briefings should clearly demonstrate the impact on the department and on the organisation as a whole of achieving or failing to achieve agreed targets; and

§         Briefings to the senior management team on the results of measures.

8.        Conclusion

This paper has established a framework and process for the implementation of a performance measurement system in an IT Department within the context of the UK Health Service. Performance measures have been identified, the rationale underpinning the selection of measures has been explained, a Performance Measurement Record Sheet has been used to define how those measures would work in practice, and approaches to implementation, monitoring and reviewing measures have been considered. As an outcome of fully implementing a performance measurement system within an IT department the following benefits may be achieved:

§         Through deciding what to measure will encourage the IT team to focus on and clarify what is important for the department within the context of what is important for the organisation as a whole;

§         When the measures have been identified and are in place the managers within the IT department will have a means of communicating to the team and senior management a clear framework for working towards the department’s goals;

§         In turn this will provide a means of influencing behaviour, and ensuring the right things are being done;

§         Having established this “route map” the IT department can check on an ongoing basis whether or not objectives are being achieved; and

§         Finally, the measurement data can be used to challenge the department’s strategy and how well it is integrated with the organisation’s mission.

Of paramount importance is to see each measure in terms of what Kaplan and Norton (2000, p69) describe as a “strategy map” i.e. each measure as “embedded … in a chain of cause-and-effect logic that connects the desired outcomes from the strategy with the drivers that will lead to the strategic outcomes”.

References

• Bourne, M and Bourne, P Understanding the Balanced Scorecard in a Week, Hodder and Stoughton, London (2000).

• Department of Health, Delivering 21^st century IT Support for the NHS (July 2002). Available from: http://www.dh.gov.uk/assetRoot/04/07/16/84/04071684.pdf

• Department of Health, Information for Health: An Information Strategy for the Modern NHS 1998-2005 (September 1998). Available from: www.doh.gov.uk/ipu/strategy/index.htm

• Department of Health, Building the Information Core: Implementing the NHS Plan (January 2001). Available from: http://www.nhsia.nhs.uk/pdf/info_core.pdf

• Department of Health, The NHS Plan: A Plan for Investment, A Plan for Reform (July 2000). Available from: http://www.nhs.uk/nationalplan/nhsplan.pdf

• Eccles, R G ‘The Performance Measurement Manifesto’, Harvard Business Review, (January-February 1991) pp 131-137.

• Kaplan, R and Norton, D P ‘The Balanced Scorecard – Measures that Drive Performance’, Harvard Business Review, (January-February 1992) pp 71-79.

• Kaplan, R and Norton, D P ‘Using the Balanced Scorecard as a Strategic Management System’, Harvard Business Review, (January-February 1996a) pp 75-85.

• Kaplan, R and Norton, D P Translating Strategy into Action: The Balanced Scorecard, Harvard Business School Press, Boston, MA. (1996b).

• Kaplan, R and Norton, D P The Strategy Focused Organisation – How Balanced Scorecard Companies Thrive in the New Business Environment, Harvard Business School Press, Boston, MA. (2000).

• Neely, A Measuring Business Performance – Why, What, How, Profile Books Ltd, London (1998).

• Neely, A, Richards, H, Mills, J, Platts, K and Bourne, M ‘Designing Performance Measures: A Structured Approach’, International Journal of Operations & Production Management, Vol. 17, No. 11, (1997), pp 1131-1152.

• Neely, A D, Mills, J F, Gregory, M J, Richards, A H, Platts, K W, Bourne, M C S Getting the Measure of Your Business, Findlay Publications, Holton Kirby (1996).

• Olve, N and Sjostrand, A The Balanced Scorecard, Capstone Publishing, Oxford, UK (2002).

• Olve, N, Roy, J, and Wetter, M Performance Drivers: A Practical Guide to Using the Balanced Scorecard, John Wiley & Sons, Chichester, UK (1999).

• Paulk, M, Curtis, B, Chrissis, M and Weber, C ‘Capability Maturity Model for Software, Version 1.1’ IEEE Software, Vol. 10, No. 4, (1993), pp 18-27.

• Wanless, D  Securing Our Future Health: Taking A Long-Term View, HM Treasury (2002). Available from:  http://www.hmtreasury.gov.uk/Consultations_and_Legislation/wanless/consult_wanless_final.cfm

Table 4: Performance measurement record sheet for IT performance measures

EJISE is published by Academic Conferences Limited
Curtis Farm, Kidmore End, Nr Reading RG4 9AY, England
Tel: +44 (0)1189 724148, Fax: +44 (0)1189 724691, Email: info@ejise.com

Send mail to info@academic-conferences.com with questions or comments about this web site.
Copyright © 2002-2006 Electronic Journal of Information Systems Evaluation
Last modified: September 29, 2005
ISSN 1566-6379