1.         Introduction

The dramatic expansion of the higher education sector in the UK has contributed to a significant increase in competition among organizations within the sector. With resources increasing at a much lower pace, the need to harness the effectiveness of core business processes has brought to the fore the importance of continuous monitoring of IT function through the systematic evaluation of computer based systems. There is a relatively small number of studies reported in the literature evaluating IS implementations in the higher education sector. Furthermore, in view of the lessons learned from the research project reported in this paper, it has become apparent that, rather than a reaction to a major crisis – non-adoption of a business critical IT system – evaluation methods should be used as a norm. Making it an incremental life-long learning experience is likely to help optimize IT investments for organizations in the sector. 

This case study focuses on the post implementation evaluation of a business critical IT system at Salford University. Based in the North of England, Salford University is one of the largest universities in the UK with regards to student numbers and programs of study. The purpose of the system is to manage student information throughout key business activities, that is, recruitment, admission, registration, invoicing, accommodation, assessment, progression, graduation and careers. This is a large centralized computer based system that can be accessed by all areas of the University. In response to the strong dissatisfaction expressed by potential users from across the organization, and their general reluctance to adopt the system, an in-depth evaluation of this substantial IT function was launched by Senior Management.

The project aim was to elicit users’ perceptions and overall attitude to the effectiveness of the Student Information System. It has been recognized in the literature that user information satisfaction significantly affects the success or failure of information systems. The methodology used to carry out the investigation was based on the Whyte & Bytheway framework for assessing an information system effectiveness. In a detailed analysis of user needs, they proposed twenty-one system attributes which most influenced users’ perceptions of the overall effectiveness. They also specified three elements relevant to the assessment of a system – the Product, the Service, and the Process (the management of the project).

The Product-Service-Process grid was used to analyse the Student Information System. The attributes associated with each element of the grid were then evaluated by the users. As part of the research method, a Gap Approach was taken to the measurement of those system attributes (importance and performance). Data were collected through a four-part questionnaire, mostly closed questions. Open questions were also included to ascertain users’ perceptions of the business case for developing the system in terms of achieving organizational objectives. A comprehensive evaluation of the system was then constructed by examining the individual attribute scores, the free responses, and the respondents’ profile. The sample included a wide variety of users across the University, who were then grouped into types. Gap analysis was performed on the whole sample as well as on each of the groupings, yielding interesting comparisons. The evaluation project was conducted over a period of six months during the year 2002.

One of the most compelling implications of the case study findings concerns the overall impact on IT project management in general within the organization. How well the users’ business needs are met by the Product and the Service attributes is dependent upon the Process. The research outcome suggests that the organization would greatly benefit from moving to a continuous participative evaluation activity integrated into the project management process.

2.         User evaluation of an information system – Conceptual framework

Projects are more likely to be successful where users do not become disillusioned, from having overly high expectations of a system that cannot be met. In order for users to perceive an Information System as a success, it is important for their expectations and perceptions to be managed effectively (Clegg et al, 1997) (Marcella & Middleton, 1996) (Lim & Tang, 2000). Whyte & Bytheway (1996) proposed a holistic approach to IS evaluation by specifying three core elements to a system: the Product, that is, hardware, software, and training provided to users; the Service, that is, how users are responded to; and the Process by which the Product and Service are provided. How well the business needs of the user are met by product and service attributes of the system is dependent upon the Process, i.e. the Management of the Project.

2.1       The product

Aspects of the product which users value are the quality information held, the design of the 'front end', the level of functionality, quality of training and quality of documentation.

2.1.1   The quality of the information held

The information held in an information system needs to be reliable and accurate, if the users are to have any faith in the system. When users cease to trust the data held in a system, they either stop using the system and create their own smaller systems, or they spend time checking the information outputs of the system against other sources. Miller & Doyle (1984) were able to identify: completeness, accuracy, flexibility and relevance of information outputs as major success factors for users.

2.1.2   The design of the front-end

The 'front-end' of the system, consists of the elements of the system that the users see and interact with, i.e. the forms, screens and reports. A well designed 'front-end' is likely to result in successful use of an information system. Users evaluate the design of the 'front-end' according to: its affect, efficiency, ‘learnability’, helpfulness and levels of Control (Oulanov & Pajanillo, 2001).

2.1.3   The level of functionality

The functionality of a system relates to how it performs various processes concerned with the business needs of the users. Clegg et al (1997) determined functionality and its consequential impact on business processes as one of the principle variables that influence users reactions to a new Information System.

2.1.4   The quality of training

The quality of training is vital to project success. It is the main way users learn about the system, and its quality will affect not only their successful use of the system but also their attitudes towards it. Riley & Smith (1997) observed how insufficient training could contribute to users 'resistance to change', and Clegg et al (1997) credited training as core to successful technological change.

2.1.5   The quality of documentation

User documentation of an information system usually takes the form of training manuals and user instructions. To achieve documentation of a high quality and of use to users, it should take a user perspective, by satisfying the varying skill levels among users, orientating and reassuring users (Nahl, 1999).

2.2       The service

Elements to the service provided which are important to users are concerned with user involvement, communication and response to their needs.

2.2.1       User involvement

Oulanov & Parjrillo (2001) emphasised the importance of user participation in system planning and design as significant factors in their perceiving project success. Cicmill (1999) cited a case where the non-inclusion of users in the requirements gathering process lead to suspicion and lack of co-operation from users, resulting in a system that was unsuccessful in meeting their information needs. Conversely Middleton (2000) was able to cite cases of comparatively successful projects, where project workers and users shared the same office space, there by enabling a development of trust and shared vision between them.

2.2.2   Communication and response

Communication and responsiveness are extremely important in an information technology project as they affect users perceptions and expectations, allow them to keep up to date with project developments and provide an environment where their views can be heard. Collyer (2000) cited the example of a successfully designed and implemented business processes and IT system at Guinness, where the IS director of the Guinness Global Support Team said communication was "key" and "leadership from top was excellent". (Middleton, 2000) observed successful projects, where supportive atmospheres were created allowing communication to flourish and users to express themselves freely.

2.3       The process

The process by which a project is managed is vital to successful projects, regardless of whether users directly perceive its affect on themselves and the project. Management of projects is not a simple process, often the change achieved is more emergent than planned and although managers may have goals at the beginning of a project, the actual outcomes are usually quite different (Boddy, 2000). Managers need to be aware of the issues surrounding users. When introducing new technology into a volatile environment, there is a need for flexibility (Boddy, 2002) and for service quality standards to be set (Pitt et al, 1998). Whittaker (1999) reported how poor project planning, especially a weak project plan and inadequate risk management, caused project failure.

2.4       Attributes related to product-service-process

Whyte & Bytheway (1996) were able to carry out a more detailed analysis of User Needs and to determine a set of attributes related to the Product, the Service and the Process discussed previously. In this study, Whyte & Bytheway’s grid of attributes was adapted and subsequently used in determining a comprehensive measurement of the users’ evaluation of the Student Information System. Some of Whyte & Bytheway’s attributes were split into several other more detailed attributes, a few were discarded as deemed irrelevant, and others were renamed to match the current users’ jargon. The grid of attributes used in the system’s evaluation is presented in Table 1.

Table 1: Grid of attributes used in the system’s evaluation

Attributes 1-5 are associated to the Process, attributes 6-16 are associated to the Product, and attributes 17-24 are associated to the Service.

User information satisfaction can be measured by comparing the level of support users expect from a system against their perceptions of the actual level of performance of the system, that is, analyzing the distance between the two (gap analysis). It is a holistic approach as it considers the various elements of the information system (Remenyi et al, 2003). With gap analysis, when there is a positive gap performance has exceeded the perceived importance of the individual elements of the system, and it could be argued that there has been a waste of resources. When there is a negative gap, the performance of the different elements of the system do not meet the perceived needs of the users and, therefore, can be identified as areas for improvement and development. For these reasons it is a reliable method of assessment of an Information System, as it is able to identify the areas where the system is failing (Remenyi et al, 2003).

Pitt et al (1998) argued that managers could gain much from adopting a service gap approach in the management of user information satisfaction. Wisniewski (2001) reported how Parasuraman et al (1985, 1986, 1988, 1991, 1993, 1994) were the forerunners in the development of Gap Analysis with their SERVQUAL model. This uses twenty-two statements to assess service quality by using each statement twice, first to measure expectations and second to measure perception. Parasuman et al claimed that SERVQUAL the information gained from the service gaps could help managers decide where performance improvements need to be targeted.

In this study, twenty-four statements were employed to assess users’ information satisfaction. Each statement was used twice, first to capture users’ ratings of importance, and second, to measure users’ ratings of performance. The system’s attributes were thus rated in importance and in performance, the gap (difference) then being computed. The terms ‘importance’ and ‘performance’ were used in the data collection, rather than the terms ‘expectation’ and ‘perception’. This is because, as it was felt, the terms ‘expectation’ and ‘perception’ are not specific enough for conveying the users’ evaluation of the system.

3.         Data collection

The data was collected through a questionnaire. It was initially piloted to a small group of users before being distributed for data collection. In total 84 users responded to the questionnaires. The final version of the questionnaire had four sections. Section A presented twenty-four statements, each one related to a system attribute. Users were asked to rate the importance of each system attribute on a five-point scale. In Section B, which presents the same twenty-four statements, users were asked to rate the performance of each system attribute on a five-point scale. The scales are described in Table 2.

Table 2: Scale for evaluating the system’s attributes

In Section C, users were asked to rank the overall performance of the system on a five-point scale (the performance scale). They were also given the opportunity to add their own personal comments in response to an open question. In Section D, users were asked to indicate their profile in terms of position at the University, division of affiliation, number of years of computer experience, frequency and range of use of the system. Table 3 and Table 4 summarize the users profile.

Finally, also in Section D, users were asked to answer two open questions. The first question was to ascertain users thoughts on why the Student Information System was being implemented. The second question was to investigate the overall reaction to the system. Questions were deliberately broad in scope (“What do you think the Student Information System is?”) to encourage a free and indirect response.

A framework based on Grounded Theory (Glaser, 1967) was used to analyse the rich data collected from the open question in Section C on the users overall opinion of the system. References to the individual attributes were given text and number labels against the transcripts of the question answers and coded according to whether they were positive or negative statements. The frequency of the negative and positive codes were then calculated to give an indication of the attitudes users had towards the different elements of the system and which they were most concerned about. Themes relating to the users positive or negative answers were then coded and grouped against the attribute statements according to whether they were a cause of or result of the users negative or positive answer. The frequency of these codes were then calculated as an indicator of the prevailing causes of and results of the user opinion.

Table 3: Users Profile

Table 4: Range of system use across functional areas

The data on the users’ understanding of the system was collected through two open questions in Section D. A framework based upon Grounded Theory was again used to analyse this data. Themes were established by breaking the data down into its component parts and labeling the emerging issues against the transcripts of the question answers. The emerging themes were then examined for their relationships and groupings. The text labels were then given number codes and the frequency of the codes calculated to give an indication of the themes dominantly occurring.

4.         Data analysis

Table 5 shows the mean score of importance for each attribute, the mean score of performance for each attribute, and the resulting gap between the two. The standard deviation around the mean was computed as an indicative measure of the users consensus on a particular score. The relative rank of each attribute score on importance and performance were also included in the table, as an aid to assess the gap.

The mean Importance score for all attributes was 3.6, which is interpreted as between 'Don't Know' and 'Important' on the Importance scale. The mean Performance score for all attributes was 2.3, which is interpreted as between 'Poor' and 'Average' on the Performance scale. This evaluation did not meet the level of consensus among users that was expected, but relatively convergent in the light of users’ profile.  Given that the standard deviation ranged between 1.3 and 1.8 on Importance scores, and between 1.3 and 1.6 (with one exception computed at 5) on Performance scores, it indicates a variation of two points on a scale of five used to evaluate the system’s attributes. A very high standard deviation of 5 was found for the ‘User Involvement’ attribute score. This lack of consensus, when analyzed in the light of the rich data (open questions), pointed to the need of managing users’ expectation during the project.

Table 5: Attribute Importance Mean and Attribute Performance Mean for all respondents