Introduction
This paper proposes a framework for the evaluation of Information System (IS) innovation projects. By innovation projects we mean those projects that involve adopting technology which allows an organization to do things in new ways and where the organization's prior experience of the IS products or services is low. These projects exhibit a tight, iterative coupling between IS design and use and closely follow an innovation process. Our observation of these types of project revealed that the methods used in their evaluation differed in several respects from those described in the current literature. Evaluation processes described in the literature are broadly external to the project; yet, we observed processes which were both tightly integrated within the project and highly dynamic - a kind of "bootstrapping" process of learning. Evaluation in the literature implies that once a system is developed and implemented it is "good" if it requires little subsequent change; our observations suggested that in innovation projects it is often not until practical experience in using the system has been gained that the requirements are really understood. The evaluation literature says little about changes in the underlying technology products, implicitly assuming them to be stable; we observed that not only were products changing but that organizations sought opportunities to influence the direction of change to be favourable to themselves.

In the following sections, a framework for the evaluation of IS innovation projects is developed and applied to a case study of electronic data capture for clinical trials at Glaxo Wellcome Research & Development.

Key features of an "IS innovation project"

There are three key features of an IS innovation project:

· The project follows an innovation process, with its three phases of initiation, adoption and implementation (Rogers, 1983). (The key point here is the need for evaluation techniques to place emphasis on the implementation phase - traditional techniques tend to view implementation as the final step in the process which is hardly elaborated upon).

· There is tight coupling between the design and use of the IS product or service (Orlikowski 1992).

· The organization's prior experience of the product or service is low.

We do not restrict innovation to early adopters: if the technology applied is new to the organization, it is considered to be an innovation even if it is well established elsewhere (Rogers 1983; Swanson 1994). However, we do restrict the scope to the consideration of new IS products or services: new IS work technologies or administrative arrangements are not included.

The purpose and timing of evaluation in IS innovation projects
Evaluation in innovation projects may be carried out for the same purposes as in conventional projects, i.e. for justification, as part of project control or for the purposes of learning for future projects or for future evaluation (see for example Farbey et al. 1993). However, in IS innovation projects, the purposes are extended to include evaluation of the development of organization-technology fit - this type of evaluation can be thought of as a mechanism for coupling between design and use and a tool for continuous learning and improvement. Whilst project justification remains important, a more incremental approach is taken with a number of separate decisions, such as those associated with funding feasibility studies, setting up of pilots, initial implementation and subsequent roll-out. Decisions are also likely to be subject to review as circumstances change.

Project monitoring and control and post-project review will both be similar to that for conventional IS projects but the underlying project process will, of course, be different.

The evaluation life-cycle for traditional and innovation projects is similar in the very early stages and during routine operational use. However, significant differences occur during the development and implementation phases which are separate in the traditional life-cycle but tightly integrated in an IS innovation project.

Figure 1 summarises the purpose and timing of evaluation in IS innovation projects. Each type of evaluation is discussed in more detail in the next sections of the paper.

Developing organization-technology fit

Initial evaluation
Initial evaluation occurs at the agenda-setting stage of the IS innovation process and may be driven either from the perceived need for a solution to a specific business problem or from the awareness of a potentially valuable new IS product or service.

In the problem-focused case, the first target of evaluation is likely to be the development of a clearer understanding of the problem. Thereupon, an active search for solutions can be undertaken and the focus of evaluation will then move to the development of "awareness knowledge". This consists of "principles knowledge" and "how-to knowledge" for each potential solution (Rogers, 1983).

In the technically-focused case, awareness of a new IS product or service may occur:
· through routine monitoring of technological advances
· as a result of publicity or contact from potential providers of a particular product or service
· through monitoring of competitors or as a result of routine professional contact

Figure 1: The purpose and timing of evaluation in IS innovation projects

The focus will then move to identifying and developing an understanding of relevant business problems that the product or service may address.

In both of these cases, the information gathered may be used to justify funding for the next stage of the project, possibly allowing for trialing of the most promising solutions. Evaluation is likely to be iterative, with the emphasis shifting between internal and external foci as knowledge of both the business problem and potential solutions increases.

Assessment, comparison and selection of options

At the matching stage, the fit between organization and technology starts to be evaluated. Again, the focus is on information gathering. In the problem-focused case, a number of possible solutions may be matched against the business problem. In the technically-focused case, potential solutions may be matched against a number of problems to determine which, if any, should be further investigated.

Evaluation will assess the feasibility of each solution, its flexibilities and constraints, risks and opportunities and any obstacles to implementation. Stakeholder groups and potential costs and benefits will need to be identified and understood. The purpose of evaluation will be to eliminate infeasible options and to compare alternative viable options to select the best to take forward to the adoption phase. Piloting can be particularly valuable.

Although at a high level the IS product or service to be implemented will have been selected by the end of the initiation phase, it is likely that for some sub-components of the project alternative options remain. Thus, this type of evaluation is likely to continue, but at a more detailed level, into implementation. There may also be subsequent re-evaluation of options in the light of experience in use.

Continuous learning and improvement

As the project moves into the redefining/restructuring stage of implementation, the fit between the organization and the technology continues to be evaluated and developed. Information is actively sought as more “principles” and “how-to” knowledge is acquired through experience in use. Not only may the product or service be adapted to better fit the organization, but the structure of the organization and its ways of working may have to be changed to accommodate the new product or service: evaluation can be thought of as a feedback mechanism between use and design.  If significant gaps in fit are found (these could arise through changes either internal or external to the project or be a result of unanticipated problems experienced in the field), it may be necessary to re-evaluate the options.

At the clarifying stage, as the product or service is put into wider use, the fit between organization and technology should be well defined, although there may be minor improvements to be made. The focus moves to supporting larger scale use. Any misunderstandings, unwanted side effects or barriers need to be quickly identified and assessed so that corrective action can be taken.

At the routinising stage, the product or service is in wide use within the organization. By this stage, evaluation for  the purpose of developing organization-technology fit will have effectively ceased. 

Project justification and decision-making

Much of the information gathered in the development of organization-technology fit will be a source of input to project justification and decision-making. This is an on-going process recognising that the information upon which decisions are made will change over time.

In the agenda-setting and matching stages, prior to any decision to adopt a particular solution, decisions to experiment with or trial particular options will be made on limited information. Justification for a particular course of action may be primarily based on information buying or risk reduction at relatively low cost. As the project progresses, however, and more information becomes available this will be used to develop the business case. 

By the end of the initiation phase, all necessary information should have been gathered to support the decision on adoption of an IS product or service, to plan for its initial implementation and to justify the necessary funding. The primary purpose of evaluation at this point is to provide the necessary support to management to enable them to take the adoption decision. This is not necessarily a straightforward choice between adoption and rejection of the IS product or service for several reasons:

·        Although one course of action may be preferred and recommended, alternatives may need to be considered

·        It may be felt that the time is not right to move forward into implementation. This may be because more information is needed or because the plans for implementation are not yet sufficiently developed i.e. the project effectively moves back to initiation. Alternatively, if the products being considered are undergoing rapid changes, a “wait and see” strategy may be better.

·        There may be choices in scale of implementation - too large and the project can become difficult to manage and control; too small and the full benefits will not be realised.

The timing of the adoption decision is critical to success. If it is taken too early, with insufficient knowledge, there is a risk that the product or service will be misused and will consequently fail leading to disenchantment and loss of confidence. If the decision is taken too late, the costs will be higher and the achievement of potential benefits delayed.

An important difference between IS innovation projects and conventional IS projects is that adoption decisions are likely to be subject to subsequent confirmation.  This may be a pre-planned review, a response to changing circumstances external to the project or as a result of problems during implementation. As a consequence, evaluation will continue for the purpose of justification of the project. As the project moves through the implementation phase, performance targets may be set and a performance measurement programme introduced. As experience grows and evidence is gathered, earlier assessments of costs and benefits will be refined to strengthen the business case.

Project monitoring and control and post-project review

Evaluation for the purpose of monitoring and control of the project will start in the matching stage, as experimentation with or trialing of potential solutions is undertaken and associated costs start to be incurred. An important aspect of evaluation at this stage is to ensure that the specific objectives of experimentation are met and that the necessary principles knowledge is gathered.

At the redefining/restructuring stage, monitoring and managing the implementation process is particularly critical: making sure that information is communicated to the right people, that processes and working practices are correctly followed, technical aspects run smoothly, etc. Performance targets and measurement programmes may be instigated.

At the clarifying stage, the focus moves to monitoring performance targets and supporting the management of larger scale use. Any unwanted side effects or barriers to meeting targets need to be quickly identified and assessed.

At the routinising stage, the product or service is in wide use within the organization and the project will be drawing to a close. Once the project has been completed, a post-implementation review is likely to be conducted. As with conventional IS projects, this will be primarily concerned with product and process review (Earl 1989) .

An illustrative example: Electronic Data Capture for Pharmaceutical Clinical Trials

To illustrate the evaluation life-cycle for IS innovation projects we present a case study carried out in Glaxo Wellcome Research and Development (GWRD) and, prior to the merger in 1995 of the Glaxo Group with Wellcome plc, in Glaxo Research and Development.

The case study describes the Electronic Data Capture (EDC) project, a major project to implement electronic data capture tools in clinical trials. Clinical trials data have traditionally been and still are largely collected on paper, using case record forms (CRFs), which are completed by the investigator, or diary cards, which are usually completed by the patient. Speeding up the clinical trials process is a key goal for Glaxo Wellcome, in common with most other pharmaceutical companies and EDC is seen as an important factor in achieving this goal.

An overview of the EDC project

The start of the EDC project can be traced back to 1993 in Glaxo Research and Development, prior to the merger with Wellcome, when a small team was set up in the UK Medical Operations Division to review options for electronic data capture.

One of the first actions of the team was to commission an in-house review of emerging technologies. The purpose of this was two-fold: firstly, to identify available options and, secondly, to gain an insight into the way other companies were implementing these technologies. Subsequently, the team focused on fax technology and combined CRF/entry screen design as areas offering most potential for significant business benefit. One electronic CRF product, based on a pen-computer, was demonstrated  but it was felt that the technologies used, in particular handwriting recognition, were not sufficiently mature at that time. The team reported their findings in early 1994 and recommended that implementation proceed only for fax-based CRF return but that a small pilot of an eCRF system also be considered. A pilot implementation of the pen-computer based system was subsequently approved.

The clinical trial chosen for the eCRF pilot was conducted during the second half of 1994 in two European countries at 13 investigator sites. The pen-based system was found to be unsuitable for use in its current form but the feasibility and potential benefits of eCRF systems in principle were clearly demonstrated.

At this time, a process improvement project was also underway in Medical Operations. Electronic data capture was identified as an important target and was given a high priority. The decision to go ahead with the next phase of the data capture project was taken at a senior level before the pilot had been completed. In October 1994, the emphasis was on putting in place a ‘dynamic framework’ of data capture tools, based on off-the-shelf components. Bespoke systems development, either in-house or contracted out, had been ruled out early on, so commercially available products were evaluated and the most promising were implemented and evaluated individually in live clinical trials.

By August 1995, when interviews for this case study commenced, the focus of attention had moved to eCRF systems, which were seen as offering the ability to shorten clinical development programmes by several months. Products from three different vendors were being used in four clinical trials, ranging from a short, single-site clinical pharmacology study in the US to a major, 18-month-long Phase III clinical trial conducted in 50 centres in six European countries. The major objective at this stage was to determine how to select and scale up on one eCRF system for ‘production use’ (the term used by the EDC project team for routine use within the organization).

In October 1995, a meeting was held to review the key lessons learnt from the project’s eCRF experience to date, to select a preferred software vendor and to plan the move into large-scale production use of eCRFs. It was also decided to contract out the hardware services associated with configuring, installing and supporting portable PCs in the field. Following the review meeting, a proposal for funding for the first two years of the production phase of the project was prepared and finally approved in May 1996.

A contract for the provision of software and software services was finalised with the preferred eCRF software vendor in May 1996. The hardware service provider was selected in August 1996 and the contract was signed in November 1996.

In August 1996, the first of a series of joint US/UK project meetings was held to prepare for large-scale implementation of the selected eCRF system. At the centre of this was the development and documentation of international processes and working practices for the use of eCRFs. Subsequent meetings in November 1996 and March 1997 extended the objectives to include the development of training and support programmes and a programme for routine evaluation of EDC.

Also in the second half of 1996, the first clinical trials were selected to use the contracted-out software and hardware services and the new processes and working practices. Initially it was anticipated that the selected eCRF system would be implemented in 100% of Phase I and 50% of Phase II, III and IV trials by the end of 1997. This was reassessed as both the internal and software vendor capabilities for scaling up became apparent and the target for the first year was revised to 50 clinical trials in 1000 sites. Experience was captured on an ongoing basis both through informal communication between members of the EDC team and the project teams and through the routine collection of metrics and user feedback.

At the end of 1996 the next phase of the Medical Operations process improvement project was initiated and EDC became a component project within Core Process Redesign (CPR). This transfer of responsibility prompted a review of EDC within the broader context of the redesign framework: the project justification was reconsidered and the validity of strategy for implementing the eCRF system was re-examined and confirmed.

Over 1997, the view of the selected eCRF solution changed as experience grew and its limitations were recognised. The strategic focus within the organization moved to “next generation” EDC and the investigation of future EDC solutions became a separate project. 

The primary focus of the EDC project became the implementation of the selected eCRF solution as tactical tool for use in the short term and as a way of building skills and knowledge in preparation for the piloting and subsequent scaling up of future EDC technologies. Implementation proceeded steadily over the year and by December 1997 the eCRF solution was implemented in between 10% and 15% of clinical studies.

In parallel with the implementation activities, an EDC Transition Team was set up in May 1997 to make recommendations on how the management and use of the selected eCRF solution should move from the EDC project team to integration within Glaxo Wellcome line functions. The team reported in January 1998 and its recommendations were approved.

Overall, fourteen major evaluation exercises were identified in the EDC project and these have been mapped against the primary evaluation purpose in Figure 2.

Evaluating organization-technology fit

Of the major evaluation exercises identified, nine were primarily concerned with the evaluation of organization-technology fit. Although in most cases the information gathered was also used to support evaluation for other purposes, this still gives a clear indication of the importance of this kind of evaluation.

**Figure 2: Mapping of EDC evaluation exercises against evaluation purpose


Evaluation exercises:

Initial evaluation

The initial review of emerging technologies was driven primarily from the identification of the capture and entry of clinical data as the remaining target for potentially useful exploitation of IT, given that data processing and reporting for clinical trials had already been addressed. There was also an awareness that the majority of other major pharmaceutical companies were already active in this area and that a number of commercial products capable of conferring significant business benefit were becoming available. The purpose of the exercise was “to review options and to produce a proposal for the integrated use of appropriate technologies” within a time frame of 5 months, to fit in with the organization’s internal budget cycle. This involved identifying and developing “awareness knowledge” of potential solutions, either commercially-available products, in-house development or collaborative ventures with other pharmaceutical companies.

An important component of the review was understanding how other companies were using these technologies and learning from their experiences.

Assessment, comparison and selection of options

There were four major evaluation exercises which were initially conducted for the purpose of assessing, comparing and selecting options within the project: the evaluation of pilot use of the pen-based eCRF system, the piloting of alternative eCRF products, the “production review” and the selection of the hardware service provider.

The objectives of the pen-based eCRF pilot were again focused on information gathering:

·        to identify by how much the technology could shorten time to market;

·        to assess the effects of the technology on working practices;

·        to determine the acceptability of the technology in the clinic; and

·        to consider options for pen-based CRF processing and their associated costs and benefits.

In the second evaluation exercise, three different eCRF systems were piloted in four clinical trials. The focus of these pilots was on identifying potential issues around scaling up.

The primary objective of the production review, which drew heavily on the information gathered in the previous pilots, was to select one or more preferred eCRF software vendor for “production use” of EDC. Other objectives included the development of scenarios for working processes and use of technology, determining implementation plans, considering the further evaluation of new technology (especially for non-CRF data) and reviewing the key lessons from the pilots. The production review was held jointly with US and UK representatives and a great deal of time was spent ensuring that there was a common view of the technology within the team through sharing information and agreeing selection criteria.

Following the production review and the identification of a preferred eCRF software vendor, the process of selecting a hardware service provider got under way. Initially, the service requirements were defined at a fairly high level, but these were specified in greater detail as the selection process progressed and understanding increased through discussion both within the EDC team and with the potential service providers.

Finally, there was one additional evaluation exercise, the identification and assessment of electronic systems for the capture of diary card, quality of life and other questionnaire data, which was conducted in a sub-component of the project determining EDC options for non-CRF data. This was carried out at a later date when the implementation of eCRFs was well under way, with the objectives of defining requirements for different types of diary cards and questionnaires and of identifying the most promising potential solutions. 

Continuous learning and improvement

As the EDC project moved into production use, a programme for the routine collection of standard metrics and user feedback from eCRF studies was set up and data was collected on an ongoing basis for more than a year. Two metrics reviews were conducted which drew on and summarised the evaluation data. These exercises were all strongly focused on learning for improvement. Objectives for the routine collection of metrics and user feedback included measuring the impact of eCRFs on clinical trial timelines and resources, collecting feedback from various stakeholders and making recommendations for improvement. The objectives of the first metrics review were to take a critical look at the current status of eCRF implementation, to demonstrate the successes and to identify the issues being addressed with the overall aim of providing a baseline against which improvement in the use of eCRFs could be measured over time. The second review had similar objectives, concentrating on changes since the first review.

Later, as the EDC Transition Team was planning for the transfer of the eCRF system into routine use, helpdesk and hardware and software service provision was reviewed with the objectives of revising the services to match current needs, which had changed significantly since they were first implemented.

Project justification and decision-making

An incremental approach was taken to justification of the EDC project.  Much of the supporting information came from the exercises concerned with the development of organization-technology fit.

In the early stages of the project, justification of the early fax and pen-computer pilots and the subsequent pilots of alternative eCRF products was primarily an ‘act of faith’, recognising that the necessary information was not yet available to build a solid business case. In applying for funding for the pilots, the emphasis was primarily on estimating costs, not benefits. Although some potential benefits were identified, it was not possible to quantify them and an important objective of the pilots was to acquire the information and understanding necessary to develop the business case.

The adoption of the preferred eCRF product for ‘production use’ was based on a capital expenditure proposal which was reviewed and approved through the standard internal GWRD mechanisms in May 1996. The proposal was supported by a cost-benefit analysis based on reduced clinical development time and hence earlier regulatory submission and ultimately product launch.

The transfer of responsibility for EDC to CPR prompted a review of the project in early 1997. The project justification was reconsidered and the validity of implementing the eCRF system was re-examined and confirmed. The routine evaluation of the early eCRF trials played an important part in the review by providing evidence showing that expectations were being met and an active programme for continuous improvement was in place. CPR placed particular emphasis on the need for a stronger cost-benefit case as the business climate within GWRD changed.

In the second half of 1997, the EDC Transition Team re-evaluated the eCRF business case and extended the metrics collection to develop a clearer understanding of the costs and benefits incurred in using the eCRF system compared to paper-based studies and CRO costs. The Team also developed tools that could be used to support decision-making on the use of eCRFs for future studies, drawing on the re-evaluation and on earlier work on eCRF study selection criteria.

Evaluation for justification and decision-making occurred throughout the life-cycle of the project; as more information became available, understanding of the implications of EDC increased and the business case developed to reflect this.

Project monitoring and control and post-project review

Much of the evaluation conducted primarily for the development of organization-technology fit also provided information which was used to support project monitoring and control. This was particularly true of the routine collection of standard metrics, which included data on resources, timelines and costs incurred as well as on the realisation of benefits which could be compared with previous estimates and used to monitor project progress.

By February 1998, when the last case study data were collected, the project team was being disbanded and the recruitment of staff for the new organizational structures was under way. Evaluation was planned to continue with the objectives of routine performance monitoring and providing information and support for decision-making on data handling strategies. At this time, no formal post-project review was planned.

Conclusion

In our view, the evaluation of IS innovation projects requires a different set of methods to those used for conventional projects. This requirement stems from the fact that the underlying project processes are different.

A critical point about an IS innovation project is that often a full understanding of requirements can only be developed through practical experience gained as the project evolves - the project process is complex and iterative and evaluation is an integral part thereof. Further, the underlying technology may be changing and this, together with any opportunities for influencing the direction of change, needs to be understood and accommodated.

The development of organization-technology fit lies at the core of IS innovation projects. Evaluation for this purpose can be thought of as a coupling mechanism between use and design and as a tool for continuous learning and improvement. It also provides information to support evaluation for the traditional purposes of project justification and project control. The importance of evaluating the fit between organization and technology on an ongoing basis from project initiation through to routine operation is clearly illustrated in the case of Glaxo Wellcome’s EDC project. Of the fourteen major evaluation exercises identified in the project, nine were primarily conducted for this purpose. 

This leads us to the conclusion that in developing methods for evaluation of IS projects we need to draw more extensively on the established body of academic literature in the field of innovation, and extend the existing methods of evaluation to reflect more specifically the requirements of IS innovation projects.

Acknowledgement

The authors would like to thank Glaxo Wellcome R&D for their support in sponsoring this research.

References

• Earl MJ (1989) Management Strategies for Information Technology. Prentice Hall, London.
• Farbey B, Land F and Targett D (1993) How to Assess your IT Investment. A Study of Methods and Practice. Butterworth Heinemann, Oxford.
• Orlikowski W (1992) The duality of technology: rethinking the concept of technology in organizations. Organization Science 3(3), 398 - 427.
• Rogers EM (1983) Diffusion of innovations. 3rd edition. The Free Press, New York.
• Swanson EB (1994) Information Systems Innovation Among Organizations. Management Science 40(9), 1069 - 1092.

EJISE is published by Academic Conferences Limited
Curtis Farm, Kidmore End, Nr Reading RG4 9AY, England
Tel: +44 (0)1189 724148, Fax: +44 (0)1189 724691, Email: info@ejise.com

Send mail to info@academic-conferences.com with questions or comments about this web site.
Copyright © 2002-2006 Electronic Journal of Information Systems Evaluation
Last modified: November 25, 2004
ISSN 1566-6379