1.     Introduction

There is sufficient evidence in the Information Technology (IT) literature to suggest that IT system users are being excluded from the IT evaluation process. Research by Ballantine and Stray (1998) and Lycett and Giaglis (2000) suggests that most IT investment decisions use quantitative or financial based IT evaluation methods. Such methods usually have a limited definition of stakeholders, typically target direct tangible costs and benefits, and are based on accounting and financial instruments such as Net Present Value and Internal Rate of Return (Farbey et al., 1995). Also, there is evidence to suggest that IT investment decision-making frequently result in ‘gut feel’ or ‘acts of faith’ type investments, ignoring the use of any IT evaluation methods (and IT system users) (Kaplan, 1984). The problem here is that traditional methods tend to ignore human and organisational components of IT system users, focusing instead on the technical side and direct costs (Serafeimidis and Smithson, 2000). Such IT evaluation practice run the risk of not identifying all the hidden costs and 'softer' benefits generated from IT system users, which may push the overall IT budget to amplify up to four times the initial investment (Hochestrasser, 1992). In addition, undetected benefits will not allow the system to reach its full potential (Pennington & Wheeler, 1998). Significantly, such practices put the organisation at risk of partial or complete IT failure (Pouloudi and Whitley, 1997). However, growth in global IT spending, $3 Trillion forecasted by 2004, representing a 33.3% increase from last year (WITSA, 2000), has added pressure on decision makers to better justify their IT investment decision, thus, fuelling the interest in IT evaluation.  

This paper argues for a comprehensive IT evaluation process that includes the participation of IT system users. To that end, IT evaluation is defined and discussed to reflect the important role of this process in the organisation, followed by an overview of stakeholder theory, and haw it reflects on IT evaluation. This paper also includes a case study recently conducted by the authors. It focuses on the organisational use of IT evaluation and how it incorporates the participation of IT system users.  It is important to note that IT evaluation in the case study represents pre-implementation evaluation. This paper recognises that there are several plausible definitions of stakeholders. However, for the purposes of this paper, primary stakeholders are defined as system users, and secondary stakeholders are defined as clients, suppliers, and organisation staff not using the system.

2.     IT evaluation

IT evaluation may be defined as establishing by quantitative and/or qualitative methods the value of the IT to the organisation (Farbey et al., 1993; Willcocks and Lester, 1994). Although this general definition is widely accepted in the IT industry, the very same authors argue that value is a troublesome term to define. The debate is most evident form the many IT evaluation methods developed to represent different interpretations of IT evaluation. For example, Renkama (1994) listed 60 evaluation methods available for the organisation, which were developed over the last 30 years. Farbey et al., (1993) classified a number of IT evaluation that range for the most basic quantitative methods that use tangible/direct costs and benefits, to purely qualitative methods that account for intangible/indirect cost and benefits form the organisational and human perspective. Furthermore, she indicates that IT evaluation can be used in the justification stage of the IT investment (pre-implementation evaluation), after the system has been implemented (post-implantation evaluation), and during the entire life cycle of the organisation (continuous evaluation). A taxonomy of IT evaluation methods was also proposed by Irani et al. (1997). Understanding the many roles of IT in the organisation helps researchers better define the meaning of IT evaluation,

It is widely accepted in the IT industry that no single IT evaluation method can be applied to all situations (Khalifa et. al., 1999).  Instead Farbey et al. (1993) suggests that IT evaluation can contribute to the success of the IT system when the appropriate method is applied to the appropriate organisational context. Although, she admits there is no comprehensive agreement between researchers on the definition of appropriate, Farbey proposes a matrix that matches the project to an IT evaluation method. It is worth noting that Farbey et al does not claim that this matrix would actually work in industry, but nevertheless argues for its importance.  Pouloudi and Serafeimidis (1999) also support the view of matching the IT evaluation approach with the culture of the organisation in order to achieve successful development of the IT system.

 

Development and proposed application of IT evaluation by researchers methods continues to be a point for discussion, driven by debate of the many roles that IT can assume in an organisation (Willcocks and Lester, 1999). For instance, evaluation can be used to rank alternative IT systems under consideration (Clemons, 1991), is used for planning and strategic IT decision making (Hawgoog and Land, 1988), and has feedback and learning applications (Baker, 1995; Walsham, 1993). Furthermore, IT evaluation can be used to ascertain by qualitative and/or quantitative techniques the value of the system to the organisation (Willcocks, 1994), is used to gain commitment in political surroundings for legitimisation (Powell, 1992; Farbey et al., 1995), and provides insight of the interface between the technology and the fundamental organisational procedures, customs and politics (Symons, 1993).

With such a wide array of roles, IT evaluation is debatably one of the key issues in the IT academic and business arena, and will continue to draw discussion due to the following factors:

• IT systems are taking on greater roles in the organisation, being applied closer to core functions, and have the potential to change the business process (Lycett and Giaglis, 2000)

• A large portion of new invested capital is used for IT investments, increasing pressure on decision makers to better justify their position (Strassman, 1985)

• To improve efficiency and competitiveness management needs to assess the impact of the IT system on its organisational performance (Clemons, 1991, Farbey et al., 1992)

• Evaluation provides feedback and facilitates organisational learning (Smithson and Hirschheim, 1998)

• Finally, benchmarking can be a product of evaluation, useful for future project development and as a measure of success (Farbey et al., 1992)

3.     Stakeholder theory

Donaldson and Preston (1995) indicate that the most complete definition of stakeholder theory should describe the organisation as having an individual set of values that provide an environment from which cooperative and competitive interests exist. Also, stakeholder theory should link (when possible) stakeholder management organisational goals. Finally, stakeholder theory should accept that stakeholders represent a person (or groups) that has a real stake in the organisation and its well being. It is important to note that as with IT evaluation, no single stakeholder theory can fit all organisational situations. Therefore, while defining stakeholders of the IT system, IT evaluation may assume one definition of stakeholders over another, or ignore one group (IT system users) and involve another (Ex: top management). Table 1 lists different stakeholder definitions developed over the years from both the management perspective and the IT perspective to reemphasis the differences and similarities between them.

The wide array of stakeholders presented in table 1 reflect the importance of having an open mind to which stakeholder groups should be included during the IT evaluation process (Daniel et al., 2000. Lyytinen (1988) argue for a stakeholder definition that includes IT system users during IT evaluation. This researcher claims that the exclusion of IT system users form the design of the system will result in user failure. According to Lyytinen (1988), such a system will suffers from serious defect, performance problems, or lack sufficient functionality. Furthermore, she suggests that user failure will take place if the system is implemented but not used. In addition, Remenyi (1997) suggested that people failure can occur if there are serious difficulties in using the system, or if the morale among users is low and staff turnover is high, or if the organisation only allocated one or two days a year and is mostly technical in nature.

Table 1: Normative Stakeholder Definitions form Management and IT Perspectives

Management Literature Perspective	Reference	IT/IS Literature Perspective	Reference
‘A stakeholder in an organisation is (by definition) any group or individual who can affect or is affected by the achievement of the organisation’s objectives’	Freeman, 1984	‘All those who have a partial concern for the effective application of new technologies, and who are in a position to take of influence decisions about why and how they are used’	Boddy and Buchanan, 1986
Those that’ will be directly impacted by decisions’	Friend and Hickling, 1987	‘People who will be affected in a significant way by, or have material interests in the nature and running of new computerized systems’	(Willcocks and Mason, 1987
‘Groups of constituents who have a legitimate claim on the firm’	Hill and Jones, 1992	‘ Every thing within the organisation that lies beyond the borders of the IS department’	Lederer  & Mendelow, 1990
Those who ‘hold a stake’ in the decisions of the organisation	Wallace, 1995	‘The stakeholders are a group of people sharing a pool of values that define what the desirable features of an information system are and how the should be obtained’	Ahn and Skudlark 1997

 

Beynon-Davies and Blyth  (2000) claim that IT failure is commonplace, and that it would occur if the system did not meet stakeholder (specifically: system user) expectations. Furthermore, they claim that such failure would be avoided if stakeholders were involved more in the systems development process. Consequently, the work of Lyytinen, Pouloudi, and Remenyi (among others) point to the need of including stakeholders, and specifically system users, in the IT evaluation process. Research into the nature and role of such stakeholder groups is quite extensive, and has produced several models to classify them including Gilbert et al. (1988), Earl (1989), and Farbey el al. (1993). These models explain why, and preach that system users are a vital part of system design and evaluation. The models are included in appendix 1 to 3 respectively.

4.     IT evaluation and stakeholders

Research (Sauer, 1993; Flowers, 1997; Butler and Fitzgerald, 1999) aimed into IT has continuously endorsed the concept of a social/technical (sociotechnical) system. Such a view holds that an IT system has hardware and software components, but also contains characteristics of the human intellect, as well as social organisational structure, that must be addressed by IT evaluation (Walsham, 1999). Such research promotes the direct participation of end-users in the design and development of the IT system (Whyte and Bythway, 1996; Butler and Fitzgerald, 1999), gaining support from senior management to the IT project (Sauer, 1993; Clegg, 1996), providing appropriate user training (Flowers, 1997), and establishing effective communication between stakeholders (Whyte and Bythway, 1996). Brooks and Kiddle (2000) further argue for the involvement of stakeholders, describing it as a primary activity for the success of the IT system. However, they do acknowledge that it is a troublesome task that IT evaluators tend to avoid, and requires skilful management for it to work properly.  In addition, Jurison (1994) suggested at least two reasons why stakeholder should be involved in the evaluation process. They are summarised in table 2.

Table 2: Reasons to Involve IT System Users in the Evaluation Process

Stakeholder Involvement	Implication
In order to calculate the IT investment budget, the evaluators need to (as much as possible) account for the cost and benefits for all primary and secondary stakeholders.	Failing to do so will result in an underestimation of the budget for the project.
Benefits of the system need to be identified (by stakeholders)	If the benefits are not accounted for through evaluation, they will not be managed (by stakeholders) and consequently will be lost

 

Moreover, case studies on IT evaluation further emphasise the position of stakeholder involvement. For example, Willcocks and Lester (1993) found that most IT evaluation was carried out during the feasibility stage, no evaluation was performed over the life cycle of the system, and that organisational learning was not promoted. Willcocks and Lester concluded that IT evaluation in such organisations would improve if stakeholders had a larger role in the process during the life cycle of the system. Similarly, Farbey et al. (1995) conducted case based research and concluded that evaluation should be conducted through the life cycle of the system and that benefit management is necessary to exploit the system, pointing to the necessity of stakeholder involvement. Pouloudi and Serafeimidis (1999) reported the participation of stakeholders in the evaluation process. They found that organisational learning was damaged as a result of the continues rearrangement between the duties of two primary stakeholders: the person who is responsible for setting-up the rules and regulation of the evaluation, and the person who actually carries out the evaluation. Such case studies indicate that the design, development, and deployment of an Information Systems (IS) require attention to both social and technical aspects of the system (Boland and Hirshheim, 1987; Orlikowski, 1992; Serafeimidis and Smithson, 2000). These authors support the idea that evaluation is a social process and therefore must keep different stakeholder groups in mind.

The position to involve stakeholders in the evaluation process is not a new concept. Mumford and Weir (1979) suggested that end-user participation in the development and implementation of the IT system is a necessary factor for the success of the project. In the late 1980s and early 1990s the concept of ‘evaluation party’ was recommended to help primary stakeholders express their views and endeavour to influence the investment decision (Gregory and Jackson, 1992; Guba and Linclon, 1989). Currently, stakeholder identification in the IT evaluation process is considered common practice, but is mostly limited to users, developers, and managers, which is far short of potential primary and secondary stakeholders suggested by Pouloudi and Whitley, (1997). They argue that evaluation fails to interoperate the role of stakeholders, potentially jeopardising the system. Several authors (including: Walsham, 1999; Serafeimidis and Smithson, 2000) have argued for interpretive evaluation, which promotes the heavy participation of different stakeholder groups in the design of the system. Such research promotes that interpretive evaluation IT evaluation can serve to enhance stakeholder participation in the organisational activities, and has the ability to link management to IT goals. The view here is that the new system will likely impact different stakeholder groups in the organisation, affecting departmental processes, and shifting the political structure. Such systems are designed to accommodate the changing organisational context as viewed by the participating stakeholders.

The next section looks at the issue of involving stakeholders (specifically end-users) in the design process of the IT system, through the discussion of a case study involving IT evaluation with qualities of an interpretive epistemology stance. As indicated above, for the purposes of this paper, primary stakeholders are defined as system users, and secondary stakeholders are defined as clients, suppliers, and organisation staff not using the system. Furthermore, IT evaluation in the case study represents pre-implementation evaluation.

5.     Case study

The next section will report the results of the empirical case study conducted by the authors of this paper. The case study aims at examining decision making in an IT department, and its effect on IT evaluation. It starts with starts with a description of the methodology used. This is followed by a basic presentation of the IT department and how it was formulated during recent organisational changes. The operation of the system and decision-making is next described to illustrate the need for IT evaluation. Finally IT evaluation as a practice is described analysed.

5.1     Methodology

The case study employs an interpretive epistemological approach that employs qualitative research methods. This allows for face-to-face contact, as well as the use of field notes, written documents and archives to support the understanding of the context. Face-to-face contact is essential here as the case study aims to uncover drivers of decision making in the IT department, especially with regards to IT evaluation. There are numerous supporters of such a methodology.  Yin (1994), advocates the use of a case study when a phenomenon needs to be studied in its real life context. Cavaye (1996) suggests that a case study is able to gather rich primary data, and permits the description of a complex structure. To conduct the case study, an interview agenda was developed for the purpose of collecting specific data during a semi-structures interview. The agenda was designed to collect data on the decision making in the organisation as a whole, and in the IT department in specific. The agenda was also created to investigate IT evaluation practices in this organisation. Ultimately, Data collected would be used to establish the level of stakeholder (system user) participation through IT evaluation. Data would also be used to establish the nature and role of Interpretive IT evaluation in this organisation.

The agenda is divided into five parts, and with open-ended questions (apart from for section A) that do not require any form of quantification. To guarantee the accuracy and quality of the information, triangular data collection was employed (Yin, 1994).  To that end, a total of four employees where interviewed. Three from the IT department, including the Chief Information Officer, a Project Leader, and a Principle Business Analyst. Their participation contributed to data regarding IT Evaluation practices in the department as well as general decision making in the IT department. The fourth interviewee was the Deputy of Finance (a senior user) who contributed to the same nature of information, but from the user side. The complete list of questions is included in appendix 4 and addresses the following:

·            Establish general company information: to determine the broad organisational context

·            Establish the decision making structure in the organisation

·            Establish the decision making structure in the IT department

·            Investigate which IT evaluation method was used, and how was it selected

·            Establish the overriding factor(s) for choosing/ excluding IT evaluation methods

5.2     Organisational context

The case study organisation was established in 1995 to serve approximately 110,000 existing customers. It is the result of a merger between four other organisations. Consequently, the IT department of this organisation was constructed from four different IT departments, with different operating platforms and different organisational cultures. During the planning stage in 1994, the recently appointed Chief Information Systems Officer (CISO) formed a project group of IT professionals to investigate the suitability of keeping or discarding legacy IT systems. This initial justification process had a specific set of criteria, which was developed by the CISO and top management. The criteria included the following:

·            Upgrading the function of the system to accommodate future organisational needs

·            Limiting the cost of establishing new systems to be within the overall budget allocated by the board for the IT costs relating to re-organisation

·            The storage capacity of the system had to be large enough to accommodate data from all four organisations

·            Consider migration issues from one system to another

·             The lease on some of the hardware and software licensing agreements, which was coming to an end. This meant that it was not always possible to continue with the current system.

·            Consideration also needed to be given to both training needs for users, and also to the social aspects of adopting the new system.

The new organisation was set to achieving its new operation within a specified period, therefore the new IT department was set to be up and running to accommodate the overall organisational schedule. That process resulted in several a new system that came on-line in April 1995.

5.3     Systems operation

Since they were implemented in 1995, there have been significant changes and upgrades to the systems. The current IT department has 34 IT specialists, whose role is to service, support and manage approximately 1500 PC users in 22 departments across 59 locations. The new systems are in part also inherited, as one of the old organisations was locked into a contract with an IT supplier until the year 2000.

The functionality of the systems can be split into two categories. The first category has one corporate system that is used by all the departments in the organisation to support its accounting functions. The other category includes several different departmental systems operating in isolation to support specific requirements of departmental business processes. These systems are not used across departments. Furthermore, their utilisation is the responsibility of the respective department. Also, the department pays for any upgrades or modifications to them. They are, however, maintained by the IT department, which also offers limited training to system users, as well as installing all new hardware/software that has not been contracted out. The organisation views all its systems as support systems and has no immediate plans to use them for strategic purposes. No outsourcing was entertained during the fist stage of operations, with exception to maintenance of complex software that was parched from vendors.

5.4     Decision making practices

The decision making structure is very well established in the IT department (as perceived by the interviewees). Rules and regulations are seldom misunderstood, and therefore the chain of command seems to be holding. For example, when the procurement of the new system is required, the CISO forms a team consisting of IT specialists and key users. A partnership is formed to arrive at an agreed solution. On the one hand, technology and the alignment with organisational strategy is the responsibility of the IT department. On the other hand, functionality, and alignment with the business process, is the responsibility of the users. The findings of that process would need to be approved by CISO in consultation with the senior user. For large acquisitions, typically great than £100,000, a request is made to the board of Directors who ultimately sanction the final decision.  Although all procurements need to be approved by the IT department, the partnership process described above only takes place for acquisitions greater than £1000.

Systems are typically replaced, modified, or purchased for one of two reasons. First, if the current system is no longer appropriate as the result of functionality problems, technical limitations, government legislation, or if the business process has changed. Second, if the department has no IT system and would like to invest in technology to improve the operational efficiency and effectiveness of the department. In both cases the change is driven by key users, IT officers or a combination of both that recognise the weakness in the system or scenario. The organisation looks at cost, functionality, alignment with business process, opinion of users, and compatibility with current technology strategy as criteria for all new procurements.

For its decision-making, the IT department relies on its CISO and the IT Management Team who share the three main responsibilities of the department (servicing, supporting, and managing the system). In addition, an IT Client Group, consisting of senior management from all user departments, meets with the IT department on a quarterly basis to discuss the IT strategy, new procurements, and other major IT decisions. All decisions with significant impact on the organisation are finally discussed and approved, or discarded, at the board of director level, in consultation with the CISO. The general strategy of the IT department comes about as the result of a series of processes initiated by the suggestion of the CISO and IT Management Team in consultation with the IT Client Group. The organisation feels that the IT department can develop such a strategy because it is exposed to the business processes of all the departments and it thus has a solid understanding of their departmental functions, corporate responsibilities and IT needs.

5.5     IT evaluation

Officially, the IT department does not conduct any form of IT evaluation, apart from a technical evaluation of how the system functions, via performance measures.  This technical evaluation is concerned more with the technical performance of the system, than the performance of the system to deliver benefits at the desired costs. IT indicators include systems availability, response times, transaction processing times, network traffic However, in 1998 the IT department started participating in a benchmarking club that compares organisational IT indicators with IT indicators of other organisations. In 1998/1999 the organisational IT costs were just about the average. The organisation applied the results of the benchmarking practice to measure the performance and effectiveness of the IT department. Informally, the IT department performs a number of activities that are considered partially driven for the purpose of IT evaluation. For example, the IT department annually sends out a survey to its users to assess the general performance of the IT department. The survey looks at a number of issues ranging from the availability of the system and response time, to support from the company that the system was bought from. User input is also sought by the IT department during the quarterly meetings with the Client Group, on the successfulness of the IT system in terms of its ability to support the business process. The IT department pursues this activity until a consensus view is achieved with the Client Group. Also in the way of an informal IT evaluation, a service level agreement (SLA) is negotiated annually and SLA meeting occur on a quarterly basis with senior department members to establish an agreement on the level of satisfaction with the service provided.  It is also apparent from the case study that political barriers inside the organisation complicate formal IT evaluation. For example, the case study shows that the systems acting in isolation have common features that can be used across departments, but that this is rarely done because departments are reluctant to share the functionality or the information even though this would result in sharing the cost for running them. This practice further limits the resources of the IT department, as there is duplication is certain areas. This reduces its ability to perform IT evaluation. A final observation is that this organisation has been investing in IT systems as a reactive measure to periodically servicing problems. Although there is also a pro-active approach in terms of maintaining IT infrastructure and formulating IT strategy.

5.6     Evidence of interpretive IT evaluation

The study has identified 1500 PC user as primary stakeholders (system users), and well as 11,000 customers as secondary stakeholders of the IT systems. The supplier of the old IT systems may also be considered as a secondary stakeholder. As noted above, formal IT evaluation is currently represented in this organisation by a benchmarking practice which partially evaluates the performance of the IT department against other departments in similar organisations, publishing performance statistics and an annual survey distributed by the IT department to assess key user perception of the performance of the IT department. All three measures are available for the consideration of management in the IT department, as well as key-users of departments. Therefore, these measures may be considered a form of interpretive evaluation.

Informally, Evidence of Interpretive participation from key-users is noticeable. For instance, the several different systems operating in isolation are the responsibility of their user department(s). With the exception of maintenance that is either outsourced or provided by the IT department, modifications to the systems, key-users are accountable for utilising the system. They are also encouraged by the IT department to offer any advice on future modifications or upgrades to the system. Moreover, user departments of the isolated system are billed the cost of upgrades or modifications to the system. During procurement, key-user share the responsibility with the IT department to assure that the system bought will align with the business process. This gives users both the ability to conduct internal evaluations, and empowers them to voice their opinion and influence decisions with regards to future changes to their IT system. As for the corporate   system, the IT department works hand-in-hand with the Client Group (which consists of senior management from user departments) to establish parameters of the functionality of the system, general IT strategy, and appraise the successfulness of the IT system in terms of its ability to support the business process.

5.7     Further analysis

The case study provided valuable insight into the organisational practice of selecting IT evaluation methods. By looking at the history of the IT department, it was clear that formal IT evaluation was not a priority during the first period of the department operation. Priority was focussed upon IT provision to support service departments business processes. The organisation decided on the initial IT investment budget after the CISO proposed a business plan to establish the IT department. That plan came about from the assessment of IT issues e in the four merging IT departments at that time. In that first period, that is, 1994 to 1996, senior management was more interested in merging these four IT Departments as successfully as possible, given that there were other tasks that required greater attention. It might therefore be said that Departments were operating more in the sense of fire fighting than in planning effective strategies for their future. Since then, the IT department has been able to implement add-on systems aimed at improving the efficiency and effectiveness of specific departments. However, these add-on systems were not formally evaluated (from the IT perspective) but were justified as commonsense since they signified improvement to the organisation. Such evaluation practices often result in the choosing of a system that best fit user functionality requirements within the budget available for the department. The employment of an accounting type feasibility report that targets initial and ongoing costs is not uncommon here. The contractor supplying the system is also evaluated for their reliability to maintain the system and train/support employees during the life cycle of the system.  Furthermore, the case study demonstrates that it was common practice to invest in a system, or add to a current system, if the department had extra funds at the end of the budget cycle. Departments would typically do this to avoid returning the extra funds to the organisation and risk reducing their allocated budget during the next cycle.

The interviewees commented that lack of organisational resources, especially cash, was (and still is) the primary factor for not conducting formal IT evaluation when investing in new projects. The reasoning here is that lack of cash prohibited the employment of qualified IT personnel to conduct proper evaluations. A great deal of time was also stated as a resource not available for the IT department. That is to say, the IT department does not have the capability to secure the participation of key users in the IT evaluation project because of time constraints on the IT and user departments. Furthermore, the unavailability of a formal methodology for evaluation makes it difficult to commit users to this process. This prevents the evaluator from leading key users through the process. Lack of political capability can also be argued as a factor here. The final, and perhaps most obvious, reason given by the interviewees for not conducting a formal IT evaluation is that it was not required from the IT department.

5.8     Learning points from the case study

Having carried out an extensive case study the authors have identified and extrapolated key learning point. Although these learning point in no way claim generality, they do offer others the opportunity of drawing parallels against the key points below. As such, the following represents a comprehensive frame of reference.

Learning issue 1: It appears that case study organisation regards IT evaluation as a luxury rather than a priority when faced with intense organisational pressures. This may be due to a number of factors such as lake of knowledge, the unavailability of financial recourses, the unavailability of labour skills to perform the evaluation, or the unavailability of the time of the participants in the IT evaluation. Therefore, senior management need to mandate IT evaluation as part of all business processes relation to IT.

Learning issue 2: This IT department will not perform IT evaluation as it was not required by the organisation’s top management. It is plausible that top management did not require it due to lake of knowledge, commitment to other IT related work, of for reasons of self-preservation.

Learning issue 3: Financial return forms the IT investment remains on the mind of top mangers in this organisation. Therefore they might be more induced to spend additional funds on IT related projects (such as IT evaluations) if unless a clear connection can be established between such a project and cash savings. It remains the duty of the IT department to establish this savings link for them.

Learning issue 4: Client satisfaction remains the most popular method of informal IT evaluation for this It department, simply because it comes with the job. That is to say, if the client is not complaining, the system must be working right. Therefore, if other IT evaluation methods can be integrated into the job, then the organisation might be more inclined to accept the idea of IT evaluation.

Learning issue 5: The formulation of this IT department required a team of employees with detailed knowledge of the organisational work process, organisational culture and characteristics, as well as updated knowledge on information technology. If the organisation did not pursue these requirements form the IT staff, the organisation might have had a system that did to meet its functionality needs.

Learning issue 6: during the merger of the four organisations, the adaptation of a new IT system required the identification of critical success factors for each department, the identification of all relevant stakeholders, and proper support form top management. All of this knowledge is essential learning points for the IT department, to be used for future IT projects.

Learning issue 7: In the absence of formal IT evaluation in this organisation, informal IT evaluation can be conducted via organisational performance surveys, statistic on user complaints, and response time of the IT department to requests for technical assistance. The IT department has been forced to use unconventional means of IT evaluation to evaluative its performance.

Learning issue 8: User participation in the design of this IT system is crucial, especially were no formal IT evaluation exists, and were user training is negligible. Involving end-users of the system helped the IT department reduce user resistance, and moderately compensates for the absence of the formal IT evaluation.

Learning issue 9: It appears that case study that this organisation is more interested IT evaluating efficiency rather than effectives. It asks doer the system work properly, rather than is the system is delivering value. The problem here is that this organisation can have a fully functioning system, but it does not meet its original business purpose.

6.      Conclusions

This paper has argued for the inclusion of IT system users in the IT evaluation process, essentially to improve the functionality of the system, and to give organisation improved value on their investment.  To that end, a literature review was conducted on IT evaluation, stakeholder theory, and their relation to each other. Evidence from that review show that organisations continue to increase spending on new IT systems, but still ignore the value of a comprehensive IT evaluation process that includes the input of end-user of the system. This contradiction between spending and questionable decision making has drawn the attention of many researchers, who maintain that IT system user can contribute to the overall success of the IT system if includes from the initial design stages of the IT system.  In support of this argument, IT failure was used to illustrate the potential negative outcomes form the exclusion of system user in the IT evaluation proves. Moreover, social/technical aspects of the IT system were discussed via the many roles that IT can perform in the organisation. Both arguments are clearly established by research, but fail to explain the current trend in IT evaluation practice in the general industry. Therefore, a case study was conducted to investigate IT investment decision making in industry. The study employed interpretive epistemological approach that employs qualitative research methods, manly to allow face-to-face contact with the interviewees.  Face-to-face proved useful in gaining perspective on the organisational context, and IT decision making in particular. The main finding of the study was that when no formal IT evaluation process exists, informal organisational statistics could be used as an informal IT evaluation tool. In such an organisation (formal IT evaluation) participation of system users is the design and operation of the IT system is essential to insure its appropriate factuality, and smooth operation.

References

• Ahn H and Skudiark E (1997) ‘Resolving conflict of interests in the process of an information system implementation for advanced telecommunication services’, Journal of Information Technology, 12: 3-13.
• Baker B (1995) ‘The role of feedback in assessing information systems planning effectiveness’, Journal of Strategic Information Systems, 4(1): 61-80.
• Ballantine J and Stray S (1998), ‘Financial appraisal and the IS/IT investment decision making process’, The Journal of Information Technology, 13(1): 3-14.
• Benbasat I and Goldstein K and Mead M (1987) ‘The case research strategy in studies of information systems’, MIS Quarterly, 11(3): 368-386.
• Berghout E and Renkema T (1994) ‘Methodologies for information systems evaluation at the proposal stage: a comparative review’, Research Report EUTIBDKI69, Delft University of Technology.
• Beynon-Davies P and Blyth A (2000) ‘IS Failure, Evaluation and Organisational Learning’, Proceedings of the 5^th UKAIS Conference, University of Wales Institute, Cardiff, 26-28 April, pp. 444-452.
•  Boland J and Hirschheini A (1987) Critical Issues in Information Systems Research. John Wiley & Sons, Chichester.
• Boddy D and Buchanan A (1986) Managing New Technology, Oxford: Basil Blackwell.
• Brooks L and Kiddle L (2000) ‘Critical Success Factors (CSF’S) and Information Systems Development’, Proceedings of the 5^th UKAIS Conference, University of Wales Institute, Cardiff, 26-28 April, pp. 516-527.
• Butler T and Fitzgerald B (1999) ‘ A review and Application of the Critical Success Factors for Research on the Information Systems Development process- The Next Generation’, Proceedings of the 4^th UKAIS Conference.
• Cavaye M (1996) ‘Case study research: a multi-faceted research approach for IS’, Information Systems Journal, 6(3): 227-242.
• Clemons K (1991) ‘Evaluation of strategic investments in information technology’, Communications of the ACM, 34(1): 24-36.
• Clegg C (1996) The Performance of Information Technology and the Role of Human and Organisational Factors, A report to Economic and Social Council.
• Daniel E, Wilson H and Sutherland F (2000) ‘Factors Determining the Successful Development of Marketing Information Systems’, Proceedings of the 5^th UKAIS Conference, University of Wales Institute, Cardiff, 26-28 April, pp. 92-98.
• Doherty N, Banerjee S and Parry M (2000) ‘ Factors Affecting the Successful Outcome of Systems Development Projects’, Proceedings of the 5^th UKAIS Conference, University of Wales Institute, Cardiff, 26-28 April, pp. 375-383.
• Donaldson T and Preston E (1995) ‘The stakeholder theory of the corporation: concepts, evidence, and implications’, Academy of Management Review, 20(1): 65-91.
• Earl J (1989) Management strategies for information technology, Prentice-Hall, USA.
• Farbey B, Land F and Targett D (1992) ‘Evaluating investments in IT’, Journal of Information Technology, 7(2): 109-122.
• Farbey B, Land F and Targett D (1993) ‘How to Assess you IT Investment’ Management Today and Butterworth-Heinemann Ltd, UK.
• Farbey B, Land F and Targett D (1995) ‘A Taxonomy of information systems applications: the benefits evaluation Ladder’, European Journal of Information Systems, 4: 41-50.
• Flowers S (1997) ‘towards predicting information systems failure’, Edited by Avison D, Key in Information Systems,  Maidenhead: MeGraw-Hill, pp. 215-228.
• Freeman E (1984) Strategic Management: A Stakeholder Approach, Balbnger, Publishing Company, Cambridge, MA.
• Friend J and Hickling A (1987) Planning Under Pressure: the Strategic Choice Approach. Oxford: Pergamon Press.
• Gregory J and Jackson C (1992) ‘Evaluation Methodologies: A System for Use’, Journal of Operational Research Society, 43(1): 19-28.
• Gilbert R, Hartman E, Mauriel J and Freeman E (1988) A Logic for Strategy, Ballinger Publishing Company, Cambridge, Mass, pp. 110.
• Guba G and Lincoln S (1989) Fourth Generation Evaluation, Sage Publications, Newbury Park, California.
• Hawgood J and Land F (1988) ‘A Multivalent Approach to Information Systems Assessment’, Information Systems Assessment: Issues and Challenges, Edited by Andersen and G.B. Davis), North Holland, Amsterdam, pp.103-124.
• Hochstrasser B (1992) ‘Justifying IT investment’, Proceedings of the Advanced Information Systems Conference; The new technologies in today's business environment, UK, pp.17-28.
• Hill L and Jones M(1992) ‘Stakeholder-Agency Theory’, Journal of Management Studies, 29 (2): 131-154.
• Irani Z, Ezingeard N and Grieve J (1997) ‘Integrating the costs of an IT/IS infrastructure into the investment decision making process’, The International Journal of Technological Innovation and Entrepreneurship (Technovation), 17(11&12): 637-647.
• Kaplan S (1984) ‘Yesterday's accounting undermines production’, Harvard Business Review, 62(4): 95-101.
• Khalifa G, Irani Z and Baldwin L.P. 2000. ‘Factors impacting selection of IT/IS decisions’. Proceedings of the Fifth UKAIS Conference, [CD Proceedings], 24-26th April, Cardiff, UK, pp. 539-554.
• Jurison J (1994) ‘Measurement and evaluation of IT benefits: A stakeholder-based approach’, Proceedings of The First European Conference on IT Investment Evaluation, pp.248-255.
• Lederer L and Mendelow L(1990) ‘The Impact of the Environment on the Management of Information Systems’, Information Systems Research, 1(2): 205-222.
• Lee S (1989) ‘A scientific methodology for MIS case studies’, MIS Quarterly, 13(1): 32-50.
• Lycett M and Giaglis G (2000) ‘Component-Based Information Systems: Towards a Framework for Evaluation’, Proceedings of the 33^rd Hawaii International Conference on Systems Sciences, pp.1-10.
• Lyytinen K (1988) ‘Expectation failure concept and systems analysts, view of information system failures: Results of an exploratory study’, Information and Management, 14(1): 45-56.
• Nisbet J and Watt J (1980) ‘Case study, Rediguide 26’, University of Nottingham, School of Education, UK.
• Muniford E and Weir M (1979) Computer systems in work design, the ETHICS method: Effective Technical and Human Implementation of Computer Systems: a work design exercise book for individuals and groups. London: Associated Business Press.
• Orlikowski W (1992) ‘The Duality of Technology: Rethinking the Concept of Technology in Organizations’, Organization Science, 3 (3), August, pp.398-427.
• Pennington D and Wheeler F (1998) ‘The Role of Governance in IT Projects: Integrating the Management of IT Benefits’, Proceedings of The Fifth European Conference on IT Investment Evaluation. pp.25-34.
• Pouloudi A and Serafeimidis V (1999) ‘Stakeholders of Information Systems Evaluation: Experience form a Case Study’, Proceedings of The Sixth European Conference on IT Investment Evaluation, 4-5 November, Brunel University, Uxbridge, UK, pp. 91-98
• Pouloudi A and Whitley A (1997) ‘Stakeholder identification in interorganizational systems: gaining insights for drug use management systems’, European Journal of Information Systems, 6(1): 1-14.
• Powell P (1992) ‘Information technology evaluation: Is it different?’ Journal of the Operational Research Society, 43(1): 29-42.
• Remenyi, D, Money A and Twite A (1997) Effective Measurement & management of IT Costs & Benefits, Butterworth-Heinemann, Oxford.
• Remenyi D and Sherwood-Smith M (1999) Maximise Information Systems Value by Continues Participative Evaluation’, Logistics Information Management, 12(1/2): 14-31.
• Sauer C (1993) Why Information Systems Fail: A case study approach, Henley-On-Thames, Alfred Waller.
• Serafeimidis V and Smithson S (2000) ‘Information Systems Evaluation in Practice: a case study of organisational change’, Journal of Information Technology, 15(2): 93-105.
• Smithson S and Hirschheim R  (1998) ‘Analysing information systems evaluation: another look at an old problem’, European Journal of Information Systems, 7(3): 158-174.
• Strassmann P (1985) Information Payoff, Free Press, New York.
• Symons J (1993) ‘Evaluation and the failure of control: information system development in the processing company’, Accounting, Management and Information Technology, 3(1): 51-76.
• Ward J and Taylor P and Bond P (1996) ‘Evaluation and Realisation of IS/IT Benefits: An Empirical Study of Current Practice’, European Journal of Information Systems, 4(4): 213-233.
• Wallace W (1995) ‘Balancing conflicting stakeholder requirements’, Journal for Quality and Participation (March), pp.84-89.
• Walsham G. (1993) Interpreting Information Systems in Organizations, John Wiley & Sons, Chichester, England.
• Walsham G (1999) Beyond the IT Productivity Paradox, Edited by Willcocks L and Lester S, Wiley, pp. 363-379.
• Whyte G and Bytheway A (1996) ‘Factors affecting information systems success’, International Journal of Service Industry Management, 7(1): 74-93.
• World Information Technology and Services Alliance (2000) Digital Planet 2000: The global Information Economy, http://www.witsa.org/.
• Willcocks L and Lester S (1993) Evaluating the Feasibility of Information Technology Investment, Oxford Institute of Information Management, Research and Discussion Paper, RDP93/1.
• Willcocks L and Lester S (1994) ‘Evaluating the feasibility of information systems investments: Recent UK evidence and new approaches', Information management: The evaluation of information systems, Edited by Willcocks L, Chapman and Hall, UK, pp.49-77.
• Willcocks L and Lester S (1999) Beyond the IT Productivity Paradox, Edited by Willcocks L and Lester S, Wiley.
• Willcocks L Mason D (1987) Computerising Work: People, Systems Design and Workplace Relations. London: Paradigm.
• Yin R (1994) ‘Case Study Research: Design and Methods’, Applied Social Research Methods Series, Sage, USA.