1.     Introduction

Both in practice and scholarly theory the topic of ‘information technology (IT) evaluation’ receives extensive attention. With the growing importance of IT in organisations (strengthened by the growing general interest in E-commerce), senior managers seek better ways to manage and control the costs and benefits of their information systems (IS). From a scholarly point of view, attention is focused traditionally on the typical characteristics of IS, and the criteria by which they can be evaluated. More recent, attention is given not only to these contents of evaluation practices, but also to the evaluation process and context (Symons and Walsham 1991, Serafeimidis 1997).

In this article we will look at IT evaluation from a critical point of view. The basis for this critical stand can be found in the evidence (Yan Tam 1992, Ballantine and Stray 1998) that organisations keep using evaluation methods which are improper for IT evaluation, despite their awareness of the limitations of these methods and the development of an extensive list of IT-specific evaluation methods (see Renkema and Berghout 1997) which is available for the purpose of IT evaluation. Using arguments from critical management and critical accounting we propose that the failure to adopt these IT evaluation methods could be found in the failure to address the issues of changing power relationships and sense of (employment) security, which are affected by the introduction of new evaluation methods. We start by outlining some perspectives on the research area of IT evaluation, and then focus on the critical perspective.

2.     IT Evaluation & IT Cost Benefit approach

Senior management as well as IS professionals recognise IT/IS evaluation to be one of the important unresolved concerns in information management (e.g. Farbey et al. 1993, Grembergen and Bloemen 1997). Evaluation of IT/IS investments is problematic not only because of the inherent difficulties of evaluation (such as making estimates for future situations), but also owing to typical characteristics of such investments. Whereas IT/IS investments used to be financially justified on the grounds of efficiency gains (e.g. the time and money saved by automating manual labour), current investments are more focused on effectiveness.

The objective of effectiveness projects is not simply to reduce costs of performing existing tasks, but to do tasks completely differently to better achieve the desired results (Fitzgerald 1998). The justification for these projects must be based on effectiveness criteria such as increased functionality, product quality, enhanced competitive advantage, etc. The benefits of IT/IS investments which aim to improve quality, avoid costs and reduce risks are hard to quantify. Furthermore the actual organisational benefits of an IT/IS investment depend on the secondary effects that take place due to changes that originated from the investment. For example the increased number of customers of an organisation (secondary benefit) due to an investment in better quality products (primary benefit) is dependent on whether the customers acknowledge the quality improvement to be worthwhile. This not only results in benefits at some distant point in the future, but also makes it harder to link perceived benefits directly to the specific IT/IS investment. Similarly investments in IT infrastructure will only be worthwhile if the infrastructure is exploited in an effective manner. Because of the difficulty in putting a price on the secondary benefits, they are said to be intangible.

Further problems with IT/IS evaluation are related to the difficulties of IT project risk assessment and uncertainty of investment results, the fact that IT/IS investments have organisational impacts which are hard to quantify, and the fact that evaluation is intrinsically subjective, based on individual value judgements (including political considerations). In addition systems do not have clear definitions for ‘success’ and ‘failure’ and investment objectives change over time due to evolving user requirements (Keen and Scott Morton 1978), which makes comparison between prior expectations and eventual outcomes difficult.

To address the problems related to IT/IS evaluation, numerous methods and techniques have been developed to aid in managing and controlling IT costs and benefits (Wolfsen and Lobry 1998). However in practice few of these methods are used (Yan Tam 1992, Willcocks 1996, Ballantine and Stray 1998, Bacon 1992).

By far the greater part of IT/IS evaluation literature is devoted to the evaluation of IT/IS investments, mainly discussing different methods to address the intangible benefits of the investments using various criteria for evaluation. Wolfsen and Lobry  (1998) give a good overview of some of the techniques and methods developed for this purpose. Considering over 65 methods for IT/IS evaluation, Renkema and Berghout  (1997) conclude that the available non-financial evaluation methods are rarely underpinned by theory, and are usually based on single case studies. Furthermore, the methods focus on the evaluation criteria rather than the evaluation process by which the evaluation takes place.

However, research (Yan Tam 1992, Willcocks 1996, Ballantine and Stray 1998, Bacon 1992) shows that the traditional appraisal techniques, such as Cost Benefit Analysis, Payback time and Return on Investment, which do not account specifically for IT/IS characteristics, are still dominant in IT/IS evaluation. Hochstrasser  (1994) shows that only 16 percent of companies use rigorous methods to evaluate and prioritise their IT investments. Kumar  (1990) shows that only 30 percent of organisations perform a post implementation evaluation of a majority (75 percent or more) of their information systems. He concludes that post implementation evaluation is performed on only a small fraction of the systems developed.

Taking an interpretive stance, which recognises information systems to be more social systems than technical systems, it can be concluded that most IT/IS evaluations concentrate on the technical rather than the human and social aspects of the systems (Hirschheim and Smithson 1988). However, interpretive researchers claim organisations are complex social and political entities which defy purely objective analysis. Because information systems are part of organisations, they cannot be viewed in isolation, but should also be considered as social systems. Many examples in the literature show success or failure of an information system to be determined not by technical aspects but by ‘people problems’  (Symons and Walsham 1991). Thus, argue interpretive researchers, a comprehensive information systems evaluation must be significantly broader in scope than methodologies such as cost/benefit analysis, value analysis and decision analysis. Information systems are social systems and an analysis which treats them as distinct from their infrastructure and context will lose correspondingly in richness of understanding. Historical, social and political issues may be of equal or greater importance than the technical and economic dimensions. An interpretive approach to IT/IS evaluation methodology is proposed by Serafeimidis  (1997), who employs the concepts of content, context and process of the evaluation. He broadens the scope of conventional evaluation methods to go beyond just the content of the evaluation, and include also the context in which the evaluation takes place and the process by which the evaluation is performed.

From a critical theoretical perspective, derived from the critical theory of Habermas, an emancipatory approach can be promoted. Where interpretivism seeks understanding of the phenomenon through description, critical studies consider inequalities and power relations within organisations. They propagate emancipation of all individual stakeholders from prevailing systems, which can for example mean that all stakeholders are given a genuine chance to express their arguments and views on the phenomenon under evaluation and that they all be regarded as equal partners. Thus a joint understanding can be aimed for with a consensus between all stakeholders concerned with the phenomenon.

In the field of evaluation, interpretive and critical studies promote deeper understanding by the stakeholders of the object under evaluation, and do so by stimulating discussion among stakeholders (Guba and Lincoln 1989, Avgerou 1995). Critical theory however goes further, acknowledging the existence of political inequalities and barriers between stakeholders. Though critical theory does not pose a solution to break down these barriers, it stimulates an explicit and critical reflection on these inequalities. Thus it perceives this recognition of inequalities to be a first step in the right direction of emancipation (Walsham 1993). It is argued here that critical aspects of evaluation methods cannot be addressed in the content of the method, but should be recognised in the context and process part of evaluation; this means that the social and political properties of evaluation are difficult to address in the evaluation method, but rather in the process of applying the evaluation method. It is however doubtful if (and how) an evaluator who facilitates an evaluation can stimulate emancipation, since he/she is part of the different power relationships within the organisation.

To focus further on critical theory and how it can benefit the improvement of IT cost benefit management, we will discuss the concept of critical management and critical accounting in the next section.

3.     Critical perspectives

Critical management, which stems from critical theory, takes a critical view on organisational management, its methods and its ways of working, related to the goals it is pursuing. Emancipating all stakeholders and social groups in social issues is one of the major concerns in critical theory. Furthermore, a critical view on power relationships, inequalities among stakeholders and the traditional formal-rationalistic approaches is endorsed. From the standpoint of critical theory, management is too potent in its effects upon the lives of employees, consumers and citizens to be guided by a narrow, instrumental form of rationality. The theory never aims simply at an increase in knowledge as such, but its goal is man’s ‘emancipation from slavery’ (Alvesson and Willmott 1992) by the initiation of social change. Critical theorists argue that people cannot fulfil their potential owing to constraints imposed on them by prevailing systems of economic, political, and cultural authority, constructed both socially and by material conditions (Orlikowski and Baroudi 1991). Such a critical perspective can be found in many other research fields and is not confined to critical management.

From a critical management perspective theories about ‘professional management’, based on the use of neutral and objective techniques and methods, are challenged. It emphasises the political nature of what is seemingly neutral and technocratic. Critical theory seeks to encourage a questioning of taken-for-granted assumptions about social reality and the models for the satisfaction of human needs and wants that are widely assumed in contemporary society. To obtain the ideal emancipation from the domination of instrumental rationality Alvesson and Willmott  (1992) suggest two approaches. The first approach is always to be autonomously critical of the ideologies and false consciousness of society. Thus the task of critical theory becomes pointing out deficiencies in society, and also identifying unrealised potential within existing institutions. The second approach is to validate speech acts (linguistically), by investigating their claims with respect to:

1.      their comprehensibility

2.      their accuracy

3.      their honesty

4.      their normative appropriateness (correspondence with accepted cultural values)

As such, the task of critical theory is to counteract distortions of jargon (1), misrepresentation (2), deceit (3) and unlawfulness (4).

Lyytinen and Klein  (1985) propose applying the critical theory of Jurgen Habermas to information systems instead of the natural scientific research theories which have dominated IS research. They stress the importance of the social characteristics of information systems, the recognition of IS development as social act, the emancipation of individuals and interest groups, the participation of all stakeholders in IS development decisions and processes, and the need for achieving consensus in the goals of IS development through extensive communication. Furthermore, IS methodology should be enhanced so that each IS design should meet the criteria of comprehensibility, accuracy, honesty and appropriateness  (Lyytinen 1992), which have been mentioned above.

Knights and Murray  (1994) take a critical stand when they argue that the process of organisational politics is central to the development and deployment of information technology. Organisational politics can be seen as the struggle of individuals and collectives to achieve and reproduce a sense of material and symbolic security in the world. Within management, Knights and Murray argue, this struggle centres on the individual pursuit of career and the symbolic and material achievement of success. The pursuit of career and success is a process that takes place in competitive conditions that vary in their intensity over time and place. However, such pursuits need not necessarily be in conflict with the goals of the organisation. Career achievements may well be interdependent with organisational goals. Nevertheless, contemporary society prohibits explicit display of the personal career goals, and so they are concealed behind acknowledged organisational goals. Western society still wants to see managers as acting rationally, not politically (Walsham 1993). By hiding the organisational politics, managers try to uphold the myth of rational decision making. This provides a possible explanation why critical theory has not been applied widely to the research area of IT evaluation.

As well as critical perspectives on information technology, in this paper we are also interested in a critical perspective on cost benefit management. This topic is addressed in critical accounting research. Accounting, defined as a technique for quantification of calculation within organisations, has been traditionally seen as instrument to give a neutral representation of the economic and financial reality of an organisation in terms of financial costs and benefits. 

Critical accounting research has emerged as accounting researchers have begun to challenge the classic philosophical assumptions that underlie mainstream accounting research. A reason for this challenge is that the mathematical models used in accounting are limited to a few variables and statistical tests, constrained by the available data. They fail to take account of the rich social and organisational practices in which accounting is embedded. Critical accounting argues that accounting has many social aspects and thus rejects the traditional technological view on accounting. Moreover, it argues that accountants are not value-free or neutral, but exist in the broader social context of the organisation.  It argues that there is no independent economic reality, but accounting is creating that reality (Power and Laughlin 1992). Since accounting information is used for decision-making purposes, it becomes an instrument for organisational politics. We will take a similar view on IT/IS evaluation methods in IT cost benefit management.

As a consequence of the broader understanding of accountancy as related to society, politics and organisational function, a wide spectrum of theoretical approaches has been advanced by the critical accounting researchers (see Lodh 1996). This does not make it easy to define a single critical approach. But although there exist dissimilarities among these diverse perspectives, ‘a common feature amongst the authors of this tradition is that they share a common feeling for accounting research in that it needs to be considered within a broader societal context, and that the development of theory needs to be considered open and refutable’ (Lodh 1996).

4.     Critical view on IT cost benefit management improvement

Many researchers have criticised the discounted cash flow (DCF) methods, which originate from accounting, as a means for IT evaluation. Both from a practical point of view (e.g. hard to quantify intangibles, the favouring of quick-return projects, lacking consideration for typical characteristics of IT projects, etc.) as well as from an ontological point of view (information systems are primarily social systems, not technical systems which can be evaluated  in an unbiased, value-free and objective way) objections are raised against applying these methods to evaluate IT. A critical theoretical perspective adds another critique, namely the inability of the methods to capture an independent reality, but rather to create a reality. The methods obscure the actual personal objectives pursued by decision-makers behind a rationalised myth and the overt organisational goals (Knights and Murray 1994).

Nevertheless, some writers argue that these methods are not to be abandoned blindly (Symons and Walsham 1991), though they are a symbolic expression of belief in rational management. Evaluation may be seen as a ritual to perpetuate an image of rationality, efficacy and accountability (Walsham 1999). In a same way, evaluation can be seen as a ritual to close finished projects (Kumar 1990), rather than an activity with a practical purpose. Walsham  states:

Symbolism and ritual in human affairs are very important, not least in business organisations, and ritualistic evaluation exercises should not therefore be condemned out of hand (page 368).

Through this symbolism and ritual a sense of security and reassurance is gained. But Walsham  argues that there are two reservations to be made in using DCF methods as a ritual. First, they may become a device to suppress the less powerful in organisational terms. Secondly, DCF methods can be a major hindrance to innovative organisational change, because they tend to favour projects with quick financial results, rather than strategic opportunities.

In conclusion, although the methods may serve a ritualistic purpose, DCF approaches are highly deficient in generating real understanding of the costs and benefits of a computer-based system and its human and organisational consequences. Walsham  (1993) therefore proposes an interpretive approach to obtain a real understanding of the costs and benefits of systems. Such an approach has been constructed by Serafeimidis  (1997 ). A large number of other evaluation methods based on other approaches have been proposed by others (see for example Wolfsen and Lobry 1998 for an overview of methods).

However, as seen in a previous section, these methods are not widely adopted in organisational practice. Even the interpretive approach in IT evaluation, recognising the social aspects of evaluation and taking into account all views of the stakeholders (as promoted by Guba and Lincoln 1989), in practice has fallen into disuse (Serafeimidis and Smithson 1995). We argue therefore that the adoption of IT/IS evaluation methods in organisations might not in the first place depend on the content and characteristics of the evaluation method, or its assumptions about how evaluation takes place. Instead we propose a shift in focus to the context in which the evaluation method in introduced, thereby not concentrating on the object under evaluation (the information system), but rather on the evaluation method itself, within its context. A critical theoretical perspective may offer insight why evaluation methods are not adopted to improve the management of IT costs and benefits.

McCabe et al.  (1998) discuss from a critical perspective a case study where the concept of Total Quality Management (TQM) is introduced in an organisation. TQM is introduced by management to unite the organisation and secure employee commitment. However, their main argument is that TQM can be used to transform social relations, but does so based on existing social relations. Consequently, while addressing some problems, TQM reconstitutes organisational inequalities and existing power relations. In doing so it (re)creates many of the problems it is intended to resolve. TQM is usually depicted as a moral and politically neutral set of techniques to manage an organisation more effectively by improving the quality of its products and services. The researchers however argue that TQM does not remove organisational politics. As it resolves older tensions, in fact it creates new political anxiety and stress. TQM promises empowerment, but instead gives a rationale for cost-cutting and labour-saving, thus resulting in employment insecurity.

In the case study of a medium-sized UK Bank McCabe et al.  show that TQM is to blame for anxiety in job security, because TQM goes hand in hand with restructuring (and lay-offs). This results in tensions in the organisation, creating an atmosphere where people (as understood from a critical political point of view) try to survive and start blaming each other for inefficiencies. The case shows that TQM only works in this company for the people who do not have to be afraid of unemployment. The people who are in a more uncertain position try to make it more secure by only doing the work that is demanded of them, and trying not to accept extra responsibilities from the people who are in the better position.

One could think that the politics engaged in, owing to employment insecurity by the employees in the TQM case, are undesirable and should somehow be removed. But these politics are not unfavourable, so say the critical researchers, but a rational and central activity 'and benefical for organisations: it is through political manipulation and manoeuvring that individuals are able to secure their sense of self and identity. In doing so individuals may derive meaning from their work which could otherwise be absent. Critically, such expressions of self may serve to curb more violent forms of resistance should management attempt to remove them' (page 123). The researchers conclude that TQM either has to be accepted as a concept that will not deliver its promises and therefore should not be employed, or TQM should be extended to also take into account the issues of power, structure, inequalities and (employment) security.

We argue that similar reasoning can be applied to the introduction of improved IT cost benefit management through adoption of a new IT evaluation method. The introduction of a new IT evaluation method changes employees’ roles, responsibilities and commitments and thereby disrupts the ‘secure identities and positions they have created for themselves’ (Knights and Murray 1994). Moreover it can be seen as an indirect threat to employment security, since improved cost management can result in reductions of staff.

Reasoning further along these lines it seems unlikely that a change in IT cost benefit management will be welcomed by the IS department in an organisation if it does not address the issues of power, structure, inequalities and security.

Let us for example assume that general management initiates the introduction of the new IT/IS evaluation method to get a deeper understanding of the actual costs and benefits of IT applications within the organisation. In particular the chief information officer (CIO) wants to use the evaluation method to demonstrate the value of IT to the business and gain control over the rising IT costs. The evaluation method requires identification and quantification of all financial costs and benefits, in addition to the identification of intangible costs and benefits.

IS management is not satisfied with the new evaluation method. Although it recognises the merits of the method to demonstrate the benefits of IT to the business and thereby create greater awareness of the importance of the IS management, it perceives the method as a vehicle for cost reduction. The cost reductions will translate into staff reduction in the IS department, which means tensions in the department. The IS department fears the consequences that will be occur when the method exposes the high tangible costs of IT and fails to demonstrate the intangible benefits in financial terms. Therefore IS management considers the introduction of IT evaluation methods no improvement, but rather a reduction of its power.

Moreover, business managers are not eager to adopt the evaluation method either. The method demands from them to quantify their estimates of uncertain future benefits. These quantified estimates will most likely be used to evaluate the investment afterwards. Failure to obtain the estimated benefits will result in reproving questions from general management and ultimately maybe even in job replacement.

Similar to the TQM case study from McCabe et al.  (1998), IT/IS evaluation methods are usually believed to be politically neutral instruments to improve IT cost benefit management, both for business as for IT managers. The introduction of rational evaluation methods should result in reduction of organisational politics and increase transparency in IT investment decisions and IT control. However, rather than empowerment the example described above shows that the introduction creates political restlessness and insecurity for stakeholders.

Although the example is based on experiences from previous case studies (Nijland and Berghout 2000), more case studies on the adoption of IT/IS evaluation methods in organisations are needed. Current research in progress is focused on better understanding of such an adoption. The perspective that is offered by critical theory will contribute to such an understanding.

5.     Conclusion

Though organisations display great interest in IT/IS evaluation methods to improve their IT cost benefit management, in practice these methods are hardly ever adopted. The majority of organisations evaluating their IT/IS investments use a discounted cash flow method to assess them. In this paper we examined a possibility for this apparent paradox, using insights from critical management.

It was argued that introducing a new evaluation method in an organisation is a politically complex activity, which not only changes tasks and responsibilities of employees throughout the whole organisation, but moreover disrupts the sense of identity of the subjects. The identity is vital for the way they secure a safe position within an organisation (Knights and Murray 1994) and enables people to make sense of their organisational reality, from which they draw reasons for actions, responsibilities and commitments. An introduction of new ways of IT cost benefit management upsets established rules and boundaries that are required to ‘play the game’ of organisational politics. People involved will struggle to maintain and secure identity, which can result in counteracting the introduction of the evaluation method. We reasoned that failure to address the issues of power and security in the introduction of an IT/IS evaluation method will result in failure to improve IT cost benefit management.

Though the change in roles, responsibilities and power relationships, resulting in counteractive actions of employees, is one of many possible explanations for an IT evaluation adoption failure, we have gained new insights from critical theory that are valuable in understanding the effect of IT cost benefit improvement. The consideration of power structures and awareness of the personal goals pursued by managers are welcome new insights, which have been overlooked in traditional approaches in evaluation. These issues are however paramount when the aim is to understand the context of IT/IS evaluation method introduction.

We conclude by noting that the search for better IT/IS evaluation methods is futile if they are not adopted in practice. That is why future research should progress understanding why IT/IS evaluation adoptions fail. The process of implementing IT/IS evaluation within the particular context of an organisation seems to us, therefore, more important than the content details of the evaluation method being implemented.

A critical perspective on the IT/IS evaluation has shown that issues related to power inequalities and awareness of security, are essential to come to a better understanding of managers' actions in IT cost benefit management improvement. This notion will be used in current PhD research on IT/IS evaluation method adoption.

References

Alvesson, M., & Willmott, H. (1992). Critical theory and Management Studies: an introduction. In M. Alvesson & H. Willmott (Eds.), Critical management studies (pp. 1-20). London: Sage Publications.

Avgerou, C. (1995). Evaluating information systems by consultation and negotion. International Journal of Information Management,vol.15,pp.427-436.

Bacon, C. J. (1992). The Use of Decision Criteria in Selecting Information Systems/Technology Investments. MIS Quarterly,pp.335-353.

Ballantine, J., & Stray, S. (1998). Financial appraisal and the IS/IT investment decision making process. Journal of Information Technology,vol.13,pp.3-14.

Farbey, B., Targett, D., & Land, F. (1993). How to assess your IT investment : a study of methods and practice. Oxford: Boston : Butterworth-Heinemann.

Fitzgerald, G. (1998). Evaluating information systems projects: a multidimensional approach. Journal of Information Technology,vol.13,pp.15-27.

Grembergen, W. v., & Bloemen, I. (1997). Belangrijke en problematische thema's in de bedrijfsinformatica (Important and problematic topics in organisational informatics; in Dutch). Informatie,vol.6.

Guba, E. G., & Lincoln, Y. S. (1989). Fourth generation evaluation. Newbury Park: Sage.

Hirschheim, R., & Smithson, S. (1988). A critical analysis of information systems evaluation. Elsevier Science Publishing,pp.17-37.

Hochstrasser, B. (1994). Justifying IT investments. In L. Willcocks (Ed.), Information management : the evaluation of information systems investment (pp. xi, 272). London ; New York: Chapman & Hall.

Keen, P., & Scott Morton, M. (1978). Decision Support Systems: An Organizational Perspective. Massachusetts: Addison-Wesley.

Knights, D., & Murray, F. (1994). Managers divided : organisation politics and information technology management. Chichester: John Wiley.

Kumar, K. (1990). Post Implementation Evaluation of Computer-Based Information Systems: Current Practices. Communciations of the ACM,vol.33,pp.203-212.

Lodh, S. C. (1996). Critical Studies in Accounting Research, Rationality and Habermas : A Methodological Reflection, Critical Perspectives on Accounting.

Lyytinen, K. (1992). Information Systems and Critical Theory. In M. Alvesson & H. Willmott (Eds.), Critical management studies (pp. 159-180). London: Sage Publications.

Lyytinen, K. J., & Klein, H. K. (1985). The critical theory of Jurgen Habermas as a basis for a theory of information systems. In E. Mumford, R. Hirsschein, G. Fitzgerald & A. T. Wood-Harper (Eds.), Research methods in information systems (pp. 219-236). Amsterdam: Elsevier Science Publishing Company Inc.

McCabe, D., Knights, D., & Wilkinson, A. (1998). The politics of IT-enabled restructuring and the restructuring of politics through total quality management. Account Management and Information Technologies,vol.8,pp.107-126.

Nijland, M., & Berghout, E. (2000). Management of IT costs and benefits: results of a quick scan at nine financial institutions. In A. Brown & D. Remenyi (Eds.), Seventh European Conference on Information Technology Evaluation (pp.133-144). Trinity College, Dublin, Ireland.

Orlikowski, W. J., & Baroudi, J. J. (1991). Studying Information Technology in Organizations: Research Approaches and Assumptions. Information Systems Research,vol.2,pp.1-28.

Power, M., & Laughlin, R. (1992). Critical theory and accounting. In M. Alvesson & H. Willmott (Eds.), Critical management studies (pp. 1-20). London: Sage Publications.

Renkema, T. J. W., & Berghout, E. W. (1997). Methodologies for information systems investment evaluation at the proposal stage: a comparative review. Information and Software Technology - Elsevier Science B.V.,pp.1-13.

Serafeimidis, V. (1997). Interpreting the evaluation of information systems investments: conceptual and operational explorations. Unpublished Doctorl disseration, London School of Economics and Political Science.

Serafeimidis, V., & Smithson, S. (1995). The management of change for a rigorous appraisal of IT investment: the case of a UK insurance organisation. In G. Doukidis, R. Galliers, T. Jelassi & H. Krcmar (Eds.), Third European Conference on Information Systems. Athens, Greece.

Symons, V., & Walsham, G. (1991). The evaluation of information systems: a critique. In R. Veryard (Ed.), The Economics of Information Systems and Software (pp. 71-88). Oxford: Butterworth-Heinemann Ltd.

Walsham, G. (1993). Interpreting information systems in organizations. Chichester: Wiley.

Walsham, G. (1999). Interpretive Evaluation Design for Information Systems. In L. Willcocks & S. Lester (Eds.), Beyond the IT productivity paradox (pp. 363-380). West Sussex, UK: John Wiley & Sons Ltd.

Willcocks, L. (1996). The evaluation and management of information systems investments: from feasibility to routine operations. In L. Willcocks (Ed.), Investing in information systems : evaluation and management (pp. 15-34). London ; New York: Chapman & Hall.

Wolfsen, R., & Lobry, R. (1998). Automatiseren met rendement (To automate with return; in Dutch). Unpublished Doctoral dissertation, Tilburg University (Katholieke Universiteit Brabant).

Yan Tam, K. (1992). Capital budgeting in information systems development. Information & Management,pp.345-357.

EJISE is published by Academic Conferences Limited
Curtis Farm, Kidmore End, Nr Reading RG4 9AY, England
Tel: +44 (0)1189 724148, Fax: +44 (0)1189 724691, Email: info@ejise.com

Send mail to info@academic-conferences.com with questions or comments about this web site.
Copyright © 2002-2006 Electronic Journal of Information Systems Evaluation
Last modified: November 25, 2004
ISSN 1566-6379