ISSN 1566-6379

First published
in 2003

   


   

Paper 2 - Issue 1

Home Papers in this Issue Previous Issues Site Map

    .

Home
About the Journal
Scope
Editorial Board
Submission Guidelines
Call for Papers

 

For information on the European Conference on IT Evaluation, click here

Downloadable documents on this site require Adobe Acrobat Reader (free download here)

Managing the life cycle of Information and Communication Technology investments for added value
Frits (G.J.P.) Swinkels, Department of Information Management, University of Amsterdam, The Netherlands

 

1 Introduction

The proposed life cycle model is a comprehensive model used to cover the benefits obtained and burdens incurred over the entire life cycle of an ICT investment. In theory and practice, emphasis is placed on the financial justification of investment proposals. Though important, this scope is too narrow. The (implicit) view that an organization merely has to select from a number of well defined and predictably realisable ICT investments does not suit the characteristics of most ICT investments. Investing in ICT is seen as a process in which the information system along with its benefits and burdens is gradually elaborated. Thus, this process does not begin with a definitive and detailed investment proposal. The preparation of the investment proposal is an investment in itself. Usually the process begins with a hunch or a rough idea and on that basis the specifications of the system are further refined. Therefore, management must control this process to maximise the added value. This implies the application of ICT life cycle management. Benefits and burdens are not the calculated outcomes of an ICT investment proposal, they result from the management of the process. This means that the realisation of added value with ICT is seen as a management responsibility. If ICT is to be beneficial for the organization, then good management is necessary to realise these possible benefits. As Swinkels and Van Reeken wrote: "A good farmer does not just sow but also takes care to harvest" (Swinkels and Van Reeken 1996). Conversely, if ICT does not seem to contribute positively, then the responsibility for stopping or changing the project or system also lies with the manager. The sharp arrows of criticism regarding the lack of added value in using ICT should thus not be directed at the technology but at the management of ICT. One must conclude that the need is obvious for a better understanding of ICT management based on the ICT life cycle.

This article outlines the ICT life cycle management based on the WIE-approach. This approach rests on the notion that for successful application of ICT a combination is needed of content efficacy and a fluent cohesion with the decision making and managerial processes.

2 The need for a life cycle approach: IE is a management responsibility

The discipline aimed at measuring the costs and benefits of the application of ICT is commonly called Information Economics (IE). WIE started studying IE in 1990, inspired mainly by the works of Benson, Parker and Trainor (Parker et al. 1988). But gradually the scope expanded, focusing on the entire life cycle of the ICT investment and incorporating more true to life aspects such as uncertainties and the support of results by those empowered to make decisions. Because of discrepancies in organization characteristics as well as in the type of ICT investment, a best method that fits every organization is not feasible. Guidelines must be developed to determine which existing methods are suitable for a given situation. An organization needs to perform well and coherently in all of the main activities in order to realise the added value. Thus, benefits and burdens are seen as a result of management rather than of calculation. Recently, several authors have stressed the development of life cycle management (Berghout 1997, Willcocks 1996). Several life cycle models exist. However, these life cycle models are dominated by technology, i.e. benefits and burdens are ‘attributes’ of these technology. In the life cycle model presented in this paper, the application of technology results from considerations regarding benefits and burdens.

Viewing IE as a life cycle management problem immediately raised a number of interesting (research) questions. For instance: how does an abstract criterion such as ‘competitive advantage’ that has been used for the justification of an ICT investment proposal, develop during the life cycle? How can such a criterion be used for an ex post-evaluation to see if the intended benefits have indeed been realised? What organizational factors influence the ICT life cycle in a specific organization (budget constraints, impact of the investment, experience of those making the decisions etc.)? How does one identify ICT life cycle conclusion and how can the burdens for the removal of an information system (in fact legacy) be determined? How can an organization synchronise the investments in infrastructure and generic ICT facilities with the investments in application information systems? How can one assure that the responsibilities for the realisation of added value do not become obscured during the ICT life cycle? How can the uncertainties that come with ICT investments be incorporated into the decision making process? It is obvious that it is not possible to answer these and further questions all at once. Nevertheless, many building blocks are already available.

The proposed model is based on five main activities which have to be performed well in order to achieve results with ICT. The model is based on a view of benefits and burdens which employs a combination of financial aspects and quantitative and qualitative criteria. During the life cycle process the abstract criteria must be made more concrete. In view of the fact that ICT investments are often coupled with a high degree of uncertainty, one must always take a margin into account that stems from this uncertainty, both for the possible benefits as well as for the burdens. An investment is not an isolated choice. The right balance between the maximum desired system and the capabilities for its realisation, must be continuously striven for during the entire process. The final step necessary for realising ICT investments is the moorage of the life cycle management processes in the organization.

The next paragraph gives an overview of the main activities. In the consecutive paragraph we will demonstrate how the other concepts are woven into the structure of these activities.

3 The main activities

To control an investment in order to realise the expected benefits, five main activities are distinguished which have to be managed in coherence. Summarily, these main activities are as follows. Application systems that might have a positive effect on the performance of the organization must be identified (identification). Investment proposals are then elaborated and judged to determine if they meet pre-set criteria (justification). In the event of a constraint such as a limited budget, the investment proposals are compared to other (ICT) investment proposals, in order to determine priorities (justification). The selected investment proposals are then realised and implemented (realisation) in order to support business processes (exploitation). The (performance of the) information system and the processes that have led to it should be evaluated (evaluation). Thus, the main activities comprise the entire life cycle of an ICT investment, from the first rough idea right up to the system's removal. Figure 1 gives a summarised description of these main activities. Each activity and its attendant performance question is mentioned.

 

Main activity and main question to be answered Short description

Identification

What are possible interesting opportunities for investment?

Interesting opportunities can be discovered by a top down, a bottom up or a centre out approach (Earl 1989). It is possible to steer this activity in order to encourage the number of suggestions made by users or to discourage them by adding criteria that must be met by the proposals. The identification of possible systems may lead to the need for additional investments. The investment in a number of information systems may (for example) lead to investments in systems development capabilities or greater computing power and data communications capacity. This activity results in a number of rather rough proposals. Investment in new systems also often implies the removal of old existing systems. The identification of these is also included in this main activity.

Justification

Do the proposals meet the criteria and which combination of information systems contributes most to the organization?

Justification is the activity that determines the information systems to be realised now or in the future. This means the justification of a specified proposal and consequently the prioritisation of the justified proposals. If a proposal is not sufficiently clear to enable one to determine its investment desirability, further elaboration may be necessary. The criteria that are used for the weighing of proposals can be quantitative and qualitative. The result is a list of selected proposals. There may be several reiterations before the justification is completed.

Realisation

How to realise a usable information system in such a way as to maximise the added value?

Realisation comprises all the activities needed to achieve the desired change (initial investment and increments). This is thus not limited to programming and testing the software but also involves all the other activities such as administrative procedures, data conversion, organizational implementation and user training. The result of this activity is an information system in use.

Exploitation

How to realise maximum value using the existing system?

The primary target of exploitation is to get the most out of the existing system. This means minimising the negative effects of disruptions and operating costs and maximising the use of system possibilities. To fulfil this task, the use and performance of the information system have to be measured and future bottle-necks have to be identified.

Evaluation

Was the maximum possible added value realised and if not how can one learn to improve this?

Evaluation means that the processes, the system and the achieved results are critically reviewed in order to compare them with the initial targets set and with the achievements of others (inside or outside the organization). The analysis can result in proposals for system improvement or in the (decision making) processes to develop and exploit the system. In the event that the project or the system has insufficient added value, then the system or project should be terminated to stop the buildup of losses. Performing evaluations is regarded as a necessary characteristic for a learning organization.

Figure 1: Summary of main activities

The WIE approach concentrates on the main questions rather than on the techniques or procedures, due to the fact that, depending on the specific situation in an organization, there may be several ways to answer these questions. In theory, many methods exist that can be used for one or more of these activities (Berghout and Renkema 1994, Swinkels and van Irsel 1992). The majority of these methods are developed for the (financial) justification of investment proposals. Apparently, no single method supports all of the activities. When an organization succeeds in managing the activities well and learns from this experience, their information management capabilities should improve. The management of ICT benefits and burdens can be seen as a process of maturation, perhaps comparable to the growth of system development capabilities (Paulk et al. 1993).

The life cycle is described as a sequence of main activities. This is not the same as a sequence of phases consisting purely of these activities. Several system development methods exist with specific phasing. These phases are in fact specific combinations of main activities. The information planning phase for example, can be regarded as a combination of evaluation, identification and justification. Nevertheless, over time every method must support a change in emphasis compared to the above sequence of main activities. This is obvious in linear phasing models. In fact, in modern methods based on incremental development or prototyping we see a number of main activity reiterations.

In this paper we will concentrate on the investments in application information systems, rather than on investments in the underlying infrastructure or generic investments such as database management systems, computing power, network capacity or system development capabilities. The processes for these investments differ from those in application information systems because the benefits for these investments are mainly realised in the IT-domain or indirectly via the specific application information systems. An investment in infrastructure can be called reactive when the need for investment results from the requirements posed by the application information systems. An additional problem which arises from these infrastructure investments, is that not only do the individual investments need to be guarded, but also the infrastructures composition and cohesion as a whole. Investments in infrastructure can also be pro-active. This is the case when those investments are necessary to enable the development of future applications (Van Reeken 1997, Renkema 1996). Increments in infrastructure or investments that are reactive can be incorporated into the process as described for application information systems. Large scale pro-active investments in infrastructure such as a company wide enterprise resource planning system, an intranet or a large customer database, require a different approach. In these instances the investment requires a significant change in the technological infrastructure. The investment proposal can thus not be justified by the fit with the current infrastructure and requires an in depth analysis regarding the future direction of the organization and the role of ICT. These investments have a heavy impact on the future organization and are very complex. Therefore, these kinds of investment decision processes are performed using a specific decision making process. The use of scenario analyses in which scenarios are introduced for developments within the organization and the environment and for relevant ICT trends, are probably more suitable for this. Usually, this process is not integrated into the general annual investments proposal procedures for application information systems. The management of investments in infrastructure is still an area which holds the possibility for much fruitful research.

4 Benefits, burdens and uncertainties

4.1 Benefits and burdens

In order to communicate about (expected) benefits and burdens, one has to be able to describe them. A specification solely in financial terms would be preferable but is unattainable. Therefore, the burdens and benefits are usually translated into criteria. The specification of (expected) benefits and burdens is therefore always a combination of financial and other criteria.

Treacy and Index Group, Inc. developed a Cost-of-Network-Ownership model. This model "represents a practical and effective tool for managers to use in identifying and analysing network costs." (Treacy 1989). At the core lies the model wherein costs can be placed in the five "line item" categories, namely, equipment, software, personnel, communications and facilities. Each of the cost items is spread over three different phases in the network life cycle: (1) the acquisition of the network, (2) routine operation and troubleshooting, and (3) incremental changes in the network. Thus, this model can be used to identify and isolate the costs of owning and operating computer networks. Since then, the original model has been expanded by its makers and others in order to enhance the applicability which results in a broader scope, not only for networks but also for other hardware and software. An additional expansion regards the incorporation of quantitative and qualitative criteria.

The original model was used for ICT that can technologically be characterised as centralised. Many of the activities originally performed by centralised data centres are currently performed by users. This leads to higher burdens within the user departments. In order to make a fair comparison of the burdens of different investment proposals, all of the burdens (such as management planning and decision making, user participation, software development and data centre) have to be taken into account. This means that the burdens for decentralised equipment, software, personnel, communications and facilities must be added for all the line items.

In theory, the distinction between acquisition and incremental changes is clear. However, in practice this distinction is blurred by incremental software delivery, software releases and scaleable hardware. Both categories can be regarded as ‘change’ ('preparing information technology for future operations'), thus, grouping the initial investment and the incremental extension under the heading of change is recommended. Today, many organizations face legacy information systems that are disablers for the development of new products and services and thus have a negative added value. These systems also have a negative residual value due to their removal costs. Explicitly recognising and managing these burdens should be a priority. Therefore, the addition of an extra column for the burdens of removal is also recommended.

So far, the model for burden analysis has been explained. The same model will be used for benefit analysis in order to compare the two. The adapted model for both benefits and burdens looks like this.

figure 2

In a model such as that shown in Figure 2, the benefits and burdens are recorded with respect to the life expectancy of the investment. Estimating the expected period of (economical) use is essential. The quantitative financial data have to be translated into years in order to calculate the indicators 'Net Present Value' and 'Pay-Back Period'.

The criteria that will be used to describe the non financial quantitative and qualitative benefits and burdens and their relative weight within the overall investment proposal, will depend on the types of investments and on the culture within the organization (see e.g. (Van Eekeren en Heinen 1996, Farbey et al. 1994, Parker et al. 1988, Van Irsel in WIE 1994). When we examine the investment proposals for ICT, several different types can be distinguished (Van Reeken 1997). These types range from a specific application aimed at the replacement of labour hours to an investment where ICT is in fact the core of a new business. It is obvious that the complexity of the investments and of benefits and burdens increases and that there only few investments which are pure ICT investments. The typology also takes the difference into consideration between the support for existing business processes ('alignment') and between investments that result in changes in business processes due to opportunities offered by ICT ('impact'). Each investment type has its own characteristics regarding benefits and burdens and how these are translated into criteria. Note that selecting the criteria for the identification and justification of investment proposals is also a management responsibility in which the desired direction for the use of ICT is communicated within the organization.

Contrary to line items in the burdens section of the model that correspond to resources used (analytical), line items in the benefits section are related to the use of the information system as a whole (functional). The model can be used to show the benefits and burdens for a single investment, but it can also be used for the total of the ICT investments. The combined effects of the ICT investments must be expressed in the organization's financial reports. In the bulk of internal and external reports, a large degree of ICT costs go hidden under another denomination, such as personnel costs or housing. Triple entry bookkeeping seems to offer an interesting possibility for bringing effects more clearly into view. That means that in an organization for which ICT is one of the critical success factors, the financial effects of ICT are expressed in the profit and loss account as shown in the following example.

pap2-2.gif (5272 bytes)

The quantitative and qualitative criteria can not as yet be incorporated into this type of calculation. In the suggested method for giving concrete form to the quantitative and qualitative criteria, it becomes possible to further increase the visibility of benefits. Therefore, the challenge is to convert those open-standing options into financial data, thus also resulting in the visibility of these effects.

4.2 Uncertainties

The assessment of expected benefits and burdens is based on the (rough) specification of the planned system and is resultant from the probability that the specified system will be realised as planned and also from uncertainty regarding the capabilities for realising the system. (Heemstra 1989). In literature, the benefits and burdens are often treated as if they are very accurate (so called point estimates). This is not in line with experience. Uncertainty is often considered merely as a negative factor, called risk. Even investments in ICT however, do occasionally offer the possibility for some good luck. The uncertainty should be incorporated into the benefits and burdens estimates and should resemble intervals or probability distributions. Uncertainties are not objective and static labels for a project. Uncertainties can reinforce each other, which may even lead to a situation where the combination of a few minor uncertainties cause the project to lose control ('runaway projects'). The overall estimate of uncertainty must be based not only on the estimation of uncertainty factors, but also on the capabilities of the organization and on the specific measures taken to control these factors. However, the simple fact that many measures are taken does not necessarily guarantee deminished uncertainties (Swinkels and Gielen 1994).

Once one accepts the fact that benefits and burdens are not reliable calculations, but rather estimates that change in the course of the project, a first step is made to manage uncertainties and their influence on benefits and burdens. Typically, at the beginning of a project, its attendant uncertainties are high since the impact of the eventual system and (thus) the course of action is not precisely clear. Thus, uncertainties regarding benefits and burdens are correspondingly high. This means that every investment proposal should include a first estimate of the uncertainties and accordingly an estimation of the margins in benefits and burdens due to these uncertainties (e.g. by using several scenario’s). Two proposals for realising the same system by using a standard software package or by using a new system development workbench will result in different uncertainty profiles. The uncertainties must be reassessed at regular intervals and related to the benefits and burdens. In this way it becomes possible to prevent project managers from aiming at reduced uncertainties and minimised burdens without paying attention to the impact that their measures may have on eventual benefits. The integrated manner of dealing with benefits, burdens and uncertainties makes it possible for management to trade-off between a low-risk project (for example) with lower expected net values and a project that has a higher expected net value but which has a high-risk profile. This may well be one of the reasons for the popularity of standard software packages.

5 Level of ambition

As we have seen in the previous paragraph, there are several choices to be made regarding the desired information system and how to achieve it. Usually, a highly desirable result is more difficult to achieve. Therefore, management must continually assess the desirability versus the realisability. The desired result is called the level of ambition, which is an important but often neglected concept (Van Reeken in WIE 1994).

In the process of specifying and realising an information system there are many choices to be made regarding the scope, functionality, delivery date, quality and performance of the desired information system etc. Thus, management must make choices to determine an information system's level of ambition. They could choose a system with limited scope and few features, but they could also choose a large, complex state of the art system. Management must be clear regarding the desired level of ambition. A simple system might be easier and cheaper to build but could also generate fewer benefits compared to a system that has more features which fit the specific business requirements, but which also requires a higher budget. In short, there is a relation between the level of ambition and added value. However, beyond a certain level of ambition, burdens rise sharply and benefits only marginally increase or will even decrease. Balancing the ambition level in relation to added value is not a one time effort. During its life cycle the organization will gain added insight into the possible benefits (identification!) and constraints. Thus, the level of ambition can change as a direct result (justification!). It is only in rare cases that the desired information system is developed and implemented in one go. Most of the time, the desired system is realised using several increments and releases. This means that the level of ambition also has to be determined for each and every increment. The description of the prevailing level of ambition is translated into criteria for benefits and burdens and their values in the benefits and burdens model as described in the last paragraph.

6 Managing benefits and burdens during the life cycle

There surely is a tension between the desire to have a number of generic criteria when weighing investment proposals (due to the use of multi criteria methods in comparing different investment proposals) and the desire to have concrete and precise measures for the calculation and evaluation of the results of an application information system. You can’t have both at the same time, as it is not precisely certain just how the information system will work out in practice, due to the abstract nature of some criteria (e.g. ‘strategic value’) or due to uncertainties. Moreover, when too many criteria are involved, decision makers will lose their overview and a detailed analysis takes time and money. Therefore, a balance must be found between a primary analysis, sufficient for justification, and between further elaboration of the quantitative and qualitative criteria during the ICT life cycle. The manager responsible has to know what ‘strategic value’ (for example) could mean for his business performance and what range of quantitative measures could be used in assessing this benefit (growth of market share, margin, customer satisfaction). Therefore, filling in the benefits and burdens model takes place gradually over time. In fact, once a budget has been allocated, the manager has been assigned a value generating potential that he or she has to define and realise during the life cycle. Thus, during the life cycle, the generic criteria for benefits must be translated into specific benefits, preferably stated in financial terms. Benefits shown at the beginning are merely 'Monopoly'-moneys, and are only the kick off for the real game. Benefits and burdens (the level of ambition) need to be continuously (re)assessed during the life cycle of an investment. Benefits and burdens are not attributes that you can procure with an investment, they have to be realised by management. Also, if further experience shows that the expected added value will not materialise, then it becomes the responsibility of management to terminate projects or to replace information systems.

The different categories of benefits and burdens lead to different measures which ensure clarity of responsibility and measurement. A performance tracking and monitoring system has to be set up in line with the benefits and burdens, as specified in the benefits and burdens model. In setting up this system, one has to take into account that the different kinds of burdens and benefits require different systems for registration, measurement and analysis. The financial charge issued back from the IT domain can be checked relatively easily as it is often based on clear agreements. One of the objectives of charge back systems is to stimulate users to make responsible use of the ICT facilities. These investments consist mainly of a large share of fixed costs that can no longer be influenced. Thus, in most cases it would be better to concentrate on the maximisation of benefits. Other quantitative criteria can be controlled via performance indicators, such as the scorecard approach (Kaplan and Norton 1992). Qualitative benefits such as ‘strategic value’ should be measured differently, as described in the previous paragraph.

7 Making it work

Reviewing the added value of an investment proposal is usually done for the organization as a whole. Looking at proposals on a more detailed level, often shows that this is a summation of benefits and burdens for several underlying processes and departments. This does not mean that every department is better off with the investment. A proposal could (for example) lead to more data entry in department X but will diminish the work in department Z. Aside from the tensions regarding concerns between different user departments, tensions can also arise between the users and the ICT department. How the later source of tension is co-ordinated is partly dependant on the position of the ICT department (is the latter a cost centre, a service centre or a profit centre?). However, benefits for the IT-domain should (indirectly) lead to benefits for the business domain due to lower costs charged to the business domain or due to enhanced IT-capabilities available to the business.

Simply concentrating on the rational aspects of decision making, such as the measurement of objective criteria, is seldom sufficient. The decisions should also be supported by the people involved and thus subjective criteria will also come into play. Information economics decision making always involves different people with different backgrounds and responsibilities (managers, users, information systems professionals, data processing people). These characteristics have to be taken into account. Just imagine the differences in the organizational style, decision making process, participants involved and in the specifics of the investment proposal. Due to the different characteristics of main activities, a manager must frequently be ready to change his attitude. Sometimes creativity is needed to look for opportunities and then again, sometimes rational judgement based on figures is required (De Bono 1985). Preliminary evidence suggests that the focus of management on the realisation of added value diminishes with each consecutive activity (Leideman 1997). This is also due to the fact that the responsibilities for (value added by the) information systems are not clearly defined. Thus, this responsibility should be clear for every main activity and every information system.

The benefits and burdens of an investment should be divided between and allocated to the responsible managers. A cost pool is a common concept for the allocation and measurement of the quantifiable costs. This pool is charged with the ICT user costs. But where does the benefit pool come in? Such a pool should represent a consistent part of the organization which can be clearly distinguished with regard to the products and the activities of the pool and with regard to the managerial responsibility concerning a share of the benefits and burdens that attend the use of this particular ICT investment.

8 Summary and conclusions

This paper gives an outline of the ICT life cycle management model that is being developed by the (Dutch) Work Group on Information Economics. This model is based on an integrated treatment of benefits, burdens and uncertainties. To structuralize the activities, a set of five main activities (Identification, Justification, Realisation, Exploitation and Evaluation) was introduced.

Most of the current literature is directed at a specific part of the life cycle or at a single concept and thus neglects their relationship with the other activities and concepts and the relevance of these. That is why several questions were raised which stem from seeing IE as a life cycle management approach. Current life cycle models are dominated by technology and benefits and burdens are seen as ‘attributes’ of this technology. In life cycle management as it is presented in this paper, the application of the technology itself is based on considerations regarding benefits and burdens.

To maximise the effectiveness of ICT, the consequences that follow from the organizational setting and from the choice of investment type have to be taken into account. But most importantly, the realisation of added value with ICT must be seen as a management responsibility.

9 Literature

  • Berghout E W (1991) Comprehensive-cost-of-ownership model for information systems. Structure of the model and suggestions for use, Internal document Philips International BV Corporate Automation Department, Eindhoven, The Netherlands.

  • Berghout E W and Renkema T J W (1994) Beoordelen van voorstellen voor investeringen in informatiesystemen: begrippenkader en vergelijken van methoden, Delft University of Technology, Faculty of Mathematics and Informatics, Delft, The Netherlands.

  • Berghout E W (1997) Evaluation of Information System Proposals, design of decision support method, Doctoral Dissertation, Delft University of Technology, Delft, The Netherlands.

  • De Bono E (1985) Six Thinking Hats, Key Porter Books.

  • Earl M J (1989) Management strategies for information technology, Prentice Hall, Ltd., United kingdom.

  • Eekeren P L M van and P M Heinen (1996) Kosten en baten van computernetwerken, Kluwer Bedrijfswetenschappen, Deventer, The Netherlands.

  • Farbey B and Targett D and Land F (1994) The Great IT Benefit Hunt, European Management Journal, Vol. 12, nr. 3, Elsevier Science Ltd., p.270-279.

  • Heemstra F J (1989) Hoe duur is programmatuur? Begroten en beheersen van software-ontwikkeling, Kluwer Bedrijfswetenschappen, Deventer, The Netherlands.

  • Kaplan R S and Norton D P (1992) The balanced score card - measures that drive performance; Harvard Business Review, January - February 1992, p. 71-79.

  • Leideman G J (1997) IT-investeringsprojecten: tussen automatiseren en strategische impact, doctoral thesis, Universiteit Maastricht, The Netherlands.

  • Parker M M and Benson R J and Trainor H.E. (1988) Information Economics -Linking Business Performance to Information Technology; Prentice-Hall Int., Englewood Cliffs (USA).

  • Paulk M C and Curtis B and Chrissis M B and Weber C V: Capability maturity model for software, Version 1.1., Software Engineering Institute, CMU/SEI-93-TR-24, august 1993.

  • Reeken A J van (1997) A Typology of Information and Communication Technology applications from a Management Point of View, paper submitted to the Fourth European Conference on the Evaluation of Information Technology, Delft, The Netherlands.

  • Renkema T J W (1996) Investeren in de informatie-infrastructuur: richtlijnen voor besluitvorming in organisaties, Kluwer, Deventer, The Netherlands.

  • Swinkels G J P and Irsel H G P van (1992): Investeren in informatietechnologie: take IT or leave IT, Compact, summer 1992, p. 3-14.

  • Swinkels G J P and Gielen L J M W (1994) Knelpuntenanalyse projectmanagement, geen mes op de keel maar vinger aan de pols, Handboek EDP Auditing B.5.1.1.1., Kluwer Bedrijfswetenschappen, Deventer, The Netherlands.

  • Swinkels G J P and Reeken A J van (1996) Een goede boer doet meer dan zaaien, NGI-magazine, Van Diemenstr. 184, 1013 CP Amsterdam, p.20-22.

  • Treacy M E and Index Group, Inc.(1989) The Cost of Network Ownership; Index Group, Inc., Cambridge (USA).

  • WIE (1994) Information Economics; Werkgroep Information Economics en PAO-Informatica, Amsterdam, publication in Dutch, 1994.

  • Wilcocks L (1996) Investing in information systems: evaluation and management. Chapman &Hall, London.

 
Copyright   © Frits (G.J.P.) Swinkels, 1999

Home Up Papers in this Issue Previous Issues Site Map

EJISE is published by Academic Conferences Limited
Curtis Farm, Kidmore End, Nr Reading RG4 9AY, England
Tel: +44 (0)1189 724148, Fax: +44 (0)1189 724691, Email: info@ejise.com

Send mail to info@academic-conferences.com with questions or comments about this web site.
Copyright © 2002-2006 Electronic Journal of Information Systems Evaluation
Last modified: November 25, 2004
ISSN 1566-6379