All over the world there is a huge amount of money spent on IT (e.g. Seddon, 2001). It is therefore important to evaluate the outcome. Evaluation is never an easy task and consequently there are a lot of suggestions for how to evaluate IT-system. Much of the literature on evaluation takes a formal-rational view and sees evaluation as a largely quantitative process of calculating the likely cost/benefit on the basis of defined criteria (Walsham, 1993). There are also interpretative approaches (e.g. Remenyi, 1999; Walsham, 1993). The interpretative perspective views IT-systems often as social systems that have information technology embedded into it (Goldkuhl & Lyytinen, 1982).

There are formative and summative approaches containing different measures or criteria. Some approaches are focusing on harder economical criteria and others are focusing on softer user-oriented criteria. According to Walsham (1993) and Scriven (1967) formative evaluation aims to provide systematic feedback to the designers and implementers while summative evaluation is concerned with identifying and assessing the worth of programme outcomes in the light of initially specified success criteria after the implementation of the change programme is completed. The criteria used are often derived from one specific perspective or theory.

All of the approaches, formal-rational, interpretative or criteria-based, are different ways and their primary message is how the evaluator should act in order to perform evaluation. Besides this “how-message” it is also important to decide about what to evaluate. When evaluating IT-systems we can think of at least two different situations that can be evaluated. In this paper, we differ between evaluation of IT-system as such and evaluation of IT-systems in use. From the questions of how to evaluate and what to evaluate we derive a matrix consisting of two dimensions “how to evaluate” and “what to evaluate”. The combination of the two dimensions results in six different evaluation types and the purpose of this paper is to, on an ideal typical level, identify and characterise each of the derived evaluation types. The aim of the matrix is to support different choices of how to perform an evaluation depending on the evaluation situation.

The different ways of how to evaluate and what to evaluate are identified from reading literature and from insights from empirical findings in evaluation projects where we have participated (Ågerfalk et al., 2002).

2.         Strategies concerning how to evaluate

We distinguish between three types of strategies:

§          Goal-based evaluation

§          Goal-free evaluation

§          Criteria-based evaluation

The differentiation is made in relation to what drives the evaluation. Goal-based evaluation means that explicit goals from the organisational context drive the evaluation. These goals are used to measure the IT-system. The goal-free evaluation means that no such explicit goals are used. Goal-free evaluation is an inductive and situationally driven strategy. Criteria-based evaluation means that some explicit general criteria are used as an evaluation yardstick. The difference to goal-based evaluation is that the criteria are general and not restricted to a specific organisational context.

2.1       Goal-based evaluation

Goal-based evaluation can be seen to be formal-rational to its character (e.g. Walsham, 1993). Walsham means that a formal-rational view sees evaluation mainly as quantitative process of calculating the likely costs and benefits. According to Patton (1990) goal-based evaluation is defined as measuring the extent to which a program or intervention has attained clear and specific objectives. The focus is on intended services and outcomes of a program – the goals. Good et al (1986) claim that evaluations should be measurable and that the evaluation should meet the requirements specification.

One common criticism of the formal-rational view is that such evaluation concentrates on technical and economical aspects rather than human and social aspects (Hirschheim & Smithson, 1988). Further Hirschheim & Smithson means that this can have major negative consequences in terms of decreased user satisfaction but also broader organizational consequences in terms of system value. We agree with the criticism of Hirschheim & Smithson, but when analysing goal-based evaluation in an ideal typical way there is no imperative relation between a focus on technical and economical aspects and goal-based evaluation. Of course, the stated goals can be of a human or organisational character. However, the traditional way of understanding goal-based evaluation is often related to harder measurable goals.

Further, there is no imperative relation between a goal-based approach, and a quantitative process. A judgement of, if the goals have been fulfilled can be evaluated with a qualitative process. As we see it, the differences between a quantitative and qualitative strategy is that the quantitative strategy aims to decide if the goals are fulfilled and which goals that are fulfilled. The fulfilment of the goals will be expressed in quantitative numbers. There are also goals of more social or human character. The fulfilment of these types of goals is preferably expressed in qualitative terms. The qualitative process has also, besides the if- and which questions, a better possibility to describe how the goals are fulfilled. This means that the qualitative approach aims at achieving richer descriptions. The goals that are used for evaluation are derived from an organisational context. That means that they are situationally applicable, which means that they act like specific business goals.

The basic strategy of this approach is to measure if predefined goals are fulfilled or not; to what extent and in what ways. The approach is deductive. What is measured depends on the character of the goals and a quantitative approach as well as qualitative approach could be used. In this paper we adopt the concept of goal-based evaluation from Patton (1990) in order to identify this approach.

2.2       Goal-free evaluation

The second identified approach is a more interpretative approach (e.g. Remenyi, 1999; Walsham, 1993). The interpretative perspective views IT-systems as social systems that have information technology embedded into it (Goldkuhl & Lyytinen, 1982). The aim of interpretive evaluation is to gain a deeper understanding of the nature of what is to be evaluated and to generate motivation and commitment (Hirschheim & Smithson, 1988). The involvement of a wide range of stakeholder groups is often considered essential to this approach of evaluation. This can also be a practical obstacle where time or resources for the evaluation are short. Patton (1990) uses the term goal-free evaluation. Goal-free evaluation is defined as gathering data on a broad array of actual effects and evaluating the importance of these effects in meeting demonstrated needs (Patton, 1990, Scriven, 1972). The evaluator makes a deliberate attempt to avoid all rhetoric related to program goals; no discussion about goals is held with staff; no program brochures or proposals are read; only the program’s outcomes and measurable effects are studied. The aim of goal-free evaluation is to (Patton, 1990):

1)         avoid the risk of narrowly studying stated program objectives and thereby missing important unanticipated outcomes

2)         remove the negative connotations attached to discovery of unanticipated effect: “The hole language of side-effected or secondary effect or even unanticipated effect tended to be a put-down of what might well be a crucial achievement, especially in terms of new priorities.”

3)         eliminate the perceptual biases introduced into an evaluation by knowledge of goals; and

4)         maintain evaluator objectivity and independence through goal-free conditions

In this paper, we adopt the concept of goal-free evaluation from Patton (1990) in order to identify this approach. The basic strategy of this approach is inductive evaluation. The approach aims at discover qualities of the object of study. One can say that the evaluator makes an inventory of possible problems and that the knowledge of the object of study emerges during the progress of the evaluation. As will be shown later in this paper, a goal-free evaluation does not need to be performed with a “high degree of involvement”.

2.3       Criteria-based evaluation

The third identified approach is a criteria-based approach. There are lot of criteria-based approaches around such as checklists, heuristics, principles or quality ideals. In the area of human-computer interaction you can find different checklists or heuristics (e.g. Nielsen, 1994; Nielsen, 1993, Shneiderman, 1998). What is typical for these approaches is that the IT-systems interface and/or the interaction between users and IT-systems acts as a basis for the evaluation together with a set of predefined criteria. More action oriented quality ideals and principles for evaluation can be found in Cronholm & Goldkuhl (2002) and in Ågerfalk et al (2002). The basis for these action-oriented ideals is to understand if and how the IT-system support the actions performed in the business (see discussion of IT-systems in section 3.1)

The criteria used are grounded in and derived from one or more specific perspectives or theories. For example, the criteria in Nielsen’s (1994) checklist are derived from cognitive science and computer science. The action-oriented ideals are mainly derived from language action theory but also inspired by usability issues. Using criteria means to set focus on certain qualities that according to the perspective is important to evaluate. At the same time the attention according to the criteria also de-emphasize other qualities. The criteria chosen governs the evaluator’s attention and thereby the kind of knowledge the evaluator achieves.

Another difference in comparison to goal-based evaluation is that the criteria that are used are not derived from a specific organisational context. That means that they are more general applicable (see section 2.1). Ideal typically, the basic strategy of criteria-based evaluation is deductive. The word criteria is often used in relation to pre-ordinate designs, and the use of this term has a ‘hard’ scientific feel which supports the tendency to prioritize technical and quantitative data (Walsham, 1993). Ideal typically this view is too limited. A criteria-based approach does not exclude of softer criteria; confer for example Ågerfalk et al (2002).

3.         Strategies concerning what to evaluate

We distinguish between two strategies; evaluating:

§          IT-system as such

§          IT-system in use

IT-systems can be viewed from many different perspectives. Our framework for IT evaluation is not dependent on any particular perspective.

3.1       IT-systems as such

Evaluating IT-systems as such means to evaluate the IT-system without any involvement from users. In this situation there are only the evaluator and the IT-system involved. The data sources that could be used for this strategy is the IT-system itself and possible documentation of the IT-system. How the evaluation is performed depends on the “how-strategy” chosen. Choosing to evaluate “It-systems as such” does not exclude any of the strategies of “how to evaluate”. The evaluator could use a goal-based, goal-free or criteria-based strategy.

The outcome of the evaluation is based on the evaluator’s understanding of how the IT-system supports the organisation. This strategy is free from user’s perceptions of how the IT-system benefits to their work. The evaluation object is the IT artefact per se. There is no study of real users’ situational use of the system. The study object is the system and of course its intended use through its implemented functionality. The evaluator explores what is possible to do with the system.

Figure 1: Two possible data sources for IT-systems in use

3.2       IT-systems in use

The other strategy of “what to evaluate” is “IT-systems in use”. Evaluating IT-systems in use means to study a use situation where a user interacts with an IT-system. This analysis situation is more complex than the situation “IT-systems as such” since it also includes a user, but it also has the possibility to give a richer picture.

The data sources for this situation could be interviews of the users and their perceptions and understanding of the IT-system’s quality, observations of users interacting with IT-systems, the IT-system itself and the possible documentation of the IT-system. Compared to the strategy “IT-systems as such” this strategy offers more possible data sources. When there are high requirements on data quality the evaluator can chose to combine all the data sources in order to reach high degree of triangulating. If there are fewer resources at hand the evaluator can chose one or two of the possible data sources.

Figure 2: Four possible data sources for IT-system in use

An argument for choosing the strategy “IT-systems in use” is presented by Whiteside & Wixon (1987). They claim “… usability becomes a purely subjective property of the interaction between a specific user and the computer at a specific moment in time”. There are always subjective perceptions such as the users attitude towards an IT-system that are more difficult to measure.

How the evaluation of “IT-systems in use” is performed depends on the “how-strategy” chosen (see section 2). Ideal typically, it is possible to choose any of the three strategies goal-based, goal-free or criteria-based when studying “IT-systems in use”. The outcome of this evaluation is not only based on the evaluator’s understanding of how the IT-system support the organisation. It is also based on the users perceptions of how the IT-system supports their work.

4.         Characterisation of six generic types of evaluation

Combining the three approaches of “how to evaluate” and the two approaches of “what to evaluate” gives a matrix of six generic types of evaluation (see table 1).

Table 1: The matrix of six generic types of information systems evaluation

In this section, we will characterize the six different types in order highlight similarities and differences. The characterization is done through classifying the evaluation types according to some general criteria:

§          Main perspective

§          What to achieve knowledge about

§          Data sources

§          Deductive or inductive

§          Who will participate

§          When to chose this type

Besides characterising each type, we also give a suggestion of how to perform the evaluation process. We recommend always to start with planning the evaluation. This means to decide upon the following issues: the scope, the level of detail of the evaluation, time, resources and who should evaluate. It also includes identifying the assigner of the evaluation, the data providers and the client (user) of the result. Planning of the evaluation should include other preparations for the evaluation. The evaluator’s pre-knowledge of the context needs to be established. This can be done through a description of the business processes and an overview of the IT-system. The evaluation plan works as a base for choosing one or a combination of several evaluation types (see figure 3). The evaluation process should end with conclusions. The conclusions should consist of a summarised problem analysis and strength analysis. The most salient problems should be highlighted in a problem priority list. If possible, the conclusions should include recommendations for change measures. The specific evaluation process for each type is described in section 4.1-4.6.

Figure 3: The general evaluation

4.1       Type 1 - Goal-free evaluation of IT-systems as such

This type is a combination of goal-free evaluation and IT-systems as such. This combination means that the evaluation is performed more open minded and that the object of evaluation is the IT-system as such (see 0). Using an open mind means that the evaluator explores different qualities of the IT-system. The strategy is inductive (see section 2.2). The data sources for this type are the IT-system itself and descriptions of the IT-system. This type has a low degree of participation and is normally chosen when an introductory study is desired. One aim of the evaluation is to increase the evaluator’s pre-knowledge of the evaluation context.

Table 2: Characterization of goal-free evaluation of IT-systems as such

In order to perform a goal-free evaluation of IT-systems as such we suggest the evaluator describes the functionality of the IT-system and walks through the IT-system in an explorative way. This is done by studying the IT-system itself, the documentation of the IT-system and by interviews with the system owner. The explorations should be evaluated in terms of strengths and problems. This means that the evaluator performs an introductory open minded and problem finding evaluation. We think that this type should in most cases be combined with another evaluation type(s). Therefore, a result from using this type should be a recommendation of a continued evaluation of some other type.

4.2       Type 2 - Goal-based evaluation of IT-systems as such

This type is a combination of goal-based evaluation and IT-system as such. This combination means that the evaluation is performed according to some predefined business goals and that the object of evaluation is the IT-system (see 0). According to the literature the perspective of goal-based evaluation often is technical and economical. We mean that the perspective depends on the character of the goals (see section 2.1). Using a goal-based evaluation combined with studying IT-systems as such means that the evaluator wants to decide if the goals have been fulfilled. The strategy can be seen to be deductive, since the evaluator uses the goals as hypotheses to be “verified” as working and fulfilled. The data sources for this type, besides the IT-system is goal descriptions, requirement specifications and descriptions of the IT-system. This type has a low degree of participation and can be chosen when a focused evaluation according to the stated goals is wanted or when there are fewer resources at hand.

In order to perform a goal-based evaluation of IT-systems as such we suggest that the evaluator describes the business goals and the functionality of the IT-system. Business goals can be found in documents such as strategy or goal documents or can be identified through interviews with business managers. The next step is to describe the functionality of the IT-system. This is done by studying the IT-system itself, the documentation of the IT-system and by interviews with the system owner. When the business goals and the IT-system’s functionality are described a goal-based evaluation of the IT-system can be performed. This means that evaluator decides if the IT-system’s functionality meets the business goals.

Table 3: Characterization of goal-based evaluation of IT-systems as such

4.3       Type 3 - Criteria-based evaluation of IT-systems as such

This type is a combination of criteria-based evaluation and IT-system as such. This combination means that the evaluation is performed according to some predefined general criteria and that the object of evaluation is the IT-system (see 0). The evaluation perspective depends on the character of the criteria (see section 2.3). Using a criteria-based evaluation combined with studying IT-systems as such means that the evaluator wants to decide if the IT-systems’ quality meets the criteria used. The strategy is deductive. The evaluator wants to check if the system fulfils the evaluation criteria.

The data sources for this type are the IT-system itself, descriptions of the IT-system and descriptions of the criteria. This type has a low degree of participation. This type could be chosen when a focused evaluation according to the chosen criteria is desired or when there are fewer resources at hand.

Table 4: Characterization of criteria-based evaluation of IT-systems as such

In order to perform a criteria-based evaluation of IT-systems as such we suggest that the evaluator starts with choosing appropriate criteria. The criteria chosen depends on the desired evaluation perspective. During the evaluation, the criteria both support focusing and limitation. Different perspectives are discussed in section 2.3 above. The next step is to describe the functionality of the IT-system. As mentioned above, this is done by studying the IT-system itself, the documentation of the IT-system and by interviews with the system owner. When the criteria and the IT-system’s functionality are described a criteria-based evaluation can be performed. This means that evaluator decides if the IT-system’s functionality meets the criteria.

4.4       Type 4 – Goal-free evaluation of IT-systems in use

This type is a combination of goal-free evaluation and IT-systems in use. This combination means that the evaluation is performed open minded and that the object of evaluation is the IT-system in use (see 0). Using an open mind means that the evaluator discovers different qualities of the IT-system independent of existing goals. The strategy is inductive since the evaluator strives to be free from predefined values (see section 2.2).

One major difference between this type and type 1 is that there are more data sources at hand. Besides the IT-system and descriptions of the IT-system there is interaction between users and the IT-system. This interaction can be observed. Interviews of the users’ understanding, attitudes and opinions are another possible data source. This type also offers possibilities for a broader participation than type 1 does. Besides the evaluator there are also users involved. This type could be chosen when a broad and deep evaluation is desired and when there are more resources at hand.

In order to perform a goal-free evaluation of IT-systems in use we suggest that the evaluator describes the functionality of the IT-system, a description of users (their pre-knowledge, IT-maturity, role, responsibility) and the interaction between the users and the IT-system. Describing the IT-systems functionality is done by studying the IT-system itself, the documentation of the IT-system and by interviews with the system owner. The user interaction can be observed using a technique called “think aloud” (Monk et al, 1993). The observations could be combined with follow-up interviews. This type is also explorative to its character. The explorations should be evaluated in terms of strengths and problems.

Table 5: Characterization of goal-free evaluation of IT-systems in use

4.5       Type 5 - Goal-based evaluation of IT-systems in use

This type is a combination of goal-based evaluation and IT-systems in use. This combination means that the evaluation is performed according to some predefined business goals and that the object of evaluation is the IT-system in use (see 0). The evaluation perspective depends on the character of the goals (see section 2.1). Using a goal-based evaluation combined with studying IT-systems in use means that the evaluator wants to decide if the goals have been fulfilled. The strategy is thus deductive.

One major difference between this type and type 2 is that there are more data sources at hand. Besides the IT-system, goal descriptions, requirement specifications and descriptions of the IT-system there is an interaction between users and the IT-system to be studied. This interaction can be observed. This interaction depends also on the user’s understanding and attitudes regarding the IT-system. Interviews of the users’ understanding, attitudes and opinions are another possible data source. This type also offers possibilities for a broader participation than type 2 does. Besides the evaluator there are also users involved. This type could be chosen when a focused evaluation according to the stated goals is wanted or when there are more resources at hand.

Table 6: Characterization of goal-based evaluation of IT-systems in use

In order to perform a goal-based evaluation of IT-systems in use we suggest that the evaluator begins with describing the business goals and the functionality of the IT-system. As mentioned above, business goals could be found in documents such as strategy or goal documents or can be identified through interviews with business managers. In the next step the evaluator describes the functionality of the IT-system. Since this type also includes a use situation there are other data sources. The evaluator should observe the interaction between the users and the IT-system and the evaluator should also interview the users about their attitudes towards the IT-system. Furthermore, the evaluator should describe the users’ pre-knowledge, IT-maturity, role and responsibility. When the goals, the IT-system’s functionality, the interaction and the users’ pre-knowledge are described a goal-based evaluation can be performed. This means that the evaluator decides if goals are met.

4.6       Type 6 - Criteria -based evaluation of IT-systems in use

This type is a combination of criteria-based evaluation and IT-system in use. This combination means that the evaluation is performed according to some predefined general criteria and that the object of evaluation is the IT-system in use (see 0). The evaluation perspective depends on the character of the criteria (see section 2.3). Using a criteria-based evaluation combined with studying IT-systems as such means that the evaluator wants to decide if the IT-systems’ quality meets the criteria used. The strategy is deductive.

One major difference between this type and type 3 is that there are more data sources at hand. Besides the IT-system and descriptions of the IT-system, there is an interaction between users and the IT-system. Similar to type 4 and 5 this interaction can be observed. This interaction also results in an understanding and attitudes of the user regarding the IT-system. Interviews of the users’ understanding, attitudes and opinions are another possible data source. This type also offers possibilities for a broader participation than type 3 does. Besides the evaluator there are also users involved. This type could be chosen when a focused but not limited evaluation according to the criteria is desired or when there are more resources at hand.

Table 7: Characterization of criteria-based evaluation of IT-systems in use

In order to perform a criteria-based evaluation of IT-systems in use we suggest that the evaluator begins with choosing appropriate criteria. As mentioned above, the criteria chosen depends on the evaluation perspective wanted. Next, the evaluator describes the functionality of the IT-system. Since this type also includes a use situation there are other data sources. The evaluator should observe the interaction between the users and the IT-system and the evaluator should interview the users about their attitudes towards the IT-system. The evaluator should also describe the users’ pre-knowledge, IT-maturity, role and responsibility. When the goals, the IT-system’s functionality, the interaction and the users pre-knowledge are described a criteria-based evaluation is performed. This means that the evaluator decides if the chosen criteria are met.

5.         Conclusions

In this paper, we have discussed and characterised six generic types of evaluation based on three strategies of how to evaluate and two strategies of what to evaluate. The evaluation types are generated from studying literature but also abstracted from empirical findings in evaluation projects. We have analysed the six evaluation strategies as ideal types.

We conclude that the ideal typical analysis has contributed to question some tight couplings between a goal-based evaluation and a quantitative process. In the literature, goal-based evaluation is traditionally often described as formal-rational, quantitative and focusing on measurable goals that often are of more technical and economical character. Our analysis shows that this must not be the case. It is possible to use a goal-based approach and a qualitative process (see section 2.1 and 4.2). We think that the confusion has to do with traditional ways of looking at evaluation. Concepts like measurable, quantitative, goals and deduction are often clustered and related to positivism. On the other side, concepts like interpretation, qualitative and induction are often clustered and related to hermeneutics.

In this paper, we have tried to nuance the different approaches of evaluation. Metaphorically one can say that we have shown that it is possible to mix eggs from different baskets. Further, we have identified, explained, characterised and exemplified different evaluation types. The evaluation types are derived from a matrix and the purpose of the matrix is to support different choices of how to perform evaluation depending on the evaluation situation.

These different evaluation types can in a practical evaluation situation be utilised together in combined ways. They can be used in different stages, where evaluation results from the earlier stages inform the later ones. It might also be possible to combine some of these types in an integrative way with instant changes in attention. Goal-based, goal-free and criteria-based can be seen as three fundamental attitudes which can applied in evaluative situations. An alternation between these attitudes might be possible to do during a combined evaluation process.

As further research we propose an analysis of how the different evaluation types can be combined (for triangulating purposes). There is also a need to refine and operationalize process models for each evaluation type and to complement with description techniques. Recommended documentation techniques for each process model support the understanding of how to use the evaluation types. These generic types might also be possible to use in order to characterize and assess existing evaluation approaches and methods. The generic types might also be helpful (as a conceptualisation of evaluation) when one is trying to integrate and combine different evaluation methods. 

References

• Cronholm S & Goldkuhl G (2002) “Actable Information Systems - Quality Ideals Put Into Practice”, presented at the Eleventh Conference On Information Systems (ISD 2002). 12-14 September, Riga, Latvia.
• Goldkuhl G & Lyytinen K (1982) “A language action view of information systems”, proceedings of third Intl. Conference on Information systems, Ginzberg & Ross (eds), Ann Arbor
• Good M, Spine T, Whiteside J & George P (1986) “User-derived impact analysis as a tool for usability engineering”, Human Factors in Computing Systems (CHI´86 Proceedings), New York: ACM Press
• Hirschheim R & Smithson S (1988) “A Critical Analysis of Information Systems Evaluation”, in IS Assessment: Issues and Changes (eds N Bjorn- Andersen & G B Davis), North-Holland, Amsterdam
• Monk A, Wright P, Haber J, Davenport L (1993), Improving your Human-Computer Interface, Prentice Hall, New York
• Nielsen J (1993) Usability Engineering, San Diego, California, Academic Press
• Nielsen J (1994). How to Conduct a Heuristic Evaluation, [online], http://useit.com/papers/heuristic/heuristic_evaluation.html
• Patton M Q (1990) Qualitative Evaluation and Research Methods, second edition, Sage Publications, London
• Remenyi D & Sherwood-Smith M (1999) “Maximise Information Systems Value by Continuous Participative Evaluation”, Logistics Information Management, Vol 12 No 1/2 pp 14-31
• Seddon P (2001) “IT Evaluation Revisted: Plus Va Change”, proceedings of Eight European Conference on Information Technology (ECITE), Oxford, United Kingdom
• Scriven, (1967 ) The Methodology of Evaluation, Rand McNally, Chicago
• Scriven M (1972) “Pros and Cons About Goal-Free Evaluation”, Evaluation Comment 3:1-7, in Philosophical Redirection of Educational Research: The Seventy-First Yearbook of the National Society for the Study of Education, Thomas L G (ed), University of Chicago Press
• Shneiderman B (1998) Designing the User Interface: Strategies for Effective User Interface Interaction, 3rd ed., Reading, MA, Addison Wesley Longman
• Strauss A, Corbin J (1998) Basics of Qualitative Research, Techniques and Procedures for Developing Grounded Theory, Sage Publications, Beverly Hills, California
• Walsham G (1993) Interpreting Information Systems in Organisations, Wiley & Sons
• Whiteside J, Wixon D (1987) “The dialectic of usability engineering”, in Human-computer interaction - interact ‘87, Bullinger H-J, Shackel B (eds), Amsterdam, North-Holland
• Ågerfalk P, Sjöström J, Eliasson E, Cronholm S, Goldkuhl G (2002) “Setting the Scene for Actability Evaluation: Understanding Information Systems in Context”, Proceeding of ECITE 2002, 15-16 July, Paris , France

EJISE is published by Academic Conferences Limited
Curtis Farm, Kidmore End, Nr Reading RG4 9AY, England
Tel: +44 (0)1189 724148, Fax: +44 (0)1189 724691, Email: info@ejise.com

Send mail to info@academic-conferences.com with questions or comments about this web site.
Copyright © 2002-2006 Electronic Journal of Information Systems Evaluation
Last modified: September 29, 2005
ISSN 1566-6379