The development and expansion of evaluation theory and practice is at the core of several different disciplines. It is important to scrutinize theories, approaches, and models used in evaluation (research) as well as evaluation research approaches’ philosophical underpinnings. Stufflebeam (2001) provides a good example. He identified and evaluated twenty-two different generic program evaluation approaches. In the Information Systems (IS) field, IS evaluation and IS evaluation research have been stressed as critical means in advancing the field (Bjørn-Andersen & Davis, 1988). Generally, IS evaluation is concerned with the evaluation of different aspects of real-life interventions in the social life where IS are critical means in achieving the interventions’ anticipated goals. IS evaluation research can be considered a special case of evaluation research. The aim of IS evaluation research is to produce ever more detailed answers to the question of why an IS initiative works (better) for whom and in what circumstances.

Based on the philosophy of critical realism, we suggest that realistic IS evaluation research can be a means to advance IS evaluation research. Critical realism has become an important perspective in modern philosophy and social science (Archer et al. 1998, Robson 2002), but critical realism is to a large extent absent in IS research. We argue that IS evaluation research based on the principles and philosophy of critical realism overcomes some of the problems associated with “traditional” IS evaluation research approaches. Realistic IS evaluation research sees the outcome of an intervention—like the implementation of an Enterprise Resource Planning (ERP) system—as the result of generative mechanisms and the context of those mechanisms, and focuses both structure and agency. A focus on the generative mechanisms entails examining the “causal” factors that inhibit or promote change when an IS intervention, for example, an ERP implementation, occurs. This is missed in “traditional” evaluation research approaches. The strength of an IS evaluation research approach depends on the perspicacity of its view of explanation.

The way critical realism and realistic IS evaluation research address the agency/structure “dilemma” means that they avoid the “fallacy of central conflation”: the tendency to see structure as so closely intertwined with every aspect of practice that

 “…the constituent components [of structure and agency] cannot be examined separately. ...In the absence of any degree of autonomy it becomes impossible to examine their interplay” (Archer 1988).

Realistic IS evaluation research is supportive of: 1) the use of both quantitative and qualitative evaluation methods, 2) the use of extensive and intensive research design, and 3) the use of fixed and flexible research design.

Drawing on “general” evaluation research, this paper also discusses traditional IS evaluation research approaches. We present four traditional IS evaluation research approaches, point out their major strengths and weaknesses, and compare them with realistic IS evaluation research.

The remainder of the paper is organized as follows: the next section presents critical realism as the underpinning philosophy for realistic IS evaluation research. This is followed by a presentation and discussion of realistic IS evaluation research. Section 4 briefly contrasts realistic IS evaluation research with four traditional IS evaluation research approaches. The final section presents concluding remarks and suggests implications for further research.

2.         Critical realism

Critical realism was developed as an alternative to traditional positivistic models of social science as well as an alternative to postmodern approaches and theories and constructivism. The most influential writer on critical realism is Roy Bhaskar (1978, 1989, 1998). Archer et al. (1998) and Lòpez and Potter (2001) contain chapters focusing on different aspects of critical realism, ranging from fundamental philosophical discussions to how statistical analysis can be used in critical realism research.

Critical realism can be seen as a specific form of realism. Its manifesto is to recognize the reality of the natural order and the events and discourses of the social world. It holds that “we will only be able to understand—and so change—the social world if we identify the structures at work that generate those events and discourses … These structures are not spontaneously apparent in the observable pattern of events; they can only be identified through the practical and theoretical work of the social sciences.” (Bhaskar 1989). Bhaskar (1978) outlines what he calls three domains: the real, the actual, and the empirical (Table 1). The real domain consists of underlying structures and mechanisms, and relations; events and behavior; and experiences. The generative mechanisms, residing in the real domain, exist independently of but capable of producing patterns of events. Relations generate behaviors in the social world. The domain of the actual consists of these events and behaviors. Hence, the actual domain is the domain in which observed events or observed patterns of events occur. The domain of the empirical consists of what we experience; hence, it is the domain of experienced events.

Table 1: Ontological assumptions of the critical realistic view of science (Bhaskar 1978).

Xs indicate the domain of reality in which mechanisms, events, and experiences, respectively reside, as well as the domains involved for such a residence to be possible.

Bhaskar argues that “…real structures exist independently of and are often out of phase with the actual patterns of events. Indeed it is only because of the latter we need to perform experiments and only because of the former that we can make sense of our performances of them. Similarly it can be shown to be a condition of the intelligibility of perception that events occur independently of experiences. And experiences are often (epistemically speaking) ‘out of phase’ with events—e.g. when they are misidentified. It is partly because of this possibility that the scientist needs a scientific education or training. Thus I [Bhaskar] will argue that what I call the domains of the real, the actual and the empirical are distinct.” (Bhaskar 1978). Critical realism also argues that the real world is ontologically stratified and differentiated. The real world consists of a plurality of structures that generate the events that occur and do not occur (these structures are called generative mechanisms). From an epistemological stance, concerning the nature of knowledge claim, the realistic approach is non-positivistic which means that values and facts are intertwined and hard to disentangle.

Layder addresses how to do empirical and theoretical research from a critical realism perspective. In general he is sympathetic to constructivism, and even to some of the ideas in grounded theory, although he is critical to “pure” grounded theory. Layder is also, in part, sympathetic to ideas from middle-range theory (Merton, 1967). Layder says: “Put very simple, a central feature of realism is its attempt to preserve a ’scientific’ attitude towards social analysis at the same time as recognizing the importance of actors´ meanings and in some way incorporating them in research. As such, a key aspect of the realistic project is a concern with causality and the identification of causal mechanisms in social phenomena in a manner quite unlike the traditional positivist search for causal generalizations.“ (Layder, 1993).

Layder suggests a stratified or layered framework of human action and social organization. The framework includes macro phenomena, like structural and institutional phenomena, as well as micro phenomena, like behavior and interaction. Figure 1 depicts Layder’s framework and describes levels (elements/sectors) of potential areas of interest in IS research.

We will briefly present the different elements and, for convenience, start with the self and work towards the macro elements. The first level is self, which refers “... primarily to the individual’s relation to her or his social environment and is characterized by the intersection of biographical experience and social involvements.“ (Layder, 1993). Self focuses on how an individual is affected by and responds to social situations. In encountering social situations individuals use strategies and tactics, based on their ”theories” (mental models), to handle the situations. In general, the self and situated activity have as their main concern

“...the way individuals respond to particular features of their social environment and the typical situations associated with this environment.“ (Layder, 1993).

Figure 1: Research map (adapted from Layder, 1993).

In situated activity the focus is on the dynamics of social interaction. The area of self focuses how individuals are affected and respond to certain social processes whereas situated activity focus on the nature of the social involvement and interactions. This means that interactions and processes have features that are the result of how the participating individuals’ behaviors intermesh and coalesce.

The focus in setting is on the intermediate forms of social organization. A setting provides the immediate arena for social activities. A setting can be things like the culture of the organization, artifacts like ICT-based IS that are used in situated activities, power and authority structures. It should be stressed that setting is not just a particular patterns of activity. The wider macro social forms that provide the more remote environment of social activity are refereed to as the context. Although there is not a clear border between settings and context and some social forms straddle the two elements it can be fruitful to distinguish them. In general, context refers to large-scale and society-wide features.

Viewing the design, development, implementation, and use of IS as layers of human activity and social organization that are interdependent has two major advantages. It enables an evaluation researcher to be sensitive to the different elements with their distinctive features. Critical realism and Layder’s framework stress that the layers operate on different ”time scales”. This means that an evaluation researcher has to view the operation of the elements not only vertically but also horizontally.

Critical realism has influenced a number of social science fields, e.g., organization studies—see, for example, Tsang and Kwan (1999), Tsoukas (1989), Reed (1997, 2001), and Ackroyd and Fleetwood (2000a, b). Critical realism has also influenced real world research (Robson 2002) as well as evaluation research (Pawson & Tilley 1997, Kasi 2003). With few exceptions, critical realism is almost invisible in the IS-field. Mutch (1997, 2002), Dobson (2001), and Carlsson (2004) argue for the use of critical realism in IS research and discuss how critical realism can overcome problems associated with postmodern approaches and theories as well as problems associated with constructivism. Mutch (2002) notes how critical realism can overcome problems in actor-network theory. Mingers (2001) used, in part, critical realism to argue for the use of pluralist methodologies in IS research. He also used an approach influenced by critical realism in reviewing the use of multimethod research in the IS literature (Mingers 2003).

3.         Realistic information systems evaluation research

Driving realistic IS evaluation research is the aim to produce ever more detailed answers to the question of why an IS initiative—IS, types of IS, or IS implementation—works for whom and in what circumstances. This means that evaluation researchers attend to how and why an IS initiative has the potential to cause (desired) changes. Realistic IS evaluation research is applied research, but theory is essential in every aspects of IS evaluation research design and analysis. The goal is not to develop theory per se, but to develop theories for practitioners, stakeholders, and participants.

A realistic evaluation researcher works as an experimental scientist, but not according to the logics of the traditional experimental research. Said Bhaskar: “The experimental scientist must perform two essential functions in an experiment. First, he must trigger the mechanism under study to ensure that it is active; and secondly, he must prevent any interference with the operation of the mechanism. These activities could be designated as ‘experimental production’ and ‘experimental control’.” (Bhaskar 1998). Figure 2 depicts the realistic experiment.

Figure 2: The realistic experiment (Pawson & Tilley, 1997, p 60)

Realistic evaluation researchers do not conceive that IS initiatives “work”. It is the action of stakeholders that makes them work, and the causal potential of an IS initiative takes the form of providing reasons and resources to enable different stakeholders and participants to “make” changes. This means that a realistic evaluation researchers seek to understand why an IS initiative (IS implementation) works through an understanding of the action mechanisms. It also means that a realistic evaluation researcher seeks to understand for whom and in what circumstances (contexts) an IS initiative works through the study of contextual conditioning.

Realistic evaluation researchers orient their thinking to context-mechanism-outcome pattern configurations—called CMO configurations. This leads to the development of transferable and cumulative lessons from IS evaluation research. A CMO configuration is a proposition stating what it is about an IS initiative (IS implementation) which works for whom in what circumstances. A refined CMO configuration is the finding of IS evaluation research—the output of a realistic evaluation study.

Realistic evaluation researchers examine outcome patterns in a theory-testing role. This means that a realistic evaluation researcher tries to understand what are the outcomes of an IS initiative (IS implementation) and how are the outcomes produced. Hence, a realistic evaluation researcher is not just inspecting outcomes in order to see if an IS initiative (IS implementation) works, but are analyzing the outcomes to discover if the conjectured mechanism/context theories are confirmed.

In terms of generalization, a realistic evaluation researcher through a process of CMO configuration abstraction creates “middle range” theories. These theories provide analytical frameworks to interpret differences and similarities between types of IS initiatives (IS implementations).

Given that the goal is to develop theories—construct and test context-mechanism-outcome pattern explanations—for practitioners, stakeholders, and participants, realistic IS evaluation researchers need to engage in a teacher-learner relationship with these IS practitioners, stakeholders, and participants.

Realistic IS evaluation research design employs no standard formula. The base strategy is to develop a clear theory of IS initiative mechanisms, contexts and outcomes. Given the base strategy, a realistic evaluation researcher has to design appropriate empirical methods, measures, and comparisons. Realistic IS evaluation research is supportive of the use of both quantitative and qualitative evaluation methods or in other words it is supportive of the use of both intensive and extensive approaches.

Realistic IS evaluation based on the above may implemented through a realistic effectiveness cycle (Figure 3).

Figure 3: The realistic effectiveness cycle (Pawson & Tilley, 1997; Kazi, 2003)

The starting point is theory. Theory includes proposition on how the mechanisms introduced by an IS invention into pre-existing contexts can generate outcomes. This entails theoretical analysis of mechanisms, contexts, and expected outcomes. This can be done using a logic of analogy and metaphor. The second step consists of generating “hypotheses”. Typically the following questions would be addressed in the hypotheses: 1) what changes or outcomes will be brought about by an IS intervention, 2) what contexts impinge on this, and 3) what mechanisms (social, cultural and others) would enable these changes, and which one may disable the intervention. The third step is the selection of appropriate data collection methods—as stressed, realists are committed methodological pluralists. In this step it might be possible to provide evidence of the IS intervention’s ability to change reality. Based on the result from the third step, we may return to the programme (the IS intervention) to make it more specific as an intervention of practice. Next, but not finally, we return to theory. The theory may be developed, the hypotheses refined, the data collection methods enhanced, etc.

4.         Traditional IS evaluation research approaches

This section briefly reviews the “traditional” IS evaluation research approaches and points out their major strengths and weaknesses. The approaches are: 1) the experimental approach, 2) the pragmatic approach, 3) the constructivist approach, and 4) the pluralist approach. The review is done in light of the aim of IS evaluation research as stated in Section 1: “The aim of IS evaluation research is to produce ever more detailed answers to the question of why an IS initiative works for whom and in what circumstances”.

4.1       Experimental IS evaluation research

The experimental IS evaluation research approach is the oldest IS evaluation research approach and it builds on the logic of experimentation: take two more or less matched groups (situations) and treat one group and not the other. By measuring both groups before and after the treatment of the one, an evaluator can get a “clear” measure of the impact of the treatment (Table 2). To exemplify, the purpose is to evaluate the effects—outcomes—of a specific Decision Support Systems (DSS) used for supporting bank personnel in deciding on loans. Ideally the experimental and the control groups are identical. Hence, it is only the application (use) of the DSS that differs and is responsible for the outcome differences.

Table 2: Experimental IS evaluation research

Evaluation researchers have recognized the practical difficulties in doing pure experimental evaluation research, and thus the idea of quasi-experimental evaluation research was developed (Campbell & Stanley 1963). Quasi-experimental evaluation research does not meet the experiment requirements and therefore does not exhibit complete internal validity.

Early IS evaluation research was to a large extent based in the experimental approach—for good examples, see the “Minnesota experiments“ (Dickson et al. 1977) and Benbasat (1989). There are two major problems with experimental IS evaluation research. First, the studies are to a large extent a-theoretical and non-theoretical. The studies do not answer the question of why an IS (or type of IS) works for whom and in what circumstances. In discussing DSS evaluation research—especially presentation formats in DSS—Carlsson and Stabell conclude: “As we see it, part of the problem is research without a suitable theory, at time without any theory. Typically such work does not present a coherent theoretical argument for how alternative presentation formats might make a difference in the decision context considered.” (Carlsson & Stabell 1986). Second, to meet the experiment requirements an experimenter (evaluator) must in most cases create an unrealistic situation and reduce intermediary variables that might affect the outcome. In other words, experimental IS evaluation research tries to minimize all the differences, except one, between the experimental and the control groups. This means stripping away the context and yielding results that are only valid in other contextless situations.

4.2       Pragmatic IS evaluation research

Pragmatic IS evaluation research was developed, in part, as a response to the problems associated with the experimental IS evaluation research approach. The pragmatic evaluation research approach represents a use-led model of evaluation research, stressing utilization: the basic aim of IS evaluation research is to develop IS initiatives (implementation of IS) which solve “problems”—problems can be organizational problems like reduced competitiveness or far from good customer services. The problems addressed in an intervention and the intervention’s goals are not given, but are politically colored and defined by stakeholders.

Following Patton’s (1982, 2002) view on evaluation, this approach stresses that the test bed is whether the practical cause of IS intervention is forwarded or not. It is not a question of following certain epistemological axioms. The pragmatic IS evaluation research approach has a toolbox view on research methods. Pragmatic evaluation research is comprised of standard research tasks. Evaluation research success is depending on a researcher’s sheer craft and this craft is primarily learned through exemplars. In doing evaluation research a researcher selects the appropriate tools and measures from the available toolbox. The rule of thumb is that the evaluation mandate comes from the stakeholder(s) responsible for the development, implementation, and use of the information systems. The more explicit the mandate is, the more compressed and technical is the evaluator’s role.

There are two major problems with pragmatic IS evaluation research. First, the studies do not answer the question of why an IS initiative (IS implementation) works for whom and in what circumstances. Second, since the evaluation mandate is coming from stakeholders this can lead to “evaluation (evaluation researcher) for hire”.

4.3       Constructivist IS evaluation research

In line with the general development in many social sciences during the 1970’s, phenomenology, hermeneutic, and interpretative approaches influenced evaluation research. This meant that focus came to be on social processes. The constructivist evaluation approach argues that IS initiatives should not be treated “…as ‘independent variables’, as ‘things’, as ‘treatments’, as ‘dosages’.” (Pawson & Tilley 1997). Instead all IS initiatives are “…constituted in complex processes of understanding and interaction” and an IS initiative (IS implementation) will work “through a process of reasoning, change, influence, negotiation, battle of wills, persuasion, choice increase (or decrease), arbitration or some such like.” (Pawson & Tilley 1997). Following Guba and Lincoln (1989) it can be argued that the social world is fundamentally a process of negotiation and so are IS initiatives. Hence evaluation research is a process of negotiation and evaluators are the “orchestrators” of negotiation processes.

The major problem with the constructivist IS evaluation approach is its inability to grasp those structural and institutional features of society and social organization which are in some respects independent of the agents’ reasoning and desires but influence (affect) an IS initiative and the negotiation process. To develop theories of why an IS initiative (IS implementation) works for whom and in what circumstances requires a researcher to generate some means of making independent judgments about the institutional structure and power relations present in an IS initiative. This is something not possible in constructivist IS evaluation research, but institutional structure and power relations affect—working as constrainers and enablers—an IS initiative and the negotiation process.

4.4       Pluralist IS evaluation research

Having presented three “traditional” IS evaluation research approaches and noted their strengths and weaknesses, one can imagine the attractiveness of developing an approach combining the strengths of the three approaches: an approach combining the rigor of experimentation with the practice of pragmatism, and with the constructivist’s empathy for the voices of the stakeholders. The pluralist IS evaluation research approach was developed more or less on these premises. The major problem of the approach is that it does not address what it is with an IS initiative which makes it worth. It also lacks an ontological position.

To summarize: This section presented four major and traditional IS evaluation research approaches. They can all be used in evaluation research, but given the aim of IS evaluation research they have major drawbacks. Realistic IS evaluation research approach overcomes the drawbacks noted with the four traditional evaluation approaches.

5.         Concluding remarks and implications for further research

The strength of an IS evaluation research approach depends on the perspicacity of its view of explanation. Realistic IS evaluation research has a different view of explanation than the traditional IS evaluation research approaches. This is a major advantage compared with the traditional approaches. Realistic evaluators orient their thinking to context-mechanism-outcome and this leads to the development of transferable and cumulative lessons from IS evaluation research. Realistic IS evaluation research is an approach being real, realistic, and realistic. It is real and deals with a stratified reality. It follows a realistic methodology and produces realistic evaluation.

We have outlined a new IS evaluation approach. It needs to be refined and also needs to be applied in actual IS evaluation research. The use of realistic evaluation has proved to be fruitful in other areas (Pawson & Tilley 1997, Mark et al. 2000, Kasi 2003) and promise to be a way to advance IS evaluation research.

References

• Ackroyd, S. and S. Fleetwood (Eds)(2000a). Realistic Perspectives on Management and Organisations, Routledge, London.
• Ackroyd, S. and S. Fleetwood (2000b). Realism in contemporary organisation and management studies. In Realistic Perspectives on Management and Organisations, S. Ackroyd and S. Fleetwood (Eds), Routledge, London, 3-25.
• Archer, M. (1988). Culture and Agency: The Place of Culture in Social Theory. Cambridge University Press, Cambridge, UK.
• Archer, M., Bhaskar, R., Collier, A., Lawson, T. and A. Norrie (Eds)(1998). Critical Realism: Essential Readings, Routledge, London.
• Benbasat, I. (1989). The Information Systems Research Challenge: Experimental Research Methods. Harvard Business School, Boston, MA.
• Bhaskar, R. (1978). A Realistic Theory of Science. Harvester Press, Sussex.
• Bhaskar, R. (1989). Reclaiming Reality. Verso, London.
• Bhaskar, R. (1998). The Possibility of Naturalism. Third edition, Routledge, London.
• Bjørn-Andersen, N. and G.B. Davis (Eds)(1988). Information Systems Assessment: Issues and Challenges. North-Holland, Amsterdam.
• Campbell, D. and J. Stanley (1963). Experimental and Quasi-Experimental Evaluations in Social Research. Rand McNally, Chicago, IL.
• Carlsson, S.A. (2004). Using critical realism in IS research. In Handbook of Information Systems Research, M.E. Whitman and A.B. Woszczynski (Eds.), Idea Group Publishing, Hershey, PA, 323-339.
• Carlsson, S. and C.B. Stabell (1986). Spreadsheet programs and decision support: a keystroke-level model of system use. In Decision Support Systems: A Decade in Perspective, E.R. McLean and H.G. Sol (Eds), North-Holland, Amsterdam, 113-128.
• Dickson, G., J.A. Senn and N. Chervaney (1977). Research on management information systems: the Minnesota experiments. Management Science, 23(9), 913-923.
• Dobson, P.J. (2001). The philosophy of critical realism—an opportunity for information systems research. Information Systems Frontier, 3(2), 199-201.
• Guba, Y. and E. Lincoln (1989). Fourth Generation Evaluation. Sage, London.
• Kasi, M.A.F (2003). Realistic Evaluation in Practice. Sage, London.
• Layder, D (1993). New Strategies in Social Research. Polity Press, Cambridge, UK.
• Lòpez, J. and G. Potter (Eds)(2001). After Postmodernism: An Introduction to Critical Realism. Athlone, London.
• Mark, M.M., Henry, G.T. and G. Julnes (2000). Evaluation: An Integrated Framework for Understanding, Guiding, and Improving Public and Nonprofit Policies and Programs. Jossey-Bass, San Francisco.
• Merton, R (1967). On Theoretical Sociology. Free Press, New York.
• Mingers, J. (2001). Combining IS research methods: towards a pluralist methodology. Information Systems Research, 12(3), 240-259.
• Mingers, J. (2003). The paucity of multimethod research: review of the IS literature. Information Systems Journal, 13(3), 233-249.
• Mutch, A. (1997). Critical realism and information systems: an exploration. The 7th Annual BIT Conference, Manchester.
• Mutch, A. (2002). Actors and networks or agents and structures: towards a realistic view of information systems. Organization, 9(3), 477-496.
• Patton, M.Q. (1982). Practical Evaluation. Sage, Beverly Hills, CA.
• Patton, M.Q. (2002). Qualitative Research and Evaluation Methods. Third edition, Sage, London.
• Pawson, R. and N. Tilley (1997). Realistic Evaluation. Sage, London.
• Reed, M.I. (1997). In praise of duality and dualism: rethinking agency and structure in organizational analysis. Organization Studies, 18(1), 21-42.
• Reed, M.I. (2001). Organization, trust and control: a realistic analysis. Organization Studies, 22(2), 210-228.
• Robson, C. (2002). Real World Research. Second edition, Blackwell, Oxford.
• Stufflebeam, D.L. (2001). Evaluation models. New Directions for Evaluation, 89(Spring), 7-98.
• Tsang, E.W. and K.-M. Kwan (1999). Replication and theory development in organizational science: a critical realistic perspective. Academy of Management Review, 24(4), 759-780.
• Tsoukas, H. (1989). The validity of idiographic research explanations. Academy of Management Review, 14(4), 551-561.

EJISE is published by Academic Conferences Limited
Curtis Farm, Kidmore End, Nr Reading RG4 9AY, England
Tel: +44 (0)1189 724148, Fax: +44 (0)1189 724691, Email: info@ejise.com

Send mail to info@academic-conferences.com with questions or comments about this web site.
Copyright © 2002-2006 Electronic Journal of Information Systems Evaluation
Last modified: September 29, 2005
ISSN 1566-6379