1.        Introduction

The aim of this paper is to contribute to information systems evaluation by introducing narratives in the evaluation process. The potential with narratives is that they can convey meanings, interpretations, and knowledge (learning) about the system, which can be used for further action. The aim of the paper should be viewed in relation to some practical issues in IS evaluation, such as evaluations is a problematic (Irani and Love 2001) and complex process (Jones and Hughes 2001), which becomes more difficult with increased complexity of IS (Farbey, Land and Targett 1995), and the growing concern that information systems do not deliver business value (Irani and Love 2001).

The literature describes several roles for IS evaluation. Serafeimidis and Smithson (2003) described control, sense-making, learning and exploratory orientations in IS evaluation. The view taken on evaluation of information systems in this paper is pragmatic: Once a particular system has been implemented, the focus of evaluation should be on continuously improving the benefits received. The pragmatic approach is based on three assumptions about evaluation in this context.

§       Firstly, evaluations should form the basis for action: do not measure if you cannot act on the measurement.

§       Secondly, post evaluations ought to be carried out with the goal of improving the use of system, not only assessing the worth of a system.

§       Thirdly, narratives can grasp the complexity of information systems better than traditional evaluation approaches, such as return on investment or total cost of opportunity.

The paper builds on research from interpretive research in information systems, such as Klein and Meyer (1999), Serafeimidis and Smithson(2000) and Walsham (1993; 1995; 1999); post-modern institutional theory (Meyer and Rowan 1977); and narratives in , accounting (Llewellyn 1998) knowledge management (Snowden 2002; Swap, Leomard, Shields and Abrams 2001), and requirements engineering (Jarke, Bui and Caroll 1998). The first two are primarily used as theoretical ground for the use and relevance of narratives whereas the latter is used as a source of practical and methodological inspiration. Thus, the aim of this paper is not try to improve the understanding of the evaluation process as such, which is common in interpretative research (see for instance Walsham 1993), but to provide inputs to how we conceptually can advance IS evaluation.

The paper is organised as follow. The following section addresses and discusses evaluation and role of measurement in evaluation. Following this is the theory of action and learning is described, which is the theoretical ground for improvement driven evaluation. Narratives and the use of narratives, which should be interpreted as the means for action, in evaluations are then explored and discussed. The paper ends with a discussion and conclusion on the practical implications for information systems evaluation.

2.        IS evaluation and measurements

While it is reasonably easy to evaluate tangible implementation costs, e.g. software license, hardware, consultancy, and training, other intangible cost are much more difficult to measure and evaluate (e.g. productivity dip and resistance to change). As a response to these difficulties both practitioners and academics have developed a number of methods and tools to support the process of determining the costs and value of IS. These issues and others have lead to extensive research into IS evaluation. The remainder of this section highlight some research contributions, which are important to understand the role of narratives in IS evaluation.

Evaluation is not a simple and straightforward process (Jones and Hughes, 2001) and may have many purposes, e.g. control projects, govern change management, communication, improvements, resource allocation, motivation, and long term planning (Sinclair and Zairi, 1995). In addition, Remenyi and Sherwood-Smith (1999) described two practical and very relevant issues in IS evaluation. The first is the so-called evaluation gap. This occurs when the evaluator distance themselves from the project and lose sight of the business objectives. The second concerns that business objectives of the IS project, e.g. organisational change and change management, are often forgotten or superficially attended to in evaluations.

As the field of IS evaluation has matured the view of IS evaluations has changed over the past years. Today there is an increased awareness about the importance of an ongoing evaluation process. There is also a growing number of IS researchers arguing that IS evaluation should focus on how IS supports businesses – the use of system, i.e. a formative evaluation process (Remenyi and Sherwood-Smith, 1999). However, a practical issue is that the purpose of IS evaluation is often to close the project (Kumar, 1990; see also Seddon et al., 2002), with emphasis on ROI (Murphy and Simon, 2002). Hirschheim and Smithson (1998) offer an explanation for this. They claim that there is a widespread belief that IS are fundamentally a technical system. Consequently, this has lead to ”a more ”technical” interpretation of evaluation” (p. 402) with a focus on tools and techniques and thereby omitting the social domain. This makes it unlikely to produce a “true” or meaningful evaluation picture (Hirschheim and Smithson, 1998).

A common factor in all evaluations is the use of measurements (Venkatraman and Ramanujam, 1986). Strassman (1985, p. 100) stresses that: “You cannot measure what is not defined. You also cannot tell whether you have improved something if you have not measured its performance”, i.e. the need of an operational definition. This is, however, difficult since “figures never are facts” (Hoebeke, 1990). Hoebeke (1990) made this comment in relation to a discussion regarding the use of measurements based on calculations, e.g. financial accounting. Measurements are invariably used in complex sense-making processes where both translations and interpretations take place, usually several times in different steps (figure 1). This, as Hoebeke (1990) points out, makes the concept of relevance a lot more important than objectivity - there has to be a shared meaning of the interpretations and their impacts by those who play a role in the process of collecting, translating and interpreting the measures, as well as those who act on decisions based on said measurements. Hoebeke’s main point is that in organisational sense-making processes such as evaluation it is impossible to have a fit between measure and action, because of the interpretations taking place.

Figure 1: Chain of activities that take place in organisational sense‑making

Farbey et al. (1995) implicitly addressed sense-making problem. They proposed a model which is based on the perception that it is possible to stratify different types of organisational change and connecting it to different IS. Their model consists of eight levels, and while the classification is not rigid, it still implies that higher levels of change increase the potential benefits, but also increase the uncertainty of outcome. Potential benefits and level of uncertainty are both cumulative, thus systems classified on a certain level may have all the benefits (and accumulated uncertainty) from any or all the levels below. Farbey et al. (1995) conclude that for the implementation of systems on the 8^th level (business transformation) “… benefits stem from the transformation as a whole. IT provides only one component of what is often a complex series of changes. It is not possible to attribute a portion of the benefits gained to any one factor” (p. 49). It would then be highly unlikely that any two implementations will have identical requirements or consequences, even if they are based on the same generic software packages. While the potential benefits might be articulated, it makes the actual benefits from implementing an IS hard to define, predict, and evaluate.

In summary, we argue that evaluation should be used as a basis for action. IS implementations are only limited in time in the most abstract sense and governed by other organisational activities; project plans, budget years, top management changes and organisational strategic decisions. Implementing a complex IS are deeply interconnected with organisational change and for the lifetime of the system, it remains a large part of the organisations formalised internal framework. Evaluation in this context should be performed continuously, combined with process reviews and organisational development, aiming to get the desired alignment and fit over time. To get the most out of a large organisation, the result of evolution has to be evaluated and new requirements formulated in an iterative process. The key to getting the most out of any IS is the use; use in this perspective is regarded in the broadest sense, including other use, miss-use, abuse and non use of system by humans and connected IS. To achieve the intended use of the system in the organisational context, organisational goals and beliefs have to be communicated to members of the organisation. Organisations use myths and narratives to make sense of equivocal situations and they are used as precedents for future actions, serving as “blue-prints” for desirable behaviour (Alvarez and Urla, 2002). The following section presents theory of action as theoretical ground and justification for the use of narratives.

3.        Theory of action and narratives

Human actions are not always what they seem to be. Humans invariably employ text rich documents to propose ideas, argument cases and give verdict on actions taken. Therefore, by including narratives in the continuous IS evaluation process, we predict that business managers and IS users can increase their learning capacity and increase the value of the investment.

One source of theoretical foundation of our ideas can be founding Argyris and Schön (1974), who suggest that people act in accordance with their mental maps rather than the theories they espouse. People are not aware of the mental maps or theories they do use (Argyris 1980). One could say that there is a split between theory and action. Argyris and Schön suggest that two theories of action are involved. “When someone is asked how he would behave under certain circumstances, the answer he usually gives is his espoused theory of action for that situation. This is the theory of action to which he gives allegiance, and which, on request, he communicates to others. However, the theory that actually governs action is theory-in-use” (Argyris and Schön 1974, p. 6-7, our italics). As humans invariably think and express themselves using the full depth of language - by telling “stories” - it is therefore conceivable that unless external observations “verifies” the stories. Stories told do not reflect theory-in-use but espoused theory.

A model of the processes involved is required to appreciate fully theory-in-use. Argyris and Schön (1974) initially looked to three elements: Governing variables, Action strategies and Consequences. Argyris (1976) proposed the double loop learning theory, which concerns changing underlying values and assumptions, i.e. learning. The focus of the theory is on solving problems that are complex and ill structured and which change as problem-solving advances. Typically, interaction with others is necessary to identify the conflict. There are four basic steps in the action theory learning process: (1) discovery of espoused and theory-in-use, (2) invention of new meanings, (3) production of new actions, and (4) generalization of results. Double loop learning involves applying each of these steps to itself. In double loop learning, assumptions underlying current views are questioned and hypotheses about behaviour tested publicly. The result of double loop learning should be increased effectiveness of action and better acceptance of failures and mistakes.

Narratives are used to persuade, convince, and make people act and behave in certain ways – a tool for learning and action. The alternative in organisation for reasoning, learning, and persuading is to use numbers and calculations, including financial statements, investment calculations, and time reports (Llewellyn 1998). Clausen (1994, p. 45) states that: “Using narratives in the system development process seems to be a way in which designers will be able to come up with the kind of descriptions that are asked for.” Besides everyday life narratives that are used by all people, a theoretical ground for narratives can be found in institutional theory. Meyer and Rowan (1977) suggests that 'rationalised myths' contribute to the understanding of organisations. Narratives or stories that convey myths are powerful tools that make the irrational become rational (Llewellyn, 1998).

In accounting, management, human computer interaction (HCI), knowledge management, strategic management, and software engineering narratives are common. Llewellyn (1998) discusses how narratives are best understood, constructed, and used in accounting and management research. In HCI, research narratives are used to improve the communication between end-users and developers for designing user interfaces, task modelling and prototyping, and supporting the specification of user interfaces (Bødker 2000). The role of narratives as conveyer of tacit knowledge is explored in knowledge management (Swap et al. 2001). In strategic management, scenarios are used to explore future alternatives where scenarios are “tools” in the “strategists arsenal” (Porter 1985, p. 481). Software engineering on the other hand uses narratives and scenarios to gather and validate requirements (Antón and Potts 1998; Jarke et al. 1998). It should be noted that there are different underlying philosophical reasons for the use of narratives. Llewellyn (1998), Bødker (2000), and Clausen (Clausen 1994) represent an interpretive research tradition, whereas Porter (1985) together with Jarke et al. (1998) and Antón and Potts (1998) can be classified as belonging to a more positivistic research tradition.

In IS research narratives have also emerged as an alternative approach. Hirschheim and Newman (1991) use the concept of myth to interpret social processes during information systems projects. Clausen (1994) develops a model for how information system designers can use narratives to make descriptions of information systems that people understand, cf. traditional methods such as structured languages and formal specifications. Brown (1998) examines the use of narratives to explain and create meaning in power struggles in information systems implementations. Dube and Robey (1999) analyse stories, by competing groups in information systems development project, as symbol of organisational values or myths to gain insights into the interpretation of management styles. Alvarez and Urla (2002) describe the use of narratives in requirements specification of ERP systems. Finally, Alvarez (Alvarez 2002) examines the role of myths to construct an ERP system as an integrated system and to elaborate the existing organisational values.

Alvarez and Urla (2002) describe three sources of benefits from narratives in relation to IS. First, narratives may provide a pragmatic view of the systems, i.e. how the users perceive the system and offer insights into how the system is actually used. This may reveal institutionalised work practice, inefficiencies of the system, and how users manage those inefficiencies. This type of narratives may be used to convince consultants or managers about necessary changes in the system. Secondly, narratives functions as mediums to convey that the system is a part of the larger organisation and not an isolated thing. Thirdly, narratives are especially for complex IS, since complex IS often affect organisational function by imposing process logic on the organisation. Other large affects imposed through complex IS may also be communicated within the organisation through narratives, e.g. integration, standardisation of work processes, implementation of business rules.

4.        Integrated model and discussion

In this section we present a model narrative based evaluation. The model integrates organisational sense- making and double loop learning. The model is depicted in Figure 2 and the logic of the model is the following. Organisational sense-making, which forms the overall process in evaluation consist of four iterative steps: action, narratives, interpretation and decision. Action refer to the use of IS, which can be individuals, groups, organisations, or society. The four groups of IS users are based on four out of five groups defined by Seddon et al.’s (1999). External parties, i.e. independent observers, are excluded since they are not users. The next step in the model is narrative, which refer to the task of producing and diffusion of narratives, which can be performed by any user group, stakeholder or external auditors. Note that we have replaced Hobeke’s measurement with narratives. Narratives should be perceived as one tool among many others, see for instance Deschoolmeester et al.’s (2004) excellent summary of different evaluation tools and methods. The third step in the model is interpretation of narratives. Interpretation can be done by any one having decision power or potential of influencing the behaviour of IS user. It can be the individual user or the management. Decision is the last step and the beginning of a new sense-making process refers to the decisions which are guided towards changing behaviour, i.e. action.

Double loop learning is integrated into the model in two ways. First, interpretations influence an affects the narratives leading to modifications of existing narratives. Modification refers to both reflection and learning in process of making sense of narratives prior deciding on which actions to take. Second, the narratives produced include criteria of success which are used in the decision process of which actions to take. Narratives make the case that effectiveness results from developing congruence between theory-in-use and espoused theory, i.e. both the creation and interpretation of narratives. Reflection is a key tool to reveal the theory-in-use and to explore the gulf between espoused theory and theory-in-use or in bringing the later to the surface. Provided the two remain connected, and then the gap creates a dynamic for reflection and for dialogue. As humans invariably think and express themselves using the full depth of language - by telling “stories” - it is therefore conceivable that unless external observations “verifies” the narratives. Narratives told do not reflect theory-in-use but espoused theory.

Figure 2: Proposed schematic evaluation process

Narratives used could range from talks among employees to published business cases describing the “good” use of IS. Thus, the actual implications for practice might not be so great, since narratives are used in business to convey meanings and to persuade people to act.

Due to their nature, narratives can carry much information and handled with care they can be used to reach procedural results and process improvements at the same time. Used as for evaluation, they are perhaps the best-suited tool for doing continuous improvements of both subject and evaluation process. The narratives interpreted and may lead to action (decisions) or be modified. New action creates the foundation for new narratives which include both the formal success criteria and the users perceived success stories. Narratives and stories may take role of communicating how to use IS better, e.g. to spread “best practice” use. Social and complex tasks are difficult to convey by other means than stories, which may develop and grow through face-to-face communication. Stories might inspire users of information systems, within the organisation or in other organisations, to investigate the possibilities in systems and ultimately change their mental models, which can lead to better use of information systems. Narratives is a communication medium with a high degree of media richness, which a suitable approach when there is a high degree of uncertainty and equivocality (Daft and Lengel 1986; Daft, Lengel and Trevino 1987). Other stories might reveal institutionalised work practice, inefficiencies of the system, how users manage those inefficiencies and to be used to convince consultants or managers about necessary changes in the system, where narratives can be used as tools to make the irrational become rational.

The use of narratives as an approach to measure and evaluate information systems might involve paradigm shifts in the existing control system and how to evaluate information systems. Changes of existing norms, behaviours, and procedures are difficult (Weick 1996). Resistance to change is likely to come. This is a rational behaviour for those who are affected by any change and not a dysfunctional behaviour (Markus 1983). The formalisation of narratives might also create new positions, such as chief storyteller. Storyteller might become the new power position. Narratives are powerful tales that can be manipulated by different stakeholders, such as managers, project leaders, and storytellers. The process of changing the organizational culture to accept narratives will be a challenging process for most organizations and business managers. Besides changing the culture, there is a need for developing support tools, e.g. story boards, and procedures to create, store, and spread stories (Snowden 2002; Swap et al. 2001).

A last issue discussed in this section is the validity of the proposed model. The validity of the evaluation model can be assessed by three particular criteria: the integration of the model (logical coherence), its practical and theoretical relevance, and relative explanatory power.

The evaluation model steps are casually inter-related; including not just the tools and methods, but also the impacts and consequences of the proposed use of narratives in evaluation. An important aspect of the model is the feed-back loop, modify and employ and change success criteria, i.e. the learning process. Failure to interpret narratives might lead to less learning and no improvement of system use. Potentially, this also clarifies some of the practical problems with action theory and what it is that should be learned in relation to system use.

The evaluation model is characterised by an integration of various theoretical perspectives, and addresses the interdependency between the evaluation and the actions taken based on the evaluation. There are other studies addressing the same issue. One notable study is Remenyi and Sherwood-Smith (1999) who proposed a formative evaluation process, which in idea is similar to the proposed one. The main difference that Remenyi and Sherwood-Smith (1999) evaluation approach address evaluation during the development of IS, whereas the proposed model address evaluation during use of system. Another difference is that the proposed model suggest narratives as a tool in formative evaluations.

5.        Conclusion

The literature (e.g. Hirschheim and Smithson 1998; Walsham 1993) have proposed interpretative approaches to information systems evaluation. Hirschheim and Smithson (1998) suggest that interpretative approaches are a way of gaining a deeper understanding of the process itself. Symons (1991) supports this by suggesting that evaluations means understanding the different perspectives of individuals and Walsham (1993, p. 179) states that “interpretative evaluations designs focus on learning and understanding” however, none has yet explicitly mentioned narratives as an interpretive evaluation approach. Neither has “doing better next time” (which would be typical process improvement) been replaced by “doing better all the time”. Learning continuously about the possibilities and difficulties we believe is the key to successful understanding of an information system before and during and installation, and use after an installation. Organisations use myths and narratives to make sense of equivocal situations and they are used as precedents for future actions, serving as “blue-prints” for desirable behaviour (Alvarez and Urla 2002). Narratives appear to be aptly suited as tools in furthering understanding of and support for improved system.

Still there are research implications and opportunities by acknowledging narratives as an evaluation approach of information systems. The implication is that narratives should not only be used as inputs in research, which is the traditional input in case research, it should also be a research output (Llewellyn 1998), where narratives can be used to reason and learn. Quantitative data can also form the bases of narratives. For instance, instead of stating the correlation between X and Y in statistical term it can put in words. The consequence of narratives as research outputs is that researchers have to be able to interpret and evaluate stories in the same way as other research output.

Thus, we conclude that information systems evaluation is highly contextual and measurements of impact on organisations from large and complex information system on business performance are interchangeable with measurements of general improvements. For such measurements to be relevant they have to be used over a long period of time and the measuring would have to be initiated before the implementation is started (Hoebeke, 1990), e.g. in the requirements specification phase. As Strassmann (1985) claimed, we argue that we need to measure impact of information systems with the same measurements that are used on a specific organisation before the implementation, and that any changes recorded are attributable to a combination of information system implementation, organisational changes and changes in method of measurement. Thus, measuring or quantifying isolated impact from information systems implementation is close to impossible. Instead of trying to measure the impossible, we propose the use of narratives as the means to spread knowledge, which ought to lead to better action and improvements during the implementation and usage of systems.

References

• Alvarez, R. (2002) "The Myth of Integration: A Case Study of an ERP Implementation," in: Enterprise Resource Planning: Global Opportunities & Challenges, L. Hossain, J. Patrick and M. Rashid (Eds), Idea Group Publishing, Hershey, PA, pp 17-42.
• Alvarez, R., and Urla, J. (2002) "Tell Me a Good Story: Using Narrative Analysis to Examine Information Requirements Interviews during an ERP Implementation," The DATA BASE for Advances in Information Systems, Vol 33, No. 1, pp 38-52.
• Antón, A., and Potts, C. (1998) "A Representational Framework for Scenarios of Systems Use," Requirements Engineering, Vol 3, No :3-4, pp 219-241.
• Argyris, C. (1976) Increasing Leadership Effectiveness Wiley, New York, NY.
• Argyris, C. (1980) Inner Contradictions of Rigorous Research Academic press, New York, NY.
• Argyris, C., and Schön, D. (1974) Theory in Practice: Increased Professional Effectiveness Jossey-Bass, San Francisco, CA.
• Brown, A. (1998) "Narrative, Politics and Legitimacy in an IT Implementation," Journal of Management Studies, Vol 35, No 1, pp 35-59.
• Bødker, S. (2000) "Scenarios in User-centered Design: Setting the Stage for Reflection and Action," Interaction with Computers, Vol 13, No 1, pp 61-76.
• Clausen, H. (1994) "Designing Computer Systems from a Human Perspective: The Use of Narratives," Scandinavian Journal of Information Systems, Vol 6, No 2, pp 37-44.
• Daft, R.L., and Lengel, R.H. (1986) "Organizational Information Requirements, Media Richness and Structural Design," Management Science, Vol 32, No 5, pp 554-571.
• Daft, R.L., Lengel, R.H., and Trevino, L.K. (1987) "Message Equivocality, Media Selection, and Manager Performance: Implications for Information Systems," MIS Quarterly, Vol. 11, No 3, pp 355-366.
• Deschoolmeester, D., Braet, O. and Willaert P. (2004) “On a Balanced Metodology to Evaluate a Portfolio of ICT Investments”, ECITE 2004.
• Dubé, L., and Robey, D. (1999) "Software Stories: Three Cultural Perspectives on the Organizational Practices of Software Development," Accounting, Management and Information Technologies, Vol 9, No 4.
• Farbey, B., Land, F.F., and Targett, D. (1995) "A Taxonomy of Information Systems Applications: The Benefits´ of Evaluation Ladder," European Journal of Information Systems, Vol 4, No 1, pp 41-50.
• Hirschheim, R.A., and Newman, M. "Symbolism and Information Systems Development: Myth, Metaphor and Magic," Information Systems Research (2:1) 1991, pp 29-62.
• Hirschheim, R.A., and Smithson, S. (1998) "Evaluation of Information Systems: A Critical Assessment," in: Beyond the IT Productivity Paradox, L. Willcocks and S. Lester (Eds), pp. 381-409.
• Hoebeke, L. "Measuring in Organisations," Journal of Applied Systems Analysis, Vol. 17, pp 115-122.
• Irani, Z., and Love, P.E.D. (2001) "Information Systems Evaluation: Past, Present and Future," European Journal of Information Systems, Vol. 10, No 4, pp 183-188.
• Jarke, M., Bui, X., and Caroll, J. (1998) "Scenario Management: An Interdisciplinary Approach," Requirements Engineering, Vol. 3, No. 3-4, pp 154-173.
• Jones, S., and Hughes, J. (2001) "Understanding IS Evaluation as a Complex Social Process: A Case Study of a UK Local Authority," European Journal of Information Systems, Vol. 10, No 1.
• Klein, H.-K., and Meyer, M. (1999) "A Set of Principles for Conducting and Evaluating Interpretive Field Studies in Information Systems," MIS Quarterly, Vol. 23, No 1, pp 67-94.
• Kumar, K. (1990) "Post-implementation Evaluation of Computer Based Information Systems (CBIS): Current Practices," Communications of the ACM, Vol. 33, No 2, pp 203-212.
• Llewellyn, S. (1998) "Narratives in Accounting and Management Research" Accounting, Auditing and Accountability Journal, Vol. 12, No 2, pp 220-236.
• Markus, M.L. (1983) "Power, Politics, and MIS Implementation," Communications of the ACM, Vol 26, No 6, pp 430-444.
• Meyer, J.W., and Rowan, B. (1977) "Institutionalized Organizations: Formal Structure as Myth and Ceremony," American Journal of Sociology, Vol. 82, No 2, pp 340-363.
• Murphy, K. E. and S. J. Simon (2002). "Intangible Benefits Valuation in ERP Projects." Information Systems Journal, Vol 12, No 4, pp 301-320.
• Porter, M.E. (1985) Competitive Advantage: Creating and Sustaining Superior Performance Free Press, New York, NY.
• Remenyi, D., and Sherwood-Smith, M. (1999) "Maximise Information Systems Value by Continuous Participative Evaluation," Logistics Information Management, Vol 12, No 1, pp 14-31.
• Seddon, P.B., Graeser, V., and Willcocks, L.P. (2002) "Measuring Organizational IS Effectiveness: An Overview and Update of Senior Management Perspectives," The DATA BASE for Advances in Information Systems, Vol 33, No 2, pp 11-27.
• Seddon, P. B., S. Staples, S. Patnayakuni and M. Bowtell (1999). "Dimensions of Information Systems Success." Communications of the AIS 2(15).
• Serafeimidis, V., and Smithson, S. (2000) "Information Systems Evaluation in Practice: A Case Study of Organizational Change," Journal of Information Technology, Vol 15, No 2, pp 93-105.
• Serafeimidis, V., and Smithson, S. (2003) "Information Systems Evaluation as an *Organizational Institution – Experience from a case study. " Information Systems Journal, Vol 13, pp 251-274.
• Sinclair, D., Zairi, Mohamed (1995) "Performance Measurement as an Obstacle to TQM," The TQM Magazine, Vol 7, No 2, pp 42-45.
• Snowden, D. (2002) "Complex Acts of Knowing: Paradox and Descriptive Self-Awareness," Journal of Knowledge Management, Vol 6, No 2.
• Strassmann, P.A. (1985) Information Payoff: The Transformation of Work in the Electronic Age Free Press, New York, NY.
• Swap, W., Leomard, D., Shields, M., and Abrams, L. (2001) "Using Mentoring and Storytelling to Transfer Knowledge in the Workplace," Journal of Management Information Systems, Vol 18, No 1), pp 95-114.
• Symons, V.J. (1991) "A Review of Information Systems Evaluation: Content, Context and Process," European Journal of Information Systems, Vol 1, No 3, pp 205-212.
• Walsham, G. (1993) Interpreting Information Systems in Organisations Wiley, Chichester.
• Walsham, G. (1995) "The Emergence of Interpretivism in IS Research," Information Systems Research, Vol 6 No 4, pp 376-394.
• Walsham, G. (1999) "Interpretive Evaluation Design for information Systems," in: Beyond the IT Productivity Paradox, S. Lester (Ed), Wiley, Chichester.
• Weick, K.E. (1996) "Drop Your Tools: An Allegory for Organizational Studies," Administrative Science Quarterly, Vol 41, No 2, pp 301-331.
• Venkatraman, N., and Ramanujam, V. (1986) "Measurment of Business Performance in Strategy Reserach: A Comparision of Appraoches," Academy of Management Review Vol 11, No 4, pp 801-814.