The Treatment of Organisational Issues in Systems Development Projects: The Implications for the Evaluation of Information Technology Investments. Neil Doherty and Malcolm King, The Business School, Loughborough University, UK. (n.f.doherty@lboro.ac.uk -mailto:n.f.doherty@lboro.ac.uk; m.king@lboro.ac.uk)

1. Introduction

Information technology evaluation is a process that attempts to assess the contribution of an information systems investment, typically in financial or economic terms. The emphasis on the financial and economic contribution of information systems is very evident in the literature (e.g. Bacon, 1992). More specifically, Ballantine et al, (1996) note that ‘studies have tended to concentrate on the financial techniques used to evaluate investments’, and that ‘financial reviews of cost / savings’ are the most common aspect of evaluation procedures. Similarly, Farbey et al, (1999) note that the debate about systems evaluation has been primarily concerned with ‘Value for IT money’. Even with the growing recognition that it is often impossible to accurately evaluate investments in purely quantitative or financial terms, the discussion of systems evaluation still focuses primarily upon using subjective, informal or qualitative techniques to establish the worth, contribution, value, or level of benefits associated with a particular IS investment. Indeed, systems evaluation is often defined in these terms, for example, Willcocks (1992) defines systems evaluation as the ‘process of establishing by quantitative and / or qualitative techniques the worth of IS / IT projects to the organisation’. In terms of the timing of the evaluation, it is suggested that there are two key points at which the contribution is evaluated. Initially, the ‘ex ante’ evaluation is conducted prior to systems development, to justify the investment, and then the ‘ex post’ evaluation is undertaken following implementation, to confirm the value of the system (Remenyi et al, 1998; pp 55-56).

Another important aspect of the information systems literature has been the debate about the important role of human, organisational and economic issues, in systems development projects. A long stream of studies, including those conducted by Ewusi-Mensah & Przasnyski (1994), Clegg et al (1997) and Doherty & King (1998a), have all identified inadequacies in the treatment of organisational issues as being significant in cases of systems failure. Furthermore, the literature also suggests that in practice the treatment of such issues is generally very poor (Eason, 1988; Hornby et al, 1992; Clegg et al, 1997). Indeed, even where such issues are addressed, it tends be accomplished in a reactive rather than proactive manner; Clegg et al (1996) note that it has been characterised as ‘too little too late’ and ‘marginalised’. It should be noted that for simplicity, the term ‘organisational issues’ will be used as a short-hand to cover a wide range of ‘organisational, human and economic issues’, in the remainder of this paper.

To date, these two important strands from the information systems’ literature have been largely distinct; the IT evaluation literature has concentrated primarily on the proposed system’s financial contribution, whilst the organisational issues literature has focussed upon assessing the system’s human, organisational and economic impacts within the development process. Consequently, whilst both the IT evaluation and the organisational issues literature have typically been concerned with assessing the impact of systems, in non-technical terms, they have differed significantly in terms of the scope and the timing of such assessments (see table 1). However, there has been some more recent recognition that there is an important intersection between these two bodies of literature, namely the treatment of the human and organisational impact of the proposed system at the time of the ex-ante evaluation. For example, Doherty and King (1998b) note that the introduction of a new system is increasingly ‘likely to have a significant impact upon an organisation’s culture, structure and working practices’, and as a consequence suggest that such implications should be evaluated at an early stage in the project. This view is supported by Guy Fitzgerald (1998) who emphasises that an IS evaluation should also consider the system’s ‘second order effects’, in terms of the ‘social, political and organisational context’. However, whilst such contributions are important, they are comparatively rare. Consequently, there is a need for further research, especially of an empirical nature, to establish, and to classify, the full range of organisational impacts, and under what circumstances, they should be considered. This paper aims to help fill this gap by presenting the results of an empirical analysis into the treatment of a wide range of organisational, human and economic issues in systems development projects. From this analysis a number of important implications for the theory and practice of the evaluation of IT investments are presented.

Table 1: Key themes in the IT evaluation and organisational issues literature.

The paper is organised into a number of distinct sections. The following section of this paper establishes the research objectives for this project, prior to reviewing the research methods. The research results are presented in a series of tables that are discussed in the third section, and their importance is assessed in the final section.

2. Research Design

The aim of this section of the paper is to present the study’s specific research objectives, before reviewing the methods by which the research instrument was designed, validated and ultimately executed. From the literature, already referred to, it is possible to identify four distinct objectives, worth pursuing:

1.        To determine the frequency with which a wide range of distinct organisational issues are treated in practice.

2.        To determine the timing of the treatment of organisational issues, in systems development projects.

3.        To identify the factors which are affecting the treatment of organisational issues, in terms of frequency and timing.

4.        To extract implications for the practice of systems evaluation, from the results of the three previous objectives.

It was envisaged that in addressing these objectives, a number of important implications for the practice of systems development, in general, and IT evaluation, in particular, might be highlighted.

2.1 Research Instrument Development

To canvass the views and perceptions of a wide range of IT managers, on the treatment of organisational issues in systems development projects, a field survey was employed. Consequently, a draft questionnaire was developed, based primarily on issues that had been highlighted as being of interest from the literature review (in particular: Hornby et al, 1992; Clegg et al, 1997). The questionnaire included a comprehensive list of organizational issues that could be used as a framework for collecting data concerning the frequency with which each issues is typically addressed. The following definition was used to select appropriate organisational issues:

‘Those issues which need to be treated during the systems development process to ensure that the human, organisational and economic impacts of the resultant technical system are likely to be desirable.’

Implicit in the use of the term ‘treatment’, in the above definition, is the notion that the development team will have to evaluate a specific impact, prior to taking appropriate steps to ensure that the impact is desirable. Consequently, it might be necessary to modify the system’s technical specification, or initiate a programme of organisational change, to ensure that all the system’s organisational impacts are desirable. Given this criteria it was immediately possible to classify issues such as: the impact of a system on an organisation’s culture, working practices, or performance, and similarly its impact on a user’s motivation or performance, as organisational issues.

Having designed a draft questionnaire, it was further refined through a rigorous process of pre-testing and piloting. More specifically, a series of interviews and pre-tests with senior IT professionals were conducted, which focused on issues of instrument clarity, question wording and validity. The pre-tests resulted in a number of important enhancements being made to the survey. In particular, it was possible to classify and extend the list of specific organisational issues. Subsequently, the survey was also extensively piloted to gain further confidence in the research instrument’s suitability. Whilst the results of the pilot study were of sufficient interest to have been already published in their own right (Doherty & King, 1998a), there weren’t sufficient responses to explore the objectives addressed in this paper. Ultimately, a list of fifteen distinct organisational issues, sub-divided into four distinct categories (see table 2), was utilised. To determine the extent to which each of these individual organisational issues had been treated, in practice, a separate 5 point Likert scale (ranging from 5 = always to 1 = never) was used for each issue.

The second section of the questionnaire concerned the treatment of organisational issues, in terms of their timing and to a lesser extent the method by which they are typically addressed. The final section of the questionnaire addressed the factors that might affect the treatment of organisational issues. More specifically, it was envisaged that their treatment might be significantly affected by an organisation’s level of IS / IT maturity. Consequently, the published literature relating to ‘stages of growth’ models [for example: Nolan, 1978 and Galliers, 1991] was highly influential in the selection of the seven ‘IS / IT maturity’ variables (see table 3). Each of these variables was operationalised using a 5 point semantic differential scale.


2.2 Data Collection

It was recognised that only those individuals who had a high degree of managerial responsibility for systems development projects would be able to comment knowledgeably about the importance of organizational issues and the extent to which they are routinely addressed. Senior IS executives were, therefore, chosen as the ‘key informant’, as they would be able to provide the requisite perspective. A sampling frame was constructed by using the British Computer Society’s (BCS) membership list to identify a large number of UK-based, senior IT executives. The main survey was ultimately distributed to all 3500 individuals on this sampling frame. It was believed that this group would be representative of those with responsibilities for IS development projects, although their spread through the UK economy was not investigated before the mailing. Overall 593 valid responses were received from the 3500 questionnaires mailed out, representing a response rate of 17%. This rate is considered highly encouraging in terms of the normally expected responses to similar types of survey


Table 2: Organisational Issues Addressed in Survey

With any survey-based research it is important that active measures are taken to reduce the chances of bias having a dysfunctional affect on the validity of the results. In this research the content validity of the constructs has been established through the process of initially linking the variables to the research literature and then refining them through an extensive and comprehensive process of pre-testing and pilot testing. Any sample bias introduced through the loss of data from non-respondents is often harder to establish, as this data is not easily obtainable. In this study, steps were taken to approximate this bias by comparing the answer patterns of early and late respondents The results of this exercise detected no response bias within the sample and consequently, the results are reasonably generalisable within the boundary of the sample frame.

Table 3: Definitions for the ‘IS / IT maturity’ variables

3. Research results

This section presents a discussion of the research results relating to the four specific research objectives proposed earlier.

3.1 The Frequency of Treatment of Organisational Issues

An average ‘frequency of treatment’ value was calculated for each of the 15 organisational issues. These are presented in table 4, in descending order of the mean value. There is clear evidence, from this study that the most frequently considered issues are those to do with organisational contribution and that these are considered fairly often, if not very often, by the majority of organisations. It is also interesting to note that the ‘timing of the implementation’ is considered in nearly all projects. This result may best be explained in the context of the increasing pace of organisational change. Because organisations often have to manage a wide variety of change projects in parallel, there is a greater need to explicitly consider the timing of a system’s implementation to consider the impact of its interaction with other concurrent initiatives. The organisational issues that have been classified as Human-centred Issues are all in the centre of the table, except for the ‘Health and Safety Issues’ variable.  It appears that issues surrounding job redesign, users working practices and method and user motivation are all treated more often than not on average but are not addressed as frequently as issues to do with organisational contribution.  On the other hand the issues to do with organisational alignment are considered even less frequently. The frequency with which issues concerned with organisational contribution are addressed is as expected, since it would be hoped no system is developed without a clear view of how it will contribute positively to the organisation.  However, it is rather disappointing that the human centred issues and alignment issues are considered in significantly fewer cases as these are precisely the issues that have been shown to lead to systems failure if they are not addressed during the development stages.

Table 4: Frequency of treatment of Specific Organisational Issues

 
3.2 The Timing of Treatment of Organisational Issues

The breakdown of the results presented in table 5 indicates that there is a high degree of variability with respect to the treatment of organisational issues, in terms of their timing. Many organisations focus their efforts into a single phase of the system’s development process, whist others prefer to treat them throughout the project. Furthermore, the results indicate that 60% of the responding organisations adopt an explicit approach to their treatment; either as an integral part of the feasibility study, the requirements analysis or the entire development process, or through an independent organizational impact analysis. A further 27% of the responses represent those organisations that treat organisational issues implicitly, which indicates that 87% of the sample consider these issues in some form, and only 13% of organisations rarely consider them.

Table 5: Organisational Issues - The timing / method of treatment

3.3 An Analysis of the factors affecting the treatment of organisational issues

The results presented, in this section of the paper, have been split into the factors affecting the frequency of treatment, and then the factors affecting the timing of treatment.

The Frequency of Treatment: To identify the factors affecting the treatment of organisational issues more effectively, it was considered important to reduce the number of variables in the statistical analysis. Consequently, the intuitively desirable idea of compressing the fifteen distinct organisational issues into four summated measures, based upon the groupings identified in table 1, was explored. However, it was necessary to confirm that all the variables, within a specific group, are measuring the same underlying construct. This confirmation was achieved by undertaking internal reliability tests, using ‘Cronbach alpha’, from which the following results were derived: ‘organisational contribution’ (0.65), ‘transitional issues’ (0.71), ‘organisational alignment’ (0.67) and ‘human issues’ (0.71). As the reliability tests resulted in alpha values that all comfortably exceeded the threshold value of 0.60 [Hair et al, 1997; p 118], average scores were calculated for each of the four constructs.

To simplify the process of identifying important patterns in the data, the respondents were classified according to the seven ‘IS / IT maturity’ variables, as identified in table 3. Cluster analysis, based upon the six phase approach as recommended by Hair et al (1997), was used to investigate whether any meaningful grouping existed. Ultimately, two distinct, stable and meaningful clusters were derived. One group (236 cases) represented organisations whose utilisation of IS / IT could be categorised as ‘mature’, whereas the other cluster (340 cases) represented organisations whose use of IS / IT could best be described as ‘evolving’. Having reduced the data to four categories of organisational issues and two levels of IS / IT maturity, ANOVA was used to explore the relationship between these.

Table 6: Analysis of the relationship between IS / IT maturity and the frequency of treatment

 
The results presented in table 6 clearly demonstrate that the frequency of treatment is significantly associated with the maturity of an organisation’s utilisation of IS / IT. It also seems highly likely that it is the level of maturity that is influencing the frequency of treatment, as opposed the frequency of treatment affecting the level of maturity. For example, it has been noted before (Doherty & King, 1998a) that in situations where IS / IT is applied in a highly integrated, process-oriented and strategic way the resultant level of organisational impact is likely to be high. It also seems reasonable to conclude that the need to treat organisational issues will be highly dependent upon a proposed system’s envisaged degree of organisational impact. In addition to exploring the influence of the level of IS / IT maturity, a variety of alternative factors that might affect the frequency of treatment of organisational issues were also explored. For example statistical tests were conducted to determine whether organisational size, sector or the degree to which an organisation opted for tailor-made systems, might influence frequency of treatment. No statistically significant relationships were identified for these variables, and since no other variables are suggested in the literature, it seems reasonable to conclude that the degree of IS / IT maturity is likely to be the major determinant of the frequency with which organisational issues are treated.

Timing of the Treatment: To investigate whether the level of IS / IT maturity also affects the timing / method of the treatment of organisational issues, a chi-squared analysis was conducted. This analysis investigated the strength of the statistical relationship between the six categories of timing / method and the two derived categories of IS / IT maturity. The results of the chi-squared analysis presented in table 7 show that the differences in terms of the degree of IS / IT maturity are translated into a number of marked differences in terms of the timing / method of their treatment. In fact, this result is significant at the 99% level, so it is important to explore the form of the relationship further. An inspection of the data in table 7 indicates that the organisation that is a mature IS / IT user has an increased likelihood of treating organisational issues explicitly, throughout the development process, whereas the evolving IS / IT user has an increased likelihood of rarely treating such issues. 

Table 7: Analysis of the relationship between IS / IT maturity and the timing of treatment

Significant at the 1 % level; c² = 18.027 > 15.09 = c² (0.01), 5df.  

3.4 Discussion of the implications for the practice of systems evaluation.

The IS literature is very clear that the treatment of organisational issues is critical to the successful development, implementation and operation of information systems. It has also been well established that their treatment, within the development process, is generally poor with the emphasis on reactive rather than proactive approaches (Lim et al, 1992). The results of this research have added to the debate by establishing that in practice, the majority of systems development teams do ultimately treat a wide range of human and organisational issues, extending well beyond the system’s economic contribution. Furthermore, it has been shown that their treatment is conducted at a variety of points within, or indeed throughout the entire, systems development process. Bringing the themes from the literature and the findings from the empirical study together, it can be argued that the treatment of organisational issues within the systems development process now appears to be in better shape than previous accounts have suggested. Furthermore, it can also be argued that as, in practice, organisations are already treating a wide range of human and organisational, as well as economic, issues they should extend the currently limited scope of their systems evaluation procedures to address a similarly wide range of issues. Equally, since in practice, an increasing number of organisations are addressing a wide range of non-technical impacts throughout the systems development process, perhaps the evaluation procedures should also be more of a continuous process, as opposed to one that has typically focused upon ex ante and ex post perspectives. 

It has also become evident that the adoption of such an enhanced evaluation process is particularly important in organisations that are mature in their use of IS / IT. Segars and Grover (1996) have noted that the need for strategic information systems planning increases as organisations seek to ‘integrate diverse systems across the organisation, re-engineer business processes with IT and identify opportunities to enhance competitive position through IT-based strategies’. Based upon the results of this research it is suggested that the need to treat a wide range of organisational issues also increases as an organisations use of IT becomes more integrated, process oriented and strategically focussed. Moreover, it has been shown that the organisation that is a mature user of IS / IT is more likely to be adopting explicit and continuous approaches to the treatment of organisational issues. There is some indication that the need to widen the scope of systems evaluation is beginning to be recognised in the IS literature. For example, evaluation has recently been defined is as: ‘a process, or group of parallel processes, which take place at different points in time or continuously, quantitatively or qualitatively, to determine all the impacts of an IT project and the programme or strategy of which it is part’ (Farbey et al, 1999). From the perspective of this research, the key phrases in this definition are ‘continuous’ and ‘all the impacts’. It can be argued that, in their treatment of organisational issues, many of the more mature users of IS / IT are conducting an IT evaluation, as defined by Farbey et al (1999). It can, therefore, be argued that the critical contribution of this research has been to demonstrate that in practice, systems evaluation and the treatment of organisational issues are converging in some key aspects. Indeed, it might be suggested that organisations should move towards an integrated programme covering both systems evaluation and the treatment of organisational issues. Not only would such a programme be concerned with the evaluation of impacts for the purposes of justifying investments and monitoring outcomes, it would also identify where action was necessary to manage the realisation of benefits and align the system with its organisational environment. 

4. Conclusions

The work presented in this paper is important, as it uses a large-scale and rigorous empirical study of senior IT executives to help bridge the gap between two important strands of the IS literature, namely evaluation and organisational issues. From the IS / IT executives perspective, this research is significant as it highlights the importance of addressing a wide range of organisational issues as an integral part of a well balanced IS evaluation process, and underlines the circumstances under which their treatment is most important. Furthermore, the classification of organisational issues, presented in this paper, provide a useful framework for deciding which issues should be addressed.

From the IS academics perspective, this research should also be of interest, as it is one of the few empirical contributions in this increasingly important domain. Perhaps more importantly, the research suggests that in future more effort should be made to consider the IT evaluation and the organisational issues literature as being different windows into a similar set of concerns. Whilst this research provides many critical insights into the treatment of a comprehensive selection of organisational issues, during the systems evaluation process, there is still much further research to be undertaken. Of particular importance would be the use of in-depth case studies to determine the degree of overlap between the treatment of organisational issues and the evaluation of IT investments, in the practice of systems development.

References

• Bacon, C. J. (1992) “The use of Decision Criteria in Selecting IS / IT Investments”, MIS Quarterly, Vol 16, No. 3, pp 335-350.

• Ballantine, J., Galliers, R. & Stray, S. (1996) “Information Systems / Technology Evaluation Practices: Evidence from UK Organisations”, Journal of Information Technology, Vol 11, No. 2, pp 129-142.

• Clegg, C., Coleman, P., Hornby, P., McClaren, R., Robson, J., Carey, N. & Symon, G. (1996) “Tools to incorporate some psychological and organisational issues during the development of computer-based systems”, Ergonomics, Vol 39, No. 3, pp 482-511.

• Clegg, C., Axtell, C., Damadoran, L., Farbey, B., Hull, R., Lloyd-Jones, R., Nicholls, J. Sell,R. & Tomlinson, C. (1997) “Information Technology: a study of performance and the role of human and organisational factors”, Ergonomics, Vol 40, No. 9, pp 851-871.

• Doherty, N. F. & King, M., (1998a) “The consideration of organizational issues during the systems development process: an empirical analysis”, Behaviour & Information Technology, Vol 17, No. 1, pp 41-51.

• Doherty, N. F. & King, M., (1998b) “The importance of organizational issues in systems development process”, Information Technology & People, Vol 17, No. 1, pp 104-123.

• Eason, K., (1988), Information Technology and Organisational Change, Taylor & Francis, London.

• Ewusi-Mensah, K. & Przasnyski, Z. (1994) “Factors Contributing to the Abandonment of Information Systems Development Projects”, Journal of Information Technology, Vol 9, pp 185-201.

• Farbey, B., Land, F. & Target, D. (1999) “Moving IS evaluation forward: learning themes and research issues”, Journal of Strategic Information Systems, Vol 8, No. 2, pp 198-207.

• Fitzgerald, G. (1998) “Evaluating information systems projects: a multidimensional approach”, Journal of Information Technology, Vol 13, pp 15-27.

• Galliers, R. D. (1991) Strategic Information Systems Planning:  Myths, Reality and Guidelines for Successful Implementation. European Journal of Information Systems, Vol 1, No. 1, pp 55-64

• Hair, J. et. al, (1997) Multivariate Data Analysis with Readings, Prentice Hall, New Jersey.

• Hochstrasser, B. & Griffiths, C. (1991) Controlling IT Investment, Chapman Hall, London.

• Hornby, C., Clegg, C., Robson, J., McClaren, C., Richardson, S. & O’Brien, P. (1992) “Human & Organisational Issues in Information Systems Development”, Behaviour & Information Technology, Vol 11, No. 3, pp 160-174.

• Lim, K., Long, J. & Silcock, N. (1992) “Integrating Human Factors with the Jackson System Development Method: An Illustrated Overview”, Ergonomics, Vol 35, No. 10, pp. 1135-1161.

• Nolan, R. (1978) Managing the crisis in data processing, Harvard Business Review, Vol 57, pp 115-126.

• Remenyi, D., Sherwood-Smith, M. & White, T. Achieving Maximum Value from IT Investments, John Wiley & Sons, Chichester.

• Segars, A & Grover, V (1996) Designing Company-wide Information Systems: Risk Factors and Coping Strategies, Long Range Planning Vol 29, No. 3, pp 381-392.