1.
Introduction
This paper describes the research conducted by those
who have presented papers at the European Conferences on IT Evaluation
(ECITE). It is a high level overview of the proceedings of these
Conferences and is based on the papers published during the first eleven
proceedings. The purpose of this paper is to facilitate an understanding
of the thrust of research in this field of study and to reflect on its
possible future direction.
The objective of ECITE, which has remained constant
throughout the eleven-year period, is to provide a platform for academics
and professionals to join together and discuss the theory and practice of
the evaluation of information technology. The first ECITE was held at
Henley Management College on the 13th-14th of
September 1994.
The first paper in the first issue of the conference
proceedings concludes by stating: ‘…. IT costs have been growing by
rates of about 15% p.a. over the last decade.’ ‘This rate of growth is far
in excess of the growth in the underlying businesses and cannot be
sustained into the next century’ (Dier and Mooney, 1994, p.11). This
statement, although perhaps a little naïve, clearly indicated the
relevance of the research. Considering the current state of the IT
industry it also illustrates that evaluation research may be thought of as
being unable to achieve a soft landing. This overview of the 10
conferences, therefore, addresses what might be regarded as research
highlights in terms of vision and practical relevance and at the same time
it also considers problems such as industry impact.
Questions addressed in this paper are:
§
What type of researchers contributed to the IT evaluation conferences?
§
What type of research questions were addressed?
§
What were the most prominent research findings?
§
What other interesting research questions are suggested?
2.
Background of
participating researchers
ECITE has always been a small specialist Conference. It
was never intended that it should attract large numbers of papers or
attendees and thus the focus of the Conference has been retained. During
the eleven years 513 authors contributed to the conference. Of this group
458 were academics and 55 authors had an industry background (11%). A
number of authors contributed on multiple occasions to the conference. In
the second and third conference 30% of the authors had also contributed to
previous ones. As the conference grew so this ratio changed. For the last
three years this number was 9%.
The conference has been held in the UK, in the
Netherlands, in Ireland, in France and Spain.
Although the conference is referred to as a European
conference, ECITE has always welcomed contributions from authors around
the globe and there have been papers presented from 38 different
countries. Of these 38 there have been 17 different European countries as
well as the USA, Australia, South Africa, New Zealand and several Asian
counties. The largest number of contributions to the conference have come
from the UK (34,9%), Australia (9,4%) and Netherlands (9,4%) as shown in
Figure 1.
|
Country of origin |
Quantity
|
% |
|
Australia |
48 |
9.4% |
|
Austria |
5 |
1.0% |
|
Bangladesh |
1 |
0.2% |
|
Belgium |
12 |
2.3% |
|
Brazil |
2 |
0.4% |
|
Canada |
6 |
1.2% |
|
Chile |
1 |
0.2% |
|
Denmark |
5 |
1.0% |
|
Estonia |
1 |
0.2% |
|
Finland |
32 |
6.2% |
|
France |
12 |
2.3% |
|
Germany |
15 |
2.9% |
|
Greece |
5 |
1.0% |
|
India |
1 |
0.2% |
|
Indonesia |
1 |
0.2% |
|
Ireland |
30 |
5.8% |
|
Italy |
9 |
1.8% |
|
Korea |
3 |
0.6% |
|
Latvia |
1 |
0.2% |
|
Luxemburg |
3 |
0.6% |
|
Mexico |
3 |
0.6% |
|
Netherlands |
48 |
9.4% |
|
New Zealand |
3 |
0.6% |
|
Norway |
2 |
0.4% |
|
Poland |
1 |
0.2% |
|
Portugal |
2 |
0.4% |
|
Romania |
1 |
0.2% |
|
Saudi Arabia |
1 |
0.2% |
|
Seychelles |
1 |
0.2% |
|
Slovakia |
2 |
0.4% |
|
South Africa |
17 |
3.3% |
|
Spain |
9 |
1.8% |
|
Sweden |
21 |
4.1% |
|
Switzerland |
4 |
0.8% |
|
Tunisia |
1 |
0.2% |
|
Turkey |
2 |
0.4% |
|
UK |
179 |
34.9% |
|
USA |
23 |
4.5% |
Figure 1: Nationality of
researchers
With only 11% of conference attendees coming from
industry it is clear that this combination of academics and practitioners
could be improved. For two of the years the conference supported a
doctoral colloquium.
3.
How the research
papers were focused
The evaluation of IT encompasses many topics and this
diversity is reflected in the papers. A total of 356 papers were published
in the proceedings. It is interesting to note that the majority of the
papers are based on case study research, representing 52,7%.
There was a diverse range of topics and this paper
considers the research under the following categories:
§
Types of information systems;
§
Industry sectors;
§
Geographical areas;
§
Organisation of the overall information function;
§
Development of particular evaluation methods;
§
Theoretical foundation of evaluation.
3.1
Types of
Information Systems researched
A plethora of information system types have been
investigated. Examples are:
§
Executive information systems (ElKordy, et al., 1997; Carlsson, 1998).
§
Strategic information systems (Spil, 1995; Deitz, 1995; Savolainen, 2000).
§
Electronic data interchange (EDI), (Hoogeweegen, et al., 1994).
§
Manufacturing information systems (Wheeler and Chang, 1994; Bonner, 1995;
Ezingeard, 1998; Coronado, 1999).
§
Knowledge based systems (Clark and Soliman, 1996; Poon, 1999; Savory,
2001; Tyndale 2002).
§
Workflow (Kueng, 1998; Grunden, 2002).
§
Intranets (Magrill and Brown, 1997).
§
Electronic commerce (Miller, et al., 1999; Stamoulis, 1999; Lee, 1999;
Cherian, 2001; Al-Mashari and Al-Samad, 2002; Beyno-Davies, 2002;).
§
Groupware (Josefsson and Nilsson, 1999).
§
Enterprise systems (Ezingeard and Chandler-Wilde, 1999; Stefanou, 2000;
Stafyla and Stefanou, 2000; Al-Mudimigh, et al., 2001; Hillam and Edwards,
2001; Al- Mudimigh and Al-Mashari, 2002).
§
Customer relationship management systems (CRM) (Ezingeard, et al., 2001;
McCalla, et al., 2002).
§
Infrastructures (Renkema, 1997).
§
Development tools (Addison and Sutherland, 1995).
§
Evaluation of Transhuman technology (Bannister 2004)
These papers typically addressed ex ante assessments of
the above systems investigating the potential value of investment in these
areas. In general the researchers are quite successful in this type of
investigations. However, it is clear from the analysis of these papers
that the authors are also often uncertain about the correctness or
validity of their outcomes. This is probably caused by the fact that the
researchers are not often able to study the actual realisation and outcome
of the projects researched. However, most of these information system
based papers also lack a theoretical underpinning of their methodology,
which results in additional uncertainty about their measurement
instrument. Due to the labour intensiveness of case study research the
number of involved organisations was also limited.
3.2
Industries
researched
Many papers refer to particular industry sectors.
Examples are:
§
Healthcare, which is also the most often researched topic overall (Kaplan,
1995; Lock, 1996; Murray and Dhillon, 1996; Peterson and de Wit, 1997;
Salmela and Turunen, 1997; Vlug and Lei, 1999; Niss, 1999; Orr et al.,
1999; Protti, et al., 2000; Suomi and Tahkapaa, 2001; Bergamaschi and
Ongaro, 2001; Turunen, 2001; Ammenwerth, et al., 2002;Carson et al,
2004
§
Telecom industry (Peterson and de Wit, 1997; Demkes, 1997; Lampikoski and
Rusi, 2002; Cheverst et al, 2004).
§
Small and medium sized enterprises (SME), McNutt and O’Donnell, 1997;
Hillam and Edwards, 1999; Dans, 2000; Dans, 2001; Johansson and Carlsson,
2002).
§
Public sector (Newton, 1995; Worrall, et al., 1998; Oliver, 2000).
§
Financial services industry (Nijland and Berghout, 2000; Maimbo and
Pervan, 2002; Diniz et al, 2004).
It is clear that evaluation research requires insight
into specific industry practice. Without this contextual knowledge the
assessment of competitive advantage or organisational improvements is
extremely difficult. Through focusing on a particular industry sector,
researchers are better able to understand industry contexts and validate
elements of the evaluation framework. However it is interesting to note
that authors frequently do not retain the industry specific focus when
discussing their conclusions. And thus too much is sometimes made of the
finding from what is essentially a narrow focused study.
3.3
National focus
Many studies have referred to a particular geographical
area. Examples are:
§
United Kingdom (Miller and Dunn, 1997; Stansfield, et al., 2000).
§
The Netherlands (Nijland and Berghout, 2000; Stansfield, et al.,
2000).
§
Spain (Arribas, 1996; Arribas and Ingusta, 1997).
§
South Africa (Hart, 1999; Sutherland, 1994, Pather et al, 2004).
§
Australia (Cronk, 1999; Myles, et al., 2000; Singh and Byrne,
2004).
§
Belgium (Deschoolmeester and Braet, 2000).
§
Romania (Avram, 1999; 2000).
§
Greece (Mitris and Serafeimidis, 1994).
§
Denmark (Andersen, 1999).
§
Ireland (O’Donnell and O’Regan, 2000).
§
Saudi Arabia (Al-Turki, 2000)
§
Sweden (Frisk and Planten, 2004)
§
Finland (Kontio, 2004)
These papers often suggest that they provide
state-of-the-art overviews of IT evaluation in a particular country and
thus these studies do include regional specific elements of evaluation.
The way groups or cultures deal with corporate dimensions such as power,
user participation and risk are major influences on the evaluation.
Country studies typically include representative data sets.
3.4
Organisation of
the IT function
Besides evaluation of IT projects or operational
information systems, the evaluation may also refer to the overall IT
function (see, Watad, 1995; Shin, 1997). In particular, issues relating to
outsourcing is an object of study (Willcocks and Fitzgerald, 1994; Currie
and Irani, 1999; Khalfan, Gough, 2000; Lin and Pervan, 2001).
Papers in this area all refer to measurement problems
and evaluation i.e. the evaluation of an outsourcing decision requires
some form of measurable objectives and processes. These papers also
typically address the boundaries of measurement (not everything is
measurable) and also consider some of the arguments related to the nature
of the measurement activity (every measurement potentially raises another
measurement problem).
3.5
Evaluation
methods
A number of evaluation methods are especially
noteworthy. Those that received more attention then others include:
§
The Balanced Scorecard (Grembergen and Bruggen, 1997; Hillam and Edwards,
2001; Deschoolmeester and Braet, 2000).
§
Simulation as analysis tools for examining the effects of an envisioned
information system (Jong, 1999; 2000; Anderson, 2000).
§
DSDM (dynamic systems development method), as an approach to enhancing a
system development methodology with an evaluation approach (Barrow and
Maylew, 2001).
Given the focus of the conference, the number of papers
that actually develop evaluation methods is modest. The discussion about
evaluation methods has primarily been taken place outside the conference
(Parker and Benson 1988; Remenyi and Sherwood-Smith, 1997; Thorp, 1998;
GAO, 2000) and was published in book form. Perhaps this is due to the fact
that any discussion on the development of evaluation methods requires much
more detail than can be reduced to approximately 5,000 words.
These methods are all of a multidisciplinary nature.
Financial approaches receive relatively little attention except for the
option theory approach (see next Section for references).
3.6
Evaluation
theory
Evaluation is a multidisciplinary topic and many
theoretical approaches have been applied to study evaluation practices and
explain the various phenomena. Examples of theory-based approaches are:
§
Economics/accounting theory (Dier and Mooney, 1994; Dirks and Lent, 1997;
Bannister and McCabe, 1999; Maanen and Berghout, 2001; Svavarsson, 2002;);
§
Interpretative approach (Serafeimidis and Smithson, 1994; McBride and
Fidler, 1994; Abu-Samaha, 2000);
§
Critical approach (Nijland, 2001; O’Donnell and Hendriksen, 2001; Jones
and Basden, 2002);
§
Structuration theory (Vaujany, 2001, Jansen and Nes (2004));
§
Grounded theory (Jones and Hughes, 2001);
§
Contingency approach (Turk, 2000);
§
Soft Systems Methodology and process theory (Kefi, 2002; Stansfield et
al., 2000).
§
Cognitive mapping (Newman and Hang, 2002);
§
Option theory (Jong, et al., 1997; Clare and Lichtenstein, 2000;
Mehler-Bicher, 2001; Svavarsson, 2002;);
§
Social theory (Berghout et al., 1996);
§
Post-modernism (Remenyi and Sherwood-Smith, 1996).
Theoretical underpinning of IT evaluation research is a
major issue, because this very well demonstrates the level of
understanding of the topic. The variety of approaches already illustrates
that there is little consensus in this area. It is even extremely complex
to make a statement about more or less promising approaches. Option theory
has not been very successful from a practical point of view. However, it
remains noteworthy from a theoretical perspective explaining the
economical aspects of evaluation. The interpretative approach such as that
advocated by Walsham, (1993) received the widest attention. Elaborating
upon a theoretical foundation of IT evaluation research is certainly an
urgent issue.
4.
Research results
In this section the conclusions of the 298 papers are
summarised into four main research findings as follows:
§
The untangling of IT value;
§
The multidisciplinary nature of evaluation;
§
The importance of stakeholder analysis;
§
The importance of organisational learning and life cycle management.
4.1
The untangling
of IT value
The untangling of IT value is discussed, first, through
elaboration on IT cost, second, through elaboration on IT benefits, and
third, through elaboration on IT value creation.
4.1.1
Untangling IT
cost
Many of the 298 papers deal with the notion of cost as
a resolved issue. However, a number of papers illustrate that this is
perhaps not the case (Bannister and McCabe, 1999; Maanen and Berghout,
2001; Dirks and Lent, 1997; Dier and Mooney, 1994). Costs associated with
developing or operating information systems are primarily of a fixed and
indirect nature. Calculating IS cost, therefore, always implies the
allocation of cost and there are many unresolved issues regarding such
allocation. Bannister and McCabe present a list of difficulties associated
with understanding IT related costs (Bannister and McCabe, 1999):
§
Identification problems. IT is always part of something else, being a
project or departmental unit. For example, to what extent are user-cost
included in IT costs?
§
Data capture problems. Besides general accounting failures, there are
typical non-recorded costs, such as, implementation cost.
§
Overhead allocation problems. Cost accounting of IT includes many charging
issues making this a personal and political problem.
§
Accounting conventions. Different standards regarding amortisation and
capitalisation are applied between and within countries.
§
Disbenefits. This typically refers to the negative consequences of
introducing IT, such as, increased risk of fraud and decreased flexibility
of operations.
A cost-based approach also seems to be more adequate
for investment analysis than cash flows. Examples of cost that would be
unaccounted for using a cash flow analysis could be hardware cost, when
the IT investment would not directly result in a purchase of additional
hardware.
In contrast to what is suggested in many of the
conference papers, there are many issues still unresolved regarding IT
cost. There is little knowledge of the cost behaviour of information
systems and few, if any tested methodologies or even theories to manage
and control IT cost. IT costing is a complex issue, which is well suited
for further research.
4.1.2
Untangling IT
benefits
Most of the papers deal with the complex issue of
untangling IT benefits. Sometimes this is done in great detail (Bannister,
1998; Remenyi, 1999; Bannister, 2000; Lillrank, et al., 2000; Remenyi,
2002). Remenyi identifies four major problems with IT benefits measurement
and management (Remenyi, 1999):
§
Benefits such as intangible performance improvements. Unlike cost, such
benefits primarily impact processes inside an organization and seldomly
associated to goods or services sold on an outside market. Their value is,
therefore, predominantly dependent on individual judgement and not on
market prices.
§
The issue of information reach. Even for the most straightforward
application it is never simple to understand exactly what the results will
be of bringing together information about different business issues. There
will nearly always be knock-on effects, especially when such a system
results in integrating business processes.
§
Tangible and intangible benefits. Some aspects of an information system
may produce hard or tangible benefits which will directly improve the
performance of the firm, such as reducing cost and will therefore be seen
in the accounting numbers of the organisation as an improvement in profit
and perhaps in return on investment. However, other aspects of this system
will only create soft or intangible benefits, which will make life easier
in the organisation, however, will not directly lead to identifiable
performance improvements. In a competitive market cost reductions are
primarily transferred to customers and the associated prevented
competitive loss may also not show up in the accounting numbers of the
firm.
§
Benefit evolution. Many information systems will have some easy to
identify or obvious benefits which will be sustainable over a period of
time. However, as the development of the project proceeds and the
ramifications of the system more fully understood, new ideas about
potential benefits will also become apparent. This will have been due to
the process of creative dialogue between the principal stakeholders, which
will bring to light new business processes and practices. In short,
potential benefits should not be seen as being static, but rather evolve
as a greater understanding is gained of the organisation and the role
which the system will play in this. Given this observation, some
researchers developed techniques to actively manage benefits (Lillrank et
al., 2000; Remenyi and Sherwood-Smith, 1997).
Notable is that most researchers define value in a
multidisciplinary perspective and very few restrict themselves to a purely
financial analysis. There is, however, no consensus about the
operationalisation of the measurement and management of benefits.
4.1.3
Untangling IT
value creation
Given the problems encountered with untangling IT cost
and benefits, it may be expected that there will also be difficulties with
untangling the notion of IT value, and this is certainly the case.
However, some common understanding has also developed in this area.
Process models have been developed to illustrate the
process of value creation at a higher level of detail. Well-known examples
of such models from outside the conference are Trice and Treacy, (1986),
Weill (1990), Soh and Markus (1993). Conference papers regarding process
models include McKeen et al. (1996), Jurison (1999), McAuley (2000),
Remenyi (2002).
Similar to the discussion around benefits there is a
prominent role for individual perspectives and the multidisciplinary
nature of value.
Almost all studies presented at the conference evaluate
the value of IT on an organisational level and, remarkably, few on higher
levels, such as, governments or countries (Shu, 2001, Bannister and
Remenyi 2003). There are also few publications regarding the lower
individual level (exceptions are Bannister, 2002 and Hughes, et al.,
2002). The country studies that have been presented focus on the use of
evaluation techniques in a particular industry or organisation and not on
the effects of its impact upon the country or the society. Although, the
mission of the conference explicitly includes macro economic studies,
these papers still have to find their way to the conference.
4.2
The
multidisciplinary nature of evaluation
There has been a special interest in the
multidisciplinary nature of evaluation since the first conference, see for
example the papers McBride and Fidler (1994), Serafeimidis and Smithson
(1994), Brown (1994), Jurison (1994). Section 3.6 already provided an
overview of the various theories that have been suggested or used.
However, there is no common understanding regarding the methodology
necessary to understand a multidisciplinary field of study. Difficulties
with finding appropriate data sets and issues related to how handle these
have yet to be resolved.
4.3
The importance
of stakeholder analysis
Regardless of the research approach or theoretical
underpinning, an analysis of the objectives and influence of the various
stakeholders has become a common part of evaluation studies (see for
instance Serafeimidis (1999), Hughes and Jones (1999), Jones and Hughes
(1999), Khalifa, et al. (2000), McAuley and Doherty (2001). Although a
stakeholder analysis appears to be an essential element of an evaluation
study, the methodological consequences remain relatively unexplored
(Grembergen and Bruggen, 1997; Deschoolmeester and Braet, 2000; Hillam and
Edwards, 2001). First studies that contain a methodology to include
stakeholder perspectives are Jones and Hughes (1999), Barrow and Mayhew
(2001), Remenyi (2002). These approaches all focus on a continuous value
management: the more traditional accounting viewpoint of pre and post
implementation studies is abandoned. These types of continuous management
are typically suited to managing scope dynamics of projects and emergent
values (Remenyi, 2002). These approaches seem to be, however, far from
robust.
4.4
The importance
of organisational learning and life cycle management
In line with the multidisciplinary nature of evaluation
and stakeholder analysis is the importance of organisational learning and
life cycle management. The organisational and strategic impact of IT
primarily depends on organisational and market characteristics, such as
the ability to adapt new working methods, as well as, the willingness to
pay for additional product information. Information about already
completed projects is, therefore, essential to improve ex-ante or up-front
assessments (Alsen and Linde, 1994; Kaplan, 1995; Ward, et al., 1995;
Serafeimidis and Smithson, 1995; Swinkels, 1997; Nijland and Berghout,
2000).
Many researchers observe problems in this learning
process (see Nijland and Berghout, 2000). Several reasons are given:
§
The project objectives were defined in an uncontrollable fashion.
§
The priorities of other tasks are higher.
§
The outcome differs from the initial perspective and the various
stakeholders try to avoid discussing who is to blame for this.
As a consequence evaluation researchers are confronted
with the situation that evaluation practice is relatively poorly
developed, case studies are unable to validate the more advanced elements
of evaluation methods and practitioners are reluctant to use untested
methods.
There is also the issue of diminishing value of
additional evaluations. Although continuous evaluation and life cycle
management do seem to be an obvious route for the development of
evaluation methods, there is always the risk over ‘over-measuring’. This
issue has, so far, not been explored at the conference.
5.
Perspectives
In this Section three issues are presented that we
consider particularly interesting for further study. Given the complex
nature of this field of study and the relatively short time it has been of
interest to academic researchers the number of topics that could be
explored is, of course, almost endless. However, based on the observations
in the previous Sections we conclude that some issues might have a higher
priority than others. These issues are:
§
Theoretical underpinning of IT evaluation.
§
Improved methodological understanding of which data sets are appropriate
and how they may be used in this type of research.
§
Establishing a more common core of concepts.
5.1
Theories used to
underpin IT evaluation
As presented in Section 3, the number of different
theories used in the studies of the past nine years is significant and
there is certainly no commonly accepted understanding of which of these
might be most appropriate.
Accounting theory and interpretive analysis are most
frequently applied. Interpretive analysis has been supported since the
first days of the conference (Serafeimidis and Smithson, 1994). In this
case IT evaluation is viewed as a socially embedded process including
formal and informal procedures and where actors try to make some sense of
their situation. Other researchers encouraging this approach are
(Abu-Samaha, 2000; Jones, 2001; Agerfalk et al., 2002; Kefi, 2002).
Several researchers refer to Habermas’ theory of social
action, where the “world” is explained as a whole (Jones and Basden,
2002). Habermas’ theory is centred on communication and seeks to clarify
the conditions, means, content, constraints and objectives of socially
organised human behaviour (Habermas, 1984). Researchers that encourage
this approach are O’Donnell and Hendriksen (2001) and Jones and Basden
(2002).
Glaser and Strauss’s Grounded Theory is referred to by
Jones and Hughes (2001) as a more structured approach that provides some
form of synergy between the more quantitative interpretive research and
more qualitative positivistic research.
Other noteworthy frameworks include critical theory,
referred to by Nijland (2001) and Giddens’ structuration theory, referred
to by Vaujany (2001). Furthermore, option theory is another referred to
quantitative approach (Jong, 1997; Clare and Lichtenstein, 2000;
Meher-Bicher, 2001; Svavarsson et al., 2002).
Clearly there is no single, superior, theoretical
underpinning for research in this field of study. And this issue in itself
is a major problem which deserves much attention.
5.2
Improved data
sets
Business research is primarily built on empirical data
and often in the form of case studies and questionnaires. This is a
logical situation given the relative immaturity of this research.
Comparable with medical sciences, most medical breakthroughs historically
originated from observations of initially successful patients and were not
the result of double blind testing. However, the internal and external
validity of some case studies is a matter of concern to the rigorous
researcher.
The issue of what data is appropriate and how to use it
in reaching conclusions is by no means agreed and needs much more
