Editorial
Dan Remenyi
dan.remenyi@tcd.ie

The Eleven Years of the European Conference on IT Evaluation: Retrospectives and Perspectives for Possible Future Research pp81-98
Egon Berghout¹ and Dan Remenyi^2
1University of Groningen, The Netherlands
²Trinity College Dublin, Ireland
e.w.berghout@eco.rug.nl, dan.remenyi@tcd.ie

This paper provides an overview of the papers that have been presented at the European Conference on IT Evaluation during the past eleven years. It considers the main issues, and learning themes addressed in papers presented to these Conferences. The paper also reflects on the possible future direction, which this research may take and three major research themes are suggested. Some 356 papers have been presented at ECITE. Over the eleven year period it is clear that the level of understanding as reflected in the papers has significantly increased. Themes, which were particularly well addressed, include IT and IS value, the multidisciplinary nature of evaluation, the importance of stakeholder analysis, organisational learning and life cycle management. Three issues are identified as particularly important for further research. These are, the theoretical underpinning of IT evaluation, improving the data sets for research and establishing a more common core of concepts.

2

Enterprise systems have been widely sold on the basis that they reduce costs through process efficiency and enhance decision making by providing accurate and timely enterprise wide information. Although research shows that operational efficiencies can be achieved, ERP systems are notoriously poor at delivering management information in a form that would support effective decision-making. Research suggests managers are not helped in their decision-making abilities simply by increasing the flow of information. This paper calls for a new approach to researching the impact of ERP implementations on global organizations by examining decision making processes at 3 levels in the organisation (corporate, core implementation team and local site).

3

For nonprofit organizations (NPO’s), ICT is crucial to fulfil their social objectives. However, it is rare that ICT investments have monetary returns; ICT also has indirect impact on the social activity of NPO’s. So it is very difficult for them both to decide about ICT investments and to evaluate their contribute to performance. NPO’s should therefore define an appropriate evaluation framework, to understand if, where, what and how much to invest in ICT, to better achieve their mission. The evaluation framework described in this paper is based on the peculiar characteristics of nonprofit organizations, on the multidimensional evaluation criteria and on the balanced scorecard, adapted to the specific nature of nonprofit activities.

4

The purpose of information systems post-evaluation ought to be to improve the use of systems. The paper proposes the use of narratives as a tool in post-evaluations. The potential in narratives is that they can convey meanings, interpretations, and knowledge about the system, which may potentially lead to action. The paper offer three main suggestions: 1) evaluations should form the basis for action; 2) narratives makes evaluation more relevant; and 3) post-evaluations should be done with the aim of improving use. Narratives should be viewed as a complement to traditional evaluation methods and as a way of making evaluation more formative and thereby moving away from the more common summative perception of evaluation. The conclusion of the paper is that narratives can advance IS evaluation and provide a richer evaluation picture by conveying meanings not included in traditional evaluations.

5

Information systems and databases in six Finnish organizations are evaluated in this multiple case study research. The main idea of the research was to describe the main gaps in information systems in the case organizations. In each case the gaps are presented with authentic descriptions. The research identified altogether seven different categories of gaps. These are first abstracted to four common categories of gaps: 1) data, 2) infrastructure, 3) turning data into information and 4) people working with the information systems. Finally, the four categories are further abstracted to two common categories of gaps: 1) information and 2) infrastructure.

6

Measuring the Quality of Electronic Journals pp133-142
Maricela López-Ornelas, Graciela Cordero-Arroyo and Eduardo Backhoff-Escudero
Autonomous University of Baja California, México
ornelas@uabc.mx, gcordero@uabc.mx, backhoff@uabc.mx

This paper presents the methodology developed to create a system to evaluate academic electronic journals. This methodology was developed in two stages. In the first stage, a system to evaluate electronic journals was created. The criteria framework and the indicators for assessment for academic electronic journals were selected and defined. According to this framework, several questions were designed to measure each indicator and, as a result, an instrument to evaluate academic electronic journals was built. In the second stage, this instrument was validated by 16 editors of electronic journals of different countries and different areas of knowledge that were considered as judges to evaluate clarity, importance, relevance and coverage of each question, indicator and criteria. This instrument was distributed by e-mail. The opinions given by the judges were processed and then used to help in the construction of a new instrument that is ready to be presented to the Mexican Council of Scientific Research in order to evaluate Mexican academic electronic journals.

7

EJISE is published by Academic Conferences Limited
Curtis Farm, Kidmore End, Nr Reading RG4 9AY, England
Tel: +44 (0)1189 724148, Fax: +44 (0)1189 724691, Email: info@ejise.com

Send mail to info@academic-conferences.com with questions or comments about this web site.
Copyright © 2002-2006 Electronic Journal of Information Systems Evaluation
Last modified: September 29, 2005
ISSN 1566-6379