1.
Introduction
Technology Based
Information Systems (IS/IT) Evaluation issues have generated much
interest among researchers and practitioners. A wide range of studies
exist, drawing on multiple evaluation perspectives, such as (1) the
technical perspective: monitoring, data quality management,
technological viability, risk evaluation; (2) the financial and
economic perspective: ex-ante and/or ex-post assessment of IS/IT
contributions to performance, productivity ratios, return over
investment ratios, financial auditing; (3) the strategic perspective:
IS/IT value chain, IS/IT-based competitive advantages; and (4) the
organizational perspective: IS/IT contribution to organizational
effectiveness, IS/IT-enabled organizational change. Most of these
studies rely on variance models and cross-sectional quantitative data
(Markus and Robey, 1988). In this paper, we focus on the
organizational perspective of IS/IT evaluation because we consider it
offers a deep framework analysis that fits with a longitudinal
approach allowing the study of two major phases: (1) the introduction,
design and implementation of technology and (2) the use, appropriation
and impacts of that technology. In this perspective, the organization
is considered both as the context of development and use of the
technological tools and the mediating interface that generates the
impacts of these tools (Foray and Mairese, 1999). We propose a
process-oriented evaluation model that explicitly takes into
consideration context variables and then apply it in an in depth
field study related to a DataWarehouse (DW) implementation project in
a financial organization.
2.
Methodology
The approach we
adopt here is affiliated to General Systems Theory (Von Bertalanffy,
1968 LeMoigne, 1977) and Soft Systems Methodology (Checkland, 1999;
Checkland and Scholes, 1990). It consists of constructing a
theoretical model to represent and describe a problem. This model will
be used as a diagnostic tool that aims to produce a clear picture of
this situation and provides knowledge to prescribe corrective actions.
Two levels of analysis are used: the positivist (checking the validity
of the model and the accuracy of the causal relations it recognizes )
and the interpretive (checking the explanatory power of the model).
The core question
we investigate is: “what are the critical elements that conduct and
converge over time to the success (or the failure) of the DW
implementation project within this organization and are there any
solutions to prescribe in order to promote success and avoid failure?”
The longitudinal
case study conducted is then based on the assumption that the adoption
and use of the DW should be conceptualised as a form of organizational
change, and that such perspective allows to anticipate, explain and
evaluate different consequences following the introduction of such a
tool in the organization (Orlikowski, 1993). We argue that these
consequences occur at multiple levels: the IS/IT level, the individual
level and the organizational level. The focus is then to develop a
context-based, process-oriented description and explanation of the
phenomenon, rather than on objective and static description
(Orlikowski, 1993; Kannelis, Lycett and Paul, 1998).
This approach
draws on Pettigrew’s (1990) conceptualisation of the organizational
change study within the organization in a manner that considers both
the content and context of change over time. Multiple research
techniques will be used to draw a clear picture of the reality we are
describing, from multiple view points and using multiple data sources,
including questionnaire, structured and semi structured interviewing,
documentation review and observation.
3.
Theoretical
development
We consider three
dimensions of what Markus and Robey (1988) call a good theory, that
means “theory that guides research, which when applied, increases
the likelihood that information technology will be employed with
desirable consequences for users, organizations and other interest
parties” (p. 583). These dimensions are: causal agency, logical
structure and level of analysis.
·
The causal agency refers to the
nature of causality relationships between the two major components of
the theory: IS/IT and Organization. Here, we have chosen to cut with
the frameworks largely used in this kind of research areas, such as
the studies investigation the organizational impacts of IS/IT, or
those related to IS/IT development and implementation literature.
These frameworks consist of causal and variance models that proceed to
the identification of dependent and independent variables, and define
the relationship between the technology and the organization, in terms
of causal unidirectional relations, based on the deterministic
assumptions of either a technological or an organizational imperative
(Markus and Robey, 1988).
The framework
proposed here is different from those identified above. It is based on
a structurational perspective (Giddens, 1987; Orlikowski, 1992, 1993,
2000; DeSanctis et Poole, 1994; Swanson and Ramiller, 1997) that
emphasizes the dual nature of technology comprising a social and a
technical interrelated components, and considers a dialectic relation
between technology and organization, via diverse attitudes and
behaviours of human agents during tasks execution. Three main
categories of agents are identified: the decision makers (or
managers), the users and the conceptors/developers. The institutional
properties of the organization are considered in this framework as:
(1) the context that affects the ways in which these agents interact
with technology; and (2) the field among which the impacts of these
interactions are shaped.
·
The logical structure concerns the
hypothesized relationships between causes and effects or outcomes
among the components of the theory and how they are shaped over time.
That means whether causes are related to effects in an invariant,
sufficient and necessary relationship (variance model) or in a recipe
of sufficient conditions occurring over time.
We argue that a
process model for IS/IT evaluation is more appropriate to investigate
over time the conditions of success or failure of an IS/IT project
through out its life cycle, considering the content of the impacts it
generates and the context in which these impacts occur.
Process-oriented
IS/IT evaluation models supposing longitudinal studies are recommended
in many studies, such as those of King and Rodriguez (1978), Hamilton
and Chervany (1981a, 1981b) and Delone and McLean (1992).
·
The level of analysis refers to
the entities about which the theory poses concepts and relationships.
We adopt a multi level perspective for IS/IT evaluation. These levels
are considered over time, beginning with the (IS/IT) tool and his
intrinsic characteristics: technical and social ones, then we consider
the impacts it generates at the individual level: the users considered
individually, then on the organization as a hole.
As such, our
process-oriented and multi-level evaluation approach has some
similarities with the Delone and McLean Model (1992). However, we
propose an extension to this model by cutting with the simple
unidirectional causes-effects relationships it advocates and that have
been mostly criticized (Ballantine, et al. 1998). The structurational
perspective allows us to build an interactionist and recursive
relationship between the three levels: the technology, the individual
and the organisation considered, in this order over time, and the
institutional context. Evaluating the performance and effectiveness
of a technology based information system within this framework
requires a global understanding of the phenomenon incorporating the
strategic and organizational context, and more specifically the
context of development and use, and the interactions between actors
during the system life cycle. We argue that the complexities related
to all these aspects have not to be ignored or underestimated.
Now, we explicitly
define the major components of our theoretical model, according to the
literature background, then we establish our research hypotheses that
characterize their causal relationships.
3.1
The strategic and organizational context
Organization’s
size, corporate strategy, structure, culture, role of the IS/IT
function in the organization, IT strategy, role of the leadership in
the decision making concerning IS/IT are some of the most cited
factors related to the strategic and organizational context of IS/IT
implementation projects. In most of the studies, they are considered
in a contingency perspective in the evaluation approaches (Saunders
and Jones, 1992; Myers, Kappelman et Prybutok, 1998).
A deeper stream of
research is based on an inductive perspective aimed at identifying,
exclusively through data collected and analysed in the field, the
critical elements that shape the impacts associated with the
implementation and use of IS/IT tools (Orlikowski, 1993).
We propose to
begin our investigation of the strategic and organizational context by
considering three aspects largely cited in the literature: (1)
corporate strategies, (2) structure and culture, (3)
role of IT function.
3.2
The context of development and use
The approaches that focus on the
context of development and use are based on theories of attitudes and
behaviour and try to identify the situational factors that lead to
intentions to IS/IT utilization and acceptance. They argue that
increased utilization and acceptance lead to positive performance
impacts. Within these situational factors, Zmud (1979) focus on the
construct of Individual Differences; Ives and Olson (1984), and
Kappelman (1995) on User Involvement; Barki and Hartwick (1994) on
User Participation, User Involvement and User Attitude; Goodhue et
Thompson (1995) on the characteristics of the task and the individual
and the concept of Task-Technology-Fit.
In these studies,
the subtleties between these approaching concepts are determined and
their impacts on IS/IT performance studied either as independent
variables or as mediating ones.
3.3
The IS/IT level: perceived performance
evaluation
This component of
our model is related to the perceptual approaches of IS/IT evaluation
that try to identify “the dependent variable” for the IS/IT
effectiveness or success and focus on the users perception of utility
and satisfaction to measure it. The so called user satisfaction or
user information satisfaction construct is the main surrogate measure
for this variable. It is the theme of a wide range of theoretical and
empirical studies, such as: Franz and Robey (1986) measure of
perceived usefulness; Doll and Torzadeh (1988) measure of end-user
computing satisfaction; Bailey and Pearson (1983) measure of user
information satisfaction; Davis (1989) measure of perceived usefulness
and perceived ease of use. And more recently the instruments
developed by Straub, Limayem et Karahann-Evaristo (1995); Limayem,
Bergeron et Richard (1997); Garrity and Sanders (1998); Ishman (1998).
We draw on this
stream of research to establish our first hypothesis:
H1:
The IS/IT quality assessed in terms of Task-Support-Fit, Ease of Use
and Interface Quality, jointly by the Degree of Usage of this tool,
determine its perceived impacts in terms of: Global user satisfaction,
Impacts on productivity, Decision making impacts and Quality of Work
life Satisfaction.
3.4
The impacts of development and use
context on the perceived performance of the IS/IT
The key issue here
concerns the nature of the relationship between the factors related to
this context and the determinants of the perceived performance of the
IS/IT defined above.
Following Goodhue
and Thompson (1995); Barki and Hartwick (1994), we consider that: Task
characteristics and UsersParticipation in the development process as
the determinant variables of Perceived IS/IT Performance.
Where as, Users
involvement (a priori and a posteriori), Users Attitude towards IS/IT
tools (mostly the degree they consider themselves as experimented and
skilled in manipulating these tools) and finally, Collaboration
relationships between Users and Conceptors/Developers, are mediating
variables that relatively affect the way previous determinant
variables shape the ultimate perceived performance (Saleem, 1996;
Mckeen and Guimares, 1997). Our second hypothesis is then the
following:
H2:
Task Characteristics and User Participation in the development process
have an impact on the perceived performance of the IS/IT. This impact
is moderated by mediating factors related to User Involvement,
Perceived Experience in IS/IT use and Collaboration Relationships
between Users and Conceptors/Developers.
3.5
The impacts of the strategic and
organizational context on the perceived performance of the IS/IT
These impacts are
not deeply investigated in the previous literature. We can however
build upon some works (Lucas, 1973; Franz et Robey, 1986) that
support the intuitively accepted hypothesis according to which:
H3:
The strategic and organizational context contribute to shape the IS/IT
development and use context and accordingly have an impact on the
perceived performance of the IS/IT. The strategic and organizational
impact can also directly affect that perceived performance.
3.6
The individual level: identifying
appropriation types
Different
interactions schemes may occur between the users and their IS/IT tool.
These are what we call appropriation types. According to DeSanctis and
Poole (1994) and the Adaptive Structuration Theory, the appropriation
concept results from studying the structurational impacts of an IS/IT
at the individual level, that means situating the user as the analysis
unit.
More precisely,
the studies on the concepts of Infusion, Diffusion and Routinization
(Cooper and Zmud, 1990; Saga and Zmud, 1996) adopt a process-oriented
perspective to describe the human-machine subsystem in its ongoing and
day-to-day practices. These result from a series of events:
implementation, use, acceptation.
Saga and Zmud
(1996) establish a utilization taxonomy at the individual level: the
extended use, the integrated use and the emergent use; and explain
each of them using different determinant factors, such as the IS/IT
Function maturity or the leadership role.
We have adopted
this point of view to establish our fourth hypothesis:
H4:
Multiple appropriation types at the individual level emerge over time.
They are affected by the perceived performance of the IS/IT and by the
prevailing context.
3.7
The organizational level: investigating
changes
The approaches
that focus on the organizational level to evaluate IS/IT are based on
the assumption that the information system, its technological support
and the human being using it to perform his (or her) tasks is a
subsystem within the organizational system and that the performance
evaluation of this subsystem has to be assessed via its contribution
to the achievement of the organizational objectives, mainly those
related to organizational effectiveness (Chakraborty, 1994; Mirani et
Lederer, 1998) and IT-enabled organizational change (Grover, Jeong and
Teng, 1995; Kettinger, Teng et Guha, 1997). While most of the studies
that investigate these aspects follow a causal, variance and
deterministic framework, those based on the structurational model
identified above try to go further and argue that the implementation
and use of IS/IT should be conceptualised as a form of organizational
change (Orlikowski, 1993; Orlikowski, 1996; Orlikowski and Hofman,
1997).
These studies
offer a deeper framework to emphasize the criticality of the
organizational context in shaping technology use in organizations and
in describing the changes occurred among them recursively. Hence, they
support our hypotheses H5 et H6:
H5:
Organizational impacts defined in terms of organizational changes:
planned change, improvisational change and/or emergent change, appear
over time. They are affected by the different individual appropriation
types and by the prevailing context.
H6:
These organizational changes (H5) potentially produce incremental or
radical changes among the pre-existing context.
Finally, our the
theoretical model consists of two components:
·
The first component is a process-oriented
evaluation instrument of a technology based information system that
determines: (1) the perceived performance
of this system by
its users; (2) its individual impacts on users by identifying the
resulting appropriation types; (3) its organizational impacts defined
in terms of organizational changes.
·
The second component of the model
displays the context factors that affect the perceived performance of
the IS/IT and its individual and organizational
impacts. These
factors are related to the organizational context, the strategic
context and the context of development and use.
According to the
structurational model defined above, the two components are
interrelated recursively (figure 1).
Figure 1: The process-oriented
and Context-based Model of IS/IT Evaluation recursively (figure 1).
4.
The conduct of the study
A longitudinal
case study was conducted in a multinational financial organization
specialized in assets management. The IS/IT under study is related to
a Datawarehouse implementation project conceived as a decision support
system and aimed to be an organization-wide information system that
supports all the activities of assets performance analysis and client
reporting.
The study has been
conducted during a period of 17 months. It consisted of three main
phases, which not only differed in time and purpose, but also with
respect to the research methodology and techniques used.
4.1 Phase 1:
(5 months)
·
Purpose: study of the strategic and
organizational context and the context of development and use
(including the DW project specificities).
·
Methodology: qualitative data collection
and analysis using: documentary analysis (content analysis),
observation and interviewing of the three categories of actors
(decision makers, IS/IT conceptors/developers and users (6
semi-structured interviews and 15 structured interviews have been
conducted)
4.2 Phase 2: (5
months)
·
Purpose: constructing and applying an
instrument that measures the DW performance and relates the
performance criteria used to some specificities of the development and
use context.
·
Methodology: questionnaire construction
and administration: a total of 101 usable questionnaires were
collected and analysed using statistical data analysis.
4.3 Phase 3: (7
months)
·
Purpose: (1) identifying appropriation
types at the individual level, relating them the performance scores
obtained in phase 2 and the characteristics of the user-machine
characteristics during all the observation period; (2) describing the
organizational changes over time, relating them to the identified
appropriation types and the prevailing context.
·
Methodology: qualitative data collection
and analysis using: documentary analysis, observation, interviewing
(11 structured interviews) and participative intervention (in
collaboration with the actors) that aims to solve problems related to
usage dysfunctions identified for each appropriation type.
5.
Analysis and Results
As stated above,
distinct research phases have been conducted. Data analysis performed
in one phase can be interpreted and reused in the following phases.
The results obtained in phase 1 concerning context properties are used
and actualised (identifying the changes) during all the research
process. The results of phase 2 are used and reinterpreted in phase 3.
5.1 Phase 1:
Context Properties
We have focused on
the strategic and organizational context and identified the
characteristics of the company concerning:
5.2 Phase 2: DW
Perceived Performance
The questionnaire
(see annexe) has been adapted from previous literature (Ishman, 1998;
Kappelman, 1995; Goodhue and Thompson, 1995; Saleem, 1996; Myers and
Kappelman, 1997; Doll and Torzadeh, 1988; Davis, 1989; Franz and
Robey, 1986; Doll and Torzadeh, 1988, Baroudi and Orlikowski, 1988).
A pre-test has
been realized in order to adapt the original items to our specific
case. Then, all the adapted items have translated in French. The
questionnaire has been established in two versions: the French and the
English ones and then administered in the two sites where the DW is
functional, in Paris and London.
The 101 usable
questionnaires obtained represent a response rate of 53,2 %. We used
the statistical data analysis package SPSS 10.0 to execute the
following treatments:
·
A descriptive data analysis
·
A confirmative data analysis through a
series of factor analyses
·
A series of hierarchical regression
analyses
All of them aim
to test the hypotheses H1 and H2.
5.2.1
Descriptive data analysis
It concerns two
categories of variables:
·
The performance criteria variables
(dependent variables): global satisfaction, impacts on productivity,
impacts on quality of work life satisfaction, impacts on decision
making, usage (frequency / regularity and perceived dependence),
task-technology-fit, ease of use and interface quality.
·
The context variables (independent
variables): including users participation in the development process,
characteristics of the human-machine subsystem (user-task), users
involvement a priori (during the development process), users
involvement a posteriori (during the use process), collaboration
relationships between users and conceptors/developers and finally
users perceived experience in IS/IT.
We have calculated
aggregated and detailed performance scores for the first category of
variables and, using frequency analyses performed upon the second
category of variables, determined groups of users presenting the same
context of development and use specificities.
2.
Confirmative data analysis
As presented in
table 1(representing the results of the VARIMAX analysis), factor
analysis produced satisfactory results in terms of reliability of
data: internal consistency (Alpha Cronbach’s > 0.7 for most of the
variables), content and convergent/discriminant validity. 24 items
over a total of 26 load to their respective factors (the variables
they are assumed to measure in the theoretical model).
Table 1: Factor Analysis results
The hierarchic
regression we executed on the basis of the previous results aim to
examine how the different categories of variables behavein terms of
causalrelationships. We consider that the ultimate dependent variables
are the perceived impacts of the DW: Global user satisfaction, Impacts
on productivity, Decision making impacts and Quality of Work life
Satisfaction. We establish a regression equation for each of these
variables where we test the determinant power of those variables
related to Task-Support-Fit, Ease of Use, Interface Quality,and Degree
of Usage, introduced progressively (hypothesis H1). Then, we test the
determinant power of the context variables (hypothesis H2), also
introduced progressively.
The results
obtained strongly confirm the first hypothesis H1 (p<0.01 and
satisfactory correlations). Global user satisfaction, Impacts on
productivity, Decision making impacts and Quality of Work life
Satisfaction are effectively determined by the DW quality assessed in
terms of Task-Support-Fit, Ease of Use and Interface Quality, and by
the Degree of Usage of this tool, in terms of frequency and regularity
of use and perceived dependence of users.
Concerning, the
second hypothesis H2, evidence showed no significant determinant power
of all the context variables for the impacts criteria related to
global user satisfaction, impacts on productivity and impacts and
Quality of Work life Satisfaction. Where as, Impacts on decision
making are effectively related to human-machine characteristics, Users
involvement and their perceived experience in IS/IT.
5.3 Phase 3:
Individual appropriation types and organizational impacts
We have to notice
here that the third hypothesis H3 has not been tested using the
previous questionnaire because it is not frequent to find items
related to strategic and organizational context variables in
satisfaction questionnaires in IS/IT usage addressed to a population
of users belonging to the same organization. We argue that these
context specificities have to be assessed and related to the results
obtained during the evaluation process, using qualitative data
collection techniques, such as observation and interviewing.
For this concern,
we found out that the role of IS/IT function in this firm as a global
entity producing standardized solutions is the main explaining factor
of a context of development and use with no significant participation
of users in the development process, except for those who will use
the DW in decision making tasks.
Now, concerning
the individual level of analysis, the results obtained due to the
questionnaire, interviewing of the actors (users, conceptors and
managers) and observation have converged to define four appropriation
types that describe different interactions schemes between the DW and
the users, depending on the role they played in the development
process, their own characteristics, mainly their perceived experience
in IS/IT and their collaboration with conceptors during the
development and use of this tool. These appropriation types are
defined regarding two axes:
These results
provide support to the hypothesis H4.
We have also
noticed that each appropriation type is characterized by homogenous
performance scores and present the same utilisation problems related
to data quality or task support misfit.
A collaborative
working group including the researcher and the representatives of (1)
the DW conceptors/developers team, (2) each group of users, (3)
managers of users departments, and (4) managers of the IS/IT
function, has been established in order to identify these problems,
conceive solutions, deliver them and provide all the support and
assistance to implement them.
This collaborative
group is a temporary structure (maintained during three months) and
has not been institutionalised. Where as, some of the changes it
helped create can be defined as organizational changes: the DW HOT
LINE implementation within the IS/IT function, routinizing the
collaborative use of the DW between the users departments are some of
the most significant one and thus, provide support for our fifth
research hypothesis H5.
Regarding the
observation period (only 7 months for the phase 3), can we detect the
recursive impact of the DW implementation process on the prevailing
institutional context ?
What we can say is
that role of the IS/IT function as a global entity delivering
standardized solutions to all the sites of the group has been
reconsidered to adjust the implemented solutions to the real needs of
the users. Thus, we can conclude that this recursive impact is also
confirmed, and so is the final hypothesis H6.
6.
Summary, implications and conclusion
In this paper, a
theoretical model is developed as a process, where three levels of
evaluation are considered over time: the IS/IT level by measuring the
perceived quality of the application and of the information, the
degree of use of that application and its perceived impacts by users,
in terms of satisfaction for example; the individual level by the
identification of individual appropriation types; and the
organizational level by the study of the organizational changes. The
whole process is integrated among the organizational and strategic
context and the context of development and use.
This model has
been applied in an in depth field study conducted over 17 months and
related to a DW implementation project in a financial institution.
Multiple research techniques have been used, including documentary
analysis, observation, cooperative intervention, interviews and a
questionnaire.
Results suggest
that the perceived performance of the DW, jointly with the context
factors related to the characteristics of the human-machine subsystem
characteristics and the role of the IS/IT function in the
organization, determine multiple appropriation types. These involve,
to be effective, a process of organizational change over time.
We finally suggest
that this study can be reproduced in other research contexts,
including multiple organizations and multiple sectors, in a
comparative perspective, and/or to provide generalization and external
validity to the results. It can also be more extended in time in
order to investigate more deeply the recursive impacts of an IS/IT
implementation project on its institutional context.
References
-
-
-
-
-
-
Checkland, P. and Scholes, J. (1990): Soft Systems
Methodology in Action. John Wiley & Sons Editions.
-
-
-
-
-
-
Foray, D. and Mairese, J. (ed.)
(1999): Innovations et Performances.
EDHESS editions.
-
-
-
-
-
-
-
-
-
-
-
-
-
-
LeMoigne, J-L. (1977): La théorie
du système général: Théorie de la modélisation.
Editions PUF.
-
-
-
Markus, M.L. and
Robey, D. (1988):“Information Technology and Organizational Change:
Casual Structuring in theory and research”.
Management Science,
34(5), pp. 583-598.
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
Von Bertalanfy, L.( 1968):
General systems theory. George Braziller NY Editions.
-
