|
1. Introduction
At the present time there is more
and more international commerce and global organisations emerging. In
order to execute and support more functions in the globalised
environment, information technology has been highly developed and is now
required to be further and more complexly developed. However, it is very
difficult to develop or deliver these large and complex products through
a single company. Therefore, the Virtual Enterprise (VE)/ Virtual
Corporation is now accepted in high technology industries as a way to
integrate a single company’s specialised knowledge and resources into
multi-partner collaborative projects (Camarinha-Matos et al 2002,
Davidow et al 1992). It forms as a temporary and voluntary co-operation
between a numbers of legally independent organisations (Andriessen,
2003).
A major difference between single
user system evaluation and collaborative system evaluation is that when
evaluating groupware, one is trying to do more than traditional
usability testing (Knutilla et al. 2000). Most collaborative/
cooperative systems have high development costs, it is therefore
important to measure the incremental progress towards developing useful,
large and complex collaborative systems. Ramage (1997) pointed out that
such evaluation is generally regarded as a highly complex task. However,
in a review of the literature on groupware systems, Pinnelle & Gutwin
(2000) found that a very high proportion of reports contained no
evaluation whatsoever and that only about one-quarter of the evaluations
involved a practical setting. This result implies that there is a
requirement and scope to pay more attention to evaluation of
collaboration systems. In addition, there are many approaches for
evaluating collaboration systems, however, there are no clear guidelines
or frameworks existing to support and determine what approach should be
applied in order to conduct an effective evaluation. Furthermore, there
is a need to conduct more evaluations across multiple projects using a
common framework where possible, in order to support the development of
a body of theory (Newman, 2001; Raybourn & Newman, 2002).
One framework based on the
evaluation lifecycle was therefore developed to support evaluation of
collaborative systems in multi-site and multi-partner projects. This
paper outlines the underlying concepts in the lifecycle-based approach.
For the purpose of proving the concept a case study of the
lifecycle-based approach is carried out within DIECoM (Distributed
Integrated Environment for Configuration Management), an EC Framework V
project. It then introduces a set of heuristics which is being developed
from the findings and during the process of using this approach.
2. Lifecycle-based approach
At present, rapid technology
development results in a diversity and volatility of circumstances
surrounding IT. It demands that evaluation methodologies need to be
flexible enough to be amended and take new requirements and changed
contexts into account. Farbey et al (1994) advise that in choosing an
approach to evaluation one should explicitly consider ‘what is being
evaluated, for what purpose, by whom, and for what audience’. In the
review of the literature, the traditional evaluation, which assumes a
stable context with a short term view, is not able to validate the
impacts of the system development. This brings out a need for a
continuous judgment of evaluation methodologies in order to act as a
continuous monitoring mechanism and be able to evaluate the entire
performance. It also implies that the choice of a suitable evaluation
method will be influenced by the organisations and social contexts.
This paper therefore attempts to
extend the traditional systems development life cycle to a model which
considers the organisational contexts and social circumstances.
Attention is particularly addressed to a continuous evaluation for
maximising the systems contributions instead of just deciding how to
begin the development, e.g. feasibility studies.
An approach therefore has been
developed within the DIECoM project to support evaluation of
collaborative systems in multi-site and multi-partner projects. This
approach follows the developmental lifecycle and aims to assist in
selecting appropriate instrumentation for evaluation. A traditional
developmental lifecycle is an iterative process evolving from
feasibility and user requirements capture through initial specifications
and definitions, then through to the analysis, coding/ design,
development/ testing and maintenance of the final product. Referring to
the traditional developmental lifecycle, this lifecycle-based approach
derives a sequence: defining evaluation plan, user requirements analysis
and specification, designing instruments for evaluating collaborative
systems against functional and non-functional requirements, implementing
the evaluation instruments and using feedback from users to modify the
instruments. This approach is described by the connected boxes shown in
diagram 1.
Typically, the lifecycle-based
approach starts off with defining the evaluation plan. Willcocks (1992)
identified a number of problems which are frequently encountered during
evaluation practice. One of them is failure to devote time and effort to
evaluate major capital assets. Thus, it is essential to define a
relevant milestone in the evaluation plan. Moreover, there are a range
of techniques which are possible contenders for gathering user
requirements, i.e. observation, questionnaires, interviews,
focus-groups, use of video and so on (Pinelle & Gutwin 2000). This
approach was built as a sequence, therefore in this step techniques for
collecting user requirements need to be defined.
Diagram 1: The Life-cycle based
approach
Andriessen (2003), in explicitly
addressing CSCW evaluation, commented that “the people and theories that
can be used to provide this much-needed user orientation are scattered
over many disciplines and communities”. Moreover, a review of recent
literature reveals awareness of the range of financial and non-financial
factors that need to be addressed into collaborative systems evaluation,
while the result of an evaluation is to be expected to have an impact on
the various stakeholder groups within the organisation. A deeper
understanding of perspectives of individuals and groups can reveal the
human and political aspects. Checkland (1981) introduced the concept
that “‘Hard’ systems thinking is appropriate in well-defined technical
problems and the ‘Soft’ systems thinking is more appropriate in fuzzy
ill-defined situations involving human beings and cultural
consideration”.
Hence, stakeholders who are
involved in a project need to be defined prior to user requirements
analysis. Evaluating the performance and effectiveness of collaboration
systems requires comprehensive understanding of the corporate strategic,
organisational context, role of the leadership in the decision making,
the perspective of development and the interactions between project
partners during the developmental lifecycle. Kefi (2002) stated that
multiple evaluation perspectives can be categorised into four groups as
below.
§ The technical
perspective: monitoring, data quality management, technological
viability, risk valuation
§ The strategic
perspective: IS/IT value chain, IS/IT based competitive advantage
§ The organisational
perspective: IS/IT contribution to organisational effectiveness and
IS/IT enabled organisational change.
§ The financial and
economic perspective: ex-ante and/ or ex-post assessment of IS/IT
contributions to performance, productivity ratios, return on investment
ratios, financial auditing
Thus, despite the goal and
objectives defined prior to the project starting, some additional
classic questions asked in the user requirements analysis stage include
‘who are the stakeholders in your organisation’, ‘what benefits are
expected from this project’, ‘how will you use this research project
result’, ‘what is the nature of decision making in your organisation’,
‘in your view, what is the most important characteristics of a
successful system in this project’ and ‘what other characteristics of
the project will affect its value to your organisation’?
In the next stage of the
lifecycle-based approach the focus is on designing instruments for
evaluating collaborative systems against user requirements in terms of
functionality and non-functionality (i.e. user satisfaction).
Consequently, this stage acts as iterative development through the
feedback from industry partners in order to be more feasible and
appropriate to fit into the project specifically.
As previously mentioned,
collaborative systems are highly complex which results in a common
phenomenon that the system is decomposed into several stages to be
implemented. Therefore, once part of the collaborative system is
developed and shifted to validation stage, the proposed evaluation
instruments can be put in front of users to assess the system
functionality in more detail. Consequently, user requirements will be
validated and the progress of the whole project will be shown according
to the percentage of functionalities achieved.
In the final stage, output or
feedback from the earlier stage can be fed into this stage to modify
these evaluation instruments. In this approach, the steps can be as a
loop iteratively from the step of designing to modifying instruments.
The whole approach and evaluation instruments then are completed prior
to the final validation being conducted.
The lifecycle-based approach is
available for designing formative and summative evaluation instruments
to support and improve collaborative system development and validate the
system performance. It can be employed either in organisations or
research projects. In order to prove the concept, a case study of the
lifecycle-based approach is carried out within DIECoM in the following
section.
3. Empirical evidence from a case study
3.1 DIECoM background
DIECoM is a Framework V project
which aims to address a need in both the European aerospace and
automotive industries to model, validate and provide tool support for
cross-organisational product configuration management (PCM). DIECoM has
developed a generic business process model for PCM and to address a
unified approach, through the development of a XML-based PCM meta-data
format, to represent and manage product, electrical and software data (Jiang
et al. 2003). The approach is realised by the development of a pilot
demonstrator, providing the basis for an Internet-enabled virtual
enterprise. Two scenarios are being used to validate the research, one
from each of the industry sectors.
3.2 The processes
In this case study, the
lifecycle-based approach is applied to design formative and summative
evaluation instruments. A basic, widely-recognised distinction between
these evaluations is that formative is conducted during the development
or improvement of a programme or product and summative is conducted
after completion (Scriven 1981).
In the case of the formative
evaluations, the DIECoM pilot demonstrator is being evaluated, and the
purpose of the evaluation is to help to improve the demonstrator and any
further products that build upon it. In the case of the summative
evaluation, the aim is to evaluate the impacts for the complete product
/ services value chain/ organisation value taking into consideration not
only the technical aspects but also economical, socio-organisational,
legal, etc. aspects.
Step 1: defining an evaluation plan
As discussed in the previous
section, several items need to be defined in this step, which include
evaluation of milestones, identifying stakeholders and techniques for
collecting user requirements. In DIECoM, the evaluation schedule has
been defined based on the project milestone and might be changed in
parallel while the project schedule changed. Semi-structured interviews
with open ended questions were mainly employed in order to allow
participants to give their opinions. There are three industry partners,
three IT vendors and one academic partner in the DIECoM project.
Different perspectives from project members need to be addressed
therefore each project partner in DIECoM are defined as the stakeholders
and interviewees for collecting user requirements.
At the present time, the
stakeholder concept represents a move away from a managerial and
ownership-based view of organisations towards a more inclusive
perspective (Avgerou, 1995; Ramage & Fides, 1996). In general four
different stakeholder groups are discerned in information system
science: users, managers, developers and outsiders (Turunen & Talmon
2000). Most work on IT evaluation is oriented towards evaluating
development and/or implementation projects, rather than research
projects. In the case of applied research projects with part-public
funding, the “outsider” stakeholder groups are highly significant,
including both the funding body and the external users of those results
that will be placed in the public domain. Thus one of the tasks of
summative evaluation is to demonstrate to those external stakeholders
the contribution that the project has made to scientific and
technological knowledge. In the DIECoM project, the external
stakeholders were identified, which included customers, suppliers,
developers, European Research project, European R&D community and
European public, i.e. commission and Council.
Step 2: user requirements analysis
In DIECoM, different stakeholders’
perspectives are emerging and dependent on the organizational context.
The proof of concept and lessons learned will be that most of the
partners expect to obtain benefits from DIECoM. They believe the concept
and lessons gained from this project will contribute to other projects
and encourage innovation. Only one industry partner considers using the
results for organisational reengineering in terms of organisational
change and improving effectiveness. IT vendors expect to improve their
products and service according to the research results and feedback from
other partners. Some IT vendors also consider it as a market strategy if
the pilot demonstrator is developed successfully. From a technical
perspective, all partners are addressing the development of a generic
business process model. They all agree that the most important
characteristic of a successful system in the project is a successful
pilot demonstrator which covers all use cases and scenarios. Moreover,
industry partners intend to use the result as an alternative choice for
future IT solutions to improve competitiveness and the ability for
innovation. However, there are fewer requirements related to financial
and economic perspectives as the financial result of the DIECoM project
is confidential to each organisation.
As a result, DIECoM partners have
stated that the evaluation instrument should be simple and easy to use
by end users. It needs to cover the functional character which includes
technical, business and organisation requirements. It also needs to
cover non-functional character in terms of organisation added value and
cost & effectiveness.
Step 3: designing evaluation
instruments
According to the project partners’
perspectives, evaluation instruments were designed against functional
and non-functional requirements in DIECoM. The formative evaluations
gather data by means of two scoreboards for evaluating general and
technical characteristics. The summative evaluation will be based on
formal questionnaire data with informal communication with project
partners.
The coverage of the general
scoreboard and the technical scoreboard are related to functionality
evaluation. The scoreboards contain a list of evaluation criteria which
are generated from business scenarios, requirements and use cases. Each
criterion in scoreboard has been given a weight based on the priority
identified by project partners. The scoreboards require users to give a
score in relation to the satisfaction of functionalities achieved.
The questionnaire is designed to
evaluate non-functionality items such as cost benefits. However, the
financial result of the DIECoM project is confidential to each
organisation. It is also difficult to get the accurate financial figures
in the short term. Therefore, in the DIECoM evaluation, we were using
the concept of Multi-Objective, Multi-Criteria methods, Cost Benefit
Analysis and Information Economics to evaluate the financial benefits.
For the purpose of evaluating business and organisation added value
associated with the above concepts, model and limitations of DIECoM, the
questionnaire was designed in a comparison of the pre- & post-DIECoM
status in the end users environments. The questionnaire was categorised
into different groups, which was based on four generic requirements in
the design of collaboration (and other) software systems: efficiency,
flexibility, ease-of-specification and reuse.
Moreover, Markus and Soh (1993)
stated “Typically IT investments do not provide full benefits for
several years. This is not only because systems take time to develop,
but also because even after implementation, it takes time for an
organisation to learn to use them effectively and to modify
organisational processes to accommodate the computerisation of work
flows.” In addition, McKeen et al. (1999) indicated that business
revenue reflects the volume of business activity (i.e. workload), then
business revenue expressed per capita and analysed over time should
capture the effects of both revenue growth and increased productivity
and therefore, constitute an appropriate indicator of organisational
performance. In general, cost benefit analysis emphasises final business
revenue/ outcomes and extended effects over a longer period. Cost
effectiveness evaluation emphasises intermediate outcomes over the short
term. Having regard to the length of the DIECoM project, cost
effectiveness evaluation is more economic and easier to measure in terms
of cost saving, time saving and better quality…etc. Thus, cost
effectiveness was also taken into account for evaluating business value
added and estimated financial benefits.
In reviewing the literature, there
are some models used to assess the value of IT. Based on the Synthesized
model which was developed by McKeen et al (1999) and along with the
DIECoM conditions and the concerns as previously mentioned, the model
could be modified as shown in diagram 2 below.
Step 4: implement and modify the
evaluation instruments
The actions of implementing the
evaluation instruments and using feedback from users to modify the
instruments are carried out in this step. There were 4 phase validations
and implementations in DIECoM. In each validation, two scoreboards,
general and technical, for functionality evaluation would be employed.
Based on the feedback from users, those scoreboards are required to be
broken down in further detail in order to provide sufficient information
for real end users’ evaluation. The questionnaire acts as a summative
evaluation instrument and will be only applied at the end of project to
collect the end users opinions. Therefore, it will not receive any
feedback during system implementation. However, during project meetings
and discussion via emails, the questionnaire was able to be modified
according to the suggestions from project partners. The evaluation
instruments, scoreboards and questionnaire, in DIECoM will be frozen
prior to the final phase evaluation.
3.3 The outcome
The validation of the DIECoM
project was divided into 4 phases taking into consideration the pilot
implementation plan. The feedback on the series actions mentioned above
from project partners was positive after phase 1 validation. The
comments about the evaluation instruments were easy to understand as
they have been discussed several times and generated an agreement during
meetings. Two scoreboards fully covered the criteria for functionality
evaluation. The result of validation was able to show the progress of
the project and pointed out which areas needed to be addressed or
improved. However, two scoreboards are still needed to be broken down
into further detail towards user-oriented evaluation tools. According to
the feedback given from project partners, two scoreboards were amended
for further improvement.
At the last validation phase in
DIECoM, user partners were requested to fill in the revised general
scoreboard, technical scoreboard and questionnaire for evaluating the
pilot implementation taking into account the performance of the overall
implementation of the DIECoM pilot demonstrator. The validation results
of the DIECoM project underlined that there are more than 75%
functionalities implemented in the DIECoM Pilot Demonstrator and it was
considered a good result. However, it requires longer time to affect
business value with the result from the DIECoM project in each
organisation. The questionnaire was therefore completed based on
assumptions in each organisation. Consequently, it will be valuable
using the questionnaire to revisit and track the impact of DIECoM in
each organisation after certain period, i.e. six months.
4. Discussion of the case study
The result of the case study shows
the lifecycle-based approach is useful for selecting appropriate
evaluation instruments. It also contributes to monitor the progression
of project. In addition, the result of this case study echoes the
literature where evaluation can serve multiple roles in addition to
ensuring the cost-efficient use of resources: it acts as a feedback
mechanism applying a critical spirit to actions taken and results
achieved (Hirschheim & Smithson, 1988; Farbey et al., 1993; Serafeimidis,
1997). Furthermore, IS evaluation potentially contributes to
organisational learning and understanding (Serafeimidis & Smithson,
1999). Moreover, a set of heuristics is developing from the findings and
during the processes of using this approach and is discussed as below.
4.1 Knowledge Management
As discussed above, the
stakeholders in DIECoM focus on the proof of concept and lessons learned
in order to enhance the ability for innovation. Willfort (2001)
indicated the company-internal innovation processes can be regarded as
an existing crucial success factor. Moreover, as Wheatley (1992) has
pointed out “Innovation arises from ongoing circles of exchange, where
information is not just accumulated or stored, but created. Knowledge is
generated anew from connections that weren’t there before”. These
implicate that successful companies manage knowledge as a foundation for
innovation.
In the past, visible assets such as
products and equipment were the main elements to contribute to the
returns on investment. At the present time, knowledge is also recognised
as one of key assets as long as they are effectively used to generate
benefits (Carneiro, 2000). Indeed, transferring the new knowledge into
products and services performs as an essential element to maintain a
competitive edge in business. No doubt the proof of concept, lessons
learned and evaluation results will contribute to knowledge management.
However, Hislop (2003) stated that the implementation of large scale,
enterprise wide innovations will involve not only the integration of
internal with external knowledge, but also the integration and
reconfiguration of different bodies of specialised internal knowledge.
It indicates a need to address how organisations manage knowledge and
how the innovation process takes place.
4.2 Potential risk
Potential risk to the project
during the process has required attention. Arguments from different
authors about problems emerging within evaluation have driven a lot of
discussions. The problem with evaluation can be summarised as: poor
evaluation and lack of understanding the full range of costs including
relevant and opportunity costs, not fully investigating risks,
neglecting intangible benefits, the narrow nature of much evaluation,
failure to devote evaluation time and effort, not covering all
stakeholders’ perspectives, boundary of knowledge, difficulty of
information requirements and high evaluation costs (Hirschheim &
Smithson, 1988; Willcocks, 1992; Farbey et al., 1999; Knutilla et al.
2000; Savory, 2001).
However, a virtual enterprise acts
similarly to an organisation therefore the organisational problems are
specifically addressed. Ballantine et al. (1996) introduced three
organisation problems which were encountered during the evaluation
process: lack of time, lack of data/ information and lack of interest.
However, virtual enterprises involving collaborative systems in
multi-site and multi-partner projects have more knowledge and
information related to limitations than normal organisations, especially
in financial aspects and in terms of confidentiality. Consequently, few
potential risks in DIECoM have been identified through the discussions
with project partners and observation during the process.
Moreover, Tyndale (2003) introduced
top eight critical success factors for IT projects, which affect the
outcome of IT project including: top management support/ champion, user
satisfaction, communication, clear goals and objectives, project
management, time/ cost/ quality, organisation and culture. In addition,
user participation, education, trust, organisational culture and a
receptiveness to change are also pointed out. These critical success
factors however can act as a checklist to identify the potential
surrounding risk of IT projects. As a result, based on the critical
success factors listed above, potential risks in DIECoM are grouped and
described as follows.
4.2.1 Communication
Tyndale (2003) stated that
communication acts a mechanism for exchanging information concerning,
project goals, changes in policies, change s of procedures, status
reports and as a feedback mechanism for monitoring. In this aspect, the
negotiation issues between different perspectives are emerging in DIECoM.
The DIECoM pilot demonstrator will
be implemented based on scenarios which were identified at the beginning
of the project and also have been given different priorities. This is
due to different opinions regarding pilot implementation. For instance,
IT vendors only addressed highly prioritised scenarios because of the
time limitations. In contrast, industry partners wanted to implement all
scenarios. This issue also reflects time allocation and management and
it echoes the need for considering time constraints more carefully while
submitting the proposal. Moreover, there are strong characters within
some participants in this project who may have stronger influences to
direct the project to their interests. However, a conclusion given at
WETICE 2003 ECE workshop (Newman et al. 2003), highlighted that
different perspectives from group members need to be addressed and
negotiated.
4.2.2 Project management
There is a major issue associated
with project management and identified in the DIECoM project: unclear
meeting purpose.
In a virtual team context, the
face-to-face communication will positively moderate the relationship
between internal collaboration and team performance, innovativeness, and
member satisfaction. Moreover, it provides an opportunity which allows
full discussion and can eliminate the arguments after a critical
decision has been made.
The DIECoM project holds management
(steering) meetings and technical meetings regularly. However,
discussion at technical meetings sometimes covers both purposes instead
of having separate management meetings. The people who attend technical
meetings are usually in more junior positions and not authorised to make
decisions. The risk of having technical meetings discuss management
issues can result in bringing out disagreement about project management
due to different industries having different concerns and priorities.
4.2.3 Time/ Cost/ Quality
In DIECoM, an essential comment
from different partners is “this is a huge project and we need more
time” as there are many opinions and different requirements from
industry partners. Moreover, due to a replacement of one of the project
partners, it was also the reason for the project being delayed.
However, there are many reasons
which could result in a lack of time in any project. Cost and time can
be understated in terms of size of IT budget which is used to increase
the opportunity of proposal acceptance. Moreover, not fully
investigating the resource which is required in a project will also
result in failure to estimate/ predict time scales. In addition, a
virtual enterprise usually involves multi-partners from different
organisations. It is also time consuming to reach a compromise with each
other and build a level of trust to work on the project.
4.2.4 Organisation
A virtual enterprise acts similarly
to an organisation. The purpose of structuring organisations is to
divide the work among the members of groupware and to coordinate the
member’s activities so that they can be directed towards the goals (Tyndale,
2003). In terms of organisation, there are two issues which have been
identified.
1. Problems of compromising
between different organisations and different industries
In DIECoM, it is a challenge to
build a generic business model which integrates two different
industries, aerospace and automotive, although both are related to
transportation. It is also a tough test to compromise two organisations’
needs even though they are in the same industry. Different industries
and different organisations have different priorities. For instance,
partners in aerospace industry emphasise more on change management,
however another partner in automotive industry focuses more on after
sales. This makes project progress difficult. Moreover, one of the
partners in aerospace industry felt that the pilot demonstrator was
focused more on the other aerospace partner.
2. Potential issue of
resource allocated in multi-sites
In the DIECoM project, one of the
partners makes their contribution through multi-sites support. It may
bring a potential issue on how to allocate their cost, workload and
collaborate with each other smoothly without causing any impact or delay
to the project.
4.3 Summary
Having regarding to the critical
success factors, the potential issues in terms of communication, project
management, time/ cost/ quality and organisation have been identified in
the DIECoM project. However, in a virtual enterprise which involves
multi-organisations and different stakeholders’ perspectives, it
requires more efforts to overcome the organisational culture and trust
between project partners. In addition, there are also some issues needed
to be addressed in each project partner, such as top management support,
user satisfaction and participation, which may have indirect impact to
the virtual organisation.
5. Conclusion and further work
In this paper, a lifecycle-based
within stakeholder-oriented approach is presented to support development
of a virtual enterprise. With the purpose of providing a conceptual
understanding of evaluation of collaborative systems and in order to
prove the concept, a case study is carried out. This lifecycle-based
approach derives a sequence: defining an evaluation plan, user
requirements analysis and specification, designing instruments,
implementing the evaluation instruments and using feedback from users to
modify the instruments. The result of the DIECoM case study proves the
value of the approach and in some respects confirms the empirical
evidence or theories.
A virtual enterprise acts similarly
to an organisation therefore the organisational problems associated with
the critical success factors are specifically addressed to identify
potential risk. This study therefore highlighted the ongoing nature of a
number of problems within the development of collaboration environment/
virtual enterprise. Compared with the problems within normal
organisations, the problems within virtual enterprise are significantly
different and even more complex. These problems are more related to
cross organisational issues and stakeholders’ perspectives. Moreover,
the potential issues within a normal/ individual organisation will also
affect the virtual organisation.
Summative evaluation results will
aim to show the project performance, but not be used to support DIECoM
development. However, the results may be useful by offering
opportunities to enhance knowledge, clarify thinking and develop future
frameworks.
The formulation of this paper has
made additional contributions by providing a clear guide for further
discussion on how to fulfil stakeholders’ perspectives, how to manage
knowledge and how to avoid potential problems in the development of
collaborative systems to support the virtual enterprise.
Diagram 2: Cost Effectiveness
analysis
6. Acknowledgements:
This work was supported in part by
the European Commission under Framework 5 project IST-2000-28270 DIECoM.
The author would like to express thanks to all those, Alenia, EADS,
Renault, SIA, IBM and Dassault Systems, who participated in this study.
The author is grateful to Caroline Parker and Julian Newman for comments
on a previous draft.
References
-
Andriessen, J H
E. (2003) Working with Groupware: understanding and evaluating
collaboration technology. London: Springer.
-
Avgerou, C.
(1995) “Evaluating Information Systems by Consultation and
Negotiation”, International Journal of Information Management, Vol.
15, No. 6, pp427-436
-
Ballantine J.
A.; Galliers R.D. & Stray S.J. (1996), “Information systems/
technology evaluation practices: evidence from UK organisations”,
Journal of Information Technology, Vol. 11, pp129-141
-
Camarinha-Matos,
L & Afsarmanesh, H (2002), “Dynamic Virtual Organizations, or Not So
Dynamic?” Proc 5th International Conference on Information Technology
for Balanced Automation Systems in Manufacturing and Services,
[online] http://culiacan.udo.mx/~gortega/Basys2002-12.pdf
-
Carneiro, A.
(2000), “How does knowledge management influence innovation and
competitiveness?”, Journal of Knowledge Management, Vol. 4, pp. 87-98.
-
Checkland, P.
(1981), Systems Thinking, Systems Practice, Chichester: Wiley.
-
Davidow W.H.&
Malone M.S. (1992), The Virtual Corporation, New York: Harper-Collins.
-
Farbey B, Land
F & Targett D. (1993), How to assess your IT investment: a study of
methods and practice, Oxford: Butterworth Heinemann.
-
Farbey B, Land
F & Targett D. (1994) “Matching an IT project with an appropriate
method of evaluation: a research note on ‘evaluation investments in
IT”, Journal of information technology, Vol. 9, pp239-243.
-
Farbey B., Land
F. &Targett D. (1999) “Moving IS evaluation forward: learning themes
and research issues”, Journal of Strategic Information Systems, Vol.
8, pp189-207
-
Hirschheim R. &
Smithson S. (1988) “A critical analysis of information systems
evaluation” in Information systems assessment: issues and challenges,
B.V.: Elsevier Science, pp17-37.
-
Hislop D.(2003)
“Knowledge integration processes and the appropriation of
innovations”, European Journal of Innovation Management, Vol. 6,
pp159-172.
-
Jiang P., Mair
Q. & Newman J. (2003) "Using UML to Design Distributed Collaborative
Workflows: from UML to XPDL", IEEE 12th International Workshop on
Enabling Technologies: Infrastructure for Collaborative Enterprises,
Los Alamitos, CA: IEEE Computer Society. (in press)
-
Kefi H. (2002)
“IS/IT evaluation: a Context-based and Process-oriented perspective”,
9th European conference on information technology evaluation, Reading,
UK: MCIL, pp249-257.
-
Knutilla A.J.,
Steves M.P. & Allen R.H. (2000) “Workshop on Evaluating Collaborative
Enterprise-Workshop Report”, IEEE 9th International Workshops on
Enabling Technologies: Infrastructure for Collaborative Enterprises,
Los Alamitos, CA: IEEE Computer Society, pp79-85
-
Markus, M. L. &
Soh, C. (1993) “Banking on Information Technology: converting IT
spending into firm performance” in Strategic information technology
management: perspectives on organisational growth and competitive
advantage. (Eds R. Banker, et al.) Hershey, PA: Idea Group Publishing,
pp375-403.
-
McKeen J. D.,
Smith H. A. & Parent M. (1999) “An integrative research approach to
assess the business value of information technology” in Measuring
Information Technology Investment Payoff: Contemporary Approaches (Ed.
M. A. Mahmood and E.J. Szewczak), Hershey, PA: Idea Group publishing,
pp5-23.
-
Newman, J.
(2001) “Validation, Evaluation and Virtual Enterprise Support”,
Proceedings of the 10th IEEE International Workshops on Enabling
Technologies: Infrastructure for Collaborative Enterprises, Los
Alamitos, CA: IEEE Computer Society, pp91-95.
-
Newman J.;
Raybourn E.M & Huang P. H. (2003) “WETICE 2003 Evaluating
Collaborative Enterprises Workshop Report”, Proceedings of the 12th
IEEE International Workshops on Enabling Technologies: Infrastructure
for Collaborative Enterprises, Los Alamitos, CA: IEEE Computer
Society. (In press)
-
Pinelle D. &
Gutwin C. (2000) “A Review of Groupware Evaluations”, IEEE 9th
International Workshops on Enabling Technologies: Infrastructure for
Collaborative Enterprises, Los Alamitos, CA: IEEE Computer Society,
pp86-91.
-
Ramage M.&
Fides M. (1996) “Cui Bono? A Stakeholder Approach to CSCW Evaluation”,
[online] http://www.comp.lancs.ac.uk/computing/research/cseg/projects/evaluation/cuibono.html
-
Ramage, M.
(1997) “Developing a methodology for the evaluation of cooperative
systems”, Proceedings of Information Systems Research in
Scandinavia,[online] http://www.comp.lancs.ac.uk/computing/research/cseg/projects/evaluation/developing.html
-
Raybourn E.M &
Newman J. (2002) “WETICE 2002 Evaluating Collaborative Enterprises
Workshop Report”, Proceedings of the 11th IEEE International Workshops
on Enabling Technologies: Infrastructure for Collaborative
Enterprises, Los Alamitos, CA: IEEE Computer Society, pp7-11.
-
Savory C.
(2001) “Co-operative inquiry as a basis for evaluation of knowledge
management tools”, 8th European Conference on Information Technology
Evaluation, Reading, UK: MCIL, pp99-107.
-
Scriven M.
(1981) Evaluation Thesaurus (3rd edition), Inverness, CA: Edgepress.
-
Serafeimidis,
V. (1997) Information systems investment evaluation: conceptual and
operational explorations, unpublished PhD thesis, London School of
Economics and Political Science.
-
Serafeimidis V
& Smithson S. (1999) “Rethinking the approached to information systems
investment evaluation”, Logistics Information Management, Vol. 12,
Nov. ½, pp94-107.
-
Turunen P.&
Talmon J. (2000) “Stakeholder Groups in the Evaluation of Medical
Information System”, Seventh European Conference on Information
Technology Evaluation, Reading, UK: MCIL, pp329-334.
-
Tyndale P.
(2003) “The evaluation of critical success factors for IT projects”,
Tenth European Conference on Information Technology Evaluation,
Reading, UK: MCIL, pp641-651.
-
Wheatley M. J.
(1992) Leadership and the New Science: Learning about Organisations
from an Orderly Universe, San Francisco, CA: Berrett-Koehler.
-
Willcocks L.
(1992) “Evaluating information technology investments: research
findings and reappraisal”, Journal of information systems, Vol. 2,
pp243-268
-
Willfort R. (2001) “Knowledge management as basis for
innovation” Second European Conference on Knowledge Management,
Reading, UK: MCIL, pp739-745.
|
