1.        Introduction

The recognition of individual contributions to group performance in student group projects is essential. Assigning the same grade to all group members implies equal effort which is often not the case. When assessing group projects equal contribution by all group members cannot be assumed, and the focus cannot be on the product alone (Cooke et al, 1997). It is therefore imperative that the assessment of group projects includes an assessment instrument that will allow the measurement of individual contributions to the group project by peers (the group members) based on specific predefined criteria. The use of peer assessments allows group members to voice their perceptions of their contributions and those of the other members in the group (Feigenbaum and Holland, 1997). As such it can be implemented to discourage social loafing, a term used by Smith (2004) to identify students who under perform in project teams.

The systems development group projects in both the 3^rd and 4^th year Information Systems (IS) courses at the University of Cape Town are capstone courses that bring together hard and soft skills and closely emulate professional practice. A multiple assessment strategy is used, comprising of formal summative assessment, formal continuous assessment and an informal formative assessment. Various methods and instruments to accomplish these assessments are used, e.g. tests and exams, checklists, questionnaires, mark sheets and scoring rubrics. In both courses, strong emphasis is placed on giving the students a real-life experience that encompasses the full systems development life cycle. Both Cooke et al (1997) and Miller (2003) point out that peer and self-assessment are skills required in industry and have become necessary for professional practice. Since students enjoy carrying out peer assessments and find it beneficial to their learning (Sluijsmans et al, 2002); the classroom provides excellent opportunities to acquire and hone these skills.

This paper will discuss the implementation of peer assessment as an assessment instrument in the 3^rd and 4^th year IS courses. The use of groups and peers within the context of this assessment strategy will be explained and the aims and benefits of peer assessment will be highlighted. The paper will detail the peer assessment process and the method used in the two courses. The way peer assessment is used as an informal formative assessments tool as well as how it is applied to make a formal summative assessment, will be described and motivated. The development and implementation of a numerical algorithm to quantify the results of the peer assessments, and the use of the quantified results to derive individual ratings will be discussed and illustrated in detail. The paper will illustrate how these ratings were used to adapt the individual marks to more adequately reflect the different individual contributions within each group. It will report on the practical implication thereof over the past two years and how it has affected individual performances within the groups.

2.        Background

Group participation and interaction is part of the real world and in most cases group participation is evaluated by peers (Cooke et al 1997). Thus the implementation of both formative and summative peer and self-assessment to assess student performance in group projects is becoming more common in higher education (Miller, 2003; Sluijsmans et al., 2002). Assigning the same grade to all members of a group gives reason for concern about equity as there is the underlying assumption that all contributed equally (Feigenbaum and Holland, 1997). Moreover, in environments where educational inequity exists and where there is significant cultural diversity, this concern is aggravated and the process of learning should be guided carefully.

Correct assessment practices can motivate students to achieve the desired learning outcomes and can be used as a valuable and effective teaching tool that ought to have its place in the classroom (Schmidtke, 2001). Evidence exists that peer and self-assessment can trigger a greater responsibility among students for independent learning (Dochy et al., 1999). Other benefits of this active involvement of students in the learning process are the obtaining of personal and academic meaning in their studies (Denicolo et al in Orsmond et al, 2002). Cooper (2000) confirms this viewpoint by stating that self and peer-assessment when used formatively incorporate feedback which facilitates student learning. For assessment to be formative, the assessment and feedback should initially be separated from grading. This allows students to develop their own judgments before being presented with the grades from other assessors (Taras, 2002).

Peer evaluation can be done by designing an instrument to objectively grade individual performance in groups. Peer assessment when used summatively provides the opportunity to grade individual contributions in a final product. Taras (2002) advocates greater emphasis on student participation through peer and self-assessment, particularly in summative assessment that “counts”.

Several methods to quantify the peer review process are discussed in the literature (Conway and Kember, 1993; Goldfinch, 1994; Li, 2001; Miller, 2003). Despite the fact that choosing between alternative methods may be a subjective process, the authors support Conway’s (1994) approach of implementing a simple, yet sufficiently fair method to assign a mark to a group member that will reflect individual effort. Goldfinch (1994) implemented a two-part peer assessment form where the first part determined which tasks were performed by each member during the course of the project. Scores allocated for the second part, where members were assessed on their group-working skills, contributed to individuals’ final mark. Conway (1993) implemented a one part form which seeks to determine the extent of each student’s participation in the tasks that make up the project.

2.1         Aims and objectives of the peer review process in general

According to the Peer Review Handbook by Christine Bruce (1997) the general aim of the peer review process is to contribute to the professional development of participants. More specifically in a systems development group project it can assist to provide equity to grade distribution (Feigenbaum and Holland, 1997).

Futhermore, according to Netpro project and project based learning (retrieved 2004), the main aims of using the peer assessment process are to:

§         Develop students’ judgment and understanding of quality.

§         Enable students to assess their own work, resulting in improving the quality of their products.

§         Enhance students’ responsibility and accountability.

§         Foster constructive communication.

2.2         Benefits

The peer review process allows students to enter into the Action Learning Cycle, a cycle that promotes continuous planning, reflection, observation and action amongst participants (Bruce, 1997). This could encourage group members to adapt to change more easily and share their learning with others. It might also act as an incentive for participation as team members are accountable to each other for individual performance (Cook et al., 1997). Amongst the benefits are also improved designs, real world experience and the development of crucial skills (Feigenbaum and Holland, 1997). Specifically, these skills in a systems development environment include both soft and hard skills.

2.3         Definitions

Within the context of the IS student group projects, groups will be used to refer to self-chosen teams of four to five members. Each group has its own organisational structure and is responsible for its own administrative tasks and project deliverables. There are typically between twenty and thirty-five groups per course.

Peers will refer to the members of a group, and the peer assessment will be limited to the particular group. During the 3^rd year, the final peer assessment will also include self-assessment.

Assessment will be used to include both measurement and evaluation as components (Scott and Van der Merwe, 2003). Du Toit et al (2001) describe measurement as a qualitative and/or quantitative grading process, and evaluation as a value judgement concerning quality – in essence an interpretation of the results through measurement.

Peer assessment is conducted by the members of a group with reference to the work done within the group. Each group member will rate the contribution to the group project of every other member in the group.

Self-assessment is done as a personal judgement by an individual group member of the value of their specific contribution to the group project relative to the contributions of other members of the group.

This paper will now proceed to discuss peer assessment as a component of the assessment strategy used for Information Systems student group projects. The value of peer assessment, the contribution to the real-life experience offered by group projects and how this process enhances deep learning, will be illustrated.

3.        The group systems development project

The complex organisational and dynamic software development environment establishes the need to equip students with a diverse set of competencies to face the challenges of the work place more effectively. The systems development group projects in both the 3^rd and 4^th year courses integrate hard and soft skills to prepare students for professional practice. This reflects the viewpoint of IS2002: An Update of the Information Systems Model Curriculum (IS2002) that the characteristics of an IS professional evolves around three major areas namely:

§         A broad business and real world perspective

§         Strong analytical and critical thinking skills

§         Strong interpersonal communication and team skills

The main deliverable for the 3^rd year course is a comprehensive web-based management system with a concise and clear business focus. Students are provided with a generic business problem, detailed specifications and functional guidelines. To expose students to the complexities of interacting with users in real organizations, they are required to use these specifications to find a “best fit”, i.e. an appropriate business problem and sponsor in industry. The sponsor supports the students with expert knowledge and experience, and performs assessment from an industry perspective. The 4^th year IS project is a carefully scoped real world project and builds on the experience gained in the similar but smaller 3^rd year project. Unlike the 3^rd year, these students have to go out in industry, and identify a need to translate a business problem, efficiently and creatively into an automated computerized system.

Both these projects equip students with crucial problem-solving abilities using object-oriented techniques and business process re-engineering; as well as with the insight and understanding required to capture business processes programmatically. The competitiveness of project teams encourages creative solutions and necessitates that students acquire advanced technical skills as well as implement the most recent technologies.

Collaboration and communication form integral parts of both these projects. 3^rd Year project groups report to a member of faculty acting as project manager, and they also have regular meetings with the business sponsor. The development process is guided by several interim deliverables and milestones, culminating in a final shrink-wrapped product and project presentation. The 4^th year groups manage their own projects and have fewer deliverables, with the project also culminating in a shrink-wrapped product and a project presentation. The final event in the calendar of both 3^rd and 4^th years is an exhibition to showcase their expertise and professionalism to industry, learners from various schools and the public.

The assessment strategy and peer assessment component as implemented in the IS group projects, are discussed in the following sub sections.

3.1         Assessment strategy

Scott and Van der Merwe (2003) advocated the use of an assessment strategy that involves multiple assessment approaches to enhance student learning and aid the objective assessment of group performance. It was argued that an assessment strategy must be adopted that will give credit to the complexities and challenges of IS group projects. This strategy is rooted in those proposed by Shepard (2000) and Pellegrino et al (2001) to reflect comprehensiveness, coherence and continuity.

The IS group projects in both the 3^rd and 4^th years are very practical in nature and have a number of deliverables requiring a wide range of abilities and skills that must be assessed. In the development of the assessment strategy, these skills and abilities were identified and assessments tools were chosen to effectively evaluate and measure them

Table 1 on the next page, adapted from Scott & Van der Merwe (2003), summarises the strategy used for the 3^rd year IS projects. The 4^th year course uses a similar strategy. It reflects the comprehensive assessment strategy, using multiple approaches to ensure coherence, and providing continuity through regular review-points, feedback and opportunities for improvement. This strategy supports the active learning process and aims to coach students into a deep learning approach thus maximising their learning experience.

One of the challenges mentioned in Scott and Van der Merwe (2003) is to give recognition to the contribution of individual members to the group project. The inclusion of peer assessment as part of the assessment strategy becomes important when meeting this challenge.Table 1: Assessment strategy

3.2         Peer assessment component

3.2.1        Use of peer assessment in IS student group projects

As noted by Miller (2003), it is becoming more common to use peer assessment for the assessment of individual student performance in group work, as it can be used for formative as well as summative purposes. Boston (2002) describes formative assessment as the diagnostic use of assessment to provide feedback, while summative assessment is described as taking place after a period of instruction and requires making a judgment about the learning that has occurred by grading or scoring a deliverable, test or exam.

Taras (2002) believes too much emphasis is placed on grading and too little on what is being learned and on student participation. For assessment to be formative, Taras (2002) asserts that assessment and feedback should initially be separated from the grading process. To create a balance between assessment, feedback and grading, she advocates student participation through peer and self-assessment in formal summative assessment.

The authors agree with this approach, and therefore peer assessment is performed at different stages in the group IS projects. During the development stages it is used as informal formative assessment, to facilitate student learning and develop skills that are needed in industry. It also prepares them to make accurate and fair assessments of their group members in the final formal summative assessment. These views are shared by Orsmond et al (2002) and Sluijsmans et al (2002). In the 3^rd year course, self-assessment is used in tandem with peer assessment for the final assessment of the group projects.

3.2.2         Instruments

The main instrument used in the assessment of the group projects, is scoring rubrics. In the development of the assessment strategy, scoring rubrics were found to be the most effective way to align the different parties involved and to stimulate discussion and debate, as well as to limit bias and convey to the students the standards against which they will be measured (Scott and Van der Merwe 2003).

The peer assessment rubric for the 3^rd year group (see Table 2) was designed to reflect both the student’s contribution to the efficient functioning of the group and the extent of each student’s participation in the various tasks of the project. This approach combines the two different focuses of Goldfinch (1994) and Conway (1993). It consists of 10 criteria that must be rated, using the assignment of a numerical score for each item. Each member must rate themselves as well as the contribution of every other member in the group. The layout of the columns gives the student a clear picture of their ratings of each group member compared to themselves and the rest of the group. This was done to assist the student in making a fair and accurate comparative assessment of each group member. How the ratings are used to assess individual contributions to the group projects, will be explained in paragraph 3.2.3 below.

Table 2: Final peer assessment including self-assessment used in 3^rd year course

The 4^th year course uses a similar instrument, although the questions were posed at a different level (see Table 3). Group members were to rate their peers considering five areas, namely Ability to Work in a Group, Amount of Effort, Dependability, Intellectual Contribution and Overall Contribution to Project, on a scale of 1 to 5.

Table 3: Final peer assessment excluding self-assessment used in 4^th year course

According to Rust et al (2003), there is a need for transparency in the assessment process. For this reason, all assessment instruments used in the student group projects, for example the rubrics or mark sheets, are made available well in advance. This helps to create awareness of the assessment criteria and associated standards. Goldfinch (1994) recognises the fact that it is educationally unsound to withhold the assessment instrument from the students until they must use it, and recommends making it available in order for students to know exactly how they will be assessed.

3.2.3        Method

Group performance is based on group cohesion, group efficacy and team building (Bahli & Buyukkurt, 2003; Smith, 2004). During the development stages, groups are lead by a facilitator in the peer assessment of the group performance, which includes reviewing group dynamics, cohesion, effectiveness and communication. This is done openly in a reflective manner, with ample opportunity for feedback.  

The final peer assessment is done secretly and includes a self-assessment component. Lejk & Wyvill (2002) concluded that secret peer assessment can be done more honestly and is therefore more accurate. There are strong arguments for and against the inclusion of self- assessment in the peer assessment of group work. While self-assessment is excluded from the final peer assessment in the 4^th year course, it was decided to include it in the 3^rd year course. For 3^rd year students, this is their first encounter with peer assessment. Given their level of experience, it was felt that their inexperience may lead to them being less critical towards their peers and over-generous in their assessment. Goldfinch (1994) has found that students who are over-generous effectively penalise themselves if the rest of the group are more conservative. She therefore argues for the inclusion of self-assessment.

It is reasonable to accept that although students bring different strengths to a team, they are encouraged to participate in most areas, for example, all members of a group are expected to contribute towards the building phase (coding) of the project. A main focus in the design of the presentation and code evaluation instruments was to objectively reflect the quality of the product rather than just the effort. The authors are of the opinion that there is not necessarily a linear relationship between the quality of a product and the effort put into the delivering of it. For this reason it was decided to use the scores in the peer assessment process as penalties to adjust imbalances within certain groups rather than just increasing or decreasing the marks of individuals.

The process involved the following steps: forms were filled out individually, sealed to ensure confidentiality and handed in with the final shrink-wrapped product. For each group the ratings obtained were entered into a spreadsheet and averages were calculated. An average for each member was obtained, based on the self- and peer ratings. The 4^th year calculations excluded the self-assessment ratings. From these averages an aggregate average across all the member’s ratings was calculated and was used to obtain differences by subtracting this aggregate average from the individual averages. From these differences a mark deduction table was developed in both courses to associate corresponding penalties to specific ranges for only those averages lower than the aggregate average.

A small committee of four academics was constituted to handle the review process for the 4^th year students, whereas the course coordinator and the course administrator executed the process for the 3^rd year students.

3.3         Results

Since 2002 peer assessment has been used effectively as an integral part of the 3^rd year course in an attempt to enhance the real world experience of the course and reap the benefits of the process as discussed in section 2.3. It was incorporated for the first time into the 4^th year course in 2003.

In the case of the 4^th year course, a 5-point Likert scale was implemented. Ratings of 1 and 2 were seen as below average, whereas 4 and 5 were above the average. The differences for each team were analysed and penalties were only applied where a team member had rankings of 2’s and 1’s. Whilst 16 students with differences greater than 0.4 were identified the committee decided that 13 students would not be penalised as their marks were deemed acceptable as they were, on average, greater than 3. Of the 3 students who were penalised, one student received a 20 mark penalty, one 15 marks and one 5 marks from the final project mark. In 2002 and 2003 the peer assessment instrument for the 3^rd year students excluded the last two categories as shown in table 2. Table 4 below summarises the peer assessment outcomes for 2002 and 2003. In 2002 10 groups were affected by the process. A severe problem, causing one group to split in two, occurred before the final evaluation process and was resolved by evaluating the project separately for the two sub-groups, and hence is not included in table 3. Three members in one group were each penalised by 5% as they contributed much less towards the project than the other two members. In another group two members were penalised by deducting 5% and 7.5% from their project score respectively. The case where 15% percent was deducted was again a severe case where the problem was already identified in the early stages of the project. In almost all the cases members admitted to not having contributed equally and the process was finalised and resolved with almost no conflict.

Table 4: Summary of peer assessment outcomes in 3^rd year course

During 2003 the technical skills transfer occurred via a series of workshops where students developed a pilot system prior to the build phase of their own projects. In 2002 this transfer occurred during a series of seminars where technical topics were addressed. Due to the additional focus on the enhancement of technical skills in 2003, all group members were expected to contribute substantially towards the building phase. Although problems were also experienced in 10 groups, they seemed less severe than in 2002, this could be attributed to the increased focus on the enhancement of technical skills and the continuous feedback from regular peer assessment sessions. The differences of 3 of the 8 students, penalised by 5%, fell just inside the specific range for this penalty. In those groups some members indicated that they would prefer their team members not to be penalised for not contributing equally.

4.        Conclusion

The authors found that the use of peer assessment during different stages of the course assisted to improve group cohesion. It also helped to identify areas within specific groups that needed attention. These might include aspects like an imbalance of workload and skills shortages. Mediation of project managers during early stages of the project aided to resolve some conflict and prepared students to effectively implement the peer assessment instruments provided. It also assisted to create an awareness of the diverse skills within each group and how these skills could facilitate learning and contribute positively to the final product. In addition this awareness prepared students for teamwork in industry.

Peer assessment as a component of the assessment strategy used for IS student group projects offers students the opportunity to become active participants in the assessment process. It also provides them with skills that are needed and can be applied in the real-world environment for which they are being prepared. As an assessment instrument it also benefits the lecturer or teacher, being an aid for a fairer and more accurate assessment of individual contributions to group projects. 

As a future enhancement, informal peer assessments can be improved to include a more structured feedback process. This will benefit the subsequent occurrences of peer assessment in this course as well as other walks of life.

References

• Bahli, B., and Buyukkurt M., (2002). “Determinants of Group Performance in Information Systems Project Teams: An Empirical Study”, in Proceedings of the 17th Annual conference of the Information Academy for Information Management, Barcelona, Spain, pp 227-235.

• Boston, C., (2002). “The concept of formative assessment”, Practical Assessment, Research & Evaluation, Volume 8, No 9.

• Bruce, C., (1997). Peer Review: A Handbook, Queensland University of Technology.

• Conway, R., Kember, D., (1993). “Peer assessment of an individual’s contribution to a group project”, Assessment & Evaluation in Higher Education, April 18, Volume 18, Issue 1, pp 45 – 56.

• Cooke, J. C., Drennan, J. D. and Drennan, P. (1997), “Peer Evaluation as a Real-Life Learning Tool”, Technology Teacher, Vol 57, No. 2, pp 23-27.

• Cooper, N. J., (2000). “Facilitating learning from formative feedback in level 3 assessment”, Assessment & Evaluation in Higher Education, September 2000, Volume 25, Issue 3, pp 279-292

• Dochy, F., Segers, M., (1999). “The use of Self-, Peer and Co-Assessment in Higher Education: A Review”, Studies in Higher Education, October 1999, Volume 24, Issue 3, pp 331-350.

• Du Toit, G., Khabanjane, E., Korff, W., Kotze, G., Masehela, K., Mostert, S. and van Tonder, F. (2001). “Assessment in OBE”, A project sponsored by the Ministry of the Flemish Government, Department of Education, Final Report.

• Feigenbaum, L., Holland, N. (1997). “Using peer evaluations to assign grades on group projects”, ASC Proceedings of the 33rd Annual Conference, University of Washington - Seattle, Washington, April 2 - 5 1997, pp 75 – 80.

• Goldfinch, J., (1994). “Further developments in peer assessment of group projects”, Assessment & Evaluation in Higher Education, April 1994, Volume 19, Issue 1, pp 29-35

• Lejk, M., Wyvill, M., (2002). “Peer Assessment of Contributions to a Group project: student attitude to hiolistic and category approaches”, Assessment & Evaluation in Higher Education, Volume 27, No 6, pp 570-577.

• Li, Lawrence K.I., (2001). “Some Refinements on Peer Assessment of Group Projects”, Assessment & Evaluation in Higher Education, Volume 26, No 1, pp 1-18.

• Miller, P. J., (2003). “The Effect of Scoring Criteria Specificity on Peer and Self-assessment”, Assessment & Evaluation in Higher Education, Volume 28, No 4, pp 383-394.

• “NetPro project and project based learning”, [online], Peer review guidelines in Network Based Project Learning in Engineering Education, http://netpro.evtek.fi/team/wp5/peerReview/prGuidelines.htm

• Orsmond P., Merry, S. and Reiling, K. (2002). “The Use of Exemplars and Formitive Feedback when using Student Derived Marking Criteria in Peer and Self-Assessment”, Assessment & Evaluation in Higher Education, Volume 27, No 4, pp 310-323.

• Pellegrino, J.W., Chudowsky, N., Glaser, R. (Eds), (2001). Knowing what students know: The Science and Design of Educational Assessment, The National Academic Press, Washington, D.C.

• Rust, C., Price, M. and O’Donovan, B., (2003). “Improving Students’ Learning by Developing their Understanding of Assessment Criteria and Process”, Assessment & Evaluation in Higher Education, Volume 28, No 2, pp 148-164.

• Schmidtke, C., (2001), “Peer Review: Fair or Foul?”, in Nade selected Conference Papers, Volume 7, pp58-62.

• Scott, E. C., Van der Merwe, N., (2003). “Using Multiple Approaches to Assess Student Group Projects”, The Electronic Journal of Information Systems Evaluation (EJIS)”, December 2003, Volume 6, Issue 2, pp 177-186.

• Shepard, L.A. (2000). “The Role of Assessment in a Learning Culture”, in Educational Researcher, Vol 29, No.7, pp4-14.

• Sluijsmans, D. M. A., Brand-Gruwel, S. and van Merriënboer, J. J. G., (2002), “Peer Assessment Training in Teacher Education: effects on performance and perceptions”, Assessment & Evaluation in Higher Education, Volume 27, No 5, pp 443-454.

• Smith, D.C. (2004). “Peer Evaluations in Information Systems Student Project Team: Pitfalls and Practice”, SACLA Conference, June 2004.

• Taras, M., (2002), “Using Assessment for Learning and Learning from Assessment”, Assessment & Evaluation in Higher Education, Volume 27, No 6, pp 501-510.

EJISE is published by Academic Conferences Limited
Curtis Farm, Kidmore End, Nr Reading RG4 9AY, England
Tel: +44 (0)1189 724148, Fax: +44 (0)1189 724691, Email: info@ejise.com

Send mail to info@academic-conferences.com with questions or comments about this web site.
Copyright © 2002-2006 Electronic Journal of Information Systems Evaluation
Last modified: September 29, 2005
ISSN 1566-6379