1.   Introduction

Software Engineering Australia (SEA) is a not-for-profit association, funded under grants and in-kind contributions from government, universities and the private sector.  Its aim is to harness Australian and international expertise and resources to deliver services that foster software engineering excellence in Australia.  In 1997, the Minister for Industry, Science and Tourism commissioned a report by the Information Industries Taskforce.  One of the report recommendations was to support the collection and dissemination of improved industry statistics and undertake regular benchmarking.  Consequently, as part of the National Industry Improvement Project, SEA Qld and the Software Quality Institute (SQI) undertook a survey in 1999 to evaluate the level of adoption of best practice by software developers in Queensland. This paper explains how the survey evaluated the software processes in use and then describes a Process Improvement Program undertaken to measure the capability of software developers.

The first part of this paper explains the background and aims of the study, the execution of the survey, and presents some interesting findings related to levels of adoption of best practice.  The questionnaire was adapted from the European Software Institute's (ESI) Best Practice Questionnaire and the preliminary findings from Queensland are compared with the ESI's results.

Further to the survey, many of the survey respondents were invited to participate in the Process Improvement Program, sponsored by SEA and conducted by SQI.  A total of 25 organisations accepted the invitation.  Based on the Software Process Improvement and Capability dEtermination (SPICE) standard ISO15504, an evaluation tool was developed and applied to enable rapid assessments of software organisations to be performed. The second part of the paper describes this evaluation tool and the method developed to assess the capability of these organisations.

2. Background

The Australian software industry makes a significant contribution to the Australian economy. In the 1995-96 financial year, income from software consultancy and maintenance services, and packaged software development totalled over $3 billion Australia-wide, with 9,673 businesses employing about 55,000 persons (ABS 1997a).  The software industry is also one of the fastest growing industries in Australia, growing at a rate of 15 per cent per annum (BSA 1998).

In formulating national policies and programs for the industry, it must be remembered that the Australian computer services industry is dominated by very small businesses - 88 percent employ less than five persons (ABS 1997b).  A similar situation exists in the USA - Fayad et al. (2000) report 65 percent of data processing companies have less than five employees.  This presents a challenge in terms of determining the current practices of industry participants, and in devising improvement initiatives which are feasible for these very small organisations.

In recent years, concern has been expressed about the need for the Australian Computer Services industry to achieve global competitiveness (Goldsworthy 1997, McKerlie 1996).  Software process improvement is recognised as having the potential to improve competitiveness by increasing productivity; reducing costs, defect and rework; and improving time to market and customer satisfaction (El Emam & Briand 1997).

The survey, which forms part of the National Industry Improvement Program, was conducted by SEA Queensland and the Software Quality Institute (SQI) of Griffith University.  The aim of the survey was to determine to what extent software developers are using best practice techniques.  The survey was conducted initially in Queensland, is currently being administered in Western Australia, and is planned to be used in all remaining Australian states.

A best practice is defined as "a management practice that is widely recognised as excellent and is recommended by most practitioners and experts in the field" (ESI 1999).  The European Software Institute (ESI) survey instrument has five sections:

·        organisational issues (8 questions) addressing project management, change control, training programmes for managers;

·        standards and processes (13 questions) covering formal assessment of benefits and risks, management reviews, control of subcontractors, independent audits, coding standards, formal handovers, test planning;

·        metrics (8 questions) such as records of actual and estimated resources, error sources, test efficiency, computer performance, project tracking;

·        control of the development process (6 questions) for example accountability for estimates and schedules, requirements management, control of code and specification changes, regression testing;

·        tools and technology (7 questions) for instance use of design notations, automated testing tools, prototyping, data dictionary, project management tools.

The ESI questionnaire was developed in 1994 by an independent organisation for the European Commission (Dutta et al. 1998).  Previous research in software process improvement and popular models such as SEI's Capability Maturity Model, Bootstrap and SPICE influenced the development of the questionnaire (Dutta & Van Wassenhove 1997).  Since 1995, the questionnaire has been distributed annually by the ESI as part of the call for proposals under the European Software and Systems Initiative.  Respondents are explicitly informed that the questionnaire is independent of the proposal review process (Dutta  et al. 1999).

3. Survey Methodology

The unit of analysis was Queensland organisations undertaking software development.  The target population was all organisations in Queensland which develop software for sale (custom and commercial off-the-shelf developers) and in-house software developers of large organisations.  Several sampling frames were used as a single list of all developers was not available.  Rather than identify a sample of the population, the aim was to reach the entire population of organisations which develop software.  As it was not possible to identify organisations undertaking software development from all organisations involved in the Information Technology and Communications industry, a list of 5,600 possible organisations was compiled, sourced from the Australia on Disk and MIS 3000 databases and contact lists from the Queensland Government's Information Industries Board and SQI. 

3.1 Survey design

To overcome constraints of time and cost, the ESI was approached for permission to customise and use the ESI questionnaire.  Permission was granted on the condition that minimal changes were made, and that the Australian results would be made available to ESI for comparison with the European data. During the pretest, concerns were raised about the section headings of the ESI questionnaire.  It was decided to arrange the questions in more of a lifecycle sequence so that organisations with very few developers would feel less threatened by the survey format, and thus respond more readily.  The new headings used were Requirements and Design (6 questions); Code and Test (13 questions); Configuration Management (5 questions); Estimates and Schedules (9 questions); and Project Management and Training (10 questions).

The format of the questionnaire was changed to appear more compact, and some questions were split to reduce ambiguity.  Two additional questions were included in the body of the questionnaire to provide information relating to the use of programming languages and development tools.  To further customise the instrument to local conventions, the ANZSIC list of industrial sectors was used in place of the European sector breakdown.

3.2 Data collection and analysis

A total of 5,600 survey forms were mailed in January 1999 with a cover letter and reply-paid envelope.  As expected, many of the organisations did not develop software and more responses were returned from non-developers (354) than developers (209).

Table 1. Breakdown of responses by type

A web site was created to enable data entry.  Microsoft Excel and SPSS were used to provide descriptive statistics, to assess normality, and to calculate correlations.

4. Survey Findings

4.1 Primary involvement in software industry

It was recognised that some organisations may have multiple roles in the software industry.  For example, most large organisations develop and use in-house systems as well as purchasing 3^rd party software.  Also, many software development companies produce off-the-shelf packages as well as providing custom-built systems to clients.  The survey included a question to determine the primary involvement of organisations in the software industry.  Although the question was worded to encourage respondents to select only one option as their primary involvement, the results (in table 2) show many respondents selected more than one option.

Table 2. Responses to primary involvement question

4.2 Adoption levels

For each response, the level of adoption of the best practices was calculated.  The number of "yes" responses to the 43 best practice questions were summed and a percentage calculated based on the proportion of "yes" responses to "yes" and "no" responses.  Blank and "not applicable" responses were ignored in this calculation.  Of the 209 valid responses, four provided demographic details but left the best practice section blank.  These four responses were excluded from the analysis of adoption of best practice.

The mean adoption level of 47.5 percent with a standard deviation of 21 percent is slightly lower than the average reported by the ESI from its 1997 questionnaire (51% s.d.21%) (ESI 1999).  Table 3 compares the leaders and laggards in best practice adoption from the ESI surveys 1995-1997.

Table 3. European Results: Leaders and Laggards

(Source: a. Dutta et al. 1998; b. ESI 1996b; c. Dutta et al. 1999)

From the survey responses, a histogram of adoption levels was produced, and rather than the expected normal distribution, a bi-modal  distribution occurred.

4.3 Adoption levels by size

Two questions relating to size were included in the survey.  The first question related to the total number of employees in the organisation and the second to the number of employees involved in software development or maintenance.  On the basis of the total number of staff, organisations were scaled as small (less than ten staff); medium (10 - 500 staff); and large (more than 500 staff).  As can be seen in table 4, adoption of best practice appears to be associated with organisation size.  Large organisations exhibited much greater adoption than medium organisations, which in turn out-performed small companies.

Table 4. Average adoption levels by organisation size

Looking at size from the perspective of the number of software developers engaged in programming or maintenance, a stronger pattern emerges.  Organisations with fewer developers reported much lower average adoption of best practice compared to organisations with a large number of software developers, shown in table 5.

Table 5. Average adoption by development team size

The ESI survey conducted in 1995 also found that the size of the software organisation influences best practice levels as shown in table 6.

Table 6. ESI Results: Size of development team and Best Practice Level

(Source: ESI 1996a)

4.4 Assessing normality

Prior to undertaking any correlation analysis to statistically prove the association between variables such as organisation size and adoption of best practice, the characteristics of the data were explored to ensure the correct statistical approach was selected.  The assumption of normality is a prerequisite for many inferential statistical techniques.

In this case, it was found that the distribution of the variable adoption level was not distributed normally.  Smaller groups of responses, based on organisation size, development team size and involvement in the software industry were checked for normal distribution. The complimentary groups of commercial off-the-shelf (COTS) and non-COTS developers were the only subsets of responses which exhibit a normal distribution for the variable adoption level, and therefore represent difference populations.  (Full details of the statistical analysis are available in Cater-Steel 2000).

A significant positive relationship was found between adoption levels of best practice and the size of the development team for both COTS and non-COTS groups (COTS: r=.3016, p<.05; non-COTS: r=.2658, p<.05).  However, the relationship between adoption levels and size of the organisation was not statistically confirmed for either COTS or non-COTS developers.

4.5 Adoption levels by primary involvement

In comparing adoption levels of COTS and custom software developers, highest adoption of best practice was reported from organisations which develop COTS software (see table 7).  According to the CEO of SEA Qld at that time, Phil Scanlan, the large number of developers (88) who saw their primary role as COTS developers reflects the large concentration of vertical niche market package developers in the Brisbane area.

Table 7. Comparison of adoption levels for COTS  and custom developers

The large difference in adoption levels between the two groups of developers begs the question: Are there specific practices which have been more readily adopted by one group compared to the other?   When adoption levels were analysed on a question by question basis, it was found that for the following five practices, COTS developers reported adoption levels of at least 17 percent higher than the non-COTS developers:

·        maintain records from which all current versions and variants of software systems and their components can be quickly and accurately reconstructed in the development environment;

·        establish a change control function for each software project;

·        log, track and analyse post-implementation software problem reports;

·        apply common coding standards to each software project; and

·        use software tools used for tracking and reporting the status of the software /subroutines in the software development library.

In fact, there was only one practice where non-COTS developers exhibited higher adoption: the use of automated testing tools.

The ESI questionnaire arranges the practices under five headings: organisational issues; standards and processes; metrics; control of the development process; and tools and technology.  As can be seen from the data provided in table 8, COTS developers exhibit much higher adoption than non-COTS organisations for every section of practices.

Table 8. Comparison of adoption by ESI sections

To gain an understanding of which software engineering practices are most used in Queensland, the highest scoring questions were collated and ranked for COTS and non-COTS developers (refer to table 9).

Table 9. Most used software engineering practices

Three of the most popular practices relate to coding and testing, the other two involve requirements and project management.  This is not surprising, as the importance of these practices has been recognised in the industry press and stressed in information systems and software engineering training courses for some time.

5. Process Improvement Program

During the last six months of 1999, 25 organisations accepted the invitation from SQI to participate in the Process Improvement Program.  This program was funded by SEA (Qld) for SEA members and was conducted at no cost to participants.  Researchers at SQI developed a procedure to enable Rapid Assessments for Process Improvement for software Development (RAPID) (SQI 1999a).  The RAPID method is based on ISO/IEC 15504 (SPICE) standard and designed to enable assessments to be performed in one day (Rout et al. 2000; SQI 1999b).

5.1 Evaluation Instrument

The ISO 15504 standard sketches out a roadmap for the implementation of best practice in software engineering by defining 40 processes, divided into five categories: customer-supplier (10); engineering (9); support (8); management (4); and organisation (9).  The process capability of each defined process evaluates to what extent the process achieves its defined purpose and objectives (SPIRE 1998 p57). Capability is measured in levels from incomplete (level 0) to optimising (level 5) as shown in table 10.  These capability levels represent milestones along the road to software process improvement.

Table 10. Capability Levels

(Source: ISO/IEC TR 15504 and SPIRE 1998)

As the RAPID assessments were restricted to one day each, rather than use the 40 processes defined in ISO 15504, only eight processes were evaluated, as listed in table 11.

Table 11. Processes and Process Categories.

Although ISO-15504 provides rating levels from 0 (incomplete) to 5 (optimising), for the RAPID assessments, only questions relating to levels 1 (performed), level 2 (managed) and level 3 (established) were included.  Also, data were collected by interviews and discussion rather than examining actual work products such as project plans.  Each evaluation was undertaken by two trained SPICE assessors, one in the role of team leader and the other as support assessor.  A set of procedures and templates were prepared covering the demographic questionnaire, assessment plan, assessment instrument, assessment report, feedback form, follow-up meeting and final report.

5.2 Evaluation Procedure

Firstly, the assessment team leader contacted the sponsor of the organisation, and sent the demographic questionnaire to the sponsor for completion.  Using the demographic information, a plan was compiled jointly by the team leader and the support assessor, and agreed to by the sponsor.  On-site interviews were conducted by the team leader and support assessor with key people involved in managing the software development effort of the organisation.  For each of the eight processes examined, the assessors followed the script of the assessment instrument to determine the extent to which the process attributes have been achieved using a four point scale: not achieved, partially achieved, largely achieved and fully achieved.  The capability level (0,1, 2 or 3) for each process was then determined, based on the achievement of the process attributes. 

A draft report was prepared by the assessment team leader and support assessor and forwarded to the sponsor at the organisation to confirm that the assessment team had accurately recorded the information discussed.  Any changes suggested by the sponsor were discussed and then the assessment report was submitted to the organisation sponsor, SEA (Qld) and SQI.  A feedback form was sent with the assessment report to the sponsor to solicit comments regarding the conduct and value of the assessment.  Six months after the assessment, a half-day follow-up meeting was planned and the final report prepared for the organisation sponsor, SEA (Qld) and SQI.

5.3 Results from Process Improvement Program

To date, 25 assessments have been conducted with 21 assessment reports submitted.  A few follow-up meetings and final reports have been completed, but many organisations requested the follow-up meeting be postponed as all their resources were absorbed implementing the new Australian Goods and Services Tax for the current financial year.

All except one of the 25 organisations were located in Brisbane.  Based on the 21 assessments reports completed, there were four small organisations (less than five developers), 14 medium size (from five to 50 developers), and three large organisations (greater than 50 developers).

Figure 1. Proportions of Capability Level Ratings by Process