1.         Introduction

The production of a business multimedia product incorporates knowledge and expertise from the business environment, software engineering, and the multimedia interface. Each of these areas uses specific terminology to describe elements of their environment. Semantic discrepancy between terms used in the various disciplines hinders the establishment of mutually understood requirements specification. Such misunderstanding has long been a feature of the interaction between software development and business with a number of frameworks and methods introduced to facilitate communication. The multimedia interface adds a further layer of complexity to this problem.

Evaluation criteria for business multimedia products need to be derived from the multiple disciplines that have come together within this application area. To produce such criteria it is necessary to establish an acceptable definition of multimedia incorporating all applicable elements from disciplines relevant to the development objectives.

The Multimedia Interface relies on usability factors and aesthetic judgement to provide the product with an interface that is appropriate for its technical and business objectives. Therefore the evaluation of the multimedia interface involves both objective and subjective indicators.

From a business perspective, the development of a multimedia product must result in a product of value that contributes to the attainment of strategic objectives. The translation of market oriented product requirements into technological tolerances suitable for software engineers and aesthetic and cognitive parameters appropriate to multimedia interface design are crucial for determination of indicators for the evaluation of design elements.

Software Engineering provides the technical basis for determining how such parameters can be implemented to provide a product that is fit for purpose. Proven methods generate the technical requirements of the end product to objective engineering tolerances e.g. reliability, portability, efficiency.

Whilst evaluation techniques for the business and software engineering areas are well established, and tried and trusted techniques exist, these are not designed for the multimedia interface. This paper examines the areas of the multimedia interface, and of business and software engineering, to tease out a set of generic design goals. These design goals are then reformulated as the framework within which the business multimedia product can be evaluated.

2.         Multimedia interface requirements

To establish a generic framework it is necessary to establish the parameters of multimedia. However the definition of multimedia is itself problematic which is why so many multimedia experts feel the need to give their own definition (Reisman 1998). This section examines the requirements of the multimedia interface from a number of perspectives in order to establish its generic requirements.

2.1       Art

Multimedia art offers an environment in which art and technology are fused, form and content are one: the medium is the message. The audience is immersed in a multi-sensory experience. A level of interaction is often present to actively engage the audience and this may be used to allow the audience to select its own apparently random way through the ‘piece’. Multimedia covers many media but it is the synergy of the different media, and of the form and content that is important. A sort of narrative exists (Packer 1999). It is not linear but it does, through interaction, enable the audience to engage in a particular set of experiences.

The criteria ‘Synergy’, ‘Hyper-Narrativity’, and ‘Interactivity’ are proposed and cover the synergistic, hyper narrative, and interactive features observed above.

2.2       Computer gaming

Synergy, Hyper-Narrativity, and Interactivity are also observed within the field of computer games. Interaction with the game becomes a natural extension of the player with games players so engaged in the activities of the game that they no longer notice the technology being used for the interface controls. Narrative structure to the game exists but players are able to move between the ‘nodes’ of the game as they choose. Synergy is evident as the game is a multi-modal activity with the technological infrastructure and game content fused. In order to prevent the player from becoming overwhelmed with irrelevant information, both in terms of location, and in terms of control mechanisms, containment mechanisms are used (Mallon & Webb 2000). The proposition ‘Leanness’ is introduced as the prevention of overload of irrelevant information and of irrelevant controls.

2.3       Hypertext and Hypervideo

Similar structures are reported within hypertext and hypervideo with threads of narrative interweaving through a network of nodes (Schneider and Smoliar 2001). The narrative consists of nodes of content with links to further nodes. A link-less node indicates closure of a path. Narrative, whilst selected by the viewer, is not random and is part of the design e.g. an interactive video may have a number of differing story lines but there is a limited number available; demonstrating hyper-narrative, interaction, and leanness. Synergy is shown by the delivery of the material itself i.e. the fusion of the delivery mechanism and the content. The system embodies the information that represents the total content domain of all the narratives that may unfold. The proposition ‘Embodiment’ is introduced to indicate the coverage of the total content material. In terms of content, Lean and Embodiment complement each other: the system should contain all of the information needed to portray the narratives but only that information.

2.4       Teaching and learning

The factors of Synergy, Hyper-Narrativity, and Interactivity, Leanness and Embodiement are demonstrated within teaching and learning environments. Typically a system would involve a number of tasks. Each task would have objectives and learning outcomes, content associated with the delivery of those outcomes, and a method for the assessment of those outcomes. Tasks would be grouped together to form lessons with students allowed to progress from one task to another by the achievement of a preset score. In some cases a task could be skipped or exited without the preset score being achieved. The environment within which these tasks are set would have a consistent interface structure and a navigation structure.

An example of such an environment is a graphics based package for teaching and learning called WordShark (WhiteSpace 2003). The student is presented with a list of available lessons and tasks based on their student profile and the required learning outcomes. The student then selects the exact lesson and tasks from this restricted list (demonstrating Hyper-Narratvity). Interaction is demonstrated by students’ engagement with the system through navigation, participation in the choice of learning material, participation in learning, and participation in reward mechanisms. Embodiment is demonstrated by the system containing all of the information necessary to dispense all of the lessons. Leanness is demonstrated by the restriction of controls and by the restriction of information to that required for any given student at any given time. Synergy is demonstrated by the multi-modal portrayal of information through a combination of audio, text, graphics, and touch, in such a way that the technology is transparent. Message and medium are one.

But is this multimedia?

This is a problem also seen in the computer gaming industry where it is often difficult to separate between multimedia-based games and graphics-based games. The use of video cannot be the decisive factor. The dividing line is really one of artistic aesthetic (Gonzalez 2000). This is an intangible quality and presumably differs from person to person. What does matter though, is high quality content and the way that it is portrayed.

2.5       Visual design

From the perspective of the visual designer, high quality content is information rich revealing both detail and complexity in an easily understood manner. Any confusion and clutter are failures of the design and not attributes of the information [Tufte 2001]. Viewers are free to concentrate on the information being presented and not on any containers or borders. Metaphor and referent are used to give context to the content. How the data is turned into such content is part of the design process and often dependent on the particular media and its grammar. The quality of the information is covered within Embodiment and its presentation and form within Synergy. The removal of ‘noise’ fits with the concept of Leanness. A description of media aesthetics, the process of examining media elements such as lighting, picture composition, and sound, the manipulation of perceptual reactions, and artistic message communication, can be found in (Zettl1999).

2.6       Web sites

Web sites demonstrate Synergy, Interactivity, Hyper-Narrativity, Leanness, and Embodiment. Sites are, in effect, based on a structure of nodes with each node comprising of one or more pages. Each node has (liberal) links to other nodes and may have links to other sites. Each node may contain multimedia objects. The site should be capable of changing over time if necessary. Its content should remain up to date and its functions should grow and shrink according to the requirements of the users (Nielsen 2000).

Interaction is demonstrated by the user being able to engage with the system navigation tools to select content; by engagement with the content; by engagement with other controls available; and by engagement with other features of the site (forms etc.). Hyper-Narrative is shown by the intentional design of a number of nodes and the links between them: there is an overall purpose to the site but the user is free to move between the nodes in a manner that matches their intention in using the site. Leanness is demonstrated if the user is only shown the information that they require; if there are no unnecessary navigation or control structures; and if only the technology necessary to meet the user profile is used. Embodiment is shown if the site has data rich high quality output and holds all of the information relevant to all of its users in relation to its stated function. Synergy is demonstrated if information is portrayed in a multi-modal way through a combination of audio, text, graphics, and touch, in such a way that the technology is transparent.

The proposition ‘Metamorphicity’ is introduced to cover the required organic nature of the product.

A web site is not necessarily a multimedia product. If it contains multimedia objects then a multi-layer model emerges where the concepts introduced must be applied, not only to all of the multimedia media components, but also to the different layers. This multi layer model is evident in other multimedia products e.g. SMIL (Bulterman 2001).

2.7       Usability theory

For a multimedia system to meet its objectives it is essential that it is usable. Usability results when the users can accomplish the tasks or needs that they have effectively, efficiently and with a high level of satisfaction (Henneman 1999). Therefore a system can be deemed usable when the system matches the requirements of all of the intended users and as such impinges on all aspects of the system with which people interact. The requirement for usability can be achieved by using appropriate metaphor, simplicity, consistency, and user feedback. A referent is also required for the navigational structure so that the user can easily locate their position within the system or narrative (Nielsen 2000).

The proposition ‘Usable’ is introduced to accommodate this factor.

3.         Business and software engineering requirements

Evaluation techniques for this area are well established.

3.1       Business requirements

The requirement from the business perspective is to produce products that add value to the business in line with its strategic objectives whilst using the least amount of resources in the process. Customers needs and requirements are ascertained and products developed to fulfill them [Tidd et al 2001].

However, methods that transform business strategy into business objectives are not solely customer focused. For instance the Balanced Scorecard derives performance indicators from the financial, the Internal business, the Innovation and Learning, and also the Customer perspective (McGregor 2002). Marketing examines the overall interests and goals of potential customers and ascertains if they can be met or modified (Borés et al 2001). This produces the user profile that acts as the basis for multimedia evaluation.

 Inhibitors of successful multimedia systems are reported as ‘very fast moving markets’, ‘uncertainty and risk associated with new technology’, ‘changes in requirements specification’ (Britton et al 1997) and ‘unclear statement of requirements’ (Barry & Lang 2001). Measures of the success of a multimedia product would therefore be: Does the product still meet the demands of the market? Does it meet its requirements specification? Is it supported by the technological infrastructure?

The specification for a multimedia product would include its scope and purpose. The content and the particular treatment of any of the content, such as video, can then be derived, as can the size, depth, functionality, and interactivity of the system. The technology required for product development and for product deployment can be stated. The level of input from subject and media experts can then be ascertained, as can the members of the development team. Any specific contractual requirements can be made explicit. And finally the budget and timescale derived. This however is not a linear process and all of the above considerations impinge on each other.

The specification details need to be positioned within the users requirements and the strategic objectives of the business taking into account market parameters, the requirements of all stakeholders, legal considerations, ethical considerations, standards, social considerations, and political considerations.

The specification of the project evolves and will trade off between all of the factors given. Trying to measure the quality of the resultant multimedia product will also be difficult, as the measure of quality will vary between the users, the client, and the development team. If members of the development team come from differing disciplines they will not even be able to internally agree as to the quality of the finished product.

To recognise this evolving nature and the many trade offs that will be made between factors, the proposition of ‘Pragmatic’ is introduced: The product should be a complete solution to the needs it is addressing and should conform to its objectives and final requirements specification exactly. Its design should be both simple and elegant. It should align with both the strategic objectives of both the business and of the client business. Requirements would be specified in terms of all of the attributes that are necessary for the success of the particular product which could be any of those given above but in particular user requirements, information content, content treatment, functionality, and interactivity. Full profiling of the users would be undertaken.

It is highly unlikely in such an evolving prototyping environment, given the extent of the trade offs, that the product will match its specification first time. Even if it does the requirements of stakeholders will change. ‘Metamorphic’ is extended to cover the changes required from a business perspective as well as those from a user perspective.

The propositions of ‘Leanness’ and ‘Embodied’ can be further clarified in light of business objectives. The product should embody ALL of the information content, and ALL of the technological infrastructure necessary for it to perform ALL of its requirements. The product should also embody any organisational image or identity. The product should use the minimum resources necessary for it to fulfil its function and should contain only that information and only that infrastructure required for it to do so. Design should be kept simple but still enable full functionality.

3.2       Software engineering requirements

Software Engineering attempts to add discipline and rigor to the software construction process by asking “Are we building the right product?” and “Are we building the product right?” (Boehm 1981). Various methodologies and development models have evolved in trying to overcome problems to relating to software development. All of the seminal works have attempted to produce an abstract model of the area to be computerised, and to provide a method whereby this abstract model could then be systematically developed into a computer system, Many criteria have been proposed over time e.g. Economy, Integrity, Documentation, Understandability, Flexibility, Interoperability, Modularity, Correctness, Reliability, Modifiability, Validity, Generality, Testability, Reusability, Resilience, Usability, Clarity, Maintainability, Portability, and Efficiency (Boehm 1988) but the essence of all the methods is to keep the design as simple as possible whilst guaranteeing the systems consistency, usability, reliability, efficiency, maintainability, and extendibility i.e. that the system will be ‘fit for purpose’.

“Web engineering is a systematic, disciplined, quantifiable approach to the development, operation, and maintenance of hypermedia applications” (Ginige 2000).

The component approach is demonstrated by a plethora of models and tools each of which has or hopes to have its own standard.

There are functional requirements, user requirements, design models, navigation models, content models, user models, programmer models, and numerous delivery models. MPEG-4 describes a media objects content and interaction (Rutledge 2001). SMIL defines how independent media objects should integrate at run time (Bulterman 2001). Content chains specify how content is derived, produced, processed and consumed (Chang 2002).

From the software engineering perspective, metrics for the full verification of a system would therefore have to include metrics for the verification of each of the components used e.g. a delivery mechanism would have to be evaluated for its performance in relation to bandwidth, synchronization, latency, delay jitter etc. Content would have to be measured against its information value, credibility, depth and timeliness.

Deshpande et al (2002) suggest a framework for the evaluation of web sites based upon usability, functionality, reliability, maintainability and scalability (metamorphicity) which they measure by examining content and link management; scalability; performance; compatibility with other systems; and usability; by auditing the web site against information from: the business; the consultants and developers; site users; other parties e.g. legal requirements; and auditors.

Mich et al (2002) suggest a framework based around multi-stakeholders (the owner, the users, the developers) using Identity, Content, Services, Location, Management, Usability, and Feasibility as the main factors. Each of these factors has sub factors and measurement is based on allocating a weighting to both the factor and the sub factor.

The framework proposed here draws on their work. However as complexity and level of detail are two of the main reasons developers report for not using methods or frameworks (Barry & Lang 2001), the framework proposed here has been kept simple in order to promote understanding.

The majority of the criteria to be applied when evaluating a multimedia product from a software engineering point of view extend the propositions given earlier. Pragmatic: the product should be fit for purpose. Usable: the product should be easy to use. Metamorphic: the product should be maintainable, extendible, flexible, modifiable, modular, portable, reusable, scalable, and compatible with other systems. Leanness: there should be a minimal simplistic approach to promote economy and clarity. No unnecessary technology should be used. Embodied: full functionality.

Two new factors are introduced here to give due prominence to the additional requirements of Software Engineering: ‘Reliability’ and ‘Efficiency’.

4.         Design goals

These have been derived from the examination undertaken within the last three sections. Hyper-Narrative has been renamed Poly-Narrative to give an easy to remembr acronym for the overall framework.

4.1       Pragmatic

The product is a complete solution to the problem for which it is being developed and performs its task exactly in terms of its final scope and final specification. It is aligned with both the business and technical objectives of the product ensuring that it meets the strategic goals of the business, adds value to the business, meets its market objectives, and takes into account legal, cultural, social, ethical, and internal organizational issues. The technological resources that the product requires in order to function are readily available. It is simple and elegant in its overall design.

4.2       Usable

The product is easy to use and fits the profile of its intended users. It adheres to all normal usability requirements, and takes into account the cultural aspects, and therefore the relevant aesthetic values, of the users and any user organisation. The product should also be satisfying to use.

4.3       Reliable

The product is secure against human error and machine malfunction. It can function in a reduced manner under abnormal conditions and protect itself against unauthorized access and modification. It is dependable and it is safe.

4.4       Efficient

The product should have an acceptable response time. Users will only use the product if it is not unduly slow in presenting the information. The definition of ‘slow’ will depend on the product being developed and the user profile it is being developed for. The software should also be efficient both in terms of its algorithmic structure and the system resources it uses. Efficiency should not come at the expense of simplicity or usability.

4.5       Synergistic

The multimedia product should

§          appear as ONE and not as the mix of its individual constituent parts.

§          provide more expression than is available just from its constituent parts.

This also applies to any constituent layers and any individual multimedia objects. The technological infrastructure should be hidden from the user and the user should be immersed in the product.

An example of such synergy would be a menu driven music cd where the user can see the video of the artist, can hear the audio soundtrack, and can interact with the ‘system’ to move to another track or get additional information on the track or artist. There may also be more than one video for a given track or more than one soundtrack for a given video clip. There would be many components to the cd and many controls to the player but they should all be singing from the same song sheet.

4.6       Interactive

There should be some level of meaningful user interaction available within the product. Users should not be passive viewers but should dynamically engage with the product. This interaction should allow the user to benefit from a non-linear approach to narrative giving the user the sense of control over the system in use. It is the interaction in the music cd system above that differentiates the product from simply being just a collection of music videos.

4.7       Metamorphic

Not only must the product be created with the right content, but it must also be able to grow and shrink in an organic fashion according to the changing needs of the stakeholders. Content should be updated as necessary to keep the product up to date.

4.8       Poly-Narrated

A loose definition of narrative is used here. A business multimedia product has a purpose and in effect this is the narrative. This narrative should be clearly seen. There should be numerous paths designed into the product. Users, through interaction, are able to determine their own path through the product to see information that is relevant to them. The path the user takes may be one of a number of pre-determined paths or via a search facility. Narrative should only be provided through the navigation features that are active to the user at the time and is based on containment, causal connections and interactivity. Users should be able to follow the overall predefined narrative if they wish to but should also be able to pursue their own interests.

4.9       Lean

The output of the product should not ‘overload’ the user. To achieve this the product should adhere to a minimalist approach. The use of the technology must reflect the need of the product and not be used for its own sake. Although the information output should be kept to the minimum required to deliver the content, it should deliver all of the content. Control and navigation interfaces should be kept simple.

4.10    Embodied

The product should embody all of the information and all of the functionality that is required to produce its objectives. Information should be of high quality to promote immersion of the user.

These general design goals can be further specified according to their lower level requirements e.g. Pragmatic: Fit for Purpose; Complete solution; Aligned with strategic goals; Adds Value to business; Meets its stated scope & purpose; Supported by technological infrastructure; Conforms to requirements specification; Conforms to market requirements; Conforms to needs of stakeholders; Meets ethical, social requirements, political requirements, cultural requirements, and legal requirements.

5.         Evaluation framework

Reforming the design goals of the last section derives the evaluation framework. The product would be evaluated for its conformance against those design factors by evaluating its actual performance against those stated in its specification.

6.         Conclusion

The framework given here provides a general framework to evaluate business multimedia products. It incorporates inter disciplinary views regarding the construction of multimedia and is constructed from the values and interrelations of the multimedia interface, of business and of software engineering. To aid use across the multidisciplinary area the individual factors are identified and constructed to be easy to remember acronym: PURE and SIMPLE. A multimedia interface is extremely complex. In order to aid development and maintenance it is important that the design of a business multimedia product be kept as simple as possible but without sacrificing the necessary functionality and form of the product.

In general terms the framework can be thought of as:

The factors drawn from the multimedia interface incorporate artistic and scientific values but they do not resolve the issue of quality in multimedia. This is left to the evaluator in relation to the users and the purpose of the system. High quality data rich multimedia presentation should be generated but there is little point in creating impressive multimedia presentations if they don’t successfully accomplish their purpose.

The framework doesn’t help in determining the boundary between graphics and multimedia in products. When is a computer game graphics based and when is it multimedia? But then does this really matter?

References

• Barry, C. and Lang, M. (2001) “A Survey of Multimedia and Web Development Techniques and Methodology Usage”, IEEE Multimedia, Vol. 8, April-June, pp52-60.

• Boehm B.W. (1988), “A spiral model of software development and enhancement”, IEE Computer, Vol. 21, No.5, pp 61-72

• Borés, C., Saurina, C., & Torres, R. (2001) “Technological convergence : a strategic perspective”, J. of Technovation, Elsevier Science 2001, Pergamon Press.

• Britton, C., Jones, S., Myers, M., and Sharif, M. (1997) “A Survey of Current Practice in the Development of Multimedia Systems”, Information and Software Technology, Vol. 39, pp695-705.

• Bulterman, D. (2001) “SMIL 2.0”, IEEE Multimedia, Vol. 8. October-December, pp82-88.

• Chang, S. (2002) “The Holy Grail of Content-Based Media Analysis”, IEEE Multimedia, April-June, pp6-10

• Deshpande,Y., Chandrarathna, A., & Ginige, A. (2002), “Web Site Auditing – First step towards reengineering”, ACM Conf. Proceedings SEKE 02, Ishia, Italy.

• Ginige, A. (2000) “Web Engineeing in Action”, Workshop Proc. Int. Conf. Software Engineering (ICSE 2000), IEEE Computer Society Press, Vols. 4-11, 2000, pp 1-9

• Henneman, R. (1999) “Design for Usability: Process, Skills, and Tools”, Information, Knowledge & Systems Management, Vol. 1, No. 2, pp133-145.

• Ivory, M., & Hearst, M. (2002) “Improving Web Sit Design”, IEEE Internet Computing, 2002, March-April, pp56-63

• McGregor, C. (2002) “Balanced Scorecard Driven Business Process Definition using XML”, Conference Proceedings 34th Hawaii International Conference on System Sciences, IEEE.

• Mallon, B., & Webb, B. (2000) “Structure, causality, visibility and interaction: propositions for evaluating engagement in narrative multimedia”, Int. J. Human-Computer Studies, Vol. 53, pp269-287

• Mich, L., Franch, M., & Gaio, L. (2003), “Evaluating and Designing Web Site Quality”, IEEE Multimedia, Vol. 10, January-March, pp34-43

• Nielsen, J. (2000) Designing Web Usability: The Practice of Simplicity, New Riders Publishing, Indianapolis.

• Packer, R. (1999) “Just What Is Multimedia, Anyway?”, IEEE Multimedia, Vol. 6, Jan-March, pp11-13.

• Reisman, S. (1998) “Multimedia Is Dead”, IEEE Multimedia, Vol. 5, January-March, pp 4-5.

• Rutledge L (2001) “Multimedia Standards: Building blocks of the Web”, IEEE Multimedia, Vol. 8, July-August, pp 13-15.

• Schneider, T., & Smoliar, S. (2001) “Description and Narrative in Hypervideo”, Conference Proceedings 34th Hawaii International Conference on System Sciences, IEEE, pp 1-8.

• Tidd, J., Bessant, J., & Pavitt, K. (2001) Managing Innovation, 2nd Ed., Wiley, Chichester.

• Tufte, E. (2001) The Visual Display of Quantitive Information, 2nd Ed., Cheshire, Graphics Press.

• WhiteSpace (2003) “Word Shark”, White Space Ltd., 41 Mall Road, London W6 9DG, UK, [online], http://www.wordshark.co.uk

• Zettl, H. (1999) “Sight Sound Motion 3rd ed, Wadsworth Publishing, Belmont, California.

EJISE is published by Academic Conferences Limited
Curtis Farm, Kidmore End, Nr Reading RG4 9AY, England
Tel: +44 (0)1189 724148, Fax: +44 (0)1189 724691, Email: info@ejise.com

Send mail to info@academic-conferences.com with questions or comments about this web site.
Copyright © 2002-2006 Electronic Journal of Information Systems Evaluation
Last modified: September 29, 2005
ISSN 1566-6379