CHAPTER 1 - OoCities · Web viewIf you consider early versions of the Microsoft MS-DOS word processors, ... ... a proposal was put forward to create a

EVALUATING THE USABILITY OF AN ONLINE LEARNING SYSTEM WITHIN A PUBLIC SECTOR ORGANISATION

ZAHID WASEEM ALI

Submitted in partial fulfillment of the requirements ofNapier University for the degree of

Bachelor of Science with Honours in Multimedia Technology

School of ComputingMay 2004

Zahid Ali, BSc (Hons) Multimedia Technology, 2004

AUTHORSHIP DECLARATION

I, Zahid Waseem Ali, confirm that this dissertation and the work presented in it are my own achievement.

1. Where I have consulted the published work of others this is always clearly attributed.

2. Where I have quoted from the work of others the source is always given. With the exception of such quotations this dissertation is entirely my own work.

3. I have acknowledged all main sources of help.

4. If my research follows on from previous work or is part of a larger collaborative research project I have made clear exactly what was done by others and what I have contributed myself.

5. I have read and understand the penalties associated with plagiarism

Signed: ________________________ Matriculation No: ____________________

Date: ______________________

2


ABSTRACT

The Internet has revolutionised many things of late, e-commerce being one of the prime examples. The use of the Internet is now being used as a means to train people via the use online learning; however one aspect of online learning that seems to be overlooked is usability.

Examples of what can affect successful use of online learning are: lack of an in-depth understanding or underestimation of the efforts, time, and resources needed for this change, lack of implementation pedagogy and quality of learning, lack of training in new technologies and a lack of tools and standards. All of these can have a vital impact on the usability of online learning.

The purpose of this dissertation is to assess the usability of an online learning system that is currently in use within a public sector organisation, in conjunction with finding the best methods to evaluate usability.

3


LIST OF TABLES

2.1 Usability attributes...................................................................................14

3.1 Comparison of usability methods............................................................34

5.1 Discussion of heuristic evaluation findings.............................................49

4


LIST OF FIGURES

2.1 A model of system acceptability attributes..............................................15

2.2 Amended diagram of system acceptability attributes..............................16

4.1 Welcome screen user is presented with upon successful login................40

4.2 Response to question regarding the ease of learning how to use the system..........................................................................................42

4.3 Response to question on having to remember specific commands to use the system....................................................................42

4.4 Response to question regarding the system giving error messages.........43

5


AKNOWLEDGEMENTS

I wish to thank my supervisor, Dr Keith Horton at the Computing School, Napier University, Edinburgh, for his important comments concerning the dissertation and for his support during my Honours Studies.

I would also like to express gratitude to all members of staff at the CSA that helped with the project, and in particular Karin Gallacher.

A final thank you to all my friends and family who supported me throughout the year.

6


TABLE OF CONTENTS

Authorship declaration...................................................................................2

Abstract..........................................................................................................3

List of Tables.................................................................................................4

List of Figures................................................................................................5

Acknowledgments..........................................................................................6

CHAPTER 1 – INTRODCUTION

1.1 Online learning........................................................................................10

1.2 The Common Services Agency..............................................................11

1.3 Research question, objectives and scope................................................11

1.4 Constraints..............................................................................................11

1.5 Content of the dissertation......................................................................12

Chapter reference...........................................................................................13

CHAPTER 2 – LITERATURE REVIEW

2.1 Introduction..............................................................................................14

2.2 What is usability?.....................................................................................15

2.3 Website usability......................................................................................16

2.4 System usability.......................................................................................17

2.5 Online learning usability..........................................................................18

2.6 Usability evaluation techniques...............................................................22

2.7 Discussion................................................................................................25


7


CHAPTER 3 – METHODOLOGIES

3.1 Introduction..............................................................................................28

3.2 Approaches to research............................................................................28

3.3 Research methodologies..........................................................................29

3.4 Secondary resources.................................................................................31

3.5 Sampling..................................................................................................31

3.6 Data collection.........................................................................................32

3.7 Data analysis............................................................................................35

3.8 Reliability & Validity..............................................................................35

3.9 Review of methodologies........................................................................36


CHAPTER 4 – CASE STUDY

4.1 Introduction..............................................................................................39

4.2 Company background..............................................................................39

4.3 The learning centre..................................................................................39

4.4 The online learning system......................................................................40

4.5 Evaluation of the system..........................................................................41


CHAPTER 5 – ANALYSIS OF FINDINGS

5.1 Introduction..............................................................................................49

5.2 Analysis of online learning at CSA.........................................................49

5.3 Analysis of evaluation techniques...........................................................51


8


CHAPTER 6 – CONCLUSIONS

6.1 Main findings...........................................................................................54

6.2 Recommendations....................................................................................54

6.3 Critical evaluation....................................................................................55

6.4 Self evaluation.........................................................................................55

6.5 Tentative requirements for online learning..............................................56

6.6 Future work..............................................................................................56

Bibliography..................................................................................................57

Appendix 1 Usability questionnaire...............................................................62

Appendix 2 Cognitive walkthrough...............................................................63

Appendix 3 Heuristic evaluation...................................................................66

Appendix 4 Coding & Tabulation..................................................................69

Appendix 5 Tabulation graphs.......................................................................71

Appendix 6 Lack of consistency ...................................................................72

Appendix 7 Project plan.................................................................................73

9


CHAPTER 1 - INTRODUCTION

1.1 ONLINE LEARNING

There is no doubt the Internet is an incredible creation; the average person is finding the Internet to be of great convenience: shopping online and paying bills online.As the Internet is developing rapidly, there is a growing demand and increase in pressure for adopting new and original approaches to the design and delivery of education, hence the introduction of online learning as a mode of study. This is partly to do with the increasing number of learners and the limited amount of resources to meet the wide range of various needs, backgrounds, skills, ages, abilities and disabilities. (Ghaoui, 2003)

The advance in the Internet and HCI (Human Computer Interaction) has made it possible to reach out to a much wider learning audience around the world. As a result the rapid development of online courses means that usability issues may have been overlooked, therefore potentially affecting the overall learnability. The purpose of this is to evaluate the usability of an online learning system used within a public sector organisation.

As new training methods are evolving, many companies are taking to the approach of training their staff via online learning.

Computer based instruction can come in many forms today but today many people are exploring the use of the Internet to deploy instructional methods. (Brown & Lu, 2001, p352)

In addition to this, training using computer-based instruction appears to be associated with consistent time savings, in some cases students learned with it more than 70% less time than students in traditional classrooms (Kulik et al, 1986 cited Parlangeli et al, 1999). This form of training is not only limited to providing IT training but can also be used to develop personal skills, such as learning a new language. Most organisations now believe online learning to be a suitable and cost effective way of training especially in times of high workloads and increased pressure.

Examples of what can affect successful use of online learning are: lack of an in-depth understanding or underestimation of the efforts, time, and resources needed for this change, lack of implementation pedagogy and quality of learning, lack of training in new technologies and a lack of tools and standards. All of these can have a vital impact on the usability of online learning.

This topic was chosen for two reasons; firstly the researcher has previously spent time working for Edinburgh’s Telford College and was involved in the development of online learning courses. Whilst developing the online materials, there was no usability being taken into consideration. Secondly, as a multimedia student, the researcher will potentially be designing applications that will be used by people of various different backgrounds, disabilities etc, so usability is a key issue in this line of work. Investigating into such matters will give a better understanding of how usability is measured/evaluated and what are the general issues regarding usability.

10


1.2 THE COMMON SERVICES AGENCY

The Common Services Agency (CSA) plays an active role in NHS Scotland by providing clinical support and advice for Scotland’s health and patient care. The CSA focus on a range of services including: blood transfusion service, environmental hazards and public health, specialist legal services, and value for money contracting and specialist commercial and technical services.

The CSA is committed to developing flexible alternative ways of learning for staff in line with NHS Scotland’s Learning Together Strategy. With this in mind the CSA has heavily invested in the use of online learning in a bid to provide all members of staff equal training opportunities.

The introduction of online learning means - Staff are able to learn at their own pace- They can access the course materials via their own workstations- The cost of sending employees to training locations is greatly reduced.- Every member of staff has equal training opportunities, regardless of their

position within the CSA.

1.3 RESEARCH QUESTION, OBJECTIVES AND SCOPE

The research question of this dissertation is:Is online learning at the CSA usable? And are traditional evaluation methods useful in assessing usability?To achieve this, the research question is broken down into the following objectives

- Provide an evaluation of literature regarding usability issues for general computer systems, and more specifically online learning systems.

- Identify the most appropriate methods that can be used to evaluate the usability of online learning systems.

- Investigate any usability issues in the current online learning system in use within the CSA.

- Make recommendations regarding usability for future applications of online learning at the CSA, and consider any general implications.

1.4 CONSTRAINTS

The major constraint I will face during this dissertation is time. The project must be finished and submitted by 5th May 2004 or earlier if possible. Other constraints lie within the Common Services Agency and the availability of staff as they are still in need to go about their daily work duties. Domestic staff start at 5pm, as a result may be unwilling to participate in any studies regarding online learning, as they will want to get on with their day-to-day duties. The Common Services Agency is also planning to re-locate to another part of Edinburgh in the first quarter of 2004; this could be problematic as all staff may not be keen on making the move to the new location. In a worst-case scenario, this would mean that the individuals involved with the online learning programme might no longer be with the CSA.

11


1.5 CONTENT OF THE DISSERTATION

Chapter 1 – IntroductionThis chapter identifies the main aims and objectives of the project, and also details why this particular topic has been chosen.

Chapter 2 – Literature reviewThis section of the dissertation focuses on reviewing literature regarding usability. The first main focus is usability of computer systems in general, followed by usability issues for online learning. The last part of the literature review looks at evaluation methods used to assess usability, followed by a discussion of the literature found.

Chapter 3 – MethodologyA discussion on the research techniques, which could be used, will be conducted with those selected to be used in the study.

Chapter 4 – Case StudyA case study of the CSA will be undertaken. The methodologies that were discussed in the previous chapter will be put into practise. This chapter also details the findings of the project.

Chapter 5 – Analysis of FindingsThis section discusses the findings and results obtained after using the Common Services Agency online learning system as a case study.

Chapter 6 – ConclusionThe final chapter of the dissertation presents the main findings of the project provides recommendations and a critical evaluation is also included.

A full bibliography can be found at the end of the document.

12


CHAPTER REFERENCES

Brown, J., & Lu, J. (2001). Designing Better Online Teaching Material. Technical Symposium on Computer Science Education, 352 – 356

Ghaoui, C. (2003). Usability Evaluation of Online Learning Programs. London: Information Science Publishing.

Parlangeli, O., Marchigianni, E., & Bagnara, S. (1999). Multimedia System in Distance Education: Effects on Usability, Interacting with Computers, 12, 37-49

13


CHAPTER 2 - LITERATURE REVIEW

2.1 INTRODUCTION

The following chapter looks at the literature associated to the research question in detail. This review is a critical evaluation of the literature, which provides an academic background to the study. The main objective of the literature review is to find out what is usability regarded as, and how it is used in conjunction to system/website design. The next section of the literature review focuses on evaluation techniques that can be used to assess the usability of a computer system, in this case the online learning system in use within the Common Services Agency. An evaluation of the material and a discussion of the findings are also presented at the end of the chapter.

2.2 WHAT IS USABILITY?

The International Organization for Standardization (ISO) defines usability as “the effectiveness, efficiency, and satisfaction with which a specified user can achieve specified goals in particular environments” (Dillon, 2001).Nielsen (1993) states that usability is a quality attribute that assess how easy interfaces are to use. This is supported by Usability First (2002) who agree that usability addresses the relationship between tools and their users. In order for a tool to be effective, it must allow the users to complete their tasks in the best way possible. This principle also applies to computers, websites and other types of software; however usability is not the only property of a user interface. It has multiple components and is normally associated with the following attributes

- Utility- Learnability- Efficiency- Retainability- Errors- Satisfaction

Usability testing of applications must include careful attention to these six attributes as the effectiveness of the application depends on accounting for these attributes of usability (Crowther et al, 2004 pp290). Table 2.1 explains these attributes further.

Attribute Usability attempts to measure: Utility How useful the application isLearnability How easy it is to learnEfficiency How much it streamlines the work

processRetainability How easy it is to remember application

operationsErrors How many errors occur during workSatisfaction How much people enjoy using the

applicationTable 2.1 – Usability attributesSource: Crowther et al (2004)

14


Nielsen (1993) claims that system usability is a narrow concern compared to the larger issue of system acceptability. System acceptability questions whether the system is good enough to satisfy all the needs and requirements of the users. Nielsen also states that the overall acceptability of a computer system is a combination of social acceptability and its practical acceptability.

Figure 2.1 – A model of system acceptability attributesSource: Nielsen, 1993, p25

Figure 2.1 shows the attributes of system acceptability, however from this it could be seen that utility is also linked to usability. The same could be applied to attributes of practical acceptability for example compatibility would have a part in usability. If a company were to update specific piece software it would need to be compatible with the older files that have been created. Reliability follows a similar path, what good is a system or a piece of software if it keeps crashing all the time? Cost is one attribute that would apply to the employer and not the user of the system, and is one of the attributes on the diagram that isn’t really linked with usability. An amended version of the diagram is shown in figure 2.2.

15


Figure 2.2 – Amended diagram of system acceptability attributesSource: Adapted from Nielsen, 1993, p25

Given that a system is deemed socially acceptable it is then broken down into various categories, one of which is usefulness. This is further broken down into the two categories of utility and usability (Gruidin 1992). Analysts from Microsoft (2000) agree that the category of usefulness is broken down into utility and usability; however they state that although the terms are related they are not interchangeable.

Utility refers to the ability to be able to perform tasks; the more tasks a piece of software is designed to do the more utility it has. If you consider early versions of the Microsoft MS-DOS word processors, such programs provided a wide range of text editing and manipulation features, however it required the user to remember dozens of keystrokes to perform them. Applications such as these can be said to have high utility but poor usability as the users have to spend as great deal of time and effort learning how to use such software. Both qualities are required for market acceptance, and both are part of the overall concept of usefulness. Obviously if a program is highly usable but doesn’t actually do anything of value, nobody will have much reason to use it. Users who are presented with a powerful program/system that is difficult to use will likely resist or try to seek alternatives.

2.3 WEBSITE USABILITY

Research conducted by User Interface Engineering, Inc., has shown that the majority of people navigating around websites has shown that people cannot find the information they are looking for. This can lead to time being wasted, lack of productivity, and an increase in frustration, which may ultimately make the user give up. Companies that rely on sales via their websites may have sales affected due to usability issues with their website.

Nielsen (1993) highlights that studies of user behaviour on the use of websites have found a low tolerance for difficult designs and slow sites. He also goes on to claim

16


that people don’t want to have to wait for a page to load or learn how to use a web page. The user has to be able to grasp the functioning of the site immediately after scanning the homepage. The fact that a certain website might be getting a lot of hits, and showing that certain parts are more popular may not be an indication of how popular the site is, but could be evidence of users getting lost after clicking on link after link.

Neil (2003) also reports that usability is often over looked in web-site development, using online retailer, Boo.com as an example of a web site containing major flaws. The fact that users refused to shop at Boo.com was too glaring to overlook even by a novice dot-com analyst. The cause of this was that usability had not been emphasised at the design stage. Nielsen (2003, cited by Neil, 2003) concludes that sites that had emphasised usability at the design stage had seen on average 135% increase in performance.

2.4 SYSTEM USABILITY

Usability of a website, software or a computer system depends on a number of factors such as how well the functionality fits the user needs, how well the flow through the application fits the user tasks. Better interfaces can be designed by designers learning key design principles and guidelines; however the designer can only create a usable system by going through a process that involves getting information from the people who will ultimately use the system (Usability First, 2002).

Trevor Bently (1991) proposes that system usability is based on two critical elements – common sense and simplicity. He states that the lack of usable systems might have been initially accepted due to the fact that the technology was not present i.e. computer memory was limited etc, however this is not the case in the present day. Bentley questions why we haven’t seen an increase in usable systems and puts this down to the fact that he believes that designers simply do not know how to make usable systems. Bently suggests the reason for designers not being able to design usable systems is that they cannot put themselves in the position of the non-computer literate person who will ultimately use the system. The study conducted by Poltrock et al (2003) was based on information sharing between design teams. Two design teams were monitored, one of which were a Microsoft team developing an online a web-based help and support service. Two usability engineers were a part of the team, which also consisted of software designers, which goes to show that usability is an issue that is being addressed from initial development.

The Boeing co is changing the way it is buying software, it is now making a products usability the fundamental purchasing criteria. Thibodeau (2002) states that they simply cannot afford to keep buying software that contains flaws. Especially when there are thousands of end users, design flaws can result in millions of dollars lost in time and productivity. There are other ways of addressing usability, such as giving the end user a degree of freedom to alter the look of a system to a certain degree; however this type of approach is also not without its shortcomings.

17


2.4.1 SOLUTIONS TO SYSTEM USABILITY

A possible solution to system usability problems could be to leave the design of the interface up the individual who will be using it. With the provision of sufficient customisation flexibility, the user will then be able to have exactly the interface they wish to have. The Intranet system that is in use within the Common Services Agency allows the site designer to have a certain degree of freedom, in which they can position links, change colours, images etc.

However studies conducted by Jørgensen & Sauer (1990) have shown novice users will not customise their interface, even when this facility is provided. Expert users were noted to have used the customisation function, but there are still reasons not to rely on user customisation as the main element of interface design. A customisation feature itself would also require some sort of user interface and thus as a result add to the overall complexity of the system and add to the users learning load.

Another problem that is then created by the ability to customise features is that the application may start to acquire an unprofessional look to it. Having users being able to customise features is one thing, however then this would require guidelines as to what is suitable content, for example one of the systems in use within the common services agency, Connect allows people to search for members of personal to find out their contact details (email address, room number, extension number etc). Connect also has the feature upload a profile picture; this is being abused in the fashion that users are uploading pictures, which may be deemed unsuitable.

System designers are human and of course use computers – both characteristics of the user. As a result it can then be tempting for designers to trust their own intuition about user interface issues (Nielsen 1993, p13).The main problem is that the system designers are different from the end users in several respects including their computer experience and knowledge of the design of the system. As they are the ones who have designed the system, and consequently have a deep understanding of it, the designers can easily fit an extra piece of information into the screen and interpret it correctly. As a result of this, a system designer may look at any given screen or error and believe it makes perfect sense. The implications of this are that a system designer could potentially go onto to design a piece of software that new users will struggle to use.

2.5 ONLINE LEARNING USABILITY

What is the definition of learning? Some definitions seem to only deal with behaviour (Schunk, 2000 cited Zaharias et al, 2002), some only with knowledge (Mayer, 1987 cited Zaharias et al, 2002) and some deal with knowledge and observable behaviour (Slavin, 1994 cited Zaharias et al, 2002, p2). What all these definitions have in common is that they all view learning as a positive change over time and that aims at improving capabilities to cognition, through a series of actions through personal observation and interaction. (Zaharias et al, 2002, p5). These researchers also find another common ground: in that they agree learning is not a process of transmitting information from someone who knows to someone who does not. Instead learning is

18


an active process that happens through direct experience, by being engaged in authentic tasks (Soloway, 1996, cited Zaharias et all, 2002 p2)

The ultimate objective for online learning is that it should be educationally beneficial, it is important in such environments to understand how usability contributes (or not) to educational goals (Jones et al, 1999). Squires and Preece (1999) argue that scholars have not considered the implications of usability features of an educational package in order to achieve educational goals. Both authors advocate that there is a need to help evaluators consider the way in which usability and learning interact.

Squires (1999) indicates the need for integration of learning and usability. He states that while there are now well-recognised traditions of research and development in educational computing and human computer interaction, workers in either of these fields very rarely speak to one another or take note of each other’s work. Educational computing literature is plagued with naive and simplistic perceptions of interface design, while many writers in the HCI field seem to be unaware of the major developments that have been made in the theories of learning.

Put another way: How can usability features be integrated with principled educational design? Different authors have addressed this in different ways: Mayes and Fowler focus on the design significance of the nature of the educational task, Squires and Preece re-cast a well known HCI evaluation paradigm in terms of socio-constructivism while Jones et al. propose a comprehensive framework for evaluating educational software which is focused on contextual significance. A socio-constructivist view of learning poses a challenge for conventional views of usability. In the most basic sense usability is seen in terms of interface design, which allows the user completion of well-defined relatively limited tasks. Socio-constructivism implies that learning environments should be context sensitive environments in which users are expected to make mistakes as a part of the learning process. (Squires, 1999, p464)

Mayes and Fowler (1999) argue that online learning requires the design of effective tasks rather than interfaces as they claim learning cannot be approached as a conventional task, as if were just another kind of work with a number of problems to solve and various outputs to be produced. This is because learning is a derivative of something else. Adopting such a task led focus has led Mayes and Fowler to propose an emerging paradigm for novel educational software (Squires 1999). This is called tertiary software, as opposed to primary software, which is more concerned with the delivery of content. Tertiary software is focused on the re-use of material, which has been produced by previous learners in the course of discussing or assessing their learning tasks. Aberdour and Smith agree that to obtain better usability, less emphasis is required on the interface. They are of the opinion that in the past software designers have tended to focus more on technical requirements and thus have placed the requirements of the user at the end of the list. Aberdour and Smith propose that a learner-centric approach should be taken to increase the usability of online learning.

Usable systems involve capitalising on the user's pre-existing knowledge. Similarly, effective educational material involves information for which a framework for extracting meaning is already in place. So both concepts (usability and learning

19


effectiveness) involve the role of prior learning. Attempting to learn material for which no previously established conceptual framework exists is, literally, a meaningless task. At the same time, trying to use technology without the assumed level of perquisite understanding can also lead to usability failure.

A study conducted by the usability company Frontend (2001) stated that unusable interfaces will cost eLearning companies billions in lost revenue simply because users will not be able to use their systems properly.

The American elearning market has a projected value of $11.5 billion by 2003 (Abernathy, 2001 cited Frontend, 2001), while the European market is expected to be worth $4 billion by this year (Information Society Commission, 2001 cited Frontend, 2001), however poor usability will contribute to disappointing results for many eLearning companies. There are several reasons for this one of which is that eLearning companies are excluding major user groups: in America alone 8% of the population has visual learning, cognitive, auditory or physical dexterity disabilities severe enough to affect their access to the Internet (ZDNET, 2000 cited Frontend, 2001). In Europe the figure is 11% and is expected to rise to 18% by 2020 (Thematic Research Initiative, 2000 cited Frontend, 2001).

Adaptive technology makes it possible for people to access computers and the Internet, however simply gaining access is not enough. People with disabilities can only use the potential of the Internet if information publishers incorporate design features that make their sites more accessible to a wider audience. Because of the multimedia nature of the Internet, combined with the poor design of websites, many users Internet users cannot make the most of it because some of them

- Cannot see graphics due to visual impairments- Cannot hear audio due to hearing impairments- Use slow connections and modems of older equipments that cannot download

large files in a sufficient time space (Arion & Tutuianu, 2003). However usability testing is not the only form of evaluation that teams can use to improve the quality of online learning. Quality assurance teams can perform several other tests, for example accessibility tests. This can ensure that colour-blind, blind, deaf and physically disabled individuals can use the online learning system, thus addressing the issues highlighted by the FrontEnd study.

2.5.1 PROBLEMS WITH LEARNING SYSTEMS

Parlangeli et al. (1999) state that a major problem when interacting with hypermedia educational applications is that the interface has to guide the student through an educational path. This means the student has to learn two things at the same time, effectively doubling the learning process: on one hand the user has to learn how to interact with the learning system while on the other hand the user has to acquire new and possibly difficult concepts.

It has been confirmed that learning is more effective when the information is presented using a multimedia application. This is supported by Kulik et al (1986) as they claim that training using computer based instruction appears to be associated with consistent time savings and in some cases students learned with it in more than 70% less time than students in a traditional classroom environment.

20


Parlangelis et al went on to conduct studies aiming at the evaluation of a distance-learning course. The results from each study have been used to determine the level of usability of the system and to give some indication of its effectiveness.

Instructors and course developers are well versed in the art of instructional design. Despite this expertise and experience, many Web-based courses suffer from weak Web design and poor usability. Often, learners can't take advantage of good instructional design because the Web environment is too problematic: content is difficult to find, course tools don't work, and navigation is inconsistent (Smulders, 2001).

2.5.2 INTERFACE DESIGN

Including options such as a help feature sometimes don’t really help the user. Help files can end up being more of a hindrance rather than supplement the user. Users can find themselves immersed in pages of text and often users do not find what they are after. Even if they do go on to successfully find the information they are after they could still misinterpret it. The addition of a help feature adds yet another set of features to a system, as a result complicating the interface by merely existing. Nielsen (1993) claims that providing help options should not be seen as an excuse to make a needlessly complex interface. It is always better if a user can operate the system without having to refer to a help system.

2.5.3 PEOPLE & TECHNOLOGY

With broadband speeds becoming more widespread, this has meant that web designers and the like have been able to make more media rich web sites i.e. incorporate more use of Macromedia’s Flash, the usage of better quality of audio and better quality sound.

How does this effect online learning usability? If designers are designing more for people with higher bandwidths this could mean a lot of online courses can incorporate high quality videos. This is all very well, and may make the course more appealing to a user and keep their attention more than them reading through screen upon screen of text however the problem is not everyone is or is able to get broadband.

Wiley and Edwards (2002) have taken another approach to this. They set the following scenario. Assume that broadband is widely available and everyone has access to it, does this solve the problems of online learning? They are of the opinion that it does not and believe that the most significant bandwidth problem in online learning has nothing to do with pushing data through pipes. The idea of “teacher bandwidth” analogizes students to data, and teachers to pipes, and formulates the problem thus: how many students can a teacher support in an online learning environment? While some distance education organizations see the Internet as an opportunity to expand their student base to hundreds of thousands of students, providing feedback and learning support for such large numbers is problematic. Traditional instructional methods were designed to support tens of students in a course, not tens of thousands.

21


2.6 USABILITY EVALUATION TECHNIQUES

Regarding usability evaluation methods, Notess (2001) states that usability testing needs additional consideration in the light of web-based learning environments.

Evaluating learning may move usability practitioners outside their comfort zone. To be effective, they need to familiarize themselves with the evaluation frameworks and methods from instructional design. Also essential are acquaintance with educational testing research, learning styles, and the rudiments of learning theory.

Furthermore Smulders (2001) proposes that with slight adjustments the usability heuristics that have been devised by Nielsen can be used to evaluate web-learning environments. Nielsen’s heuristics consist of the following: provide the user with informative feedback, create a match between the system and the real world (i.e. the system should speak the users language, and not use system orientated phrases), provide clearly marked exits, maintain a level of consistency, prevent errors, minimize the amount of information the user has to remember, provide help and documentation, and achieve aesthetic appeal of the interface.

Guidelines have previously mentioned these heuristics can be useful, however it is clear that a set of pedagogical guidelines must be created. The problem for most types of web-based training is that a set of established heuristics do not currently exist. The web design heuristics that do exist are based upon e-commerce and should be used with caution as since many assumptions about the users of e-commerce do not apply to online learners. Squires and Preece (1999) have argued that Nielsen’s (1992) software heuristics cannot be simply applied to online learning as they fail to address the specific challenges of learner centred interface design and the issue of combining usability and learning, bring this in mind Squires and Preece have proposed an adaptation of Nielsen’s existing heuristics, however this is by no means the final cut – merely a first version. The proposed set of learning software heuristics consist of the following:

- Match between designer and learner models- Navigation fidelity- Appropriate levels of learner control- Prevention of peripheral cognitive errors- Understandable and meaningful symbolic representations- Support personally significant approaches to learning- Strategies for cognitive error recognition, diagnosis and recovery- Match with curriculum

Smulders (2001) states Nielsen’s heuristics can be applied to evaluate usability in online learning, once they have been slightly adjusted.

22


2.6.1 EXPERT VS USER BASED EVALUATIONS

The term evaluation generally refers to the process of gathering data about the usability of a design or product by a specified group of users for a particular activity within a specified environment or work context (Preece, 1994). The main goal of an interface evaluation is to discover any usability problems. A usability problem may be defined as anything that interferes with the user’s ability to efficiently and effectively complete tasks (Karat et all, 1992).Crowther et al (2004) state that usability has two primary approaches: expert analysis and end-user testing. A usability professional conducts the expert analysis, identifying problems with an application based on usability guidelines or heuristics. End-user testing involves end-user representatives using the application at any stage of the development and giving feedback. Nevertheless this does not mean that normal people cannot be used for evaluations such as heuristic evaluations, but Karoulis & Pombortis (2003) are of the view that normal evaluators do not perform well and miss out many problems and expert would not.

2.6.2 HEURISTIC EVALUATION

Heuristic evaluation is a method for finding usability problems in a user interface design by having a small set of evaluators examine the interface and judge its compliance with recognized usability principles (Nielsen, 1992).

Jacob Nielsen and Rolf Molich started their research in 1988 and in 1990 they presented the heuristic evaluation. (Karoulis & Pombortis, 2003). These heuristic rules were derived from studies, criteria lists, on-field observations and prior experience. The main point is to evaluate the usability of the interface. Based on the ISO principles about usability, Nielsen (1994) stated the following heuristics, which have been modified by Karoulis & Pombortis (2003)

1. Simple and natural dialog and minimalistic design.2. Visibility of the system – provide feedback.3. Speak the user’s language: match between the system and the real

world.4. Minimize the users’ cognitive load: recognition rather than recall.5. Consistency and standards6. Flexibility and efficiency of use i.e. provide shortcuts7. Support the users’ control and freedom8. Prevent errors9. Help users recognize, diagnose and recover from errors10. Help and documentation

However this is not the only set of heuristics that have been devised to evaluate an interface, as UserDesign.com states, the validity of Nielsen’s heuristics have been questioned, and other guidelines do exist.

Non-expert users could find many usability problems when performing a heuristic evaluation, and using the simplified thinking aloud test could reveal many other problems.

23


Simeral & Branaghan (2000) claim that the main strength of using a heuristic evaluation is that they are quick to use, while the costs are relatively low. Bearing this in mind the heuristic evaluation is not without its shortcomings. Smieral & Branagham (2000) highlight that one of the main weaknesses of this method is that it can produce false positives. An example of a false positive is a minor bug that does not negatively impact the user performance or the users’ perception of product quality. The danger that this brings up is that developers may spend valuable time addressing problems that do not impact user performance or satisfaction at the expense of those that do.

A second weakness is that heuristic evaluation does not approximate the conditions under which real users would use the system, as a opposed to an evaluation such as the cognitive walkthrough. Heuristic evaluation does not take place in the context of a real user; therefore the results obtained my not be as reliable as using proper users in the evaluation process. Finally, Simeral & Branagham (2000) are of the impression that heuristics themselves are not helpful in detecting bugs. They state in their experiment that most of the evaluators relied on common sense and previous experience and one of the bugs could not even fit under the list of heuristic. Katner et al (1997) oppose this as they claim that having a heuristic evaluation undertaken by experts is a better way of finding problems within a system. Karoulis & Pombortis (2003) agree to this as they state expert evaluators are more likely to find more problems as opposed to novice evaluators. Another advantage of using experts is that once problems have been identified, the user testing can be set so that is specifically focuses on the problems found in the heuristic evaluation.

2.6.3 THINKING ALOUD PROTOCOL

The thinking aloud test involves having one user use the system one at a time while being asked to “think out aloud”. By doing this the observer will be able to determine not only what they are doing with the interface, but also why they are doing it. Using this method of usability testing is a cheap way of getting a lot good feedback during testing.Having this insight into the users thoughts can also help pinpoint interface elements that cause misunderstandings; as a result they will be able to be redesigned.

Observation is the easiest way to assess usability as it simply involves visiting more than one user, while doing as little as possible so not to disturb them while they work. Many aspects of usability can be studied by simply asking the users of the system. This is especially true for issues relating to the users’ subjective satisfaction and possible anxieties, which are hard to measure objectively. Using the questionnaire/interview approach is also useful to ascertain how users use the system, and what features they like/dislike (Nielsen 1993).

24


2.7 DISCUSSION

This chapter introduced the issues usability presents in computer systems, online learning and which methods can be used to evaluate usability. The literature that has been found has gone to show how important usability is now considered. Companies such as Microsoft now have usability engineers involved when creating new pieces of software (Poltrock et al 2003). System usability literature has shown that there are different opinions on how to address usability issues users encounter. The study that was conducted by FrontEnd is a clear indication that usability isn’t given much considerations when an online learning package is created, however there are authors such as Preece, Squires, Mayes and Fowler that have touched upon usability in an online learning environment. Having realised usability is a problem in online learning is one thing, but the next problem is how to measure the usability of online learning, after all it is only after performing usability evaluations problems will be highlighted. This is where the literature review identified gaps in this field, for example there is no concrete version of online learning heuristics. People will happily use the heuristics that have been proposed by Nielsen, or authors such as Smulders will slightly adapt them.

Once a list of heuristics specific to online learning have been created and standardized, this will make the evaluation of online learning systems easier as evaluators will all be judging by the same attributes. Without having a proper set of heuristics to adhere to, usability evaluations that have already been conducted may present misleading results.

25


CHAPTER REFERENCES

Aberdour, M., & Smith, R. (2004). Usability in e-learning. Epic Group PLC White paper 1-31.

Bentley, T. (1991) Systems Usability. Financial Management, 69(5) 12

Crowther, S. M., Keller, C. C., & Waddoups, L., G. (2004). Improving the quality and effectiveness of computer-mediated instruction through usability evaluations. British Journal of Educational Technology, 3(15), 289-303.

Dillon, A. (2001). Beyond Usability: process, outcome and affect in human computer interactions. Canadian Journal of Information Science, 26(4), 57-69.

FrontEnd.com. (2001). Why people can’t use eLearning: What the eLearning industry needs to learn about usability. 1-16.

Gruidin, J. (1992). Utility and Usability: research issues and development contexts. Interacting with Computers 4(2) 209 – 217

Neil, D. (2003). Good Design Pays off – IT Week accessed 3 May 2004 from http://www.itweek.co.uk/Features/1141003

Jones, A., Scanlon, E., Tosunoglu, C., Morris, E., Ross, S., Butcher, P., & Greenberg, J., (1999). Contexts for evaluating educational software. Interacting with Computers, 11 499-516.

Jorgensen, A. H., and Sauer, A. (1990). The Personal touch: A study of users’ customisation practice. Proceedings of Interact’90, UK, 549-554.

Karat, C., Campbell, R., & Fiegel, T. (1992). Comparison of empirical testing and walkthrough methods in user interface evaluation. Proceedings of ACM CHI’92 Conference, USA, 397-404.

Karoulis, A., Anastasia., & Pombortis, A. (2002) On Expert-Based evaluation of the Usability and the Learnability of Web-Based Open and Distance Learning Environments. Workshop on Computer Science and Information Technologies CSIT, Greece, 1-6.

Kulik, C.C., Kulik, J.A., & Shwalb, B.J. (1986). The effectiveness of computer-based adult education: a meta-analysis. Journal of Educational Computing Research, 2, 235–252.

Microsoft. (2002) Usability in software Design accessed 12 March 2004 from http://msdn.microsoft.com/library/default.asp?URL=/library/en-us/dnwui/html/uidesign.asp

26


Microsoft., (2000) Usability in Software Design retrieved 12 March 2004 fromhttp://msdn.microsoft.com/library/default.asp?URL=/library/en-us/dnwui/html/uidesign.asp

Nielsen, J. (1992). Finding usability problems through heuristic evaluation. Proceedings of ACM CHI’92 Conference, USA, 373-380

Nielsen J. (1993). Usability Engineering. San Diego: Academic Press

Notess, M. (2001). Usability, user experience, and learner experience, retrieved November 24 2003 from http://elearnmag.org/subpage/sub_page.cfm?section=4&list_item=2&page=1

Parlangeli, O., Marchigiani, E., & Bagara, S. (1999). Multimedia systems in distance educations: effects of usability on learning. Interacting with Computers, 12, 37-49.

Terry, J., & Fowler, J. C. (1999). Learning technology and usability: framework for understanding courseware. Interacting with Computers, 11 485-497

Usability First. (2002). What makes a website or piece of software usable? Retrieved 20february 2004 from http://www.usabilityfirst.com/intro/index.txl

Simeral, J. E., & Branagham, J. R (2000). A comparative Analysis of Heuristics and Usability Evaluation methods. 307-309.

Smulders, D. (2001, August). Web Course Usability, retrieved December 8, 2003 from http://www.learningcircuits.org/2001/aug2001/elearn.html

Wiley, A. D., & Edwards, K. E. (2002). Online self-organizing social systems: The decentralized future of online learning. 1-18.

Squires, D. (1999). Usability and Educational Software Design: Special Issue of Interacting with Computers. Interacting with Computers, 11 463-466.

Squires, D., & Preece, J. (1999). Predicting quality in educational software: Evaluating for learning, usability and the synergy between them. Interacting with Computers, 11 (5) 567-483.

Wharton, C., Rieman, J., Lewis, C., & Polson, P. (1994). The cognitive walkthrough method: A practitioner’s guide. In J. Nielsen & R. Mack (Eds.), Usability Inspection Methods (pp.105-140). New York: J Wiley.

Zaharias, P., Vassilopoulou, K., & Poulymenakou, A. (2002). Designing On-line Learning Courses: Implications for Usability. Scientific Journal on Applied Information Technology 1 (1) 1-12

27


CHAPTER 3 – METHODOLOGY

3.1 INTRODUCTION

The previous chapter discussed usability in general, websites and online learning systems, whilst also looking at usability evaluation techniques. This chapter describes the research methods adopted for the investigation section of this project. To be able to collect data in an efficient manner, the correct technique has to be used. There are many different ways in which data can be collected, depending on which type of data you require.

3.2 APPROCHES TO RESEARCH

There are several ways in which research methods can be classed. A common method is to make the distinction between qualitative and quantitative research (Myers, 1997)

3.2.1 QUANTITATIVE DATA

Quantitative research methods allow researchers to gather mass amounts of data. This data can allow them to compile facts and figures and be able to examine the relationships and associations between one set of data from another. The figures can then be used to create graphs, charts etc. A quantitative research method would be useful in a study that required statistics, which could be generated from distributing questionnaires/ quick answer interviews.

3.2.2 QUALITATIVE DATA

Myers (1997) describes quantitative research study associated between groups of collected data when compared with another. Quantitative research usually involves a form of statistical analysis based on a mathematical calculation; in contrast qualitative researchers try to understand people and cultural/social contexts. Usability methodologies could be classed as both quantitative and qualitative as they involve the observation of users and measure usability by using questionnaires.

28


3.3 RESEARCH METHODOLOGIES

The data required for this project was of a qualitative nature. The purpose of the research was to evaluate the online learning system in use within the CSA and ascertain how individuals felt using the system. Even though the data was of a qualitative nature, this did not strictly rule out using quantitative methods for data collection.

A research method is a strategy of inquiry which moves from the underlying philosophical assumptions to research design and data collection (Myers, 1997). The choice of research method influences the way in which the researcher collects data. Specific research methods also imply different skills, assumptions and research practices.

The research methods that will be discussed in this chapter are:- Ethnographic study- Case Study- Action Research

3.3.1 ETHNOGRAPHIC STUDY

Social and cultural anthropologists wishing to study some aspect of society or culture in depth originally developed an ethnographic approach to research. Ethnographers immerse themselves in the lives of the people they study (Lewis 1985, p. 380) and seek to place the phenomena studied in their social and cultural context.

The ethnographic approach is now no longer confined to anthropological studies after the pioneering work carried by Wynn (1979), Suchman (1987) and Zuboff (1988). Ethnography has now become more widely used in the study of information systems in organizations, from the study of the development of information systems (Hughes et. al, 1992; Orlikowski, 1991; Preston 1991) to study the aspects of information technology management (Davies, 1991; Davies and Nielsen, 1992). Ethnography has also been discussed as a method whereby multiple perspectives can be incorporated into system design (Holzblatt and Beyer, 1993) and as a general approach to the wide range of possible studies relating to the investigation of information systems (Pettigrew, 1985). This method of research involves observing users and is often the best way to determine usability requirements. Using this technique enables the researcher to share the same experience users would go through upon using a system. This approach was not chosen, as there would not have been enough time to implement it thoroughly.

29

http://www.qual.auckland.ac.nz/ethrefs.htm#Lewis,%20I.M.

http://www.qual.auckland.ac.nz/ethrefs.htm#Lewis,%20I.M.

http://www.qual.auckland.ac.nz/ethrefs.htm#Davies,%20L.J.%20and%20Nielsen,%20S.

http://www.qual.auckland.ac.nz/ethrefs.htm#Davies,%20L.J.

http://www.qual.auckland.ac.nz/ethrefs.htm#Orlikowski,%20W.J.%20

http://www.qual.auckland.ac.nz/ethrefs.htm#Hughes,%20J.A.,%20Randall,%20D.%20and%20Shapiro,%20D.


3.3.2 CASE STUDY

The Case study is one of the most common qualitative methods used within information systems (Orlikowski and Baroudi, 1991; Alavi and Carlson, 1992). Yin (2002) describes a case study as an:

Empirical inquiry that investigates a contemporary phenomenon within its real-life context, especially when the boundaries between phenomenon and context are not clearly evident (Yin, 1984. p13).

The case study research methodology is used for studies where the experiences of the subjects are important and the context within which they operate is vital (Benbasat, 1987). Bell (1993) also states that this is a suitable methodology to use within a limited time scale for one individual researcher.

However there have been many criticism of case study as a research methodology. When a single individual carries out research, the objective selection of what will feature in the final report may lead to distortion of results. There is also the problem that generalisation is not usually possible because of the lack of representative sample and the study of a single event (Bell, 1993). However, Bassey (1981) argues that the reliability of the a case study is more important than its generalisation, if case studies are carried out systematically and critically then they are valid forms of research. Yin (1994) and Benbasat et al. (1987) are advocates of case study research, whereas Walsham (1993) is an advocate of interpretive in-depth case study research.

The research project did not aim to provide a generalisation of the thesis question, more an insight into area of which there has been little study performed. Thus case study was the preferred methodology.

3.3.3 ACTION RESEARCH

There are numerous definitions of action research, however one of the most widely cited is that of Rapoport’s, who defines action research in the following way:

Action research aims to contribute both to the practical concerns of people in an immediate problematic situation and to the goals of social science by joint collaboration within a mutually acceptable ethical framework (Rapoport, 1970, p. 499).

Action research is a practical, problem solving approach to research, which is carried out over long periods of time (Bell, 1992). This research methodology has been accepted as a valid research method in applied fields such as organization development and education. This technique is useful for projects that require specific knowledge for a specific problem. It is often used as a part of a problem solving strategy along side research (Silverman, 1993). An important feature of

30

http://www.qual.auckland.ac.nz/action.htm#Rapoport,%20R.N.

http://www.qual.auckland.ac.nz/case.htm#Walsham,%20G.%20Interpreting%20Information%20S

http://www.qual.auckland.ac.nz/case.htm#Benbasat,%20I.,%20Goldstein,%20D.K.%20and%20Mead,%20M.

http://www.qual.auckland.ac.nz/case.htm#Yin,%20R.K.


action research is that it is an on going process. Even when a project is over, participants should be continually reviewing and improving practices.

Until recently action research has been largely ignored in the domain of Information Systems apart from one or two notable exceptions (Checkland, 1991). Action research was not deemed a suitable approach for this project; given that there was a limited time scale and no definite hypothesis to be tested it was decided not to implement this research methodology.

3.4 SECONDARY RESOURCES

The main sources of secondary information were derived from a range of books and journal articles obtained from sources such as the Napier University Learning and Information Services (NULIS). A literature review was conducted to provide the theory, which the analysis findings were based upon. The secondary sources were:

- NULIS Catalogue- ACM- Citceer- Internet Search Engines

3.5 SAMPLING

Sampling is a process that involves the selection of some, but not all, the members of the larger population. Using a representative sample means that a survey’s results can be generalised, however there is the possibility of some sort of error or bias when selecting samples (Fink, 1995).There are a number of ways to conduct sampling (Bell, 1992). One method is a random is the random sampling procedure, which involves the random selection of subjects from the population under study.

Stratified sampling provides a more representative selection of the general population. This method involves random sampling within restricted categories of the population in order to try and represent subgroups within the population (Bell, 1992).

31


3.6 DATA COLLECTION

The data required for the project is mainly of a qualitative nature. The main purpose of the research project is to investigate the usability of the online learning system that is currently in use within the Common Services Agency (CSA), bearing this in mind some of the methods, which are used to collect data, are usability evaluations.

3.6.1 QUESTIONNAIRES

Questionnaires are a useful way of collecting large amounts of data quickly and cheaply. They can allow collection of both qualitative and quantitative data. A main advantage of using questionnaires in usability research is that it gives direct feedback from the point of view of the user. Other advantages of using questionnaires include:

- Standardised data. The responses are gathered in a standardised way, unlike interviews

- Information can be collected from a large group of people, which would prove quicker than interviewing everyone.

Using the questionnaire approach is not without its disadvantages, as mentioned above they are a good way to acquire data from large groups of people however this doesn’t always work out in practices as returns from questionnaires are usually perceived to be low. People returning questionnaires may also be inclined to answer superficially if a questionnaire consists of too many questions.

3.6.2 HEURISTIC EVALUATION

Heuristic evaluation is a discount usability engineering method for quick, cheap, and easy evaluation of a user interface design. Heuristic evaluations are one of the most popular usability inspection methods used. It is a technique used for finding usability problems with a user interface. A small number of evaluators (typically 3 to 5) separately inspect a user interface by applying a set of heuristics or guidelines. This usability inspection method is one of the most popular used. The heuristics that have been designed by Nielsen were used in the project. This involves going through one of the online courses and measuring each screen to the heuristics that have been proposed by Nielsen. To put this methodology into use, evaluators were issued with a checklist. The checklist contains a section for each heuristic; each heuristic is given a rating based upon usability problems.

3.6.3 COGNITIVE WALKTHROUGH

Cognitive walkthrough was originally proposed and later revised by Polson et al (1992) as en attempt to introduce psychological theory into the informal and subjective walkthrough technique (Polson et al, 1992 cited by Dix et al, 2004).This technique for evaluating user interfaces pays special attention to how well the interface supports exploratory learning i.e. first time use without formal training (Rieman et al, 1995). This usability methodology can be used at any stage of design using a prototype, a conceptual design document or even the final product.

Based on a user’s goals, a group of evaluators step through tasks, evaluating each step how difficult it is for the user to identify and operate the interface element

32


most relevant to their current sub goal and how clearly the system provides feedback to that action. Cognitive walkthroughs take into consideration the user’s through process that contribute to decision making, such as memory load and ability to reason (Usability First)

For example, logging into the CSA learning website can be broken down into several levels of tasks. At a general level, it requires the user to open up the browser window and type in the address of the learning site. If they cannot remember this they would then be required to search for it via an Internet search engine or find the specific link via the company intranet.

This type of approach is intended especially to help understand the usability of a system for the first-time use or for infrequent users. For the cognitive walkthrough approach to be used in the project a list of steps had to be created, and then presented to users of the system. The evaluation process then involves observing the users walk through the steps, assistance can only be provided should the user ask for any.

33


The table below is specific to usability techniques, and shows which ones could have been used in the project. The advantages of each are listed, which can help show why certain techniques were used.

Name Advantages DisadvantagesThink aloud user testing

Very close approximation to actual individual usage

Uses less users (less expensive)

Think aloud instruction rarely works well and is unnatural for the user

Focus Groups

Produces a large quantity of possible design choices and new features

May improve customer relations

Is designed for a marketing tool

Is pseudo-scientific and is not a controlled environment

Does not test user's actual interaction with the software

Records what user's think they want, not necessarily what they would actually use.

Heuristic Walkthrough

Uses experts Gives multiple reviewers

common rules to site for justification of reviews

Reasonably fast

The validity of Nielsen's guidelines have been questioned and alternative guidelines exist

Cognitive Walkthrough

Puts the focus on the user May focus on known

problem areas

Recognition of user goals

May be tedious Tries to make the

designer the user Inherent bias because

of task selection

Does not cover entire problem space

Table 3.1 - Comparison of Usability Evaluation MethodsSource: UserDesign.com

34


3.7 DATA ANALYSIS

The purpose of collecting and analysing the data gathered was to provide a better understanding and a more accurate explanation regarding the project subject matter. The main question was to determine the usability of the online learning package in use within the CSA. The data was gathered by using questionnaires, take users through walk through and undertake a heuristic evaluation of the online learning system.

3.8 RELIABILITY & VALIDITY

Reliability and validity are important measures in research and help to ensure the accuracy of findings and results. Peräkylä (1997) states, "the issues of reliability and validity are important, because in them the objectivity of research is at stake". Enhancing objectivity ensures the accuracy of the results and findings from research.

3.8.1 RELIABILITY

Reliability is the extent to which a procedure will produce the same results under the constant conditions (Bell, 1993; Kirk & Miller 1986). In the case of this study, the reliability of the research results indicated whether or not the same findings would occur if the study was repeated in the same manner. Benbaset et al (1987) state that a clear description of the data sources and the manner in which they contribute to the overall findings of a study is an important aspect to the reliability and validity of the results.

With regard to the results from the questionnaire, there were issues of reliability that arose in the view that some people had seemed to fill in the same answer to each of the questions in the questionnaire. One of the questions in the questionnaire was deliberately included to test the validity of answers.

3.8.2 VALIDITY

Validity describes whether an item measures or describes what it is supposed to measure or describe (Bell, 1993). It is much more complex than reliability and there are many sub-divisions to which researchers can investigate in attempts at ensuring validity of their results. Bell (1992) states that researches involved in smaller projects without complex testing or measurement need not investigate the concept of validity too in thoroughly but should examine results and methods critically. Construct validity is central to the overall validity of research (Peräkylä, 1997). Construct validity is concerned with the relationship between a theoretical model and the observations made by the researcher.To increase validity and to ensure accuracy, follow up emails and phone calls were used to discuss and clarify topics of discussion.

35


3.9 REVIEW OF METHODOLOGY

During the implementation of the methodologies, there were some problems that occurred. The main difficulty being that the person in charge of the online learning, Emma Talbot had left the CSA, unknown to the researcher until a month had passed. This meant it was difficult to ascertain who was currently undertaking training via the online learning system. To do this a meeting with Emma’s replacement, Karin Gallacher had to be arranged.

Once this was done all staff that have been involved in use of the online learning system were emailed with the questionnaire and people were also asked to volunteer for the cognitive walkthrough. It was decided that the cognitive walkthrough would start from the user logging into the online learning system and access a course. The heuristic evaluation was undertaken three expert users. This was easier than the cognitive walkthrough to arrange, however there were problems logging into the system when colleagues tried to perform their heuristic evaluations. This mean the heuristic evaluation had to be put back to a later date.

The next chapter gives an overview of the CSA, goes into more detail about the online learning system in use and details the results of the evaluations carried out.

36


CHAPTER REFERENCES

Bell, J. (1993). Doing your research project: a guide for first-time researchers in educational and social science (2nd ed.). Buckingham: Open University Press

Benbasat, I. Goldstein, D.K. Mead, M. (1987). The Case Research Strategy in Studies of Information Systems. MIS Quarterly (11:3) pp. 369-386.

Checkland, P. Nissen, H-E. Klein, R, A. (1991). From framework through experience to learning: the essential nature of action research in Information Systems Research: Contemporary Approaches and Emergent Traditions, North-Holland, Amsterdam, pp. 397-403.

Davies, L.J. (1991) Researching the Organisational Culture Contexts of Information Systems Strategy, Information Systems Research in the 1990's. Amsterdam, Elsevier/North Holland.

Davies, L.J. Nielsen, S. (1992) An Ethnographic Study of Configuration Management and Documentation Practices in an Information Technology Centre. The Impact of Computer Supported Technology on Information Systems Development. Amsterdam, Elsevier/North Holland.

Dix, A., Finlay, J., Abowd, G & Beale, R. (2004) Human-Computer Interaction.

Holzblatt, K. and Beyer, H. "Making Customer-Centred Design Work for Teams", Communications of the ACM, (36:10), 1993, pp. 93-103.

Hughes, J.A. Randall, D. Shapiro, D. (1992). Faltering from Ethnography to Design, CSCW '92. ACM Conference on Computer-Supported Cooperative Work: Sharing Perspectives. New York: ACM Press, pp. 115-123.

Kirk, J., & Miller, M. L. (1986). Reliability and validity in qualitative research. London: Sage.

Lewis, I.M. Social Anthropology in Perspective. Cambridge University Press, Cambridge, 1985.

Myers, M.D. (1997). Qualitative Research in Information Systems. London: Pitman

Myers, M. (1997), Qualitative Research in Information Systems, retrieved 20 April 2004 from http://www.qual.auckland.ac.nz/

Orlikowski, W.J. (1991). Integrated Information Environment or Matrix of Control? The Contradictory Implications of Information Technology. Accounting, Management and Information Technologies, (1:1), pp. 9-42.

37


Peräkylä, A. (1997). Reliability and validity in research based on tapes and transcripts. In D. Silverman (Ed.), Qualitative research: theory, method and practice. (pp. 201-220). London: Sage.

Pettigrew, A.M. "Contextualist Research and the Study of Organizational Change Processes", in Mumford, E., Hirschheim, R., Fitzgerald, G. and Wood-Harper, A.T. (eds.), Research Methods in Informations. Amsterdam, North Holland, 1985, pp. 53-78.

P. Polso, C.Lewis, J. Rieman and C. Wharton. Cognitive walkthroughs: A method for theory-based evaluation of user interfaces. International Journal of Man-Machine Studies, 36: 741-773, 1992.

Preston, A.M. "The 'Problem' in and of Management Information Systems", Accounting, Management and Information Technologies, (1:1), 1991, pp. 43-69.

Rapoport, R.N. "Three Dilemmas in Action Research," Human Relations, (23:4), 1970, pp. 499-513.

Suchman, L. Plans and Situated Actions: The Problem of Human-Machine Communication. Cambridge, Cambridge University Press, 1987.

Usability First (2002), Methods: Cognitive Walkthroughs, retrieved 20 April 2004 from http://www.usabilityfirst.com/methods/cogwalk.txl

UserDesign.com (1998), Comparison of Usability Evaluation Methods, retrieved 5 March 2004 from http://www.userdesign.com/usability_uem.html

Wynn, E. Office conversation as an Information Medium. Unpublished PhD thesis, University of California, Berkeley, 1979.

Yin, R. K. Case Study Research, Design and Methods, 3rd ed. Newbury Park, Sage Publications, 2002.

Zuboff, S. In the Age of the Smart Machine. New York, Basic Books, 1988.

38


CHAPTER 4 - CASE STUDY

4.1 INTRODUCTION

This chapter gives an overview of the CSA, and details more information about the online learning system. The study was vital to establish whether the users of the online learning system could actually use the system without facing any major difficulties.

4.2 COMPANY BACKGROUND

The Common Services Agency (CSA) plays an active role in NHS Scotland by providing clinical support and advice for Scotland’s health and patient care.

The CSA is made up of several departments located over Scotland which specialise in individual fields The CSA focus on a range of services including:

- Blood transfusion services; tissue and bone banking services; and diagnostic products

- Screening programs and national health services- National surveillance of communicable diseases, environmental health hazards

and public health.- Health statistics, analysis and information to inform decision-making.- Value for money contracting and specialist commercial and technical services.- Specialist legal services- Family health service payments, patient registration and monitoring of clinical

standards.- Deterring, detecting and investigating fraud.- Conference facilities, event organizing and national secretariat services

4.3 THE LEARNING CENTRE

In the CSA business plan 2001-2002, a proposal was put forward to create a Learning Resource Centre at Trinity Park House. With the current development of the Agency-wide Intranet, it has been envisaged that there may be potential for further direct access to the Learning Resource Centre from other Agency sites across Edinburgh and Scotland.

The Learning Centre is especially important for individuals who have traditionally not had regular access to learning and individual development opportunities. Services and opportunities for learning include online learning, flexible learning workbooks and reference materials around:

- Corporate Induction- Health and safety and health promotion- Numeracy and Literacy development- SVQ related topic areas such as customer service and administration- Communication skills

39


- Computer-based information, communication and administration- Personal Development through Learning to learn, Career development and

Personal effectiveness- Supervisory management training- NHSS updates and general knowledge- Community Educational and Vocational subjects including languages, and

lifestyle related subjects

For example, for those staff who currently have no access to computers or who work with computers in a very limited way within their current jobs, information and communication training could be made available through a combination of online learning and group work through an accredited programme such as the European Computer Driving Licence.

4.4 THE ONLINE LEARNING SYSTEMThe online learning system has been introduced into the CSA at the start of 2003. It has been has been developed by NetG learning. Users of the learning system can access the log in page from the CSA intranet or via https://www.netglearning.com/logon_SSL.aspx. Once logged in they are presented with the following screen:

Figure 4.1:Welcome screen user is presented with upon successful log-in.

After a successful login, users are able to choose from a range of courses they can study. This can cover subjects such as IT related topics i.e. Java Programming through to people management courses. The learning system also has a feature “text to speech accessibility” feature. This requires the user to download software, which will allow them to use the audio versions of the courses. Users are able to log in from any

40


computer that has an Internet connection and a web browser; however after speaking to the person in charge of online learning development, Karin Gallacher; she stated that they were working on stopping people from logging in from their own offices. The reason being that if people are to log into the system while in their office, they will still be partial to every day distractions such as phone calls, other colleagues etc.

Administrators of the system also have additional options once they log in. They have the ability to check who has been using what course, for how long they have been using it and to check to see how many courses they have passed. The administrator of the system also has the ability to modify the page layout.

Once users have chosen a course the system remembers this in case they cannot finish it in one sitting. Once they log back into the system they have the choice of resuming where they left of or choose another course.

4.5 EVALUATION OF THE SYSTEMAn evaluation of the system was undertaken with the help of the staff at the CSA. The aim of the evaluation was to assess the usability of the online learning system.

To gather preliminary data for this, questionnaires (see Appendix 1) were handed out to staff that have previous experience of using the online learning system.The survey was conducted using a stratified sampling method, which allowed a greater variation of the population to be canvassed (from administration staff through to senior management).

Completed questionnaires were received from 17 members of staff (from a total of 30 handed out) who were split across various departments.

To further assess the usability of the system, a cognitive walkthrough was devised (see Appendix 2) and the system was also put through a heuristic walkthrough (see Appendix 3)

The questionnaire results were coded and tabulated (see Appendix 4); it was based on a CSUQ (Computer System Usability Questionnaire). Using this type of questionnaire allowed an insight into how usable the users perceived the online learning system to be. Special attention was given to the following questions:

i. Was it easy to learn this system?ii. The system gives error messages that clearly tell me how to fix

problems?iii. The organisation of the information on the system screens clear?iv. It is easy to find the information needed?v. I do not have to remember specific commands to use the system?

41


4.5.1 QUESTIONNAIRE RESULTSThe response to the questionnaire in general was quite one sided in certain areas, however there were some interesting points that were raised from the answers. From the returned questionnaires, it appears that the majority of the users found the online learning system easy to use (see figure 4.2).

Figure 4.2: Response to question regarding the ease of learning how to use the system

It appears the ease of use of the system is highlighted by the fact that the users do not have to remember any specific commands (see figure 4.3) when using a training course. 9 of the 17 users who completed questionnaires also stated that it was easy to find information, and that the information presented was in a clear and concise way.

Figure 4.3: response to question on having to remember specific commands to use the system.

42


There was a mixed view of the system when it came to the system providing effective error messages. 10 out of the 17 people questioned disagreed that the system gave error messages that clearly instructed them how to fix their problem.

Regarding the information that was provided by the system, the general consensus was that the organisation of the information was clear, only one of the users questioned begged to differ. The results were similar when users were questioned regarding the ease of finding particular information. (see Appendix 5).

The final question – The system gives error messages that clearly tell me how to fix problems? Received mixed results (see graph 4.4). This identified a possible problem the users had with the system, therefore could be explored further in the cognitive walkthrough evaluation.

Figure 4.4: Response to question regarding the system giving error messages

From the comments that were on some of the returned questionnaires it seems apparent that the users are not happy about the lack of having a tutor, as some users mentioned they did not really know who to ask when having problems with the actual learning content.

43


To ensure that the data from the questionnaires was reliable, a question regarding the provision of shortcuts was added at the end. It was anticipated that all users questioned would disagree or strongly disagree regarding this question. Those who agreed that the online learning system did provide users with shortcuts were found to have answered the same for each question on the questionnaire.

4.5.2 HEURISTIC EVALUATION RESULTSThree expert users undertook the heuristic evaluation, as the literature in chapter 2 has stated, there are no set heuristics that can be used to evaluate the usability of online learning. There is a debate regarding which set to use, some authors will happily use those already defined by Nielsen. The heuristics used in this evaluation were based on the ones devised by Reeves et al (see Appendix 3).

The evaluators were instructed to browse through the system (no specific scenarios were presented). The results of the evaluation were stated under each of the heuristics:

VISIBILITY OF SYSTEM STATUSThe overall loading time of a course was found to be relatively slow, even though the course was tested on a broadband connection. Whilst loading, it was also noted that the system did not incorporate a status bar to indicate the loading progress. This was found to rate as a low usability problem.

MATCH BETWEEN SYSTEM & REAL WORLDThe majority of the evaluators found there were no usability flaws under this heuristic. However one did note that the online learning system adopts a PC/Windows based look. This could prove problematic for those who are not computer literate.

ERROR RECOVERY & EXITINGEach of the courses tested was found to have an exit option on each of the pages regardless what section of the course the user was currently in. Regarding the course materials, the users do not have the ability to recover from errors they make in certain contexts, for example if a user tried to close the text box, a message is displayed telling the user this part cannot be closed down. If a user is undertaking an assessment and they answer something wrong they do have the chance to change their answer. When users are asked to participate in clicking on certain parts of the interface, should they make a mistake they do not have the chance to amend their action. A dialogue box appears indicating they have clicked on the wrong part of the interface.

CONSISTENCY & STANDARDSA major usability problem that was identified by all of the evaluators was that the same options are not present for each of the courses, for example one of the courses that was accessed in the evaluation initially asked the user which version of the course they would like to use – a high bandwidth version or a low bandwidth version that did not incorporate audio. Different courses were also found to have completely different user interfaces (see Appendix 6)

44


NAVIGATION SUPPORTThe navigation of the courses was found to be very simple to use. The only problem highlighted was the fact that it was not consistent throughout all of the courses. As a result of this the navigation support differed depending on which course was being used.

AESTHETICSThe welcome page was found to have a cosmetic usability problem, as the font used was not very readable. One evaluator stated that the inclusion of more graphics would have helped, as the content seems very text heavy.

HELP & DOCUMENTATIONThe way the help files are represented are again different depending on which course is in use. Evaluators agreed that the help system can provide a new user with a good starting point should they get lost. The IT courses help pages are displayed in standard HTML format. In other courses the help is an integrated part of the system. This even help system even recognises first time users and offers an overview of how to use the navigational elements as well as detail how the course works.

INTERACTIVITYInteractivity was seen to be poor in the IT based courses. Frequently users were simply required to navigate through screens of text with the exclusion of some of the assessments, which required the user to click on certain areas of the interface. Some of the non-IT related courses offered a higher level of interactivity, as they provided a virtual workplace environment in which users can put what they have learned into practise.

MESSAGE DESIGNMessage design was another heuristic that was different from course to course due to different interface designs being used. With the exception of the IT based courses all evaluators agreed that the message design was prominent.

INSTRUCTIONAL ASSESSMENTAll of the courses tested in the evaluation had a self-assessment feature. No problems were encountered measuring this heuristic.

FEEDBACKThere were no usability problems encountered in the feedback section. Self-assessment sections were noted to have good feedback sections as a comprehensive reasoning to each answer is provided should the user answer a question wrong.

45


4.5.3 COGNITIVE WALKTHROUGH RESULTSThe last method used to assess the usability of the online learning system was using the cognitive walkthrough approach. The purpose of this evaluation was to assess and observe current users navigating through the system, however in conjunction to this new users were also invited to go through the walkthrough, as Rieman et al (1995) state that using cognitive walkthrough is an evaluation technique that can be used on first time users of a system that have had no formal training.

Initial research from the questionnaire had indicated that users have found the system easy to use; however this was also a chance to assess the validity of the response to that part of the questionnaire.

The participants of the cognitive walkthrough were of various different ages, education level and computer knowledge to provide a wide a selection as possible.

Each of the participants was handed out a three-page handout prior to the walkthrough. The hand out consisted of initial general questions regarding online learning, followed by the steps for the walk through. The last section of the hand out asked questions on the system that had just been used. Users were tested one at a time, in a separate room. It was imperative that the test was conducted in this fashion as it denied the other users a chance to view the system prior to using it.

All of the users were encouraged to think aloud when going through the walkthrough. Prior to beginning the test, users were informed that they would be observed as they went through the steps and notes would be made on their level of progress. There was no set time limit in which to complete the list of tasks, however if specific tasks are taking users a long time to complete this may underline a usability flaw.

The evaluation procedure involved users navigating to the website in which the online learning system resides. Once logged into the system, the walkthrough consisted of- Accessing the course list- Choosing and loading a particular course- Changing user preferences- Load a specific topic- Navigate through to the end of a topic- Return to the main screen

46


4.5.4 WALKTHROUGH OBSERVATIONS

Eight participants undertook the evaluation, each with varying levels of computer knowledge. Out of the eight only one user took longer than 5 minutes to go through the system, however this was not due to a lack of computer knowledge. The computer the learning system was being tested on did not have a recent version of the java runtime environment. It was however noted that the learning system did not inform the user. Another user was seen to have difficulty locating the course list. When asked about this the user commented that the font used on the button is not a very clear one to make out. A couple of the users were also noted to have a few problems distinguishing the menu bar from the rest of the system due to the interface and menu consisting of the same colour. The presentation of the menu bar also posed another problem; when users were instructed to change the font size at the start of the walkthrough it was noted that the only text that actually changed size was that displayed in the information box, users commented that they expected all of the text on a page would change size.

Navigation wise most of the users managed to complete all relatively quickly, one of the users in the expert walkthrough did however note that they would usually expect navigation controls to be at the bottom of the interface rather than the top. One of the participants in the walkthrough evaluation had very little of using a computer in general, thus they did tend to struggle substantially more than the other users.

It is also possible that mistakes that were made may have been down to users being nervous about being observed while using the system.

47


CHAPTER REFERENCES

Reeves, C. T., Benson, L., Elliot, D., Grant, M., & Holschuh, D. (2002). Edmedia. 1-16.

Rieman, J., Franzke, M., & Redmiles, D. (1995) Usability evaluation with the cognitive walkthrough. Conference on Human Factors in Computing Systems, USA, 387-388.

48


CHAPTER 5 – ANALYSIS OF FINDINGS

5.1 INTRODUCTIONThis chapter provides discussion of the findings from the evaluation of the learning system. The chapter also discussed the use of usability techniques that were used to assess the online learning system. Comparisons to the literature found are also made.

5.2 ANALYSIS OF ONLINE LEARNING AT THE CSAThe questionnaires that were distributed show that the majority of users were content using the NetG learning system. Overall, the results of the questionnaire would indicate that the staff at the CSA are satisfied with the online learning system. Some did mention that they would prefer it if there was an actual person they could refer to if they were having problems in certain parts of the course. The system welcome page does mention you can ask online mentors questions, however no evidence of actually being able to do this was found. This is something that can inhibit the use of the system; people may use it initially but once they find out there is a lack of support they may be inclined to seek other training opportunities.

Table 5.1 discusses the findings from the heuristic evaluation.

Heuristic Problems found DiscussionVisibility of system status Slow loading of course,

with no indication of load progress.

The slow loading of the course was very surprising considering they were tested on a broadband connection. As the literature suggested, the connection speed can affect the usability of the system. These flaws would have been highlighted better had the system been checked on a 56k connection.

Match between system & real world

PC/Windows based feel. Problems to the user from this are that if they are not highly computer literate, they will continue to struggle through the course. As a result of this the likelihood of them learning anything would be relatively low. The literature did mention that trying to use technology without the

49


presumed level of perquisite understanding can lead to usability failure

Error recovery & exiting Not able to recover from errors in the self-assessment.

The system does not give the user to change answers in some sections of the self-assessment. This contradicts the view of Squires and Preece’s socio-constructivist approach, whereby learners should be able to make errors and learn from them.

Consistency & standards Courses do not maintain a similar design. Different depending on topic being studied.

Providing the user with different interfaces means that each time they choose to use a different course they must first familiarise their selves with the interface. This of course then takes away time from the learning process. According to the literature this presents the users with 2 problems: trying to learn from the course materials, whilst also trying to learn how to use the interface.

Navigation support No problems found. N/AAesthetics - Cosmetic usability

problems with the welcome page. - IT related courses are not visually pleasing

The aesthetics could be classed as a low usability problem as the design of the welcome page is not very adequate. Users found it difficult to read the button names on the welcome page. The welcome page is a section the CSA administrators can customise. As the literature states this is function is not always used in the best way when in the hands of novice users.

50


Help & documentation No problems found. N/AInteractivity Poor level of interactivity

in the It related coursesThis problem also related to consistency. IT based courses mainly involve the user clicking through a series of text based screens. The literature is of the opinion that learning is an active process that happens through direct experience. Lack of experience therefore effects the users learning.

Message design Not very clear on the IT related courses.

Again the problem is mainly on the IT based courses, messages are short and brief. In some ways this can be seen as a good thing, however it causes problems in the self-assessment function as there are not enough instructions given to the user. As a result the user would be more likely to make a mistake and click on the wrong thing.

Instructional assessment No problems found. N/AFeedback No problems found. N/A

5.1 – Discussion of heuristic evaluation findings In relation to the literature found, it seems apparent that usability of systems is being compromised at the sake of good design. Some of the courses tested in the NetG learning system are a prime example of this. Mayes and Fowlers opinions that online learning should be more task-centric rather than focus on the interface could well to lead to less usability issues arising.

51


5.3 ANALYSIS OF EVALUATION TECHNIQUES

5.3.1 HEURISTIC EVALUATIONThe implementation of the heuristic walkthrough proved to be rather problematic. As stated in the literature, a heuristic evaluation can prove to be difficult to use as at times problems can be found with a system, however they do not necessarily fit under the categories in the heuristic checklist. The literature also states that heuristic evaluations should be carried out be experts, either in the field of usability or HCI. The evaluators used in the evaluation were of a very high computer literacy (i.e. one of them being a professional web-site designer). The problem that seemed evident was that, as the evaluators were not used to doing this type of evaluation they found it hard to apply problems to certain heuristics, this could well be down to the fact that they did not understand what the actual heuristic was measuring.

It would appear that Karoulis & Pombortsis (2003) statement that novice users should not be used in heuristic evaluations appears to have a degree of truth to it. Regarding the validity of the results, the problems that were found for each heuristic were areas that were thought to be problematic.

5.3.2 COGNITIVE WALKTHROUGHThe cognitive walkthrough was the easier of the two evaluation methods to use. All it required was to consider of what the evaluator would like to see a user doing in the system i.e. navigate through different sections; see if they can find certain areas. The walkthrough provided an invaluable insight as to how participants were using the courses. Looking back on the project, it would have been better to base the walkthrough on the problem areas that would have been identified by users of the system in the usability questionnaire.

52


CHAPTER REFERENCE

Karoulis & Pombortsis (2003) – Heuristic Evaluation of Web-based ODL Programs

53


CHAPTER 6 - CONCLUSION

6.1 MAIN FINDINGS

The main objective of this research project was to determine the usability of the online learning system within the CSA. The aim was achieved using both secondary and primary information so that an extensive and detailed coverage of the topic could be obtained. A thorough review of literature regarding a range of usability issues was conducted in chapter two, this enabled the researcher to gain an understanding of what usability problems were persistent in general computer system and more specifically what usability issues lie within the use of online learning systems. Whilst covering usability, the literature review also covered evaluation methods, which can be used to assess usability. Regarding the literature of usability in online learning it seems apparent there is still a lot of work required in this area. Literature states that usability within online learning environment has been approached in different ways.Authors also appear to be divided in which way to assess the usability as different authors have proposed using different heuristics.

Methods used to collect information were described in chapter three. This was included to justify the choice of methods used during the research project, along with advantages and disadvantages of using the selected techniques. Questionnaires were used to gather initial feedback regarding the online learning system. Evaluation techniques that were put into practice to assess the usability of the online learning system were effective to a certain point, however it is clear that there is a need for a set of standardised heuristics in which usability of online learning systems can be evaluated.

The researcher is of the opinion that using different sets of heuristics to evaluate different online learning systems can present different results, as differing heuristics may focus on other areas.

6.2 RECOMDENDATIONS

- The CSA should ensure that employees that are using the online learning system are computer literate.

- NetG, the developers of the learning system must adopt a standardised approach to interface design. Each of the courses should have similar interfaces. Currently the interface changes depending on which course is being used, having different interfaces means the users have to adjust each time they use a new course.

- The methods used to evaluate the online learning usability are effective if used in the right manner. It is recommended that a set of heuristics specific to online learning are required to be created and standardised so that everyone is evaluating against the same attributes.

54


6.3 CRITICAL EVALUATION

The original aims of the research project were:- Provide an evaluation of literature regarding usability issues for general

computer systems, and more specifically online learning systems.- Identify the most appropriate method(s) that can be used to evaluate the

usability of online learning systems.- Investigate usability issues in the current online learning system in use within

the CSA. - Make recommendations regarding usability for future applications of online

learning at CSA, and consider any general implications.

The researcher believes that the aims of the research project have been answered. The findings proved to be interesting as the researcher had the initially anticipated that the system was largely flawed, however the researcher still is sceptical regarding the results of the research. It is believed that the evaluation methods that were used could have been conducted in a better manner to gain a deeper insight into the usability issues present within the CSA system i.e. the results of the questionnaires would have highlighted what aspects of the system users are having problems with. The cognitive walkthrough should then have focused on these problem areas.

Nevertheless the evaluation methods used did highlight that there are usability problems present in the system. In the eyes of a usability expert these problems could be conceived to be cosmetic usability problems e.g. those that do not require a high priority of redevelopment. From the literature regarding usability evaluations, the researcher would agree that using an evaluation method such as heuristic evaluation would be best left to usability experts. It is also acknowledged that the heuristic evaluation results may not have been recorded in the right manner. A typical record of results for a heuristic evaluation usually consists of numbers of errors found per heuristic, not a description of errors found.

If the dissertation was to be undertaken by another individual, the researcher is of the opinion that it would almost certainly produce differing results. This is largely down to the evaluation methods used. Another individual may perceive other evaluation methods more appropriate than the ones used in this project, or even choose to use different heuristics to measure the learning system by.

6.4 SELF EVALUATION

Considering the project on a whole, the actual assessment of the project had worked far better than the researcher had anticipated. There were initial problems gathering literature regarding system usability, however a small amount was found.

Retrospectively it would appear a better approach to the project would have been to tackle the project in the following manner:

1. Conduct literature review; find out what are the main issues regarding online learning and usability of computer systems.

2. Based on the findings of the literature create an online learning system

55


3. Compare and contrast to the CSA system; undertake an evaluation of the online learning system created.

This was realised mid way through the project; as a result there was not enough time to rearrange the deadlines.

The dissertation has shown how important usability is in the world of IT in general, it also indicates that in a lot of places, usability is taken for granted or compromised at the sake of better looking designs.

6.5 TENATIVE REQUIREMENTS FOR ONLINE LEARNING

For organisations thinking in using online learning as a means to provide their staff with training, it is advised they should consider the following

- To get the most out of this form of training, all users will have to be computer literate.

- The use of online learning is perhaps better used to supplement classroom-based approaches to training as from this study it appears that users are not happy about the non-presence of a tutor.

6.6 FUTURE WORK

Although the dissertation has highlighted the main points of online learning usability, there will always be room for further research.

Poor usability is not the only reason why online learning can fail. It could well be that the online learning system is instructionally unsound, or not well written enough to engage the user. An improvement of usability will mean that learners have a greater chance of successfully achieving their learning objectives via online learning.

If there was more time available, an ethnographic study could have been conducted as the literature regards this as the best way to determine usability requirements.

Usability is now becoming more prominent in the design of computer systems as companies are now starting to realise the cost of not including usability at the design stage of software.

It is hoped that the research will help to highlight how underestimated usability is in the domain of online learning, and will be useful for identifying further areas of research.

56


BIBLIOGRAPHY

Aberdour, M., & Smith, R. (2004). Usability in e learning. Epic Group PLC White paper 1-31.

Bell, J. (1993). Doing your research project: a guide for first-time researchers in educational and social science (2nd ed.). Buckingham: Open University Press

Benbasat, I. Goldstein, D.K. Mead, M. (1987). The Case Research Strategy in Studies of Information Systems. MIS Quarterly (11:3) pp. 369-386.

Bentley, T. (1991) Systems Usability. Financial Management, 69(5) 12

Bodker, S., & Buur, J. The design collaboratorium: a place for usability design. ACM Transactions on Computer-Human Interaction, 9(2), 152 – 169.

Brown, J., & Lu, J. (2001) Designing better online teaching material. Proceedings of the thirty-second SIGCSE technical symposium on Computer Science Education, USA 352 – 356.

Brown, J., & Lu, J. (2001). Designing Better Online Teaching Material. Technical Symposium on Computer Science Education, 352 – 356.

Checkland, P. Nissen, H-E. Klein, R, A. (1991). From framework through experience to learning: the essential nature of action research in Information Systems Research: Contemporary Approaches and Emergent Traditions, North-Holland, Amsterdam, pp. 397-403.

Crowther, S. M., Keller, C. C., & Waddoups, L., G. (2004). Improving the quality and effectiveness of computer-mediated instruction through usability evaluations. British Journal of Educational Technology, 3(15), 289-303.

Davies, L.J. (1991) Researching the Organisational Culture Contexts of Information Systems Strategy, Information Systems Research in the 1990's. Amsterdam, Elsevier/North Holland.

Davies, L.J. Nielsen, S. (1992) An Ethnographic Study of Configuration Management and Documentation Practices in an Information Technology Centre. The Impact of Computer Supported Technology on Information Systems Development. Amsterdam, Elsevier/North Holland.

Dillon, A. (2001). Beyond Usability: process, outcome and affect in human computer interactions. Canadian Journal of Information Science, 26(4), 57-69.

Dix, A., Finlay, J., Abowd, G & Beale, R. (2004) Human-Computer Interaction.

Frokjaer, E., Hertzum, M., & Hornbaek, K. (2000). Measuring usability: are effectiveness, efficiency, and satisfaction really correlated? Proceedings of the SIGCHI conference on Human factors in computing systems, Netherlands, 345 – 352

57


FrontEnd.com. (2001). Why people can’t use eLearning: What the eLearning industry needs to learn about usability. 1-16.

Ghaoui, C. (2003). Usability Evaluation of Online Learning Programs. London: Information Science Publishing.

Gruidin, J. (1992). Utility and Usability: research issues and development contexts. Interacting with Computers 4(2) 209 – 217

Holzblatt, K. and Beyer, H. "Making Customer-Centered Design Work for Teams", Communications of the ACM, (36:10), 1993, pp. 93-103.

Hughes, J.A. Randall, D. Shapiro, D. (1992). Faltering from Ethnography to Design, CSCW '92. ACM Conference on Computer-Supported Cooperative Work: Sharing Perspectives. New York: ACM Press, pp. 115-123.

Jones, A., Scanlon.,E., Tosunoglu, C., Morris, E., Ross, S., Butcher, P., & Greenberg, J., (1999). Contexts for evaluating educational software. Interacting with Computers, 11 499-516.

Jorgensen, A. H., and Sauer, A. (1990). The Personal touch: A study of users’ customisation practice. Proceedings of Interact’90, UK, 549-554.

Karat, C., Campbell, R., & Fiegel, T. (1992). Comparison of empirical testing and walkthrough methods in user interface evaluation. Proceedings of ACM CHI’92 Conference, USA, 397-404.

Karoulis & Pombortsis (2003) – Heuristic Evaluation of Web-based ODL Programs.

Karoulis, A., Anastasia., & Pombortis, A. (2002) On Expert-Based evaluation of the Usability and the Learnability of Web-Based Open and Distance Learning Environments. Workshop on Computer Science and Information Technologies CSIT, Greece, 1-6.

Kilby, T. (2001). The direction of web based training: a practitioner’s view. The learning Organization 8, (5)

Kirk, J., & Miller, M. L. (1986). Reliability and validity in qualitative research. London: Sage.

Kulik, C.C., Kulik, J.A., & Shwalb, B.J. (1986). The effectiveness of computer-based adult education: a meta-analysis. Journal of Educational Computing Research, 2, 235–252.

Lewis, I.M. Social Anthropology in Perspective. Cambridge University Press, Cambridge, 1985.

Marold, K., Larsen, G., Shaw, K., & Robertus, P. (1999). Usability testing for a computer skills WBT (Web based training program) Proceedings of the 1999 ACM SIGCPR conference on Computer personnel research, USA, 304 – 309.

58


Microsoft. (2002) Usability in software Design accessed 12 March 2004 from http://msdn.microsoft.com/library/default.asp?URL=/library/en-us/dnwui/html/uidesign.aspMicrosoft., (2000) Usability in Software Design retrieved 12 March 2004 from

Myers, M. (1997), Qualitative Research in Information Systems, retrieved 20 April 2004 from http://www.qual.auckland.ac.nz/

Myers, M.D. (1997). Qualitative Research in Information Systems. London: Pitman

Neil, D. (2003). Good Design pays off – IT Week accessed 3 May 2004 from http://www.itweek.co.uk/Features/1141003

Nielsen J. (1993). Usability Engineering. San Diego: Academic Press.

Nielsen, J. (1992). Finding usability problems through heuristic evaluation. Proceedings of ACM CHI’92 Conference, USA, 373-380.

Notess, M. (2001). Usability, user experience, and learner experience, retrieved November 24 2003 fromhttp://elearnmag.org/subpage/sub_page.cfm?section=4&list_item=2&page=1http://msdn.microsoft.com/library/default.asp?URL=/library/en-us/dnwui/html/uidesign.asp

Orlikowski, W.J. (1991). Integrated Information Environment or Matrix of Control? The Contradictory Implications of Information Technology. Accounting, Management and Information Technologies, (1:1), pp. 9-42.

P. Polso, C.Lewis, J. Rieman and C. Wharton. Cognitive walkthroughs: A method for theory-based evaluation of user interfaces. International Journal of Man-Machine Studies, 36: 741-773, 1992.

Parlangeli, O., Marchigiani, E., & Bagara, S. (1999). Multimedia systems in distance educations: effects of usability on learning. Interacting with Computers, 12, 37-49.

Parlangeli, O., Marchigianni, E., & Bagnara, S. (1999). Multimedia System in Distance Education: Effects on Usability, Interacting with Computers, 12, 37-49.

Peräkylä, A. (1997). Reliability and validity in research based on tapes and transcripts. In D. Silverman (Ed.), Qualitative research: theory, method and practice. (pp. 201-220). London: Sage.

Pettigrew, A.M. "Contextualist Research and the Study of Organizational Change

Processes", in Mumford, E., Hirschheim, R., Fitzgerald, G. and Wood-Harper, A.T. (eds.), Research Methods in Informations. Amsterdam, North Holland, 1985, pp. 53-78.

Preston, A.M. "The 'Problem' in and of Management Information Systems", Accounting, Management and Information Technologies, (1:1), 1991, pp. 43-69.

59


Rapoport, R.N. "Three Dilemmas in Action Research," Human Relations, (23:4), 1970, pp. 499-513.

Rieman, J., Franzke, M., & Redmiles, D. (1995) Usability evaluation with the cognitive walkthrough. Conference on Human Factors in Computing Systems, USA, 387-388.

Simeral, J. E., & Branagham, J. R (2000). A comparative Analysis of Heuristics and Usability Evaluation methods. 307-309.

Smulders, D. (2001, August). Web Course Usability, retrieved December 8, 2003 from http://www.learningcircuits.org/2001/aug2001/elearn.html

Squires, D. (1999). Usability and Educational Software Design: Special Issue of Interacting with Computers. Interacting with Computers, 11 463-466.

Squires, D., & Preece, J. (1999). Predicting quality in educational software: Evaluating for learning, usability and the synergy between them. Interacting with Computers, 11 (5) 567-483.

Suchman, L. Plans and Situated Actions: The Problem of Human-Machine Communication. Cambridge, Cambridge University Press, 1987.

Terry, J., & Fowler, J. C. (1999). Learning technology and usability: framework for understanding courseware. Interacting with Computers, 11 485-497

Usability First (2002), Methods: Cognitive Walkthroughs, retrieved 20 April 2004 from http://www.usabilityfirst.com/methods/cogwalk.txl

Usability First. (2002). What makes a website or piece of software usable? Retrieved 20february 2004 from http://www.usabilityfirst.com/intro/index.txl

UserDesign.com (1998), Comparison of Usability Evaluation Methods, retrieved 5 March 2004 from http://www.userdesign.com/usability_uem.html

Wharton, C., Rieman, J., Lewis, C., & Polson, P. (1994). The cognitive walkthrough method: A practitioner’s guide. In J. Nielsen & R. Mack (Eds.), Usability Inspection Methods (pp.105-140). New York: J Wiley.

Wiley, A. D., & Edwards, K. E. (2002). Online self-organizing social systems: The decentralized future of online learning. 1-18.

Wynn, E. Office conversation as an Information Medium. Unpublished PhD thesis, University of California, Berkeley, 1979.

Yin, R. K. Case Study Research, Design and Methods, 3rd ed. Newbury Park, Sage Publications, 2002.

60


Zaharias, P., Vassilopoulou, K., & Poulymenakou, A. (2002). Designing On-line Learning Courses: Implications for Usability. Scientific Journal on Applied Information Technology 1 (1) 1-12

Zuboff, S. In the Age of the Smart Machine. New York, Basic Books, 1988.

61


APPENDIX 1 – USABILITY QUESTIONNAIRE

Name: Date:

Position in company

Online course used

How long have you been using the Internet?

Was any training provided on how to use the online learning system?

StronglyAgree

1 2 3 4 StronglyDisagree

Overall, I am satisfied with how easy it is to use this systemIt was simple to use this systemI am able to complete my work quickly using this systemI feel comfortable using this systemIt was easy to learn to use this systemThe system gives error messages that clearly tell me how to fix problemsWhenever I make a mistake using the system, I recover easily and quicklyThe information provided with this system is clearIt is easy to find the information I neededThe information provided for the system is easy to understand The information is effective in helping me complete the tasks and scenariosThe organization of information on the system screens is clearIcons are arranged consistently throughout the interfaceThe interface of this system is pleasantI like using the interface of this systemThis system has all the functions and capabilities I expect it to haveI do not have to remember specific commands to use the systemThe system provides the usage of shortcuts

Please describe any problems encountered whilst using the online learning system Please suggest any improvements you would make to the online learning system

62


APPENDIX 2 – COGNITIVE WALKTHROUGH

ONLINE LEARNING

As the Internet is developing rapidly, there is a growing demand and increase in pressure for adopting new and original approaches to the design and delivery of education, hence the introduction of online learning as a mode of study. This is partly to do with the increasing number of learners and the limited amount of resources to meet the wide range of various needs, backgrounds, skills, ages, abilities and disabilities.

As new training methods are evolving, many companies are taking to the approach of training their staff via online learning as it can be seen to reduce training costs dramatically. Online learning also means people can study at their own pace and at their own convenience.

Name:

Level of computer experience: Beginner intermediate expert

Have you heard of online learning before?Yes no

Would you consider taking an online learning course as opposed to classroom-based training?Yes no

63


COGNITIVE WALKTHROUGH - WALKTHROUGH STEPS

Please follow the steps below. If assistance is required please ask for help.

1. Go to www.netglearning.com2. Under company type in: isdcsa3. Under username type in: emma.talbot4. Under password type in: smicer15. Click log in button6. Click on training plan button7. Expand all courses by click on the + sign, unless list is already

expanded. 8. scroll down the list until you reach MS Word 2000 Expert user9. Click on the MS Word 2000 course 10.Click on take training button11.Wait for the course to load12.Once loaded click on the menu button13.From the list choose User Preferences14.Click on font size15.Change the font size from the displayed size16.Click on Interface Themes17.Change the theme from the currently selected theme18.Click ok19.Click on Fast Track to a topic button20.From the list click on Data Tables: Linking Excel Data21.Click on start22.Use the navigation buttons at the top to go to page 523.Click on menu24.Click on start up options25.Click on exit26.Click Yes

Thank you for your time.

64


POST WALKTHROUGH QUESTIONNAIRE

How did you find using the online learning system?_______________________________________________________________________________________________________________________________________________________________________________________________________________

Was it easy to navigate through the different sections?

Yes no

Were you required to remember any specific commands while using the system?

Yes no

Having briefly used online learning, would you re-consider using it for training purposes?

Yes no

The interface is visually pleasing?

Yes no

Information displayed is short and to the point?

Yes no

It was easy to return to the initial start up screen

Yes no

65


APPENDIX 3 – HEURISTIC EVALUATION

THE HEURISTICS

1. VISIBILITY OF SYSTEM STATUS - i.e. when modules and other components of the system are downloading, is the status of the download clearly communicated?

2. MATCH BETWEEN SYSTEM & REAL WORLD - i.e. the e-learning program’s interface employs words, phrases, and concepts familiar to the learner or appropriate to the content.

3. ERROR RECOVERY & EXITING - i.e. the program allows the user to recover from input mistakes and provides clearly marked exits

4. CONSISTENCY & STANDARDS - i.e. does the program maintain an appropriate level of consistency in its design from one part of the program to another?

5. NAVIGATION SUPPORT - i.e. Does the interface speak for itself so that extensive consultation of the manual or other documentation does not interfere with the learning?

6. AESTHETICS – i.e. are font choices, colours, and sizes consistent. Does it look visually pleasing?

7. HELP & DOCUMENTATION – i.e. provides documentation that is readily accessible to the user?

8. INTERACTIVITY – i.e. does the system provide meaningful interactions with the system rather than clicking through screens of text?

9. MESSAGE DESIGN – i.e. is the most important information is placed on the screen in a way that would attract your attention?

10. INSTRUCTIONAL ASSESSMENT – i.e. does the program provide the user with opportunities for self-assessment that advance learner achievement?

11. FEEDBACK – i.e. does feedback provide the learner with information concerning his/her current level of achievement within the program?

66


HEURISTIC EVALUATION

Severity Scale Rating0 I don’t agree that this is a usability problem1 Cosmetic problem – need not fixed unless extra time is available on project2 Minor usability problem – fixing this should be given a low priority3 Major usability problem – important to fix, should be given high priority4 Usability catastrophe – Imperative this is fixed before release

1. Visibility of system status0 1 2 3 4

2. Match between system & real world0 1 2 3 4

3. Error recovery & exiting0 1 2 3 4

4. Consistency & standards0 1 2 3 4

5. Error prevention0 1 2 3 4

6. Navigation support0 1 2 3 4

7. Aesthetics0 1 2 3 4

8. Help & documentation

67


0 1 2 3 4

9. Interactivity0 1 2 3 4

10. Message design0 1 2 3 4

11. Instructional assessment0 1 2 3 4

12. Feedback0 1 2 3 4

Thank you for your time.

68

Additional comments:


APPENDIX 4 – CODING AND TABULATION

Coding Book (Usability questionnaire)Question/Variable Label

VariableName

Column Location

Value Labels

Values Missing Values

Range of values

Satisfied S 1 Strongly agreeAgreeDisagreeStrongly disagree

1234

9 1-4

Simple to use STU 2 Strongly agreeAgreeDisagreeStrongly disagree

1234

9 1-4

Complete work quickly

CWQ 3 Strongly agreeAgreeDisagreeStrongly disagree

1234

9 1-4

Comfortable using the system

CUS 4 Strongly agreeAgreeDisagreeStrongly disagree

1234

9 1-4

Easy to learn the system

EL 5 Strongly agreeAgreeDisagreeStrongly disagree

1234

9 1-4

Gives error messages

EM 6 Strongly agreeAgreeDisagreeStrongly disagree

1234

9 1-4

Recover easily RE 7 Strongly agreeAgreeDisagreeStrongly disagree

1234

9 1-4

Info provided is clear

IC 8 Strongly agreeAgreeDisagreeStrongly disagree

1234

9 1-4

Easy to find info needed

EI 9 Strongly agreeAgreeDisagreeStrongly disagree

1234

9 1-4

Info is easy to understand

IE 10 Strongly agreeAgreeDisagreeStrongly disagree

1234

9 1-4

Complete tasks and scenarios

TS 11 Strongly agreeAgreeDisagreeStrongly disagree

1234

9 1-4

Organization of info is clear

OC 12 Strongly agreeAgreeDisagreeStrongly disagree

1234

9 1-4

69


Icons are consistent

IC 13 Strongly agreeAgreeDisagreeStrongly disagree

1234

9 1-4

Interface is pleasant

IP 14 Strongly agreeAgreeDisagreeStrongly disagree

1234

9 1-4

I like the interface

LI 15 Strongly agreeAgreeDisagreeStrongly disagree

1234

9 1-4

System has all the functions expected

FE 16 Strongly agreeAgreeDisagreeStrongly disagree

1234

9 1-4

Do not have to remember commands

RC 17 Strongly agreeAgreeDisagreeStrongly disagree

1234

9 1-4

Provides shortcut keys

SC 18 Strongly agreeAgreeDisagreeStrongly disagree

1234

9 1-4

Coding Results

S STU CWQ CUS EL EM RE IC EI IE TS OC IC IP LI FE RC SC

01 2 3 2 2 2 3 2 2 3 3 2 2 2 2 2 2 2 402 4 1 4 1 2 3 3 3 2 2 4 2 1 1 1 4 1 303 3 3 4 1 1 2 1 4 1 1 1 1 1 1 1 3 1 304 4 3 4 4 3 4 4 4 3 2 4 2 1 3 2 3 2 405 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 406 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 407 2 2 2 2 2 4 2 2 2 2 2 2 2 2 2 2 2 308 2 2 3 2 2 4 4 3 3 2 3 2 2 2 2 2 2 409 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 110 1 1 1 1 1 2 2 1 1 1 2 2 2 1 1 2 2 111 4 3 9 4 9 3 3 3 3 3 3 4 2 2 3 3 2 312 1 1 1 1 1 2 2 1 1 1 1 1 2 2 1 2 2 313 2 3 2 2 2 3 4 3 3 4 3 2 2 3 3 4 4 414 4 1 4 1 2 3 3 3 2 2 4 2 1 1 1 4 1 315 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 116 4 3 4 4 3 4 4 3 3 2 4 2 1 3 2 3 2 417 2 3 3 3 2 3 2 2 3 3 2 2 2 2 2 2 2 4

70


APPENDIX 5 – TABULATION GRAPHS

Graph 4.3

Graph 4.4

71


APPENDIX 6 – LACK OF CONSISTENSY & STANDARDS

The interface a user is presented with while using an IT course

The interface of a non-IT related course.

The interface of another non IT related course.

72


APPENDIX 7 – PROJECT PLAN

73

Documents

CHAPTER 1 - OoCities · Web viewIf you consider early versions of the Microsoft MS-DOS word processors, ... ... a proposal was put forward to create a