296
COMPUTER SUPPORT FOR MULTIMEDIA CURRICULUM DESIGN Qiyun Wang

computer support for multimedia curriculum design

  • Upload
    others

  • View
    0

  • Download
    0

Embed Size (px)

Citation preview

Page 1: computer support for multimedia curriculum design

COMPUTER SUPPORT FOR

MULTIMEDIA CURRICULUM DESIGN

Qiyun Wang

Page 2: computer support for multimedia curriculum design

CIP-GEGEVENS KONINKLIJKE BIBLIOTHEEK, DEN HAAG Wang, Qiyun Computer support for multimedia curriculum design Thesis University of Twente, Enschede - With refs. - With summary in English and Dutch. ISBN 90-365-1670-0 Lay-out: Sandra Schele Press: PrintPartners Ipskamp - Enschede © Copyright, Qiyun Wang, 2001 All rights reserved. No part of this book may be produced in any form: by print, photoprint, microfilm, or any other means without written permission of the author.

Page 3: computer support for multimedia curriculum design

COMPUTER SUPPORT FOR MULTIMEDIA CURRICULUM DESIGN

PROEFSCHRIFT

ter verkrijging van de graad van doctor aan de Universiteit Twente,

op gezag van de rector magnificus, prof.dr. F.A. van Vught,

volgens besluit van het College voor Promoties in het openbaar te verdedigen

op woensdag 24 oktober 2001 om 15.00 uur

door

Qiyun Wang

geboren op 5 februari 1970 te Shandong, China

Page 4: computer support for multimedia curriculum design

Promotor : Prof.dr. J.J.H. van den Akker Assistent-promotor : Dr. N.M. Nieveen

Page 5: computer support for multimedia curriculum design

i

Table of Contents PREFACE vii

1 INTRODUCING THE CASCASE-MUCH STUDY 1

1.1 Context of the study 1 1.1.1 Curriculum development in China 1 1.1.2 Curriculum development in Shanghai 4

1.2 Origins of the study 9 1.2.1 The project of multimedia curriculum for Biology (MCB) 9 1.2.2 The CASCADE study 11

1.3 Overview of the study 12 1.3.1 Intended target users 12 1.3.2 Aims of the program 13 1.3.3 Methodology: Focus on development research 16 1.3.4 Approach of the prototyping stage 17 1.3.5 Approach of the assessment stage 19

1.4 Preview of the dissertation 20 2 MULTIMEDIA CURRICULUM AND COMPUTER SUPPORT

SYSTEMS 21

2.1 Multimedia 21 2.1.1 Definitions of media and multimedia 21 2.1.2 Media and learning 22 2.1.3 Multimedia and learning 24

2.2 Curriculum development 26 2.2.1 Concepts of curriculum 27 2.2.2 Curriculum development models 30

2.3 Multimedia curriculum development 32 2.3.1 The definition of multimedia curriculum 32 2.3.2 Multimedia curriculum modules 34 2.3.3 Multimedia curriculum development model 36

Page 6: computer support for multimedia curriculum design

ii

2.4 EPSS 37 2.4.1 The concept of EPSS 37 2.4.2 The potentials of EPSS 39 2.4.3 The design of EPSS 41

2.5 Existing computer support systems for curriculum development 43 2.5.1 CASCADE 43 2.5.2 CASCADE-SEA 44 2.5.3 Other support systems 46 2.5.4 Comparison of the support systems and implications on the

design of the study 47 2.6 Conclusions 51 3 PROTOTYPE DEVELOPMENT 53

3.1 Preliminary choices 53 3.2 Overview of the prototyping process and the prototypes 55

3.2.1 Prototyping process 55 3.2.2 Evolution of the structure and the four components 56

3.3 The first prototype 64 3.3.1 Content 64 3.3.2 Support 68 3.3.3 Interface 69 3.3.4 Expert appraisal and micro evaluation 71

3.4 The second prototype 76 3.4.1 Content 76 3.4.2 Support 79 3.4.3 Interface 79 3.4.4 Scenario 81 3.4.5 Expert appraisal and micro evaluation 82

3.5 The third prototype 88 3.5.1 Content 88 3.5.2 Support 90 3.5.3 Interface 92 3.5.4 Micro evaluation 94 3.5.5 Expert appraisal at the UT 101

3.6 The fourth prototype 105 3.6.1 Content and support 105 3.6.2 Expert appraisal at the ECNU 106

Page 7: computer support for multimedia curriculum design

iii

4 DESCRIPTION OF THE FINAL VERSION 113

4.1 Overview of the program 113 4.2 Content 115

4.2.1 Goal and usage analysis 116 4.2.2 Learner analysis 117 4.2.3 Content selection 119 4.2.4 Content representation 121 4.2.5 Content organization 122 4.2.6 Interface design 125

4.3 Support 126 4.3.1 Information 127 4.3.2 Advice 128 4.3.3 Tools 132 4.3.4 Training 133

4.4 Interface 134 4.4.1 General characteristics 134 4.4.2 Screen design 134

4.5 Scenario 139 4.5.1 General characteristics 140 4.5.2 Components 140

5 ASSESSING THE PRACTICALITY OF THE PROTOTYPE 143

5.1 Introduction 143 5.2 Design of the assessment studies 144

5.2.1 Participants 144 5.2.2 Procedures and activities 146 5.2.3 Data collection and analysis 148

5.3 Results with primary target group users 150 5.3.1 Perceived practicality of the four components 150 5.3.2 Actual use of the program 154 5.3.3 Comments and suggestions 159

5.4 Results of study with other users 161 5.4.1 Perceived practicality of the four components 161 5.4.2 Actual use of the program 164 5.4.3 Expected extension to other subjects 166 5.4.4 Other comments and suggestions 168

5.5 Conclusions 170

Page 8: computer support for multimedia curriculum design

iv

6 DISCUSSION 171

6.1 Introduction 171 6.2 Discussion of the main findings 172

6.2.1 Content 172 6.2.2 Support 175 6.2.3 Interface 177 6.2.4 Scenario 179 6.2.5 Functionality for various user contexts 181

6.3 Discussion of the development research approach 184 6.3.1 Prototyping 184 6.3.2 Formative evaluation 186 6.3.3 Design principles 188

6.4 Recommendations 190 6.4.1 Web support 190 6.4.2 A follow-up study 191 6.4.3 Implementation 192

6.5 Closing remarks 193 REFERENCES 195

ENGLISH SUMMARY 205

NEDERLANDSE SAMENVATTING 213

APPENDICES

A-1 Screen dumps of Main Frame (English and Chinese version) 221 A-2 Screen dumps of Designer's Aid (English and Chinese version) 227 A-3 Screen dumps of Edit Panel (English and Chinese version) 245 B Example of an instructional scenario 249 C Interview topic list (used during the second round of prototyping) 255 D Instruments for the micro evaluation (used during the third round

of prototyping) 263 E Instruments for the expert appraisal at the UT (used during the

third round of prototyping) 267 F Questionnaire for the assessment studies 275 G Example of interface styles 281 H Original data collected from the assessment studies 283

Page 9: computer support for multimedia curriculum design

v

LIST OF FIGURES

1.1 The administrative framework of curriculum innovation in Shanghai 6 1.2 Line of reasoning for this study 11 1.3 Multimedia curriculum development process 14 1.4 Multimedia curriculum design process with CASCADE-MUCH 15 2.1 Difference between a multimedia curriculum (left) and a conventional

curriculum (right) 33 2.2 Multimedia curriculum development model 36 3.1 The prototyping process 55 3.2 A screen dump of Designer's Aid 70 3.3 A screen dump of Designer's Aid in the first prototype 80 3.4 A screen dump of Edit Panel 93 4.1 Overall structure of the CASCADE-MUCH program 114 4.2 Overview of Designer's Aid 115 4.3 Linear content organization 123 4.4 Non-linear content organization 124 4.5 Integrated content organization 124 5.1 Means of answers given by the novice and experienced designers 154 5.2 Examples of walking routes 155 5.3 Time spent on each screen 157 5.4 Illustration of means 164 5.5 Average time spent on each screen 165 6.1 Possible extension of the program 181 6.2 Assessment of the effectiveness of the program 192 LIST OF TABLES

2.1 An example of matrix media-selection model (Allen, 1967) 24 2.2 Curriculum components proposed by various authors 28 2.3 The comparison of the compute support systems 49 3.1 Overview of the prototypes 59 3.2 The relationship among the components, quality and participants 83 3.3 Modules of a multimedia curriculum 90 3.4 Characteristics of the subject teachers (n=7) 95 4.1 The interrelationship between the analysis and the design elements 131 4.2 The explanation of the menu 136 4.3 Presentation forms and screen elements for content representation 138 4.4 The factors used for describing each knowledge unit 142 5.1 General characteristics of the participants in study 1 (n=6) 145 5.2 General characteristics of the participants in study 2 (n=13) 146

Page 10: computer support for multimedia curriculum design

vi

5.3 Overview of instruments and data analysis 148 5.4 Perceived practicality of the content (n=6) 150 5.5 Perceived practicality of the support 151 5.6 Perceived practicality of the interface (n=6) 152 5.7 Perceived practicality of the scenario (n=6) 153 5.8 Support tools utilized by the participants 158 5.9 Perceived practicality of the four components by other users 162 5.10 Support tools utilized by other users 166

Page 11: computer support for multimedia curriculum design

vii

Preface

I still clearly remember when I studied at the primary school of a small villagein China, one of our teachers said "In about 20 years, the new millenium willcome. At that time you will be around 30 years old and become talentedpersons." I could not imagine where I would be and what I would be doing inthe year 2000, but I was filled with beautiful hopes. Now, with the year 2000just passed, a hope will soon become a reality. Through more than four yearsof struggling, I have successfully finished the Ph.D. study and written thebook in my second language -- English.

Looking backward, carrying out the Ph.D. study was a real challenge for me.Without the help of others, it would have been impossible for me to finish it.Here I would like to express my sincere thanks to those people who gave mesupport during the study.

First of all, I would like to thank the two promoters: Prof. dr. Jan van denAkker and Dr. Nienke Nieveen. They supported me during the whole processof conceptualization, data collection and analysis, and writing the thesis. Theirvaluable comments and suggestions always helped me go forward in the rightdirection. Furthermore, their support and encouragement made me feelconfident to pursue the study. In particular, Nienke Nieveen gave up muchpleasant time with her lovely young daughter, but spent it in reading andmaking comments and suggestions on this thesis, so as to make it like it is.

I also would like to thank Prof. dr. Tjeerd Plomp and Prof. dr. Zhiting Zhu. Itis they who provided me with this opportunity to be able to study at theUniversity of Twente as a Master student and then as a Ph.D. candidate.During the process of the Ph.D. study, they often stimulated me and gave mesupport. Particularly, Prof. Zhiting Zhu helped me organize an expertappraisal workshop in Shanghai while he was fully engaged.

Page 12: computer support for multimedia curriculum design

viii Preface

I also want to thank Prof. dr. Pløn Verhagen. His encouragement and humorstimulated me to treasure the opportunity of the study. Several years ago, hespent his valuable time in making comments and suggestions on the initialideas for my thesis during his visit to Shanghai. A few months ago, he alsomade some constructive comments and suggestions on the final version of mythesis. I really appreciate his help.

Furthermore, I would like to thank all participants who made contributions tothe formative evaluation activities both in Shanghai and in the University ofTwente, the Netherlands. Also, thanks are going to Kevin McKenney for hiseffort to polish and improve my English writing; and Sandra Schele for herdelicate formatting of the book.

In addition, I want to thank my parents who brought me into the worldfortunately (I am the seventh and also their last child), and educated me to bediligent. Their love, encouragement, support and understanding always inspireme to push forward.

Finally, wholehearted thanks are given to my wife Qingxia Yan and my sonKevin Wang for their love. My wife always took care of my personal life andhelped me arrange everything in order so that I had enough time to work onthe study. Although my son's arrival in the world during my Ph.D. studybrought us extra work and a new challenge, she did and also does an excellentjob, taking care of our son and making him grow healthily. Cheerfully, myson's birth makes me feel happy and proud, and fills me with energy andpleasure. Playing with him after a lengthy time of working became a greatdelight in my life.

I hope that the completion of the Ph.D. study is not only a finishing act of theformer academic carrier, but also a starting point of a new life in the future.

Enschede, September 2001

Page 13: computer support for multimedia curriculum design

1

Chapter 1 Introducing the CASCADE-MUCH study

T

his chapter introduces the study on "Computer ASsisted Curriculum Analysis Design and Evaluation - MUltimedia curriculum design in CHina" (Acronym: CASCADE-MUCH). The contextual information of curriculum development

in China/Shanghai is introduced in Section 1.1. Section 1.2 provides the origins of the study. An overview of the study and a preview of the dissertation are given in sections 1.3 and 1.4 respectively. 1.1 Context of the study The CASCADE-MUCH study took place in the context of a major curriculum innovation in Shanghai, China. This section first presents the overall context of curriculum development in China, and then focuses in particular on the curriculum development within the Shanghai region. 1.1.1 Curriculum development in China Educational system

China is a socialist country, which was founded in 1949 by the Chinese Communist Party. Geographically, China is the third largest country and her population of 1.2 billion is the world largest. China is a multi-ethnic country. In addition to Han Chinese, who constitute about 90 percent of the population, there are fifty-six minority groups, each with its own language and culture. Since the economic reform and open-door policies established in 1978, an ongoing educational reform --mainly referring to the educational structure-- has been taking place in China. Nowadays, China is popularizing a schooling system following a 6-3-3 structure. Learners study at primary schools, junior secondary schools and senior secondary schools for six, three and three years respectively. The first nine years in primary and junior secondary schools are a

Page 14: computer support for multimedia curriculum design

2 Chapter 1

compulsory educational period. Learners typically start their primary schooling at six years of age; before that they usually have received four years of pre-primary education. After the nine-year compulsory education, learners can choose to continue their studies in senior secondary schools or in vocational schools. If they choose senior secondary school studies and go on to pass the university entrance examinations, they may enter universities (or colleges) for another four years (or three years) of studies. The university system has a 4-3-3 structure, composed of four-year undergraduate, three-year master and three-year Ph.D. studies. In the last twenty years, Chinese education has made great progress in terms of the number of learners who actually go to school. According to Chen (1999), minister of the Ministry of Education (MOE), by 1997, the regions making up 65% of the total population have popularized the nine-year compulsory education. The rate of entering primary schools has reached 99%; the rate of entering junior secondary schools has reached 87%; and the illiteracy rate of adults has dropped below 6% (about a 4% drop from 1992). In addition, in 1997 there were 8.5 million senior secondary school students. The rate of entering senior secondary schools has risen from 26% in 1990 to 41% in 1999. In 1997, there were about 6.1 million university students including 180 thousand graduate students. The numbers are 2.2 and 9.6 times greater than those in 1979 respectively. Administrative framework

Curriculum development in China used to be a highly centralized task carried out solely by the State Educational Commission (SEC), a ministry of the State Council. Within the SEC, the Department of Secondary and Primary Education was directly responsible for finalizing the General Teaching Outline (the national curriculum), specifying curriculum organization and the timetable arrangement for all school subjects. Once the General Teaching Outline was finalized, it was People's Educational Press (PEP) that developed and published the ready-to-use school curricula. The PEP was a specialized publishing house directly under the leadership of the SEC. The main task of the PEP was to carry out curriculum studies, as well as to develop, publish and distribute lesson materials --including both teaching and learning materials.

Page 15: computer support for multimedia curriculum design

Introducing the CASCADE-MUCH study 3

In 1998, with the organizational reform of the State Council, the SEC changed its name to MOE. According to the Educational Act, the MOE is responsible for: i) planning and managing educational matters all over China; and ii) making overall arrangements for the development of educational undertakings. The Department of Basic Education within the MOE is specifically in charge of curriculum reform planning for primary and secondary schools, as well as the evaluation of the nation-wide curricula. The PEP is still responsible for the development, publishing and distribution of lesson materials. Furthermore, each province or district has a department of education, which is mainly responsible for local management of educational matters. Sometimes they may also develop some complementary lesson materials for their local use. Main features and existing problems

According to Goodlad (1994) and van den Akker (1998), curriculum development activities often take place at three different levels: societal or system level ('macro'), institutional or school level ('meso'), and classroom level ('micro'). However, in China before 1988, the curriculum development was usually undertaken only at the societal or national level (You, 1998; Wang, 1992a). The general teaching outline was determined by the SEC, and the concrete lesson materials were developed by the PEP, both at the national level. Although the curricula were allowed to have local alterations, curriculum development activities at school and classroom levels were less common (cf. Yat-ming, 1991). Since 1988, the SEC has attempted to change this situation by encouraging curriculum development at different levels. Four categories of curricula have been planned to be developed oriented to: 1. nation-wide regions and schools with normal conditions; 2. economically more developed regions and schools with better conditions; 3. economically less developed regions and schools with poorer conditions;

and 4. minority regions. The first category of curricula is developed at the national level, while the other three categories are developed by local institutes (cf. You, 1998). Despite the attempts of the SEC, it appears that the overall curriculum development process has not changed much. Most activities of curriculum development are still carried out at the national level, and most primary and secondary schools all over the country are still using the nation-wide curricula.

Page 16: computer support for multimedia curriculum design

4 Chapter 1

However, the nation-wide curricula developed by the PEP are often criticized by local teachers or educators for having the following shortcomings (cf. You, 1998; Wang, 1992b). First, they are oriented to the average conditions all over the country; it is hard for them to cover specific needs of various regions. The imbalance of economic development between coastal and inland regions implies that these 'one-size-fits-all' curricula cannot work well. Second, almost all curricula, even starting from primary and junior secondary schools, are designed for those learners who aim to enter higher grades. The overemphasis on entering higher grades too early is not good for learners' full development. Third, the nation-wide curricula are considered to be overly catering to test-driven education. The content and instructional strategies are mostly oriented to passing tests and entering higher grades. They overemphasize (Wang, 1992b): theory but ignore application; cognitive knowledge but ignore skills; test scores but ignore creativity development; and science but ignore arts.

The development of learners' full qualities, such as physical education, morality and creative thinking skills, is not sufficiently supported in the nation-wide curricula. Generally speaking, Chinese education has made great progress in the last twenty years, particularly in the structural reform and the number of learners who can go to schools. But less progress of reform on content and instructional strategies has been achieved. Although the nation-wide curriculum can ensure academic quality in general, it has been found to have some shortcomings for specific situations in various regions as well as for quality development of learners. 1.1.2 Curriculum development in Shanghai Introduction

Shanghai is the largest city in China. According to the latest census finished in 2001, Shanghai has more than sixteen million citizens allocated in sixteen districts and four counties. It is also one of the more advanced regions both economically and educationally. In 1988, Shanghai was mandated to make a trial curriculum innovation by the former SEC. During the trial curriculum innovation process, Shanghai was responsible for development of new curriculum materials for economically more developed regions (Category 2), including Shanghai rural and suburb districts as well as some other coastal

Page 17: computer support for multimedia curriculum design

Introducing the CASCADE-MUCH study 5

regions such as in Jiangsu and Zhejiang provinces. By 1997, Shanghai had finished its first round of curriculum innovation and started proceeding with the second round. So far, the Shanghai curriculum innovation has achieved several fruitful outcomes. The newly developed curricula, curriculum standards, and learning and teaching materials for all subjects and for all grades are being fully implemented in the whole city (Zhang, 1999). However, the other economically more developed regions have not yet adopted the newly developed Shanghai curricula. One of the main reasons might be the fact that Shanghai has had the power to develop its own university entrance examinations from the beginning of the innovation. In this way, they are able to make the university entrance examinations consistent with the intentions of the new curriculum. Although the other regions are encouraged to implement the new curriculum materials, they are not allowed to adapt their own university entrance examinations. In China, university entrance examination has always been considered to be a 'baton of orchestra', playing a very important role for teaching and learning. In the regions outside Shanghai, schoolteachers are reluctant to use the newly developed curriculum materials because the students have to pass the national examinations, which are still made based on the nation-wide curricula. Administrative framework

The administrative framework of curriculum innovation in Shanghai is illustrated in Figure 1.1. The Shanghai Educational Committee is in charge of the curriculum innovation activities as well as other educational matters. It is headed by the Shanghai local government and the MOE. The curriculum innovation activities are mainly carried out by the Curriculum Innovation Committee, which is a department of the Shanghai Educational Committee. Two departments are involved in the Curriculum Innovation Committee: an office and a commission for development and evaluation of lesson materials. The office is mainly responsible for organizational work, and the commission is responsible for the development and evaluation of specific curriculum materials. For each subject, there is a development and an evaluation group. In general, the development groups consist of subject matter experts and experienced teachers, whereas the evaluation groups are mainly composed of subject matter experts and administrative leaders. In addition, within the Shanghai Educational Committee there is a test department, which is responsible for making sure that tests are consistent with the newly developed curriculum materials.

Page 18: computer support for multimedia curriculum design

6 Chapter 1

In Shanghai, the general curriculum development approach is that the Curriculum Innovation Committee develops a general curriculum standard for all subjects. The commission for development and evaluation of lesson materials makes a specific and detailed curriculum standard for each subject based on the general curriculum standard. The development group of each subject develops lesson materials based on the specific curriculum standard for that subject. The evaluation group is responsible for evaluating the quality of the developed materials.

Curriculum Innovation Committee

Office

Development group for each subject

Evaluation group for each subject

Shanghai Educational Committee

MOE (Ministry of Education)

Shanghai local government

Test department

Commission for development and evaluation of curriculum materials

Figure 1.1: The administrative framework of curriculum innovation in Shanghai

The first round of curriculum innovation

The first round of curriculum innovation started in 1988 and was completed in 1997. The main aim was to bring 'two changes and three breakthroughs' (Yuan, 1990, p. 60). The two changes referred to: a shift in the focus of the curriculum content from entering higher grades towards quality development of learners; and a shift in instructional strategies from being inert towards being more (inter)active. The three breakthroughs were to result in: lessening learners' burden, improving learners' skills, and improving learners' full qualities and personality.

Page 19: computer support for multimedia curriculum design

Introducing the CASCADE-MUCH study 7

According to Wang (1992a), director of the Curriculum Innovation Committee at that time, the basic tasks of the first round of the curriculum innovation were to develop: a general curriculum standard for all subjects and a specific curriculum

standard for each subject; new lesson materials for primary and secondary schools; and some complementary lesson materials including computer based learning

(CBL) software. In the first two years following 1988, the Curriculum Innovation Committee worked on the analysis and planning for the curriculum innovation. At the beginning of 1990, a general curriculum standard for all subjects and a specific curriculum standard for each subject were formulated. From then on, lesson materials for each subject have been successively developed and implemented. The new curricula were composed of three modules: required courses, optional courses and active courses. In order to lessen learners' burden, the new curricula reduced learning time for the required courses, and added more learning time for the optional and the active courses. In the beginning, only some trial primary schools used the new curricula. Some subjects, such as Chinese language, used two different textbook versions developed by two different development teams. Gradually, the new curricula were used in an increasing number of primary and secondary schools with the students entering higher grades, and each subject finally settled on a single version of textbook. By 1997, all primary and secondary schools in Shanghai were using the new curricula. The last task of the curriculum innovation was to develop and use computer based learning software in order to make the instructional strategies more active and/or interactive. It was expected that all district educational colleges, secondary schools, as well as universities, could make contributions to the development of CBL software. The second round of curriculum innovation

The second round of curriculum innovation has been going on since 1997. In the first two years after 1997, a plan of primary and secondary curricula for the new century (2000-2010) was formulated, based on the guideline that the second round of curriculum innovation would focus on improving learners' creative thinking skills and inquiry skills (SSCRCO & SECIRO, 1999). The plan included a general blueprint for the second round of curriculum innovation and specific curriculum innovation guidelines for each subject. For the sake of

Page 20: computer support for multimedia curriculum design

8 Chapter 1

brevity, this plan will henceforth be called the 'second curriculum plan'. According to this second curriculum plan, the innovation should have the following characteristics. First, the new curricula would be classified into three new modules: basic courses, extended courses and research courses. The basic courses refer to the courses, which can provide learners with basic knowledge and skills, including language, mathematics and science. The extended courses refer to the courses, which elaborate knowledge and skills in depth in order that learners can enhance or improve the knowledge and skills they prefer. The research courses intend to improve learners' creative thinking and inquiry skills by providing an authentic learning environment and research topics. Second, the curriculum development activities would be carried out on three different levels: system level, district level and school level. The system level refers to the city level; the curricula at this level would be used by all schools in Shanghai. The basic courses and some extended courses would be developed at the system level. The economic development and educational status in different districts/counties are not equal. Some extended courses would be developed at the district level. At the school level, different schools could develop their own curricula based on their specific needs. Some extended courses and research courses would be developed at the school level. According to the second curriculum plan, especially curriculum development at the school level would be enhanced during the second round of curriculum innovation. In addition, the Curriculum Innovation Committee decided that in the second round of curriculum innovation, more information and communication technology (ICT) should be integrated into the new curricula. According to Zhang (1999), the associate director of the Curriculum Innovation Committee, the second round of curriculum innovation will 'change the single representation form of curriculum in written text towards integrated lesson materials in both written text and multimedia, in order to improve the quality and efficiency of classroom teaching and learning' (p. 2).

Page 21: computer support for multimedia curriculum design

Introducing the CASCADE-MUCH study 9

1.2 Origins of the study The CASCADE-MUCH study derived from two sources: the multimedia curriculum project for Biology undertaken in Shanghai and the CASCADE study performed by Nieveen (1997). In this section, these two origins will be briefly introduced. 1.2.1 The project of multimedia curriculum for Biology (MCB) One expected outcome of the curriculum innovation in Shanghai was to produce new curricula for the primary and secondary schools in economically more developed regions. During the same period of time, ICT has been growing tremendously, and educators realized that ICT would play an important role in the new curricula. Actually, in both of the two rounds of curriculum innovation --especially in the second curriculum plan-- multimedia technology has been expected to be integrated into the curricula. In 1997, a project called 'Bainian shuren' (fostering personnel in a long term) was initiated. It was sponsored by the Central Audio and Video Bureau of China. One of its sub-projects was to design multimedia lesson materials for the subject of Biology for senior secondary schools. Three experienced Biology teachers developed an instructional scenario for the multimedia lesson materials. The instructional scenario included a detailed instructional plan depicting: the content to be covered by the multimedia lesson materials; the organization of the content; and the way both teachers and learners would use the materials.

Based on the instructional scenario, computer programmers created the multimedia lesson materials. During this process, several problems were found. First, the instructional scenario did not fully exploit the advantages of CBL. Compared to traditional classroom teaching, CBL usually has two advantages: i) interaction; and ii) individualization. Computers can act as a tutor, presenting teaching materials and asking questions. After learners give their responses to the questions, computers can give immediate feedback. In some applications, learners can also ask questions, and the computer gives answers. In traditional classroom teaching, because of a large number of learners in one class, this kind of interaction between teachers and each individual student rarely happens. In addition, learners can also control their pace of learning when they are learning

Page 22: computer support for multimedia curriculum design

10 Chapter 1

with a computer; they can go as fast or as slowly as they want. They can repeat activities several times until they have grasped the material. Whereas in traditional classroom teaching, one pace for all learners is popularly adopted. Nowadays, with the development of ICT, computers can also help learners with collaborative learning in addition to individual learning. Unfortunately, in the produced instructional scenario, such advantages were not sufficiently utilized. There were only limited interactive activities, and learners had very little control over their own individual learning. Lesson materials were usually organized in a linear way, not demonstrating sufficient flexibility for learners to browse. In short, the instructional strategies in the instructional scenario were similar to those used in traditional classroom teaching. Second, the produced instructional scenario was not described in sufficient detail. It focused on content description with various media, but paid less attention to interface design and interconnections between various components. Due to a lack of experience, the subject teachers did not provide enough information on what a screen should look like, what elements (such as buttons, list and textbox) should be included, and what would happen after clicking a button or a menu item. This information is usually essential for computer programmers in developing a multimedia program. Furthermore, the instructional scenario developers (the experienced teachers) also complained that they lacked support tools when they were making the instructional scenario. The teachers did have a great deal of experience in teaching the subject of Biology, but they did not have enough knowledge and skills in designing multimedia materials. That was probably the main reason for the existence of the two problems mentioned above. Therefore, a support tool seemed to be necessary to help the teachers in making usable instructional scenarios. This study was to investigate in what way computers can help teachers in developing instructional scenarios. The line of reasoning for this study is shown in Figure 1.2.

Page 23: computer support for multimedia curriculum design

Introducing the CASCADE-MUCH study 11

ICT

Need for support

Problems in the instructional scenario

The MCB project

Multimedia lesson materials

New Curricula

The curriculum innovation in Shanghai Figure 1.2: Line of reasoning for this study

1.2.2 The CASCADE study This study can be seen as an extension of the CASCADE (Computer ASsisted Curriculum Analysis Design and Evaluation) project initiated in 1993 by the Department of Curriculum, Faculty of Educational Science and Technology, University of Twente, the Netherlands. The CASCADE project aimed to learn about how Electronic Performance Support Systems (EPSSs) could contribute to curriculum development. In particular, it focused on supporting Dutch professional curriculum developers through the often-neglected process of formative evaluation (Nieveen, 1997). A more detailed description can be found in Section 2.5.1. In 1996, two follow-up studies (CASCADE-SEA and CASCADE-MUCH) were launched to explore computer support in different contexts. The CASCADE-SEA (Science Education in Africa) study investigates support of teachers in creating exemplary science lesson materials for classroom use in southern Africa (McKenney, 1999). A more detailed description can be found in Section 2.5.2. CASCADE-MUCH, the current project, examines computer support for instructional scenario development in the process of multimedia curriculum development in China, particularly in Shanghai. Actually, before this doctoral research project started, a computer support system for the

Page 24: computer support for multimedia curriculum design

12 Chapter 1

analysis phase of curriculum development had already been developed during the Master program study (1995-1996) at the University of Twente (Wang, 1996). Much experience with EPSS design has thus been accumulated. After the MCB project, carried out in 1997 as explained in Section 1.2.1, this study was to focus on the design phase of multimedia curriculum development. In 1999, another follow-up study CASCADE-IMEI (Innovative Mathematics Education in Indonesia) started to investigate how computer support can help teachers in designing mathematics lesson materials based on the realistic approach (Zulkardi, 1999). A more detailed description can be found in the web site: http://projects.edte.utwente.nl/cascade/imei/, or http://www.cascadeimei.com. 1.3 Overview of the study With the rapid growth of ICT, it is expected that more multimedia lesson materials will be produced in the future (cf. Glasgow, 1997). This study intended to investigate what computers can offer in the process of making multimedia lesson materials. By following a development research approach, the study aims to produce a computer support system for subject teachers making instructional scenarios, which will be used by computer programmers to create multimedia programs. In this section, the intended target users, the aims of the CASCADE-MUCH program, the research questions and the research approach will be introduced. 1.3.1 Intended target users As mentioned earlier, Shanghai has more than sixteen million inhabitants allocated in sixteen districts and four counties. On the city level, the Shanghai Educational Committee is directly responsible for educational matters throughout the city. Within each district or county, an educational bureau is responsible for local educational administration. In addition, an educational college in each college or county --headed by the educational bureau-- is responsible for teacher training and curriculum development in that district or county. In each educational college, each subject usually has one teaching researcher who is responsible for coordinating teaching activities, teaching research and curriculum development for that subject. Because these teaching researchers

Page 25: computer support for multimedia curriculum design

Introducing the CASCADE-MUCH study 13

used to be experienced teachers, they are often chief members and even administrators of curriculum development teams in their districts or counties. Instructional scenarios for multimedia curriculum development at district level are usually developed by those teaching researchers, sometimes together with other experienced teachers at local secondary schools. Although those teaching researchers are experienced teachers, they usually do not have competent knowledge of computer-based learning, nor sufficient skills for designing instructional scenarios. The intended target users of CASCADE-MUCH are teaching researchers who are curriculum development members at district educational colleges, as well as those subject teachers who might participate in instructional scenario development in Shanghai. They are commonly called teacher-designers in this study. As different subjects have different characteristics and usually need different instructional strategies, a selection of a small sample of subjects to start the exploration was made. For CASCADE-MUCH, the subjects of Biology and Geography were chosen as its experimental subjects, because these two subjects involve many natural phenomena which are either hard for learners to learn or hard for teachers to teach (cf. SSCRCO & SECIRO, 1999). 1.3.2 Aims of the program The development of multimedia lesson materials is a comprehensive and complex process that usually calls upon several kinds of expertise, such as expertise of subject teachers, instructional designers, multimedia designers, and curriculum specialists. This means that the development of multimedia lesson materials is usually a cooperative effort, especially when a project covers a whole course, for instance. In this study, multimedia lesson materials covering a whole course will be called a multimedia curriculum. The term of multimedia curriculum will be defined and elaborated in Chapter 2. The development of a multimedia curriculum usually starts with instructional scenario development by teacher-designers, and followed by programming by computer programmers. According to Walker's (1990) deliberative approach to curriculum development, at the beginning of a complex curriculum development project, curriculum developers usually need discussions with other stakeholders, including educators, educational administrators, teachers, learners and parents, to formulate needs, aims and possible solutions. Similarly, developing a multimedia curriculum often starts with consultations between teacher-designers and other experts in order to get consensus on the

Page 26: computer support for multimedia curriculum design

14 Chapter 1

final product (cf. Tessmer, 1993). Of course in some cases an expert may possess expertise in multiple disciplines. Based on the consultations and discussions, teacher-designers can develop an instructional scenario. Computer programmers will eventually create a multimedia curriculum based on this developed instructional scenario. Figure 1.3 shows the process of multimedia curriculum development.

Teacher-designers

Instructional scenario

Interface design expertise

Instructional design expertise

Multimedia curriculum

Computer programmers

Curriculum development expertise

Multimedia design expertise

Figure 1.3: Multimedia curriculum development process

As long as the teacher-designers get the support needed, this process works well. However, teacher-designers sometimes cannot get immediate support from the experts. The experts are usually very busy professionals, and it is impossible for them to be continuously available for the teacher-designers. That absence or delay of support may hinder the development process. In addition, if no sufficient discussions took place between computer programmers and teacher-designers, the computer programmers may have difficulties with understanding the developed instructional scenario. Furthermore, some good inputs or suggestions from the computer programmers for quality improvement of the instructional scenario may not be integrated in the instructional scenario or learned by the teacher-designers.

Page 27: computer support for multimedia curriculum design

Introducing the CASCADE-MUCH study 15

The use of CASCADE-MUCH is expected to yield the following solutions: a. It can help teacher-designers make a usable instructional scenario by

providing them with just-in-time support. b. After the instructional scenario has been made, it can make discussion or

information exchange between teacher-designers and computer programmers easier.

c. It improves teacher-designers' professional knowledge/skills for multimedia curriculum design.

With the support of CASCADE-MUCH, the process of multimedia curriculum development mainly includes three phases (see also Figure 1.4).

➂ ➁

➁ Multimedia curriculum

Computer programmers

CASCADE -

MUCH

Instructional scenario

Teacher-designers

Various expertise

➂ ➀ ➁

Figure 1.4: Multimedia curriculum design process with CASCADE-MUCH

1. On the one hand, teacher-designers can learn from the CASCADE-MUCH

program what content is covered, and how the content is represented or organized in a multimedia curriculum. On the other hand, they can also use the program to develop a tentative prototype of the instructional scenario. Of course, during this process, the teacher-designers may also need to consult with the experts to get more expertise and advice. This phase is labeled as ➀ in Figure 1.4.

2. The teacher-designers will work together with the computer programmers on the tentative instructional scenario, and make it ready for programming. During this process, formative evaluation activities will be carried out to collect comments and suggestions on the preliminary instructional scenario, and revision decisions will be made. A revised prototype will be developed by the teacher-designers based on these decisions. Also, it is possible for teacher-designers to consult with the experts as well. This phase is labeled as ➁ in Figure 1.4. This phase may take place several times until a ready-to-use instructional scenario has been produced.

Page 28: computer support for multimedia curriculum design

16 Chapter 1

3. The computer programmers will create a multimedia curriculum based on the ready-to-use instructional scenario. During this process, computer programmers will create the multimedia curriculum mostly based on the instructional scenario, but sometimes they may also need face-to-face discussions with the teacher-designers. This phase is labeled as ➂ in Figure 1.4.

Figure 1.4 also demonstrates how important an instructional scenario is in the process of multimedia curriculum development. On the one hand, it carries comprehensive information, which might be used for programming; on the other hand, it also provides a concrete platform for discussion between teacher-designers and experts/computer programmers. 1.3.3 Methodology: Focus on development research The traditional view of research used to be discovery of knowledge, and development was the translation of that knowledge into a useful form in practice (Richey, 1997). In reality, a disconnect often exists between research and practice. Either theory is too abstract to guide practice, or practice lacks suitable theory to follow. Through development research, this gap is expected to be bridged to some extent. According to Seels and Richey (1994), development research is 'the systematic study of designing, developing and evaluating instructional programs, processes and products that must meet the criteria of internal consistency and effectiveness' (p. 127). In development research, the research process and the development process are merged into one enterprise. During this joint process, development and research can contribute to each other. Van den Akker and Plomp (1993) give a functional definition to development research by specifying its two main purposes: a. supporting the development of prototypical products; and b. generating methodological directions for the design and evaluation of such

products. This definition implies that development research mainly contributes to two aspects: product improvement and knowledge growth. Product improvement aims at making a high quality product to be valid, practical and effective. Knowledge growth is preferably reflected in 'design principles' of the following format: 'If you want to design intervention X [for the purpose/function Y in context Z] then you are best advised to give that intervention the characteristics A, B, and C [substantive emphasis] and to do that via procedures K, L, and M [procedural emphasis], because of arguments P, Q, and R' (van den Akker, 1999, p. 9).

Page 29: computer support for multimedia curriculum design

Introducing the CASCADE-MUCH study 17

Van den Akker (1999) mentions that knowledge growth and product improvement are of equal importance in development research. Furthermore, in agreement with the description of development research by Richey and Nelson (1996), van den Akker (1999) distinguishes two types of development research: Formative research. Research activities are carried out during the entire

development process, aiming at optimizing the quality of the product as well as generating and testing design principles. Reconstructive studies. Research activities are conducted sometimes during but

oftentimes after the development process, aiming at articulating and specifying design principles.

This study followed a development research approach, aiming at: i) producing a valid and practical computer support system; and ii) generating methodological guidelines for the design and evaluation of such products. The main research question of the study is:

What characteristics should a valid and practical computer support system for multimedia curriculum design have in the context of Shanghai?

By following the development research approach, this study progressed through two main stages: prototyping and assessment. The overall research design of these two stages will be presented in the following two sections. 1.3.4 Approach of the prototyping stage Research question

In the prototyping stage, the main research question was:

What characteristics should a valid (and potentially practical) computer support system for multimedia curriculum design have in the context of Shanghai?

According to Nieveen (1997, 1999) and van den Akker (1999), an educational or training product can be assessed on three quality criteria: validity, practicality and effectiveness. Validity refers to the extent that the product is designed based on state-of-the-art knowledge ('content validity') and that the various components of the product are consistently linked to each other ('construct validity'). Practicality refers to the extent that users (and other experts) consider the product attractive and usable under 'normal' conditions. Effectiveness refers to the extent that the experiences and outcomes with the

Page 30: computer support for multimedia curriculum design

18 Chapter 1

product are consistent with the intended aims. Undoubtedly, during the prototyping process, an educational or training product such as the computer support system should be designed and constructed towards the direction of meeting these quality criteria. During the prototyping stage, this study mainly focused on validity and practicality. The main reason was that emphasis often shifts from validity to practicality, to effectiveness during a development process (van den Akker, 1999), and usually validity and practicality are essential preconditions of effectiveness. However, during the entire study special attention was given to indicators of potential impact of the support program on the design of instructional scenarios. Prototyping approach

In software engineering, there are two popularly used approaches to software development: the waterfall approach and the prototyping approach (cf. de Hoog, de Jong, de Vries, 1994; Sommerville, 1996). The waterfall approach separates software development activities into different phases such as requirement specification, software design, implementation and testing. After each phase is defined it is 'signed-off' and development goes on to the following phases. The prototyping approach, according to Smith (1991), is a process of producing successive trial versions of software before developing a final system. This study adopted the prototyping approach. Three benefits of using the prototyping approach were to: 1. Gradually clarify the characteristics of the innovative program

The CASCADE-MUCH program is a rather innovative endeavor, lacking much experience to learn from. It was expected that the prototyping approach could help gradually clarify the design specification and characteristics of the program.

2. Successively approximate an optimal computer support program During formative evaluation activities, some descriptive and prescriptive information could be collected. Based on this information, further improvement of the prototype could be made in a next round of prototyping. Through this cyclic prototyping process, the quality of the prototype could be successively improved and optimized.

Page 31: computer support for multimedia curriculum design

Introducing the CASCADE-MUCH study 19

3. Promote communication between the designers and users One of the main reasons why users often do not like computer programs is the lack of communication between designers and users during the program development process (Gray & Black, 1994). Smith (1991) claims that communication between designers and users becomes easier and more effective through prototypes than through other means.

The detailed description of the prototyping stage, including its formative evaluation activities can be found in Chapter 3. 1.3.5 Approach of the assessment stage Research question

After four rounds of prototyping, the prototype was judged to be valid and perceived to be rather practical. The assessment stage aimed to assess the practicality of the prototype by its users. The research question of the assessment stage was:

To what extent is the CASCADE-MUCH program practical for both primary target group users and other users in the context of Shanghai?

In addition to assessing the practicality of the prototype for intended target users, the practicality for other users, such as teacher-designers of other subjects, was also expected to be probed. Therefore two types of users were invited to attend the assessment studies: primary target group users and other users. Assessment approach

Two studies were organized in Shanghai to assess the practicality of the prototype. The first study was with primary users, and the second study was held with a group of other users. The main aim of the first study was to assess whether the prototype was practical for subject teachers who are either experienced designers or novice designers for multimedia curriculum development to make instructional scenarios. In order to collect and compare the possible differences between experienced designers and novice designers, a built-in log file was added to the prototype to trace how the users really used the prototype. In addition, several instruments such as a questionnaire and observational notes were used during the assessment process.

Page 32: computer support for multimedia curriculum design

20 Chapter 1

The main aim of the second study was to assess to what extent the prototype could be used in other subjects as well as in other contexts. The workshop was open for participants of related professionals: teachers or teacher-designers of other subjects, CBL designers in primary or secondary schools, and CBL designers in computer companies. The same instruments were used to collect data in the second study. In particular, the focus was on discussing possible extension to other subjects and to other contexts. The detailed information of the design and results of the assessment studies can be found in Chapter 5. 1.4 Preview of the dissertation The development research activities and findings of the CASCADE-MUCH study are presented in the subsequent chapters. In Chapter 2, a conceptual framework is presented. In the first part of that chapter, three closely related domains are introduced: multimedia, curriculum development and EPSS. In the second part of the chapter, some related support systems are briefly described, and their implications on the design of the CASCADE-MUCH study are given. In Chapter 3, the prototyping approach of CASCADE-MUCH is elaborated in more detail. Four prototypes are described in terms of content, support, interface and scenario. In addition, details about the formative evaluation and revision decisions are provided. In Chapter 4, the detailed description of the final version of the CASCADE-MUCH program is presented. In Chapter 5, the assessment activities, procedures and results of the study are presented. Finally, Chapter 6 discusses the main findings of the study.

Page 33: computer support for multimedia curriculum design

21

Chapter 2 Multimedia curriculum and computer support systems

T

his chapter presents the conceptual framework of the study, which is closely related to three domains: multimedia, curriculum development, and Electronic Performance Support Systems (EPSSs). Section 2.1 gives the definitions of media

and multimedia, and elaborates on the relationship between media and learning, as well as between multimedia and learning. Section 2.2 provides the definition of curriculum and some curriculum development models. Section 2.3 explains the relationship between a multimedia curriculum and a conventional curriculum, and presents a model for multimedia curriculum development. Section 2.4 introduces the concept of EPSS. Section 2.5 presents a number of existing EPSSs for curriculum design, and discusses their implications for the design of the study. Finally, Section 2.6 ends up with conclusions concerning the main issues presented in this chapter. 2.1 Multimedia Although the term multimedia is popularly used nowadays, no precise definition has been commonly accepted (Moore, Burton & Myers, 1996). This section defines the two terms: 'media' and 'multimedia' as used in this study, and explores the relationship between media and learning as well as between multimedia and learning. 2.1.1 Definitions of media and multimedia Media

A consensus definition of media is that they are the means or pieces of equipment that transmit information from sender to receiver (Verwijs, 1998). For example, Romiszowski (1981) defines media as 'the carriers of messages, from some transmitting sources, to the receivers of the message' (p. 339). In the context of education, the term media is usually defined as instructional facilities that carry messages to learners. Very often these messages have different presentation forms (Verhagen, 1992). For instance, a description of the Chinese Great Wall

Page 34: computer support for multimedia curriculum design

22 Chapter 2

in a book can use text and a picture, which are different presentation forms. Modern advances in technology have made the computer able to integrate and present these presentation forms. In the world of multimedia, these presentation forms are usually called media. In this study, the term media mainly refers to these presentation forms encoded and presented on a computer, including text, pictures, audio, animation and video. Multimedia

The term multimedia used to refer to the use of several media devices in a coordinated fashion such as synchronized slides with audiotape (Moore et al., 1996). However, the adopted definition of media in this study implies that the term multimedia particularly refers to a combination of various presentation forms presented on a single device: the computer. In addition to the terms of media and multimedia, another term, hypermedia, is also being widely used. A hypermedia system often refers to a multimedia system in which contents are decomposed into chunks of information (or 'nodes') composed of various presentation forms and connected with interrelated hyperlinks (cf. Ambrose, 1991; Park, 1991). Users can visit these chunks of information by following the embedded hyperlinks. In the extreme case, if all chunks of information are represented purely as text, a hypermedia system will become a hypertext system. Similarly, if all embedded links have been removed from a hypermedia system, it will become a multimedia system (cf. Borsook, 1997). This study does not intend to separate hypermedia systems from multimedia systems. Although the two terms are separated by the above distinctions, they are seen as interchangeable in this study. 2.1.2 Media and learning Whether media can improve learning is a controversial topic. In 1983, Clark published his famous disputing assertion:

'… media are mere vehicles that deliver instruction but do not influence student achievement any more than the truck that deliver our groceries causes changes in our nutrition' (p. 445).

This statement led to a debate in the 1980s and received a new impulse in 1994 with two special issues of the journal Educational Technology Research and Development. Two representative persons standing on these two opposite sides of the disputation are Clark and Kozma. Clark (1983, 1994) stresses that it is instructional method, not media, that affects learning. Kozma (1991, 1994)

Page 35: computer support for multimedia curriculum design

Multimedia curriculum and computer support systems 23

strongly disagrees with Clark's assertion, and claims that media and instructional method cannot be separated from each other. For a specific instructional method, there might be some most suitable media to support it. An instructional method, when used together with the most suitable media, will get better learning results than with other media. People may continue to argue whether media can influence learning. A common agreement is that a combination of some proper media can make hard-to-implement instructional methods more feasible. For example, in the subject of Biology, the process of cell division is usually hard to explain by oral explanation or still pictures, but a computer simulation with animation and sound can make the process much clearer and easier to understand. A follow-up question is 'What are the proper media?' Already several decades ago, educators have tried to find answers to this question. For example, in 1967, Allen distinguished three levels (high, medium, and low) of proficiency between learning objectives and media. Allen (1967) proposed a matrix presenting the relationship between media and learning objectives as Table 2.1. This matrix has often been used and elaborated, for example by Briggs and Wager (1981). However, to some extent many cells in this matrix need negotiation. For instance, television programs can do a fine job of presenting factual information; programmed instruction nowadays can be used efficiently to achieve many learning objectives such as teaching facts, visual identification and rules. In addition to this matrix, some other models, such as flowcharts and worksheets, can also be used for selecting appropriate media (Verwijs, 1998). A flowchart approach usually starts by asking users some questions. By answering these questions with 'yes' or 'no', the flowchart will lead the users to the most suitable media for their purpose (cf. Romiszowski, 1988). A worksheet can be described as a checklist with questions or a scheme that weights criteria selected by the users. Examples of worksheet models can be found in Reiser and Gagné (1983).

Page 36: computer support for multimedia curriculum design

24 Chapter 2

Table 2.1: An example of matrix media-selection model (Allen, 1967)

Learning objectives

Media Lea

rnin

g fa

cts

Lea

rnin

g vi

sual

id

enti

fica

tion

Lea

rnin

g p

rin

cip

les,

co

nce

pts

, ru

les

Lea

rnin

g p

roce

du

res

Per

form

ing

skill

ed

per

cep

tual

mot

or

acts

Dev

elop

ing

atti

tud

es, o

pin

ion

s,

mot

ivat

ion

s

Still pictures o + o o – – Motion pictures o + + + o o Television o o + o – o 3-D objects – + – – – – Audio recordings o – – o – o Programmed instruction o o o + – o Demonstration – o – + o o Printed textbooks o – o o – o Oral presentation o – o o – o

Note: + = high; o = medium; – = low In conclusion, media selection is a complex process, which needs to take comprehensive consideration of several factors such as instructional goals, learner characteristics, practical factors, and costs (Fenrich, 1997; Romiszowski, 1981; Verwijs, 1998), until a satisfying plan can be reached. 2.1.3 Multimedia and learning When talking about multimedia and learning, the following assumptions of multimedia can be found in literature (cf. Moore et al., 1996; Zhu, 1997): It improves effectiveness of learning since human organs may sense

information simultaneously. It provides a context-rich learning environment since multimedia lesson

materials are presented with multiple presentation forms. Non-linear organization improves learners' higher order thinking skills

because users have to analyze what they have learned and make decisions where to go. Friendly user interface may motivate learners' interest since it is attractive

and interactive.

Page 37: computer support for multimedia curriculum design

Multimedia curriculum and computer support systems 25

However, these assumptions have not been convincingly confirmed by experiments, with some experiments even showing poorer results with multimedia (Moore et al., 1996). For example, Mayer and Anderson (1991) found that coordinated presentation of narrative and animations results in better performance on tests of creative problem solving than the word-before-picture group. Nevertheless, Reynolds and Baker (1987) found that texts with graphs and texts without graphs did not differ in degree of learning effect. In addition, the presentation of materials using computers was shown to increase attention and learning. Lorch, Bellack and Augsbach (1987) noted that in television presentations, children's recall of content was comparable to audio only, visual only, or simultaneous across both modes. Fenrich (1997) states that some comparative research outcomes are not very convincing because the studies fail to compare excellent regular instruction with excellent multimedia instruction. In most cases where traditional instructional methods yield excellent results, teachers feel less need for multimedia applications. Nevertheless, he believes that at least some potential benefits of a multimedia-learning environment for both learners and teachers can be found. Some of those benefits for learners are that they can: learn at their own pace, control their own learning path, and review as

often as they wish; study when they want to at any time of the day or night; learn from an infinitely patient tutor that can adapt instruction to

individual abilities and backgrounds; and actively pursue learning and receive immediate feedback.

In addition, some potential benefits offered by multimedia applications for instructors are that they: can replace ineffective or potentially dangerous learning activities with

simulations, animations and games; are well suited for teaching many dull or routine topics; may add something exciting, innovative, and/or different to an instructor's

routine; save time through reduced need for teaching and preparation. Time saving

can, in turn, save money and increase student contact time.

Page 38: computer support for multimedia curriculum design

26 Chapter 2

Although a multimedia-learning environment may have the above-presumed benefits, some shortcomings warrant further attention. For example, the theory of multi-channel communication reminds that (cf. Moore et al., 1996): Humans can only receive and process one channel message at one time,

consequently multi-channel messages may cause human information memory overload. Multi-channel messages may negatively interfere with each other when

they are unrelated or contradictory. Ragan, Boyce, Redwine, Savenye and McMichael (1993) carried out a survey of seven major reviews of research on multimedia. Their main findings are as follows: Multimedia is at least as effective as conventional methods and has

substantial cost benefits and efficiency. Frequently, multimedia instruction is more effective than conventional

instruction. Multimedia is more efficient in terms of learning time than is conventional

instruction (30% savings). Perhaps many people do not completely agree with these findings, and even doubt how the reviewers compared the multimedia instruction with the conventional methods and how they calculated the timesavings. Nevertheless, their findings on multimedia can serve as positive evidence for development of multimedia applications. In Shanghai, teachers as well as other educators expect that on the one hand multimedia applications can make the passive instructional process more interactive; on the other hand, multimedia can present the knowledge units in a more attractive and interesting way, even though the instructional process is still passive. 2.2 Curriculum development Curriculum development is a complex task. It can take place at different levels with different representation forms. In this section, various concepts of curriculum as well as the definition of curriculum adopted in this study are presented. Curriculum components, levels and representation forms are provided. In addition, several curriculum development models as well as the model used in this study are described.

Page 39: computer support for multimedia curriculum design

Multimedia curriculum and computer support systems 27

2.2.1 Concepts of curriculum Among the most abstract terms in educational literature, the term curriculum probably belongs to the most elusive ones (van den Akker, 1998). It has many different definitions in literature. For example, Walker (1990) lists a broad range of curriculum as follows: learning activities; content; a set of events; situations; a series of things; total efforts of schools; a sequence of potential experiences; a set of abstractions; offering of valued knowledge, skills and attitudes.

This list shows that curriculum can be defined very broadly (i.e. total efforts of schools) or very narrowly (i.e. content). Different people may refer to completely different matters when they are using or talking about the term curriculum. To understand the term, it will be helpful if the intended meaning, scope and context of the term can be specified (van den Akker, 1998). In this study, the term of curriculum refers to a plan for learning as proposed by Taba (1962). The major advantage of this simple definition is that it 'allows specification for many educational levels, representations and contexts' (van den Akker, 1998, p. 421). Walker (1990) gives a more informative description about the nature of a plan for learning. He defines that 'the curriculum refers to the content and purpose of an educational program together with their organization' (Walker, 1990, p. 5). Curriculum components

When looking at the term curriculum, a number of related topics may emerge, such as aims, content, learning activities, teaching strategies, time, etc. Table 2.2 lists several components of a curriculum proposed by various authors. Tyler (1949) summarizes four basic questions (issues) for curriculum and instruction: objective, selection of learning experience, organization of learning experience, and evaluation. Taba (1962) elaborates Tyler's four elements and expands them to seven components: rationale, aims, content selection, content organization, learning experience selection, learning experience organization, and evaluation. Marsh (1991) states that a curriculum should include four basic elements: policies/plans, teaching/learning

Page 40: computer support for multimedia curriculum design

28 Chapter 2

experiences, teachers and students. Klein (1991) proposes nine curriculum elements as goals/objectives/ purposes, content, materials, resources, activities, evaluation, grouping, time, and space. Eash (1991) summarizes five components for a curriculum: assumptions about the learners and society, aims/objectives, content, modes of transaction, and evaluation. Table 2.2: Curriculum components proposed by various authors

Aim

s/go

als/

ob

ject

ives

Con

ten

t

Lea

rnin

g ex

per

ien

ces

Inst

ruct

ion

al

stra

tegi

es

Tea

cher

s

Lea

rner

s

Eva

luat

ion

Gro

up

ing,

ti

me,

sp

ace

Tyler × × × Taba × × × × Marsh × × × × Klein × × × × × × Eash × × × × × Although different authors decompose a curriculum into different components, some common components emerge. For example, all above authors agree that aim/goal/objective is a basic component of a curriculum. Most of them agree that content, learning experience, and evaluation are also essential for a curriculum. Although some components are named differently such as instructional strategies (Klein, 1991) and modes of transaction (Eash, 1991), their meaning is more or less similar. In addition, some components are closely related to each other. For example, the component of learning experiences is related to learners, and instructional strategy is related to teachers and learners. After comparing the components proposed by different authors, Nieveen (1997) summarizes five substantive components of a curriculum: i) rationale; ii) aims and objectives; iii) subject matter; iv) modes of transaction; and v) evaluation. As the component of aims and objectives is closely related to the component of rationale, in this study four basic components of a curriculum are classified as follows:

Page 41: computer support for multimedia curriculum design

Multimedia curriculum and computer support systems 29

1. Aims/Goals/Objectives In addition to regular aims, goals and objectives, assumptions about the learners and society (Eash, 1991) and rationale (Nieveen, 1997) are also included in this component.

2. Content Here content is an umbrella term which includes all subject matter materials and resources, with which learners interact as they are experiencing a curriculum.

3. Instructional strategies Instructional strategies encompass learning experiences (Marsh, 1991; Taba, 1962; Tyler, 1949), learning activities and teaching strategies (Klein, 1991), and modes of transactions (Eash, 1991).

4. Evaluation/Assessment Although evaluation and assessment sometimes have minor distinctions (Marsh, 1997), they are used interchangeably in this study. They refer to activities and procedures for determining what students are learning or have learned.

Curriculum levels

According to Goodlad (1994) and van den Akker (1998), a curriculum is usually planned on various levels (macro, meso, and micro) in various educational settings. On macro (societal or system) level, a curriculum refers to a more general curricular framework. On meso (school or institute) level, a curriculum refers to an educational program for a school or institute. On micro (classroom) level, a curriculum refers to a plan for concrete instructional activities usually taking place in a classroom. These three levels (macro, meso and micro) may be manifested somewhat differently in China, referring to national level, local level and classroom level. On the national level, a curriculum refers to the general teaching outline or the curriculum standard to be used for the whole country. On the local level, a curriculum refers to an educational program for local use such as within a province, a city, or a school. On the classroom level, a curriculum refers to a concrete plan for specific instructional activities, which usually take place in a classroom. In China, multimedia curriculum development activities (the term 'multimedia curriculum' will be explained in Section 2.3) usually take place on the local level, sometimes on the national level, but seldom on the classroom level. One of the main reasons is that educational institutes on the local level want to

Page 42: computer support for multimedia curriculum design

30 Chapter 2

develop multimedia curricula to meet their specific needs as the nation-wide curricula are usually designed for the whole country based on the average conditions. For example, the Shanghai Educational Committee wants to develop more multimedia curricula during the second round of curriculum innovation as introduced in Section 1.1.2. In addition, for teachers at the classroom level it is nearly impossible --nor is it useful-- to develop a complete multimedia curriculum covering a whole course, even though they may develop some multimedia applications covering pieces of content in their courses. Curriculum representation forms A curriculum may have various representation forms. A typology proposed by Goodlad, Klein and Tye (1979) and adapted by van den Akker (1998) illustrates well the representation forms as follows: the ideal curriculum reflects the original assumptions, visions and

intentions; the formal curriculum reflects the concrete written curriculum documents,

such as learner materials (textbooks) and teacher guides; the perceived curriculum represents the curriculum as interpreted by its users

(especially teachers); the operational curriculum reflects the actual instructional process as realized

in a classroom; the experiential curriculum reflects the curriculum as it is experienced by the

learners or students; the attained curriculum reflects the learning outcomes of learners.

In this study, a multimedia curriculum refers to the intended curriculum, which is a combination of the ideal curriculum and the formal curriculum. On the one hand, a multimedia curriculum is usually designed based on an existing curriculum document, and functions as a complement and/or extension of it. On the other hand, a multimedia curriculum may also include some elements of an ideal curriculum, which are difficult to be implemented in paper-based materials. 2.2.2 Curriculum development models Visscher-Voerman (1999) summarizes four commonly used strategies or paradigms in training and education: i) instrumental; ii) communicative; iii) pragmatic; and iv) artistic. Although these four strategies refer to broad educational design and development processes, they are also useful for

Page 43: computer support for multimedia curriculum design

Multimedia curriculum and computer support systems 31

guiding curriculum development. According to Visscher-Voerman (1999), the instrumental (or planning-by-objective) strategy usually starts with the formulation of objectives, and ends with the assessment of objective achievement. The representative advocate of this approach is Tyler, who proposes a linear and rational model composed of four basic questions for designing curriculum and instruction (Tyler, 1949): What educational goal should the school seek to attain? How can learning experiences be selected which are likely to be useful in

attaining these objectives? How can learning experiences be organized for effective instruction? How can the effectiveness of learning experiences be evaluated?

The communicative (or consensus) strategy suggests that curricula or other educational products are developed in close interaction and communication between the developers and others involved. A representative advocate of the communicative approach is Walker, who introduced a deliberative (or naturalistic) approach to curriculum development. Walker (1990) assumes that better curriculum planning and development will result when educators engaged in it understand the complexity of the process. Walker's deliberative model is composed of three stages: platform of ideas, deliberation and design. In the platform stage, individuals come together to discuss and argue about their beliefs and aims regarding the ideal curriculum. In the deliberative stage, the attention shifts to how the beliefs formulated are used in assessing actual states of affairs and possible courses of action. In the design stage, activities will be undertaken to achieve the beliefs and aims. The pragmatic (or prototyping) strategy suggests that curriculum developers create their products by building, testing, and revising several prototypes. This approach is particularly useful where goals are unclear, for example in cases of a new product or a new context (Nieveen, 1997). It originated in the field of computer software development, where it is usually called rapid prototyping. More recently, it has also been used in the field of electronic lesson materials development (Nieveen, 1997; Visscher-Voerman, 1999). The artistic (or connoisseurship) strategy indicates that curriculum developers behave like artists who, based on their own expertise and experiences, create their curricula in response to the specific situation in which they work. A representative advocate is Eisner, who considers that people who make decisions about curriculum development are acting in a manner similar to

Page 44: computer support for multimedia curriculum design

32 Chapter 2

artists who choose among an almost limitless variety of ways of representing their own view of reality (Eisner, 1985). This study adopted the pragmatic strategy based on the following considerations (see also Section 1.3.4). First, multimedia curriculum development is a complex task. At the beginning of the study, it was not clear how to design a multimedia curriculum and what kinds of support would be necessary for teacher-designers. The pragmatic strategy could help the designer clarify the design specification, and successively approximate an optimal product. Second, each prototype could stimulate communication between the designer and the various participants of each formative evaluation activity. 2.3 Multimedia curriculum development In this section, the definition and modules of multimedia curriculum, and the development model to be used for multimedia curriculum are introduced. 2.3.1 The definition of multimedia curriculum This study aims to support developing a specific form of curriculum, called multimedia curriculum. Compared to the existing term of courseware, a multimedia curriculum has some additional characteristics. First, it is broader than courseware. It usually includes the entire four components (aims/goal/objectives, content, instructional strategies, and assessment), covers the complete content of a course, and integrates several instructional strategies. Whereas a courseware application may include a part of the components, or cover a part of course content; and very often it includes only a single instructional strategy. Second, the intended target users of courseware are usually learners, who use it to explore or to review lesson materials, whereas a multimedia curriculum is usually designed for both learners and instructors. Learners can use it to learn, and instructors can also use it to facilitate their teaching process. In this study, combining the definitions of multimedia (see Section 2.1.1) and curriculum (see Section 2.2.1), multimedia curriculum is defined as:

a plan for learning where various presentation forms (such as text, pictures, audio and video) are integrated, encoded and presented on a computer.

Page 45: computer support for multimedia curriculum design

Multimedia curriculum and computer support systems 33

The relationship between multimedia curriculum and conventional curriculum can be described from two perspectives: content and media. As far as the content is concerned, a multimedia curriculum and a conventional curriculum may cover the same content. However, since a multimedia curriculum usually functions as a complement of a conventional curriculum, some extended supporting materials might be included in a multimedia curriculum. Furthermore, the content of a multimedia curriculum is relatively easier to update. Although the preparation, design and construction of the content for these two kinds of curricula may have a similar degree of complexity, once the content is ready, publication and update of a multimedia curriculum are much easier. Especially with the increase in use of the Internet, updating the content of a multimedia curriculum is becoming continually more convenient. With respect to the media used, a multimedia curriculum and a conventional curriculum may in principle use the same presentation forms such as text, pictures and audio. In a multimedia curriculum, however, these presentation forms are encoded and presented on a computer, whereas in a conventional curriculum, text printed in textbooks is usually the dominant presentation form. Even though various presentation forms are used in a conventional curriculum, they usually come from different media sources. For example, text may be from textbooks, pictures from textbooks or graph books, and audio from audiotapes. Figure 2.1 shows the difference between a conventional curriculum and a multimedia curriculum on medium.

video

videotapes

audio

audiotapes

textbook text

graph

… animation

video

audio

computer

graph

text

Figure 2.1: Difference between a multimedia curriculum (left) and a conventional curriculum (right)

Furthermore, a multimedia curriculum often offers more possibilities than a conventional curriculum. For example, a multimedia curriculum may have more flexibility and interactivity than a conventional curriculum. Users can easily jump to another knowledge unit by clicking a hotspot. They can find some related topics by keyword searching, or they can even zoom in on or

Page 46: computer support for multimedia curriculum design

34 Chapter 2

zoom out from a picture. In addition, a multimedia curriculum is no longer an inert information repository; it can receive, process, and give immediate feedback to learners. In this study, in the absence of further distinction, the term 'curriculum' refers to the conventional curriculum, whereas the focus of this study will be on the design of a multimedia curriculum. 2.3.2 Multimedia curriculum modules As explained in section 2.2.1, a curriculum is composed of the following four components: Aims/goals/objectives, content, instructional strategies and assessment. A multimedia curriculum may have the same components, but some minor differences may exist in each component, particularly in the component of instructional strategies. In order to highlight the differences, hereafter the components of a multimedia curriculum will be called modules. In this section, the modules of a multimedia curriculum will be reclassified based on the four components and their specific instructional strategies. Instructional strategies

An instructional strategy can be defined as the way in which teachers present lesson content or they facilitate learning (Ertmer & Newby, 1993; Moore, 1992). In a classroom or group-learning environment, the following instructional strategies are usually adopted (Frazee & Rudnitski, 1995): lecturing modeling role-playing discussion collaborative learning.

However, in a multimedia-learning environment, the instructional strategies usually include the following (cf. Fenrich, 1997; Zhu, 1999): tutorial drill-and-practice simulation educational games virtual reality cognitive apprenticeship case studies testing.

Page 47: computer support for multimedia curriculum design

Multimedia curriculum and computer support systems 35

Actually, some of the above instructional strategies are interconnecting or overlapping. For example, modeling in a classroom can be implemented by simulation or educational games on a computer. Role-playing in a classroom can also be implemented by or replaced with virtual reality. Generally speaking, classroom instructional strategies are more practiced by teachers to implement classroom teaching, whereas multimedia instructional strategies are more oriented towards individual or group learning. As a multimedia curriculum is usually designed for both classroom teaching and individual (or collaborative) learning (see also Section 4.2.1), the above instructional strategies, including those used in a classroom and used in a multimedia learning environment, should be supported in a multimedia curriculum. Next, some common elements/events, which can constitute these instructional strategies, are to be summarized. Instructional events

Gagné, Briggs, and Wager (1988) summarize nine events of instruction, which normally occur in the process of instruction. These events include: 1. gaining attention; 2. informing learner of the objective; 3. stimulating recall of prerequisite learning; 4. presenting the stimulus material; 5. providing learning guidance; 6. eliciting the performance; 7. providing feedback about performance correctness; 8. assessing the performance; 9. enhancing retention and transfer. These instructional events do not invariably occur in this exact order, although this is their most probable order. Any instructional strategy can consist of or focus on some of these events. For example, a tutorial strategy may include all these events, or merely focuses on the events 4, 5, 6 and 7; a drill-and-practice strategy may focus primarily on the events 6 and 7. These events can be grouped into four stages: preparation, elaboration, practice and testing. The preparation stage includes the first three events; the elaboration stage consists of events 4 and 5; the practice stage consists of events 6 and 7; and the testing stage consists of events 8 and 9. Together with the components of a conventional curriculum, a multimedia curriculum may include the following modules:

Page 48: computer support for multimedia curriculum design

36 Chapter 2

aims/goals/objectives; content preparation; content elaboration; practice; assessment.

This study focused on the module of content elaboration. Although all above modules can be well presented and supported in a multimedia curriculum, the added values for other modules are comparatively lower in this particular application. For instance, the module of aims/goals/objectives can be printed in textbooks with text; content preparation can be performed in oral explanation by a teacher at the beginning of a class; practice can be done by using learners' practice books, and assessment can be done via a paper-based examination after a period of learning. Relatively, the module of content elaboration needs more presentation forms to present the knowledge units of a curriculum in more detail. 2.3.3 Multimedia curriculum development model Multimedia curriculum has a dual characteristic. On the one hand, it is a curriculum. On the other hand, it is also a multimedia computer program. Similarly, the development of multimedia curriculum has a dual characteristic, too. Firstly, its development has the general characteristics of conventional curriculum development. Its development activities should include preliminary research, analysis, design, formative evaluation and summative evaluation. Secondly, as a computer program, its development should follow a computer program development approach. In this study, the adopted multimedia curriculum development model is shown in Figure 2.2.

Formative evaluation

Formative evaluation

Design

Analysis

Program-ming

MC IS Summative evaluation

Preliminary research

Note: IS = instructional scenario; MC = multimedia curriculum Figure 2.2: Multimedia curriculum development model

Page 49: computer support for multimedia curriculum design

Multimedia curriculum and computer support systems 37

Compared to the curriculum development model used by Nieveen (1997), the multimedia curriculum development model adds one more cycle, which is composed of programming and formative evaluation. After the preliminary research, the conditions and constraints can be clarified, and the decision to continue or stop the project will be made. During the first circle, teacher-designers will carry out needs analysis, subject matter analysis, and learner analysis, etc. Based on the analysis results, a tentative instructional scenario will be developed. After that, formative evaluation activities will be carried out to collect comments and suggestions on the instructional scenario from various experts. Based on the comments and suggestions, revision decisions will be made and a next round of prototyping starts. This cycle may take place several rounds until a satisfying instructional scenario has been developed. In the second cycle, computer programmers will create a multimedia curriculum based on the developed instructional scenario. Similarly, after a tentative multimedia program has been created, formative evaluation activities will be carried out, and a next round of prototyping starts based on the collected comments and suggestions. This process may continue through several rounds until an optimal multimedia curriculum has been created. Afterwards, the created multimedia curriculum may need a summative evaluation to judge its quality. 2.4 EPSS CASCADE-MUCH is a computer support system. It is related to the concept of Electronic Performance Support System (EPSS). In this section, the concept of EPSS, the advantages and disadvantages of an EPSS and the design of an EPSS are presented. 2.4.1 The concept of EPSS Today's computers can provide and integrate many kinds of performance support such as information, advice and tools (Nieveen & van den Akker, 1999; Stevens & Stevens, 1995). This is related to the concept of EPSS. Instead of providing performance support separately, an EPSS is an integrated computerized environment, in which performers (users of the EPSS) can get many kinds of support in a just-in-time fashion (Gery, 1991; Nieveen, 1997; Stevens & Stevens, 1995).

Page 50: computer support for multimedia curriculum design

38 Chapter 2

Gery (1991) claims that an EPSS is a useful tool for (novice) performers because it can help them to solve difficulties or questions such as 'Why do this?' 'What is it?' 'How do I do it?' 'Please show me an example.' Gery (1991) believes that the number of questions or requests is finite, and the information associated with a given question or request can be reasonably defined. An EPSS can provide the performers with a powerful tool for overcoming their difficulties. The traditional methods for answering the questions or meeting the requests are usually to provide overhead support (hotlines, help desk, support personnel), paper-based support (manuals, periodicals, brochures, memos, reference cards), or ad-hoc training (supervisors, co-workers, trial-and-error). These traditional methods often work well, but they have some limitations. For example, overhead support may work efficiently for employees, but it is expensive and inefficient for employers to place an organization's most talented personnel in these reactive roles. Paper-based support may be flexible and easy to use, but these job aids require time-consuming and expensive updates. Formal training mostly needs performers to be away from their jobs. Also, employees may have diverse background, and the one-size-fits-all approach to training no longer works well (Gery, 1991; Legent Service, 1992). As an integrated computerized support environment, an EPSS can effectively overcome the above limitations by providing performers with alternative tools on a computer. For example, overhead support can be replaced with online help; paper-based support can be replaced with online reference, and ad-hoc training can be replaced with computer-based training. In this case, support will be available anytime when it is needed; information can also be easily updated, and costs may be reduced. Gery (1991) lists several possible kinds of software found in EPSSs, including advisory or expert systems, interactive productivity software, help systems, interactive training systems, assessment and monitoring systems, etc. In addition, she mentions that an EPSS could incorporate anything else that an employee might need, such as a tutorial, a calculator, a notebook, etc. These kinds of software or components can be reclassified into the following four broad categories (cf. Hudzina, Rowley & Wager, 1997): information advice tools training.

Page 51: computer support for multimedia curriculum design

Multimedia curriculum and computer support systems 39

Information refers to the static information and/or help usually organized in a hypertext/hypermedia format, which remain the same for various users. Advice refers to heuristic support information, which is given based on users' specific needs and the embedded intelligent expertise. For instance, advisory and monitoring (Gery, 1991) belong to this category. Tools usually refer to the support aiming at helping performers carry out tasks. Such tools may include external applications such as a word processor, a calculator, etc. The productivity and applications (Gery, 1991) belong to this category. Training usually refers to the support aiming at improving performers' skills. For instance, the training and assessment (Gery, 1991) belong to this category. The key difference between training and tools is that training focuses on skill improvement of users, whereas tools focus on task performance. In this study, the CASCADE-MUCH program includes all these four categories of support. Comparatively, information, advice and tools received more attention and efforts, but training got less. In the CASCADE-MUCH program, the support of training mainly includes some wizards and a tutorial program. The wizards were developed in the second prototype, but some were removed in successive prototypes. More discussion on wizards can be found in Section 6.2.2. The tutorial program has not been completed yet. The reason is that it was too early to develop a tutorial program for (novice) users to learn the prototype before the prototype has been completely finished. However, the tutorial program will be completed gradually in the future. 2.4.2 The potentials of EPSS Compared to conventional job aids, an EPSS has a number of assumed advantages. These advantages will be elaborated from three perspectives: users, end results and designers. From the users' (or employees') point of view, an EPSS can help them perform their tasks more efficiently than in a traditional training situation, because an EPSS can provide advice, information and/or instruction immediately when needed (Gery, 1991). With an EPSS, users do not need to remember all issues related to their work, and they can consult the EPSS on the issues they want at the time they really need it. From this point of view, an EPSS can also reduce information load during task performance. In addition, an EPSS can also improve users' professional development knowledge, because they can learn information from it by interactively using it.

Page 52: computer support for multimedia curriculum design

40 Chapter 2

From the end results' point of view, an EPSS can improve the quality of the end results. During the process of using an EPSS, users can get expert advice on how to proceed with a task and/or how to improve the quality of the end result. Furthermore, some embedded or linked tools can help them perform a task easily and even improve the quality of the product as well. From the designers' point of view, designing an EPSS has the potential to increase its domain design and development knowledge, and make implicit knowledge become explicit. During the process of designing an EPSS, the designers need to collect information and intelligent expertise in order to provide users with information and advice. With the growth of the EPSS, the domain knowledge of the designers will also be enlarged (cf. Paquette, Aubin & Cervier, 1994). In addition, the content of the EPSS will be presented clearly and explicitly in the system. It will help the designers make their implicit knowledge become more explicit and transparent to the users. Furthermore, the use of EPSSS may help promote organizational learning (Nieveen, 1997). Practically every organization possesses a shared knowledge base, such as some collection of information regarding techniques, methods and procedures that are common to the work. Quite often, this knowledge base grows intuitively, and it is not formalized into a written form. The idea of organizational learning includes the notion that, on the one hand, the captured knowledge from individuals and teams will be stored in the knowledge base and will be available for the whole organization; on the other hand, the communication tools provided with the EPSSs will give the team members opportunities to exchange their individual ideas. When individuals change jobs, their knowledge doesn't leave with them; the stored information in a computer can be used for the future. Even better, less experienced individuals can use such a support system to learn from it, to communicate with other experienced members, and to become familiar with the development process. In addition to the assumed advantages above, some concerns about EPSS need to be addressed here. First of all, an EPSS may be not appropriate in some cases, for example, where successful performance depends on habitual, automatic, and seamless performance like in surgery (Nieveen 1997). Second, few exemplary systems of EPSS exist since the concept is new. It might be hard for performers, designers, as well as administrators to understand the concept without seeing and/or using an existing EPSS. Consequently, resistance may exist that may prevent shifting some resources from training to EPSS (Legent service, 1992). Third, people who lack competent computer skills may fear the use of an EPSS.

Page 53: computer support for multimedia curriculum design

Multimedia curriculum and computer support systems 41

In this study, all of the above three advantages were expected to be exploited in the CASCADE-MUCH program. Also, by taking these advantages, the design aims (see Section 1.3.2) of the program were also expected to be effectively achieved. For instance, the first advantage of helping users perform tasks and improving their professional knowledge is closely related to the aim (a) and (c) on page 15. It means that the CASCADE-MUCH program, as an EPSS, has the potential to help teacher-designer to make instructional scenarios easily and to improve their professional knowledge. Similarly, the second advantage of improving the quality of the end results is also related to the aim (a) and (b). 2.4.3 The design of EPSS Typically two approaches can be used to design an EPSS (Gustafson, 2000). The first one is a classic instructional design approach (or waterfall approach in software engineering). Analysis, design, development, implementation, formative evaluation and summative evaluation are directly applicable for designing an EPSS. The classic design approach works best when the performance process is stable, and related information is available. The second approach is prototyping. It is a process of producing successive trial versions before developing a final system. The prototyping approach works best when the aims or context conditions of the support system are unclear, complex, complicated, or still changing. In this study, the design approach adopted is prototyping. The detailed design of the study will be introduced in the following chapters. During designing an EPSS, a number of considerations should be taken into account. For example, Gustafson (2000) summarizes several major considerations when creating (or selecting) an EPSS such as: 1. Black box/glass box design

In a black box design, the inner structure (like rules, data structure, models) of a support system is invisible to the users. In contrast, a glass box design makes those underlying rules, models, algorithms, and other structures readily available, and perhaps even requires users to examine them.

2. Part-task/whole task support Some workplace tasks may be very complex or lengthy. It might be hard for an EPSS to provide support for the whole process, although support for parts of the task should be possible.

Page 54: computer support for multimedia curriculum design

42 Chapter 2

3. Embedded/linked tools Some tools have been available externally such as flowcharting and concept mapping tools. In this case, it is not necessary to create them again. Decisions must be made as to whether support tools will be embedded or linked to the support system. When embedded, the support tools become an internal part of an EPSS, and all of the pieces of an EPSS are seamlessly integrated. When linked, the support tools are readily available, but access usually requires some action on the part of the users.

4. Static/dynamic system A static system remains the same always. Users will be unable to tailor or modify it, while a dynamic system may learn from the users and grow during the process.

In addition, as an integrated computerized support system, an EPSS needs a lot of tools and technologies during its design. Gery (1991) summarized a list of tools for designing components of an EPSS, including information processing tools (such as hypertext, text management and retrieval systems), training development tools (such as CBT authoring systems, concurrent authoring systems), advisory system development tools (such as knowledge processors, expert systems), general purpose tools (such as word processors, graphical editors), special purpose tools (such as animation software), and programming environments (such as object-oriented programming languages). In this study, CASCADE-MUCH followed a glass-box objective design. By using the CASCADE-MUCH program, teacher-designers can, on the one hand, produce instructional scenarios; on the other hand, they can also learn some information from it by accessing the information and/or suggestions. It is a part-task support system, because it focused on the design phase rather than on the entire process. Several external tools, such as Microsoft Word and Mindman, were linked to the CASCADE-MUCH program. For the time being, the CASCADE-MUCH program is a rather static system. However, users can add or modify some information or tips in the text files or in the databases. In addition, in agreement with Gery (1991), developing the CASCADE-MUCH program required many tools and technologies, such as the Delphi programming platform, the communication technology between Delphi and Microsoft Word, and the technology of database management.

Page 55: computer support for multimedia curriculum design

Multimedia curriculum and computer support systems 43

2.5 Existing computer support systems for curriculum development

Over the years, a number of computer support systems for curriculum or instructional design have been produced. In this section, some existing computer support systems for curriculum development are to be introduced. These support systems include two CASCADE studies (CASCADE and CASCADE-SEA) and several other support systems. 2.5.1 CASCADE The CASCADE study was initiated by the Department of Curriculum at the University of Twente together with the Dutch National Institute for Curriculum Development (SLO) in 1993. Although the concept of formative evaluation is familiar to many curriculum developers, research has shown that such activities are often neglected in the development process. The material developers (curriculum developers) at the SLO have been, in this sense, no exception (van den Akker, Boersma & Nies, 1990). This group has indicated that such evaluations are seen to be complex and time-consuming. As a result, opportunities for small-scale efficient evaluations are often underutilized. The initial CASCADE research (Nieveen, 1997) aimed to explore ways to support relatively quick and easy evaluation efforts. It was hoped that the CASCADE program could make formative evaluation activities less daunting and more efficient to be carried out. The CASCADE study contained three main phases: analysis, prototyping and evaluation. The analysis phase was driven by the question: 'In what ways can a computer-based support system for curriculum development contribute to the optimization of development practices within the SLO?' Analyses were conducted on various forms of computer supported curriculum development systems, and an initial prototype was developed for comments and discussion with experts in the domain of computer supported curriculum development. In the prototyping phase, five successive prototypes were created and revised based on formative evaluation results. The prototyping process was concluded with a final version of the program. This was then tested to determine if the system would be useful to a larger group of SLO developers, and to see if the system would yield real effects. 'What is the practicality and effectiveness of the final version of CASCADE in the SLO practice?' was the question leading this final phase of the study.

Page 56: computer support for multimedia curriculum design

44 Chapter 2

The final evaluation concluded that the use of CASCADE could: improve consistency of evaluation plans and activities by helping to

structure decision-making as well as aiding in weighing options; motivate developers and offer reassurance in one's ability to conduct

formative evaluation activities, in part by offering an overview of such activities; save time by offering assistance in developing a framework for an

evaluation plan and by offering sample documents (such as evaluation instruments) that can be adjusted for one's own situation; support the underpinnings of decisions regarding the design and execution

of evaluation by offering explanations about the concepts used. In summary, CASCADE evolved as the name for both the line of research and a computer program. The original CASCADE program focused on the evaluation phase of the entire process. This system was designed to help professional curriculum developers conduct formative evaluations of exemplary teacher and learner materials. Among other findings, evaluation of this program indicated that such a tool may offer much to the world of curriculum development, particularly with regard to the creation of classroom materials (Nieveen, 1997). For additional information regarding CASCADE, please refer to Nieveen (1997), Nieveen and van den Akker (1999), and http://projects.edte.utwente.nl/cascade/original/. 2.5.2 CASCADE-SEA The CASCADE-SEA study (McKenney, 1999, 2001) has been set up to explore how computer supported curriculum development might be a potential solution to some of the problems encountered by materials developers in the domain of Science Education in southern Africa. Educational reform projects have been in operation in southern Africa for over 20 years, and the creation of locally relevant classroom materials has been a recurring theme among many of these projects. Various ongoing efforts stress the notion that teacher development is directly linked to curriculum development. CASCADE-SEA aids facilitator teachers through the process of translating their good ideas (about lesson materials) into a form that is valuable to others. This support system strives to promote improved task performance (better quality materials), organizational learning (among resource teachers) and improved curriculum design and development knowledge (teacher professional development). These factors could then

Page 57: computer support for multimedia curriculum design

Multimedia curriculum and computer support systems 45

contribute to additional benefits. CASCADE-SEA evolved through four prototype versions. Throughout this evolution, two main kinds of changes took place. Each prototype aimed at learning more about a specific aspect or area of curriculum development (in this case, science education materials design) and at the same time, fine-tuning design-specifications for the program as a whole was also on the agenda. The CASCADE-SEA program currently consists of two main elements: CD-ROM and a web site. Although the number of CASCADE-SEA users with Internet access is rapidly increasing, many still work with the system in an off-line setting. For this reason, the web site is a supplement to the main program. Whereas the CD-ROM aids the material designers in making personal decisions about how to create a series of lesson plans (a teacher guide), the web site aims to foster communication between material designers. The database contains a variety of completed lesson plans as well as building blocks for materials (clip art, activity ideas, etc.) which visitors are welcome to use. They are also encouraged to contribute any similar resources for biology, chemistry, physics or mathematics. The CASCADE-SEA CD-ROM aims to support those groups and individuals involved in the process of creating exemplary lesson materials or teacher guides, usually to be shared among colleagues in the same region. Toward that end, the program asks the user(s) to think about what they would like to achieve (why they are making materials, and what kinds of materials would be useful for their particular setting). If the developer already has a basic rationale in mind, then the computer helps to make this explicit and generates a 'rationale profile' that may then be used in discussion with co-developers. Should the user have difficulty-determining key issues related to the materials (s)he is about to develop, then CASCADE-SEA will recommend that the analysis section be visited. In the analysis portion of the program, support is offered in conducting a material needs and context analysis, which (when completed) will then aid in forming a rationale. Once the user has generated sufficient specifications regarding the kind(s) of materials to be developed, the design phase supports the creation of these materials. The design area helps the user to map out a lesson series, build individual lessons and to think about the layout, form and style to be applied. For users who have completed some of the development (ranging from rationale formation to a complete lesson series), support is also available for conducting a formative evaluation of the materials designed so far. The evaluation component is heavily based on the former CASCADE program, although it has been translated in terms of both language and context. For additional information regarding CASCADE-SEA,

Page 58: computer support for multimedia curriculum design

46 Chapter 2

please refer to McKenney (1999, 2001) and http://projects.edte.utwente.nl/ cascade/seastudy/. 2.5.3 Other support systems In addition to these two existing CASCADE studies, another study CASCADE-IMEI has been started and is going on now. It is related to Innovative Mathematics Education in Indonesia. For additional information regarding the CASCADE-IMEI research, please refer to Zulkardi (1999), or visit the web site: http://projects.edte.utwente.nl/cascade/imei/ or http://www.cascadeimei.com. Furthermore, some other computer support systems for curriculum/instructional design can also be found in literature, such as AGD, ID Expert, ID Library, ILCE, CEDID, ECC COCOS, GAIDA, QUEUE, SimQuest and GTE. Brief introductions of these support systems can be found in Nieveen (1997), and also in Nieveen and Gustafson (1999, 2000). In this section, a computer support system called Designer's Edge is to be introduced, because many ideas behind the Designer's Edge are similar to those of the CASCADE-MUCH study. For example, the Designer's Edge aims to design a multimedia course running on a computer, not a conventional course on paper. This aim is the same as that of CASCADE-MUCH. In Designer's Edge, the multimedia course can be a Windows-based multimedia course, or a web-based online course. It is also similar to CASCADE-MUCH, in which the final curriculum is a Windows-based multimedia curriculum. In addition, a storyboard is popularly used in Designer's Edge to help designers create screen layouts, and the designed course can be output to other authoring systems such as Quest or directly to HTML or JAVA templates for online delivery. In CASCADE-MUCH, the instructional scenario (or storyboard) is the outcome, which will be used by computer programmers to create a multimedia curriculum. The instructional scenario can be exported to Microsoft Word. More comparisons can be found in Table 2.3. According to the introduction on the web site: http://www.mentergy.com, Designer's Edge is an integrated computer support system, which focuses on the common activities of instructional design. Its creation was based on the standard instructional system design model, and it provides support for an entire 12-phase instructional development process with special emphasis on the analysis, design and evaluation of effective technology-based training.

Page 59: computer support for multimedia curriculum design

Multimedia curriculum and computer support systems 47

There are two distinct target group users for Designer's Edge. The first group consists of experienced instructional designers (trainers/educators) who are familiar with the process of creating instructional materials. For this group, Designer's Edge can be used to accelerate the instructional design process by offering design consistency in organizing data, writing reports, storyboarding and media pre-production, etc. The second group of target users consists of subject matter experts (SMEs) and human resource personnel who find themselves in a training role because of their subject-matter expertise. Subject matter experts are most effective when involved in the early stages of development, making decisions about content, objectives and presentation sequencing. But many companies have engaged in failed attempts to teach subject matter experts to become authoring system specialists. For this group, Designer's Edge can help them become competent authoring system specialists. Designer's Edge can help designers effectively perform the following tasks: brainstorming during needs analysis; writing objectives; outlining content; writing test questions; mapping out a course; storyboarding; managing instructional media elements.

In carrying out these tasks, Designer's Edge demonstrates a number of characteristics, such as: 12-phase instructional design process support; customizable interface; copying or sharing design data; full-screen and graphical storyboarding; instructional advice.

The detailed and further information regarding Designer's Edge can be found in Chapman (1998) or on the web site: http://www. mentergy.com 2.5.4 Comparison of the support systems and implications on the

design of the study Computer support systems can be compared in different ways based on different criteria. Here the comparison is made based on the scheme defined by Nieveen and Gustafson (1999), which mainly includes the following

Page 60: computer support for multimedia curriculum design

48 Chapter 2

attributes: A. Type of output

Type of output refers to both curriculum levels and characteristics of the results. With respect to curriculum levels, the output of a computer support system might be lessons, courses, or a (computer) product. With respect to characteristics of the result, the output might be oriented to learners or teachers in a form of paper-based, computer-based or web-based, and the output may be used in a site specific or generic context.

B. Intended target groups Intended target group is another element worthy of comparison. It mainly includes two perspectives: expertise of user group (professional designers, subject matter experts, teachers, or learners), and computer experience (low or high).

C. Type of development process Type of development process is mainly composed of two perspectives: paradigm for engaging in educational and training development, and underlying elements of the systematic approach to development of education and training. The paradigm can be divided into four perspectives (see also Section 2.2.2): instrumental, communicative, pragmatic, and artistic, defined by Visscher-Voerman (1999). Although various educational and training systems may involve various elements/activities, some certain key elements can be found in most literature, such as: analysis, design, development, implementation and evaluation. These are the elements used in the comparison scheme.

D. Task support Task support is the last attribute to compare, which includes types of support and adaptability of support. In the scheme, type of support includes communication aids, job aids, and training aids. Support can be tailored inside the tool, outside the tool, inside networked tool, or not at all (closed).

The comparison of the computer support systems introduced in this section is illustrated in Table 2.3, in which the design of CASCADE-MUCH is listed in the shaded column.

Page 61: computer support for multimedia curriculum design

Multimedia curriculum and computer support systems 49

Table 2.3: The comparison of the computer support systems

CASCADE

CASCADE-SEA

Designer's Edge

CASCADE-MUCH

Curriculum levels

- Lesson - Course

- Lesson - Course

- Computer program

- Computer program

A

Characteris-tics of results

- Developer-oriented

- Paper-based - Site specific

and generic

- Teacher-oriented

- Paper-based - Site specific

and generic

- Developer-oriented

- Computer-based

- Generic

- Learner-oriented and teacher-oriented

- Computer based

- Site specific and generic

Expertise of user group

- Professional designers

- Teachers - Instructional designers

- SMEs or human resource personnel

- Teacher-designers

B

Computer experience

- Low - Low - Low - Low

Develop-ment paradigm

- Pragmatic - Pragmatic - (not clear) - Pragmatic C

Elements of systematic approach

- Formative evaluation

- Analysis - Design - Develop-

ment - Implementa-

tion - Evaluation

- Analysis - Design - Evaluation

- Analysis - Design

Types of support

- Job aids - Job aids - Communi-

cation aids - Training

aids

- Job aids - Communi-

cation aids - Training

aids

- Job aids - Training

aids - Communi-

cation aids

D

Adaptability of support

- Inside tool - Outside tool

- Inside tool - Outside tool

- Inside tool - Outside tool

- Outside tool

Page 62: computer support for multimedia curriculum design

50 Chapter 2

This table shows that the two CASCADE studies (CASCADE and CASCADE-SEA) result in instructional materials in a form of lessons or courses which are paper-based, whereas Designer' Edge results in a computer-based program. The CASCADE study is oriented to curriculum developers, who have expertise of professional curriculum development. The CASCADE-SEA study is oriented to subject teachers, who have expertise of their subject domains; Designer's Edge is oriented to educational and/or training systems developers, who have expertise of instructional design, subject matter or human resource development. The two CASCADE studies are site-specific but have the potential to be extended to other generic contexts, whereas Designer's Edge is rather generic. All of the support systems require only a low level of computer experience. The development approach adopted by the two CASCADE studies are both pragmatic, whereas the development approach adopted by Designer's Edge is not clear. The CASCADE study investigates computer support for the element of formative evaluation in the context of SLO. The CASCADE-SEA study investigates computer support for the whole process of teacher-based materials development in the context of Africa. Designer's Edge provides computer support for the 12-phase instructional design process, especially focusing on the elements of analysis, design, and evaluation. All of the above support systems provide job aids, and most of these systems, with the exception of CASCADE, provide communication aids. Also, CASCADE-SEA and Designer's Edge provide training tools. With respect to adaptability of support, the support of CASCADE, CASCADE-SEA and Designer's Edge can be tailored both inside and outside the systems. CASCADE-MUCH proceeds with a similar study to the above systems, but with different focus and context. The other CASCADE studies focus on the development of paper-based lesson and/or course materials in the site-specific or generic context, whereas the CASCADE-MUCH study focuses on the development of computer-based multimedia curriculum in the context of China, particularly in Shanghai. The characteristics and design of these systems have the following implications for the design of the CASCADE-MUCH study. First, CASCADE-MUCH adopts the pragmatic development approach, which is the same as that of other CASCADE studies. This approach has been explained in Section 2.2.2.

Page 63: computer support for multimedia curriculum design

Multimedia curriculum and computer support systems 51

Second, CASCADE-MUCH focuses on two phases --analysis and design, in particular the design phase-- of the systematic development process rather than on the whole process. Among the above support systems, only CASCADE-SEA focuses on the whole process, and the other systems mostly focus on some specific phases. For example, CASCADE focuses on the phase of formative evaluation, and Designer's Edge focuses on the phases of analysis, design, and evaluation. Third, CASCADE-MUCH attempts to involve a web site (http://projects.edte.utwente.nl/cascade/much/ or http://cascade-much. 20m.com) as a communication tool to provide users with updated information and technical service. Most of the support systems provide such communication tools. Furthermore, the rapid growth of the Internet being witnessed presently allows communication tools to be easily created as well. Fourth, the CASCADE-MUCH study borrows some design perspectives and experiences from the CASCADE study, such as components and quality criteria. In addition, object-oriented technology is used to describe screen elements in Designer's Edge. In the CASCADE-MUCH study, object-oriented technology is adopted to represent knowledge units with various presentation forms. 2.6 Conclusions Although the debate on relationships between media and learning and between multimedia and learning has not yet been resolved, a consensus has been reached that some proper media can make hard-to-implement instructional methods possible. Multimedia is considered to be at least as effective as conventional methods, and frequently it can improve the motivation of learners and the interaction between learners and a multimedia curriculum. A multimedia curriculum and a conventional curriculum have in common that they both cover the same content with the same presentation forms. However, in a conventional curriculum, text is usually the main presentation form. Even when several presentation forms are used, they are mostly from different media sources. In a multimedia curriculum, usually various presentation forms are used simultaneously, and they are encoded and presented on a single medium: the computer.

Page 64: computer support for multimedia curriculum design

52 Chapter 2

Multimedia curriculum development should follow both a curriculum development process and a multimedia program development process. After an instructional scenario has been developed by teacher-designers (usually after several rounds of prototyping), computer programmers start to create a multimedia curriculum based on it. Similarly, computer programmers may also need several rounds of prototyping until a satisfying multimedia curriculum is finally created. EPSSs have some assumed advantages. CASCADE-MUCH, as an EPSS, is expected to be able to help teacher-designers develop instructional scenarios and improve the quality of the produced instructional scenarios by accessing the provided information, suggestion, tools and other supporting materials. Also, the process of designing scenarios is expected to improve teacher-designers' professional knowledge about multimedia curriculum design.

Page 65: computer support for multimedia curriculum design

53

Chapter 3 Prototype development

P

rototyping has been the main approach during the design of CASCADE-MUCH. The study progressed through four rounds of prototyping, focusing on four components: content, support, interface, and scenario. In this chapter, each

prototype will be introduced in detail. This chapter starts with the preliminary choices in Section 3.1. Section 3.2 presents an overview of the prototyping process and the prototypes, themselves. The first prototype is explained in Section 3.3, focusing on the formulation of the content, support and interface. The second prototype is elaborated in Section 3.4, focusing on the design of the content, support and scenario. The third prototype is covered in Section 3.5, focusing on the improvement of the content, support and scenario. The fourth prototype is described in Section 3.6, focusing on the revision of the content and support. In addition, each prototype underwent a formative evaluation; the design and results of those evaluations are presented at the end of each section. 3.1 Preliminary choices According to de Hoog, et al. (1994) and Nieveen (1997), a computer program can be broken down into some key components in order to make its development process more transparent. In this study, the prototype has been broken down into four components: content, support, interface and scenario. Based on the introduction of the context in Chapter 1, and the literature review and study of examples in Chapter 2, some preliminary choices for these four components were formulated at the beginning of the development process. In this section, these preliminary choices will be briefly introduced. With regard to the content, two preliminary choices were made. First, the program should be closely related to multimedia and curriculum development, since the prototype aimed at facilitating teacher-designers in designing a multimedia curriculum presented on a computer. Second, it should include both theoretical models and practical guidelines. The theoretical models should help users understand why the content was put on the screens, and it could also help them improve their professional knowledge for multimedia

Page 66: computer support for multimedia curriculum design

54 Chapter 3

curriculum design. The practical guidelines should combine the theoretical models and the practical situation in the context of Shanghai, and provide teacher-designers with more practical information. With regard to the support, the prototype should integrate many types of support, since the prototype is designed to be an EPSS to help teacher-designers develop instructional scenarios. As explained in Section 2.4, EPSS is an integrated computerized support environment in which users can get many types of support just in time. An EPSS usually integrates one or more of the four following categories of support: information, advice, tools, and training (cf. Hudzina, et al., 1997). The prototype should integrate some or all of these four broad categories of support. With regard to the interface, the following two choices were made. First, the program should be consistent. On the one hand, it should be externally consistent to some extent with other computer applications. In this regard, users should not feel confused when they first see the interface, and they should be able to start with it easily. On the other hand, it should be internally consistent. For instance, the fonts, colors, and buttons should be the same on different screens, and the same elements on different screens should be in the same positions. In this case, navigating within the prototype would be easy to learn. Second, it should be flexible. The prototype should include several parts so that different users can start with, or work on, different parts based on their specific experiences or needs. In addition, it should provide both linear and non-linear navigation tools, and most actions in the prototype should allow the user to choose between either mouse or keyboard in order to meet various users' preferences. With regard to the scenario, two preliminary choices were made. First, a scenario should include information, which might be useful for computer programmers and for stimulating discussions between teacher-designers and computer programmers, because the instructional scenario is intended to achieve these two purposes. Second, since a lot of information would be included in an instructional scenario, the information should be well structured, otherwise it might be hard for computer programmers to read and/or understand it. In order to be well structured, the instructional scenario should be in a format that can be readily exported to Microsoft Word. Microsoft Word is a powerful word processing tool and can help to organize the produced instructional scenario in an elegant style.

Page 67: computer support for multimedia curriculum design

Prototype development 55

3.2 Overview of the prototyping process and the prototypes The program progressed through four rounds of prototyping, focusing on the four components: content, support, interface and scenario. At the end of each round of prototyping, formative evaluation activities were carried out and revision decisions were made based on the comments and suggestions. Each prototype was revised by integrating the revision decisions and new design ideas in a new round of prototyping. In this section, the prototyping process and the characteristics of the structure and the four components of each prototype will be briefly introduced. 3.2.1 Prototyping process The actual prototyping process of the program is depicted in Figure 3.1. The first round of prototyping focused on the components of content, interface and support. In the end, a formative evaluation was carried out in Shanghai. The group of participants was mainly composed of intended target users, curriculum or instructional designers, and computer-based learning (CBL) designers. More detailed information of the first prototype can be found in Section 3.3.

IV III II I

Scenario Support

Content Interface

Notes: I, II, III, IV refer to the four prototypes respectively more focus less focus Figure 3.1: The prototyping process

Page 68: computer support for multimedia curriculum design

56 Chapter 3

The second round of prototyping focused on the components of content, support and scenario. At the end of the second round of prototyping, another formative evaluation activity was carried out in Shanghai. The characteristics of the participants were similar to those in the first round of prototyping. More detailed information of the second prototype can be found in Section 3.4. The third round of prototyping focused on the content, support and scenario. At the end of this round of prototyping, two formative evaluation activities were carried out in Shanghai and at the University of Twente (UT) in the Netherlands respectively. In Shanghai, three micro-evaluation workshops were organized. The groups of participants of the first two workshops consisted of intended target users. The participants of the third workshop were computer programmers, who developed multimedia programs based on the produced scenarios. After this micro-evaluation activity in Shanghai, minor modifications were made, and an expert appraisal workshop was organized at the UT. The participants were experts in curriculum development, instructional design or multimedia technology. The detailed description of the third prototype can be found in Section 3.5. The fourth round of prototyping focused on the content and support. After the expert appraisal at the UT, some content and support of the prototype were revised. In order to check whether the revised prototype was valid in the context of Shanghai, another expert appraisal workshop was carried out in Shanghai following the revisions. The participants in this workshop were also experts of curriculum development, instructional design or multimedia technology. The detailed description of this prototype can be found in Section 3.6. 3.2.2 Evolution of the structure and the four components Table 3.1 gives an extensive overview of the structure and the four components (content, support, interface and scenario) of each prototype. In this section, the evolution of the structure and the four components of the program will be described. The first prototype was composed of two parts: Designer's Aid and Edit Panel. The Designer's Aid part aimed at guiding (novice) designers of a multimedia curriculum in designing an instructional scenario step-by-step; the Edit Panel part provided an editing environment in which (experienced) designers of a multimedia curriculum could directly design an instructional scenario without following the step-by-step guidance. In the second

Page 69: computer support for multimedia curriculum design

Prototype development 57

prototype, one more part was added to the program: Main Frame. The Main Frame part aimed at providing a more comprehensive environment, in which the various parts were connected and users could easily visit other parts. In the third prototype, another part was added: Tutorial. The Tutorial part aimed at providing an interactive training environment in which novice users could learn how to use the program. More description of the relationship between these parts will be given in Section 4.1. The component of content got attention during each round of prototyping. Table 3.1 shows the evolution of some issues such as the definition of multimedia curriculum, the goals and usage of a multimedia curriculum. For example, in the first prototype, the goals of an electronic curriculum (this would be called multimedia curriculum since the second prototype) was determined to meet the needs of quality-driven education and test-driven education. Since the second prototype, the goals have been changed to be: i) basic knowledge and skills learning; ii) extended knowledge and higher-level skills improvement; and iii) attitudes development. This division of goals has remained the same to the final version. Four broad categories of support (information, advice, tools and training) were involved in each prototype, and each broad category was broken down into some specific types of support. As illustrated in Table 3.1, during the evolutionary process, some types of support were added, removed and/or changed. For example, the support of explanations and examples were separated in the first two prototypes, but they were merged in the third prototype; some wizards were added to the second prototype, but some were removed in the third prototype. The interface was basically formulated in the first prototype, but some improvements were made in the succeeding prototypes. For example, in the first prototype, each screen had a logo, but most logos were temporarily selected and they were not of the same size or with the same style. Since the third prototype, all logos have been changed to be more consistent. In addition, the interface of Edit Panel changed a lot in the third prototype. In the first two prototypes, all media formats were shown in a shared textbox, and only one media format could be shown at a time. In the third prototype, the content representation part provided a separate folder for each presentation form, and each presentation form was organized in an object-oriented style.

Page 70: computer support for multimedia curriculum design

58 Chapter 3

The component of scenario was gradually formulated over several prototypes. In the first prototype, the concept of scenario was not explicitly introduced, but a tentative idea was that all information needed by computer programmers should be included in an instructional scenario. In the second prototype, the concept of scenario was temporarily formulated based on the perceptions of some related concepts such as storyboard, super storyboard and programmer-ready materials (cf. Champman, 1998; Harrison, 1995). A paper-based scenario was developed and formatively evaluated in the second round of prototyping. Since the third prototype, the program has included the ability to produce instructional scenarios and export them to Microsoft Word automatically. The unique characteristics of these four components of each prototype will be explained in the following sections, and the detailed description of each component in the final version will be given in Chapter 4.

Page 71: computer support for multimedia curriculum design

Tabl

e 3.

1:

Over

view

of th

e pro

totyp

es

Ch

oice

sIs

sues

Pro

toty

pe

1 (§

3.3

) P

roto

typ

e 2

(§ 3

.4)

Pro

toty

pe

3 (§

3.5

) P

roto

typ

e 4

(§ 3

.6)

Par

ts

Des

igne

r's A

id

Edi

t Pan

el +

Main

Fra

me

+ T

utor

ial

- S T R U C T U R E

Con

nec

tion

bet

wee

n

the

par

ts

Fro

m D

esig

ner's

Aid

to

Edi

t Pan

el C

licki

ng th

e ed

it bu

tton

each

tim

e in

voke

d a

copy

of

Edi

t Pan

el.

+ Fr

om th

e M

ain

Fram

e pa

rt to

D

esig

ner's

Aid

, Edi

t Pa

nel,

and

Wor

d.

Inv

oked

the

sam

e co

py o

f oth

er p

arts

. -

Con

cep

ts

Elec

troni

c cu

rric

ulum

M

ultim

edia

curr

icul

um

Mul

timed

ia cu

rric

ulum

-

- -

1. I

t sho

uld

cove

r re

lated

con

tent

.

-

Prototype development 59

Mul

timed

ia fo

rmat

s P

rese

ntat

ion

form

sR

elat

ion

ship

bet

wee

n

a m

ult

imed

ia

curr

icu

lum

an

d a

co

nve

nti

onal

cu

rric

ulu

m

Goa

ls o

f a

mu

ltim

edia

cu

rric

ulu

m

Qua

lity-

driv

en

educ

atio

n

Tes

t-driv

en

educ

atio

n

Bas

ic kn

owled

ge

and

skill

s E

xten

ded

know

-led

ge a

nd h

ighe

r le

vel s

kills

A

ttitu

de

-

C O N T E N T

2. B

oth

theo

retic

al m

odel

s and

pr

actic

al gu

ide-

lines

shou

ld b

e in

clud

ed.

Usa

ge o

f a

mu

lti-

med

ia c

urr

icu

lum

C

lassr

oom

teac

hing

I

ndiv

idua

l lea

rnin

g +

Colla

bora

tive

learn

ing

-

*

To b

e con

tinue

d on

the n

ext p

age

*:

see

Sect

ion

2.3.

1

Page 72: computer support for multimedia curriculum design

C

hoi

ces

Issu

esP

roto

typ

e 1

(§ 3

.3)

Pro

toty

pe

2 (§

3.4

) P

roto

typ

e 3

(§ 3

.5)

Pro

toty

pe

4 (§

3.6

) L

earn

ing

theo

ries

of

mu

ltim

edia

cu

rric

ulu

m d

esig

n

Obj

ectiv

ism

Con

stru

ctiv

ism

- M

ove

to th

e he

lp

file

-

Sub

ject

s su

pp

orte

d

All

subj

ects

B

iolo

gy

Geo

grap

hy

- -

Lea

rner

an

alys

is

Com

pute

r ski

lls

+ Su

bjec

t kno

w-

ledg

e/sk

ills

- S

ubjec

t kno

w-

ledg

e/sk

ills

Com

pute

r ski

lls

Lea

rnin

g st

yles

Mod

ule

s of

a

mu

ltim

edia

cu

rric

ulu

m

Con

tent

ela

bora

tion

Pra

ctice

T

est

Exp

erim

ent

Sim

ulat

ion/

virtu

al re

ality

C

ase

stud

ies/

pr

oblem

solv

ing

- G

oals

Con

tent

ela

bora

tion

Res

ourc

es

Exp

erim

ent/

prac

-tic

e/ca

se st

udie

s/

prob

lem so

lvin

g T

est

Col

labor

ativ

e lea

rnin

g

Aim

s/go

als/

objec

tives

C

onte

nt

prep

arat

ion

Con

tent

ela

bora

tion

Pra

ctice

A

sses

smen

t

C O N T E N T

Dev

elop

men

t el

emen

ts

Con

tent

selec

tion

C

onte

nt a

nalys

is C

onte

nt d

escr

iptio

n C

onte

nt

orga

niza

tion

Int

erfa

ce d

esig

n

Int

erfa

ce d

esig

n C

onte

nt se

lectio

n C

onte

nt

desc

riptio

n C

onte

nt

orga

niza

tion

Con

tent

selec

tion

Con

tent

re

pres

enta

tion

Con

tent

or

gani

zatio

n

Int

erfa

ce d

esig

n

-

To b

e con

tinue

d on

the n

ext p

age

60 Chapter 3

Page 73: computer support for multimedia curriculum design

C

hoi

ces

Is

sues

Pro

toty

pe

1 (§

3.3

) P

roto

typ

e 2

(§ 3

.4)

Pro

toty

pe

3 (§

3.5

) P

roto

typ

e 4

(§ 3

.6)

Exp

lan

atio

ns

& E

xam

ple

s U

sed

the

stat

us

butto

n to

shift

. C

ould

be

selec

ted

from

the

pop

up

men

u.

- A

dded

a h

otsp

ot o

f ex

ampl

e to

the

expl

anat

ion

in th

e he

lp fi

le.

Got

an

expl

anat

ion

first

; got

an

exam

ple

from

the

expl

anat

ion

win

dow

.

-

Hel

p

The

oret

ical m

odels

P

ract

ical g

uide

lines

-

--

Hin

ts

Exp

lanat

ions

of

butto

ns

-

-

-

Tip

s I

nfor

mat

ion

abou

t so

me

oper

atio

nal s

kills

-

--

Lea

rnin

g m

ater

ials

F

ew

Sav

ed in

loca

l driv

ers

- +

Link

s to

the

Inte

rnet

-

Sugg

esti

ons

Exp

ert a

dvice

-

- + +

Inte

llige

nt e

xper

tise

Exp

lanat

ions

P

revi

ew

W

hat n

ext

- +

Exp

lanat

ions

W

izar

ds

E

ach

elem

ent h

ad

one.

Som

e w

izar

ds w

ere

rem

oved

. -

Tu

tori

al

Ver

y te

ntat

ive

- C

once

pt

map

pin

g

'M

indm

an' w

as li

nked

. -

Flo

wch

art

No

prop

er to

ols w

ere

foun

d.

-

Web

sit

e

+ ht

tp:/

/cas

cade

-m

uch.

20m

.com

S U P P O R T

1. M

any

type

s of

supp

ort s

houl

d be

inclu

ded

Tra

nsl

atio

n

Chi

nese

C

hine

se

Eng

lish

- -

To b

e con

tinue

d on

the n

ext p

age

Prototype development 61

Page 74: computer support for multimedia curriculum design

C

hoi

ces

Issu

esP

roto

typ

e 1

(§ 3

.3)

Pro

toty

pe

2 (§

3.4

) P

roto

typ

e 3

(§ 3

.5)

Pro

toty

pe

4 (§

3.6

) F

onts

S

ong

(Chi

nese

) S

mall

size

S

ong,

Tim

es N

ew

Rom

an

Sm

all si

ze

Son

g, C

omic

Sans

MS

Med

ium

size

-

Col

ors

Key

wor

ds a

nd n

ew

wor

ds u

sed

diffe

rent

co

lors

(red

& b

lue)

.

Key

wor

ds a

nd n

ew

wor

ds u

sed

the

sam

e co

lor.

- -

Bu

tton

s T

extu

al lab

els (|

<, <

, >

, >|)

E

ach

butto

n ha

d an

ico

n an

d te

xt.

The

'Tut

orial

' but

ton

was

on

the

first

scre

en.

Sho

rt-cu

t key

s wer

e ad

ded

to so

me

butto

ns.

The

'Tut

orial

' but

ton

was

mov

ed to

the

help

m

enu.

Too

lbar

T

he to

ol b

utto

ns w

ere

tem

pora

rily

loca

ted.

-

The

tool

but

tons

wer

e gr

oupe

d: o

ne g

roup

w

as fo

r all

scre

ens a

nd

anot

her g

roup

was

for

spec

ific

scre

ens.

-

Log

os o

n

scre

en

Tem

pora

ry lo

gos

Not

the

sam

e siz

e -

All

logo

s had

the

sam

e st

yle a

nd si

ze.

-

Men

u

A

main

men

u w

as

adde

d.

- +

Hom

e pa

ge a

nd 'A

bout

' ite

ms w

ere

adde

d to

the

help

men

u.

I N T E R F A C E

1. I

t sho

uld

be

cons

isten

t. 2.

It s

houl

d be

fle

xibl

e.

Ed

it

Pan

el

Con

tent

des

crip

tion

used

one

text

box.

C

onte

nt o

rgan

izat

ion

used

a ta

ble

to

illus

trate

the

relat

ions

hip.

The

con

tent

re

pres

enta

tion

part

adde

d an

indi

cato

r for

ea

ch m

ediu

m.

The

con

tent

repr

e-se

ntat

ion

part

was

or

gani

zed

in a

n ob

ject-

orien

ted

style

. M

ouse

righ

t but

ton

coul

d se

lect o

r un-

selec

t kno

wled

ge u

nits

.

-

To b

e con

tinue

d on

the n

ext p

age

62 Chapter 3

-

Page 75: computer support for multimedia curriculum design

C

hoi

ces

Issu

esP

roto

typ

e 1

(§ 3

.3)

Pro

toty

pe

2 (§

3.4

) P

roto

typ

e 3

(§ 3

.5)

Pro

toty

pe

4 (§

3.6

) F

orm

ula

tion

A p

aper

-bas

ed

scen

ario

was

mad

e m

anua

lly.

Sce

nario

s cou

ld b

e ex

porte

d to

M

icros

oft W

ord

auto

mat

ically

.

-

S C E N A R I O

1. I

t sho

uld

incl

ude

muc

h in

form

atio

n.

2. I

t sho

uld

be w

ell

stru

ctur

ed.

Fra

gmen

ts

A

nalys

is M

odul

es to

be

incl

uded

F

low

char

t I

nter

face

des

crip

tion

Des

crip

tion

of e

ach

know

ledge

uni

t

Rev

ised

desc

riptio

n of

eac

h kn

owled

ge

unit.

-

Note

: - =

No

chan

ges;

+

= N

ewly

adde

d;

(blan

k) =

not

con

sider

ed

Prototype development 63

Page 76: computer support for multimedia curriculum design

64 Chapter 3

3.3 The first prototype In order to offer flexibility to both novice designers and experienced designers of a multimedia curriculum, the first prototype was split up into two parts: Designer's Aid and Edit Panel. The first round of prototyping focused on the components of content, support and interface. The concept of a scenario was not formulated yet. In this section, the main characteristics of the content, support and interface will be explained. 3.3.1 Content Concepts and relationships

In the first prototype, the new form of curriculum was temporarily called electronic curriculum, and the regular curriculum was called conventional curriculum. According to Taba (1962), a (conventional) curriculum is usually defined as a plan for learning, mostly presented on paper. However, an electronic curriculum is often presented with many electronic formats. In this prototype, an electronic curriculum was temporarily defined to be a plan for learning with electronic formats, especially in multimedia formats on a computer. The relationship between an electronic curriculum and a conventional curriculum is depicted in Table 3.1. First, it was thought that an electronic curriculum was a new form of a curriculum, integrating more ICT (especially multimedia technology) and presented with more media formats. Second, an electronic curriculum and a conventional curriculum might include the same components, such as objectives and content, since an electronic curriculum was still a kind of curriculum. Third, compared to a conventional curriculum, the interface was thought to be an important component of an electronic curriculum, because it is also a computer program. Goals and usage

As explained in Section 1.1.1, the ongoing educational reform in China is intended to stimulate the shift from test-driven education to quality-driven education. However, any educational change is a complicated process (Fullan, 1991), which usually takes a long time. At the moment of designing the first prototype, test-driven education was still dominant in China. The goals of designing an electronic curriculum in the first prototype were to meet the needs of:

Page 77: computer support for multimedia curriculum design

Prototype development 65

quality-driven education; and/or test-driven education.

In China, classroom teaching has a long history and will continue to be used in the near future. In particular, nowadays computers are not very popular in families, and classroom teaching was assumed to remain for a long period (cf. Brock, 1994). However, with the rapid growth of personal computers, individual learning with computers outside of classrooms will become more and more popular, since it can provide a rich learning environment that enables students to freely experiment, test, and invent (Heinich, Molenda, Russell & Smaldino, 1996). Based on these considerations, in the first prototype the usage of an electronic curriculum was provisionally determined to facilitate: teacher's use for classroom teaching; and/or learner's use for individual learning.

Learning theories

Designing a multimedia curriculum was a new challenge, in particular aiming at improving quality-driven education (see Section 1.1.1). It seemed appropriate to follow some existing learning theories in order to better achieve the design goals of a multimedia curriculum. The first prototype intended to follow two learning theories: objectivistic and constructivistic learning theories. According to Jonassen (1991a), objectivists believe in the existence of reliable knowledge about the world; and the objectivistic learning theory focuses on the transfer of knowledge from instructors to learners. Also, assessment is often carried out to check the degree of learners' ability to replicate or recall the transferred knowledge. It seemed to be rather consistent with the test-driven education. However, constructivists claim that reality is more in the mind of learners; and constructivistic learning theory highlights the knowledge construction process of learners. It seemed that the constructivistic learning theory might have positive impacts on the quality-driven education. The first prototype provided a list of modules. It was expected that teacher-designers could pick up proper modules based on the learning theories to achieve the design goals. Subjects and learner analysis

Each subject may need the support of multimedia, and it would also be possible to present each subject on a computer. Consequently, the first prototype did not focus on any specific subjects. It was expected to be able to provide generic support for all subjects.

Page 78: computer support for multimedia curriculum design

66 Chapter 3

Learner analysis may include the analysis of general characteristics, prerequisite knowledge/skills and learning styles (Heinich, et al., 1996). In the first prototype, learner analysis focused on the computer skills. The teacher-designers were assumed to have enough information about the learners concerning subject-related knowledge/skills and the general characteristics, since the teacher-designers were subject teachers and the learners were rather stable. In addition, as different learners might have different learning styles, a plausible way of meeting learners' needs with various learning styles would be to include as many as possible different types of learning materials and/or activities in a multimedia curriculum (cf. Fenrich, 1997). However, teacher-designers might need further information about computer skills because they usually do not know what computer knowledge/skills the learners should have. Modules

Since an electronic curriculum is also a type of instructional software, its modules should be closely related to instructional strategies often included in instructional software. Instructional software is usually classified into tutorial, drill-and-practice, testing, micro-world, cognitive tools, case studies, and cognitive apprenticeship (Fenrich, 1997; Roblyer, Edwards & Havriluk, 1997; Zhu, 1996). Similarly, in the first prototype an electronic curriculum was classified into the following modules: 1. Content elaboration

A tutorial program often includes content elaboration and practice. In the first prototype, the practice became a separate module. Furthermore, the content elaboration part would be enhanced in an electronic curriculum since content is an important part of a curriculum.

2. Practice Practice is usually an effective way of improving learner's knowledge and/or skills. Normally a practice module starts with questions, followed by diagnosis and immediate feedback.

3. Test Compared to practice, the aim of testing is to check whether learners have grasped knowledge/skills. Usually, a test should be done within a limited time. After it is finished, a summary of learners' performance including time and score is given.

4. Experiment In some subjects such as Biology, experimentation is a very important approach to knowledge acquisition and/or skills improvement. An electronic curriculum was expected to provide a rich experimental

Page 79: computer support for multimedia curriculum design

Prototype development 67

environment in which learners can pick up virtual instruments, set up experiments and observe results.

5. Simulation/virtual reality Simulation normally presents or models the essential elements of a real or imaginary situation, in which learners can learn by manipulating on-screen models in ways that resemble real-world interactions (Fenrich, 1997). In particular, when a knowledge unit is a complex process, computer simulations may serve as an effective approach. Virtual reality, also called virtual environment, virtual world, or cyberspace (Biocca, 1992), is a more 'realistic' computerized simulated environment, in which learners can feel and see three-dimensional objects or images with electronic gloves and glasses. Several benefits of virtual reality can be found in literature, such as improvement of display, enhancement of control, and involvement of learners (Roblyer, et al., 1997).

6. Case studies/problem solving Constructivists believe that learners are best motivated by real problems in authentic situations (Coleman, Perry & Schwen, 1997; Jonassen, 1991a). Case studies and problem solving were expected to provide an authentic micro world in which learners can investigate the cases/problems by analyzing its complicated elements and relationships. It was also expected that knowledge gained from the cases or authentic problems could be easily transferred to other situations (Vockell & Schwartz, 1992).

Development elements

Here the development elements refer to the activities to be performed when designing the six modules described in the previous section. Content and interface were considered to be two basic elements since an electronic curriculum is an integration of a conventional curriculum and instructional software (cf. Moonen, 1999). Specifically, it was intended that the modules be designed through the following activities: Content selection could be carried out based on the available curriculum

standard and some criteria such as relationship with the curriculum goals, importance and difficulty, frequency included in tests, etc. Content analysis aimed at providing support for content description. The

preliminary idea was to use the Allen's model (see Table 2.1) to guide content description based on the results of content analysis. Content description aimed at representing the selected knowledge units with

various media such as text, pictures, sound, and video, etc. Content organization aimed at sequencing the selected content in a logical

and/or flexible way. In the first prototype, three styles of content

Page 80: computer support for multimedia curriculum design

68 Chapter 3

organization were outlined: linear, tree, and hypertext/hypermedia. The linear style meant that information was organized in a linear sequence. The tree style meant that information was organized into different levels in a hierarchical structure, in which users could go up or down. The hypertext/hypermedia style meant that information was organized into an interconnected network, in which users could go to any related pieces of information by following their embedded links. Interface design aimed at selecting proper interface styles for various

modules. Due to the fact that the intended target users (subject teachers) were not entirely computer literate, they might expect to encounter difficulties while designing an interface. To solve this problem, a list of representative interface styles was provided from which they could directly select a proper interface. In the first prototype, three interface styles for an electronic curriculum were summarized: classic and elegant, modern and popular, vivid and vigorous. These three interface styles remained through the final version.

3.3.2 Support In the first prototype, the following specific types of support were included: explanations/examples help hints tips materials suggestions translation.

Some key or new words needed elaboration for intended target users, especially for new users. The keywords referred to some important concepts related to multimedia curriculum design. The new words referred to some fresh words for subject teachers. In the first prototype, in addition to an explanation, each keyword had an example, but each new word did not. Clicking each keyword would get an explanation or an example depending on the status button on the toolbar. If the status button was depressed, clicking a keyword would call up an explanation, otherwise it would get an example. In addition, if users clicked keywords with the right mouse button, a pull-down menu would pop up, from which users could visit explanations or examples. Two kinds of examples were provided. One kind was shown in the help Window, and another kind was displayed in --automatically launched-- external applications. In order to be more consistent, all examples were related

Page 81: computer support for multimedia curriculum design

Prototype development 69

to the MCB project (see Section 1.2.1). Some theoretical models and practical guidelines were included in the prototype, aiming at explaining why such screen content was selected and improving users professional knowledge for multimedia curriculum design. Clicking on the help button on the toolbar could access such help. Some buttons on the screens --in particular speed buttons-- might not display enough information to indicate what the buttons were or intended to do. The hints were designed to show such pieces of information when the mouse moves over (speed) buttons or hotspots. The tips were designed to show pieces of information about how to perform tasks efficiently in the form of a 'tip of the day'. Few supporting materials such as available pictures, animation and video clips were collected and put together with the prototype. Some expert suggestions on how to select options or design an element were provided in the first prototype based on the users' profiles and the embedded expertise. The first prototype was developed in Chinese, but the translation function was designed to make the prototype usable in English. The detailed explanations of these kinds of support can be found in Section 4.3. 3.3.3 Interface When designing an interface, designers need to consider a lot of attributes such as learnability, satisfaction, memorability, error rate and efficiency (Nielsen, 1993). These attributes are mostly interconnected or affected by each other. For example, users may feel satisfied if an interface is efficient, no errors are occurring or has an attractive design; if an interface has attributes making it easy to memorize, then the error rate may be reduced. Brown (1988) summarizes some useful guidelines for interface design, including consistency; physical analogies; expectations and stereotypes; and ease of learning and use. During the process of interface design, it is necessary to establish a balance of these attributes/guidelines and emphasis may be shifted. For example, consistency should be considered from the beginning to the end, while error rate is not so serious at the beginning. Based on the above attributes/guidelines, the interface of the first prototype was formulated. Figure 3.2 shows a screen dump of Designer's Aid in the first prototype. Fonts and colors

In order to be consistent, the fonts and color for the same titles or at the same positions on different screens were kept the same. However, there might be some different fonts or colors on one screen for different purposes. For

Page 82: computer support for multimedia curriculum design

70 Chapter 3

instance, all titles on different screens used the font 'Song' (Chinese) with size of 14 and style of bold, but the normal text used the same font with a smaller size and the normal style. The background color was gray throughout. The screen elements mostly used the same shade of black, but the keywords and new words used two different colors: red and blue.

Figure 3.2: A screen dump of Designer's Aid

Buttons, toolbar and logos

There were two kinds of buttons: speed buttons and regular buttons. Most speed buttons were located on the toolbar for quick access to support, while the regular buttons were placed at the bottom of the screens and used for linearly browsing the prototype. The speed buttons were usually smaller with logos on them (without captions). However, the regular buttons were bigger and incorporated both logos and captions. In the first prototype, each screen and each button had a logo or an icon on it, but all logos were provisionally selected. Some of them were not very meaningful, and not all logos were of the same size. Also, the speed buttons on the toolbar were randomly located.

Page 83: computer support for multimedia curriculum design

Prototype development 71

Edit Panel

The Edit Panel part aimed at carrying out the detailed design of each element. The content selection could be performed through a checkbox. The content analysis could be performed by specifying the type of content (such as a fact or concept) through a list of radio buttons. The content description presented the selected knowledge units using various media formats. The process was that users first specified the media format they wanted to use (such as a picture) for describing the knowledge unit. Then, they could describe the attributes (such as colors, size, location and source, etc.) of the media format in a shared textbox. The content organization could be carried out by specifying the organizational relationships between the 'cells' of: i) knowledge units; ii) sections; iii) chapters; and iv) knowledge units, sections and chapters. The complex relationships among the above cells was illustrated in a table, in which the top row and the left column listed all selected knowledge units, sections and chapters. The interface design was carried out through selecting a proper interface style for a section and/or a chapter. In addition, the connection between the Designer's Aid and Edit Panel parts was that Designer's Aid provided a link (i.e. an edit button) to Edit Panel. Clicking the link would invoke Edit Panel to execute. However, due to technical limitations, each click would launch a separate copy of Edit Panel. This might cause two problems. First, too many copies of Edit Panel might cause Windows to freeze up. Second, users could not continue the work in the former copies. 3.3.4 Expert appraisal and micro evaluation After the first prototype had been formulated, a formative evaluation was carried out in Shanghai, focusing on the validity and potential practicality of the prototype. In this section, the formative evaluation activities will be elaborated in more detail. Research question and participants

The research question of the formative evaluation was:

To what extent is the prototype valid and will it be practical for the intended target users?

Page 84: computer support for multimedia curriculum design

72 Chapter 3

The following sub-questions were expected to be answered by the following types of participants: Intended target users (n=5)

- Will the prototype be helpful for you in designing an electronic curriculum?

- Can you learn some information from the prototype? - What are the main difficulties when you are using the prototype? Curriculum/instructional designers (n=2)

- Are the explanations of keywords accurate? - Do you think the information included in the prototype would be

helpful for the intended target users? - What information should be changed or added? CBL designers (n=3)

- Do you think the content included in the prototype will be useful for intended target users?

- Do you think the prototype is easy to use? - Do you agree with the choice of fonts, colors, buttons and other screen

elements? - What should be changed or improved?

Five intended target users took part in the initial formative evaluation. Among them, two were Biology teaching researchers, two were Geography teaching researchers and one was a Geography teacher. The two Biology teaching researchers gained experience in scenario development during their participation in the MCB project (see Section 1.2.1). The two Geography teaching researchers and single teacher had experience in multimedia instructional software design, but did not have experience in instructional scenario development. Two curriculum/instructional experts were involved in the formative evaluation, focusing on the component of content. Three students, working towards their master's degree in multimedia instructional design, participated in the formative evaluation, focusing on interface design, support and content. Approaches, instruments and procedures

The main approaches employed during the formative evaluation were expert appraisal and micro evaluation. The main instrument was a list of questions –the same as the sub-questions shown above. The formative evaluation was carried out at several times. The two Biology teaching researchers and one Geography teaching researcher did a micro evaluation as a small group, since they worked in the same district educational college. The other Geography

Page 85: computer support for multimedia curriculum design

Prototype development 73

teacher and single Geography teaching researcher did micro evaluations separately at two different times. Because the prototype was very superficial and the five intended target users were off campus, they walked through the prototype using screen dumps. Because the two curriculum/instructional designers were close by, they gave expert appraisals on a computer individually at two different times. The three CBL designers did a micro evaluation as a small group on a computer operated by one of them. At the start of each evaluation meeting, the evaluator first gave a brief introduction of the main aim of the prototype as well as the aim of the formative evaluation. It usually took about five to ten minutes. After that, the participant(s) walked through the prototype in different ways as explained above. During this process, the evaluator briefly explained the elements on screens they were visiting, answered their questions, and wrote down their comments and suggestions. This process mostly took about one and a half hours. Each session was wrapped up with a short discussion. During the discussion, the participants were invited to answer the questions from the question list mentioned above. Their answers to the questions were written down by the evaluator. In total, the duration of each formative evaluation session was about two hours. Comments and suggestions

The main points of the various participants' comments, suggestions and answers to the questions are described as follows. Intended target users a1. They agreed that the prototype would be useful for them to make

instructional scenarios, but they thought that users needed to have competent computer skills, otherwise they might have difficulties in using the prototype. Also, they suggested adding an easy-to-use tutorial, to help users learn how to use the prototype.

a2. They believed that they learned some useful information from the prototype. Three users mentioned that the style of content organization using hypertext/ hypermedia was new for them.

a3. All of them showed great interest in the embedded examples, and expected to have more added.

a4. They said that they did not find the differences among the three interface styles from the screen dumps.

Page 86: computer support for multimedia curriculum design

74 Chapter 3

a5. One participant mentioned that she expected to know the results caused by the current settings immediately.

a6. Most of them mentioned that the content description part was very important and should be enhanced. But in the first prototype, this part seemed to be too simple.

a7. They criticized some inappropriate labels such as 'often included in tests' and 'test-driven education'. They said that these labels were not consistent with the current educational reform trend.

Curriculum/instructional designers b1. Both of them thought that the prototype was designed in detail, and they

believed that the prototype would be useful for subject teachers to make instructional scenarios.

b2. They agreed that the embedded examples would be very helpful for users. b3. One instructional designer suggested not using new terms, such as 'quality-

driven education' and 'nine-year compulsory education', because these new terms were either not well defined, or not commonly agreed upon. Alternatively, they suggested using the often-used and popularly accepted terms.

b4. They thought the design aims, usage, and electronic curriculum modules were well chosen, but the rationales should be stated explicitly.

b5. They thought that the learner analysis part looked simple; more information should be added.

b6. They thought that the three interface styles were not enough; more styles should be added.

b7. They suggested that the content description part should be elaborated in more details, because it was a key and useful part for users.

CBL designers c1. They agreed that the user interface looked beautiful and consistent. The

buttons and logos were at the same positions, and the screens looked attractive. The hierarchical and linear browsers provided users with flexible ways to use the prototype.

c2. They thought the embedded examples would be useful for intended target users. They suggested adding more examples. Also, they mentioned that the tips at the bottom of each screen would provide users with useful information.

c3. They mentioned that some screen elements could be designed better. For example, the signs '<' and '>' on ' < Back' and '> Next' buttons may lead to misunderstanding. Users may doubt whether they should click the '<' button on the keyboard, or is it only a backward sign? Some button

Page 87: computer support for multimedia curriculum design

Prototype development 75

captions were not in the same language. For example, in the Chinese version, 'Close' and 'Return' were still used. The meaning of button '…' was not clear, and it should be changed to 'Details'. In summary textboxes, it was expected that more fonts/colors would be used.

c4. They asked why it was not possible to directly enter the Edit Panel part by omitting the '…' buttons on the content selection, description, organization and interface design screens, and how to guarantee that users would visit the Edit Panel part before they clicked the 'Next' button.

c5. They mentioned that users might not know what kinds of media had been used for the content in the Edit Panel part. It would be better to add an explicit indicator such as '√' to each medium used. Also, a preview window for interface styles was expected.

Revision decisions

Based on the comments/suggestions, the following revision decisions were made on: Content r1. The rationales for categorizing goals, usage and modules needed to be

made explicit [b4]. r2. New terms (such as 'quality-driven education' and 'nine year compulsory

education') would be removed or replaced with more familiar terms [a7, b3].

r3. More aspects would be added to the learner analysis part [b5]. Support r4. A tutorial (training) part would be added to the prototype, by which new

users could easily become familiar with the prototype [a1]. However, since the prototype was still unsophisticated, a tutorial would not be added to the second prototype.

r5. A prediction (or 'what next') function would be added, by which users could know what results they could expect using the current settings [a5].

r6. More examples would be added to the prototype since most participants mentioned that the examples were helpful [a3, b2, c2].

Interface r7. Each button should have both an icon and a caption. The icons should be

pictures, and the captions should be more meaningful [c3]. r8. The prototype should prompt users to visit the Edit Panel part, since they

could miss it otherwise [c4].

Page 88: computer support for multimedia curriculum design

76 Chapter 3

r9. More interface styles would be added to the prototype, or new styles could be added by users [a4, b6].

r10. The content description element would be enhanced [b7, c5]. 3.4 The second prototype The second prototype included three parts: Designer's Aid, Edit Panel, and Main Frame. The Main Frame part was added aiming at providing users with a more integrated environment, in which they could choose to enter the other parts based on their specific needs or level of experience. The detailed introduction of the Main Frame part and the relationship with the other two parts can be found in Section 4.1. The second prototype focused on the improvements of content, support and interface. Also, in this case a paper-based scenario was preliminary designed. In this section, the main characteristics of the second prototype are to be explained. 3.4.1 Content In the second prototype, some issues of content were improved and even remained through the final version, such as the goal division and usage (see Table 3.1). Some issues were kept the same as those in the first prototype, such as the learning theories and modules. In this section, the unique characteristics of the second prototype are presented. Concepts and relationships

It was found that the term 'electronic curriculum' did not precisely describe what the final product would be. In the literal sense, an electronic curriculum can use any electronic format, such as videotapes or audio tapes. Actually, the final product would be in multimedia formats, encoded and presented on a computer. So from the second prototype on, the term 'electronic curriculum' has been changed to be 'multimedia curriculum'. It was defined to be a plan for learning with multimedia formats, such as text, sound and graphics. In addition, some other terms such as 'quality-driven education' and 'test-driven education' did not appear in the second prototype based on the revision decision [r2]. The relationship between a multimedia curriculum and a conventional curriculum has been shown in Table 3.1. With regard to the content, a multimedia curriculum might cover a part (not all) of a conventional curriculum. In addition, a multimedia curriculum might also include some

Page 89: computer support for multimedia curriculum design

Prototype development 77

extended content such as in-depth explanations of some knowledge units. With regard to the media, the main medium of a conventional curriculum is text. Some other media formats might be also included, such as graphics and sound. But mostly these media are from different sources such as from videotapes or audio tapes. Whereas in a multimedia curriculum, text is mostly only one type of media, and other media materials such as graphics, audio and video might be also included. All of these media are encoded and presented using a computer. Goals, usage, subjects and learner analysis

In the second prototype, the main issues in the analysis part of Designer's Aid referring to the goals, usage, subjects and learner analysis were revised. The goals of a multimedia curriculum were divided into three broad categories: knowledge, skills, and attitudes [r1]. Combined with the practical situations in China, the goals were revised to stimulate i) basic knowledge and skills learning; ii) improvements in extended knowledge and higher level skills; and iii) attitudes development. The detailed description can be found in Section 4.2.1. With regard to the usage of a multimedia curriculum, one more usage was added: collaborative learning between learners. With the rapid growth of computer networking, computer supported collaborative learning was becoming possible (cf. Heeren, 1996; Johnson & Johnson, 1996). A multimedia curriculum might also be used for supporting collaborative learning in the future. With regard to the subjects, the second prototype aimed at two subjects (Biology and Geography) as its experimental samples to investigate how computers can support multimedia curriculum design. A broad support system for all subjects might be not very useful for a specific subject, because it cannot take all specific characteristics into account and provide specific supporting materials for all subjects. In practice, these two subjects usually encompass a lot of natural phenomena, which can particularly benefit from multimedia support. In addition to the computer skills analysis, the learner analysis added one aspect of subject related knowledge and skills analysis in the second prototype [r3]. It was expected to help teacher-designers explicitly classify what subject knowledge and/or skills the learners had and what should be supplied in advance. However, the learning style analysis was not added. It was assumed

Page 90: computer support for multimedia curriculum design

78 Chapter 3

that a multimedia curriculum should be designed to meet the needs of learners with various learning styles, not designed only for some learners with particular learning styles. Development elements

In the second prototype, the sequence of the elements (i.e. content selection, content representation, etc.) for each module was adjusted. It was thought that it might be more practical to start with interface style selection, because after an interface style had been selected, screen elements on that interface could be determined and the types of media to be used were also indicated. Based on the selected interface, content selection, description and organization would be carried out successively. In addition, it was found that the content analysis part was hard to carry out for teacher-designers, because in curriculum standards each knowledge unit was not clearly specified with a category of objective. This issue was also related to the definition of knowledge unit. How big is a knowledge unit? Is it as big as a fact, a concept, or a principle? The practical situation is that usually a textbook is organized in a chapter-section-unit structure. A knowledge unit may be as small as a fact or a concept, or as big as a mixture of several facts, concepts and procedures. In China, a knowledge unit is mostly seen as being as big as a lesson. It may include everything carried out within a lesson time of 45 minutes. In the first prototype, it was expected that the results of content analysis would provide guidance for content description, for instance based on the media selection matrix shown in Table 2.1. However, since it was hard to carry out, and as many media formats as possible were expected to be used for describing a knowledge unit, the content analysis aspect was removed in the second prototype. The elements for each module in the second prototype became: interface design; content selection; content description; and content organization.

The first three elements remained the same as those in the first prototype. The content organization element had minor changes on names. The three content organization modes were termed: linear, hierarchical, and hyperlink.

Page 91: computer support for multimedia curriculum design

Prototype development 79

3.4.2 Support In the second prototype, three kinds of support were added: prediction, wizard and translation. The predictions indicated what results would be caused by the current settings [r5]. Compared to the suggestions, which gave designers advice based on the previous settings, the predications referred to what would be affected by the current settings. The use of wizards for offering support was newly added based on the Designer's idea. A wizard is a small program that divides a complex task into several steps. By following these steps, on the one hand, users can carry out the task; on the other hand, users can also learn how to carry out the task. In the second prototype, some wizards were added to provide (novice) users with an easy way to design the elements. For example, the wizard of content description started by asking questions like 'Do you want to use text?' If the designers answered 'yes', then a follow-up question would be "Does the text already exist in a file?" If the answer was "no", the wizard would prompt the designers to input text. After that, the wizard continued to ask questions about other media formats such as picture, sound, and animation. The second prototype supported two languages. The translation tool could make the prototype be presented in either Chinese or English. In addition, most keywords had examples in the second prototype [r6]. 3.4.3 Interface There were no significant differences between the first and second prototypes regarding the interface, but some improvements were made based on the revision decisions. For example, each regular (not speed) button had both a pictorial icon and a textual caption on it [r7]. On the screens of development elements, a prompt window would pop up if users clicked the 'next' button before they visited the Edit Panel part to really perform something such as selecting content [r8]. No more interface styles were added to the second prototype because the three existing styles seemed to be representative. However, users were allowed to add some new interface styles if they wanted [r9]. In Edit panel, an indicator was added to each media format to indicate whether the media format had content or not [r10]. In addition, some other improvements were made in an effort to make the prototype easier to use.

Page 92: computer support for multimedia curriculum design

80 Chapter 3

The first change was on colors and buttons. Both keywords and new words used the same style with red color and underlines on screens. If a keyword or new word had an example, an example button would be added to the help file. The pictorial icons were rather different from the characters on the keyboard. The 'Tutorial' button was put on the first screen, with which beginners could immediately start with the training program. Often-used buttons (such as the four linear browsing buttons) were specified with short-cut keys, which users could press to perform common tasks. The second change was that a main menu was added to the second prototype, as shown in Figure 3.3, in order to be more consistent with other external programs. The menu included four items: File, View, Tools, and Help. The detailed description of the menu can be found in Section 4.4.2.

Figure 3.3: A screen dump of Designer's Aid in the first prototype

In addition, the connection between the various parts was more integrated. The Main Frame part provided access to other parts. Users could easily navigate to any parts from the Main Frame part, based on their needs. It was also possible to enter Edit Panel from the Designer's Aid part.

Page 93: computer support for multimedia curriculum design

Prototype development 81

3.4.4 Scenario In the second round of prototyping, the concept of scenario got major attention. In literature, some related labels were found, such as specification, storyboard (Kemp & Smellie, 1989), super storyboard (Chapman, 1998), and programmer ready materials (Harrison, 1995). Movie directors usually use a storyboard to formulate episodes of a story. Each frame on the storyboard represents a scene of a movie. All frames on the storyboard are usually organized in a linear way representing the episode of a story. The idea of an instructional scenario for computer programming is similar to the storyboard for movies. But the difference is that all frames on a movie storyboard are organized in a predetermined linear way, while the organization of frames in a scenario of a computer program is more flexible. A linear organization for a scenario of a computer program is usually not enough. Chapman (1998) uses the term 'super storyboard' to describe the complex relationship among the frames. On a super storyboard, all frames are not organized in a linear way, but usually refer to each other in a network. Carroll (1994) defines a scenario as a 'fairly open-ended narrative description of typical and critical situations that prospective users participate in, and which the design work may transform' (p. 65). Witkin and Altschuld (1995) indicate that a scenario is a holistic description or 'snapshot' of a future event. Tessmer (1998) classifies two types of scenario in the process of multimedia exploration systems design: introductory scenario (or goal scenario) and instructional scenario. An introductory scenario epitomizes the purpose of the instruction by concisely describing how students might use the final program. Compared to the introductory scenario, an instructional scenario contains more specifics on the type of information explored and the tools used to explore it. It details how learners will interact with the multimedia program in their intended settings; how they will access knowledge, pursue instructional goals, and chart their learning progress. In the second prototype, the term 'scenario' was defined as: A storyboard that depicts the aims, usage, characteristics of learners, and specifies interface design, content selection, description and organization for knowledge units. It aims to guide programming and to stimulate discussions between designers and computer programmers. Choi (1997) summarizes seven features which should be considered when designing a scenario for instructional simulation: 1) sequence; 2) operating procedures; 3) characteristics of participants; 4) characteristics of simulated objects; 5) level of fidelity; 6) time frame; and 7) non-computer activities. Tessmer (1998) indicates that a scenario should describe learning interactions,

Page 94: computer support for multimedia curriculum design

82 Chapter 3

usage problems encountered and solutions to them, student attitudes, instructor activities, and environmental settings. Chapman (1998) includes several fields in a storyboard such as: event name, event description, graphic description, video description, video shot list information (location, subject, props, talent, etc.), on-screen touch areas, branching logic, and programming notes, etc. In the second prototype, the preliminary idea was that a scenario might include the following components: the analysis results including the course name, the grade, the average age of

the learners, the goals and usage of the multimedia curriculum, and the designers' names, and mailing addresses; the module(s) to be included in the multimedia curriculum; a flowchart depicting the working sequence of the multimedia curriculum; a screen layout presenting the interface style to be used; descriptions of knowledge units including title, text description, picture,

sound, animation, video, content organization and notes. A paper-based scenario was formulated manually in a Word document format including the above fragments. During the formative evaluation, the paper-based scenario would be shown to the participants for comments and suggestions. 3.4.5 Expert appraisal and micro evaluation The formative evaluation was carried out in Shanghai, focusing on the four components of content, support, interface and scenario. In this section, the formative evaluation activities will be introduced. Research question and participants

The main research question was:

To what extent is the prototype valid and practical for the intended target users? Four kinds of participants took part in the formative evaluation: i) subject teachers; ii) curriculum/instructional specialists; iii) CBL experts; and iv) computer programmers. Table 3.2 shows the relationship among the four components, quality and the four kinds of participant.

Page 95: computer support for multimedia curriculum design

Prototype development 83

Table 3.2: The relationship among the components, quality and participants

Quality Component Validity Practicality Interface CBL, CPs STs Content C/I, CBL STs Support CBL STs Scenario CBL, STs CPs Note: STs = Subject teachers (n=9); C/I = Curriculum/Instructional specialists (n= 6); CBL = Computer based learning experts (n=5); CPs = Computer programmers (n=7) In total, 27 participants made contributions to the formative evaluation. Nine were subject teachers, five of whom were Geography teachers, two were Biology teachers, and the other two were teachers of other subjects. Four of the subject teachers had experience in multimedia instructional design, and five had no such experience. Six were curriculum/ instructional design specialists. Five were CBL design experts, who had experience in multimedia instructional design. Seven were computer programmers or students working towards their master's degree in (instructional) software design. Approaches, instruments and procedure

The approaches to the formative evaluation of the second prototype were expert appraisal and micro evaluation. The main instrument used for data collection was a preplanned interview topic list, which included two parts. The first part listed a number of walkthrough questions, which would be answered by the participants during their hands-on experiences using the prototype or watching a demonstration. The second part included some follow-up questions, which would be used for collecting users' general comments and/or suggestions on the entire prototype after having walked through it. The original interview topic list is shown in Appendix C. The main procedure for the formative evaluation was composed of the following three steps: 1. Introduction

Each time, the evaluator first briefly introduced the aims, the main functionality and characteristics of the prototype, the aims of the formative evaluation, and the time needed. Mostly the introduction took about five to ten minutes.

Page 96: computer support for multimedia curriculum design

84 Chapter 3

2. Walkthrough In order to find out to what extent the prototype was practical, it was expected that all participants could have hands-on experiences on the prototype. However, as there was only one computer available for the formative evaluation, it was not feasible to provide all participants with individual hands-on experiences. Nevertheless, all individual participants (5 STs, 2 C/I, 4 CBL, 3 CPs) did have hands-on experiences with the prototype. Even within three groups (2STs, 2STs, 4CPs), one member had hands-on experience on the prototype while the other(s) discussed with him/her. During this process, the evaluator observed their reactions to the system; wrote down their questions, comments and suggestions, and answered their questions. Two group of participants (2 C/I + 1 CBL, 2 C/I) walked through the prototype by watching demonstrations operated by the evaluator since they lacked sufficient computer skills or they did not want to use it. During the demonstrations, the evaluator guided the participants to the parts where he particularly needed their comments/suggestions. The evaluator wrote down their comments, probed the reasons behind the comments; and asked questions related to what was displayed on the screens. The duration of walkthrough mostly took about one and a half hours.

3. Debriefing After a participant or a group of participants walked through the prototype, a debriefing followed. The debriefing included further discussion of formerly unclear issues, characteristics of the prototype and their general comments and suggestions. The evaluator also asked them the follow-up questions. The duration for a debriefing was around half an hour.

Comments and suggestions

During the formative evaluation, a lot of comments and suggestions were collected. Next, these comments and suggestions will be summarized in terms of the four components: content, support, interface and scenario. Content The curriculum and/or instructional specialists agreed with most content in the prototype. They thought that: + The prototype was designed in detail, and the content was properly

analyzed and elaborated. + The content would be helpful for intended target users to learn how to

design a multimedia curriculum.

Page 97: computer support for multimedia curriculum design

Prototype development 85

In order to improve the prototype, much effort was put into collecting prescriptive information. Specifically, they made the following comments and suggestions: a1. The learning theories might be difficult to understand for the intended

target users. a2. The goal division was reasonable, but one participant said that it was out-

of-date, she preferred to divide it into fact, process and strategy. a3. One curriculum specialist preferred multimedia curriculum to be defined

as an experience rather than a plan for learning, because he thought that interaction was a key characteristic of a multimedia curriculum. Some participants mentioned that the relationship figure in Table 3.1 between a conventional curriculum and a multimedia curriculum was not accurate, and even misleading.

a4. The grades and subjects could be extended to primary schools and other subjects.

a5. Some modules were hard to understand, such as simulation and virtual reality.

a6. No obvious differences existed between the last two interface styles at first glance. More interface styles should be added.

a7. A concept mapping tool should be added to facilitate the content selection.

a8. The term of 'content description' should be changed to 'content representation' because the term 'content description' seemed to present content purely through text.

Support During the formative evaluation, it was found that the participants did not use many types of support when they were walking through the prototype. The reason for this was probably because the designer was on hand, and the participants could ask the designer any questions directly. However, the participants still provided some comments/suggestions on the support of the prototype: b1. Not all users liked using wizards. Most of them mentioned that the tasks

were not complex enough to use wizards. One said he did not like using wizards because he thought wizards were usually boring.

b2. The Tutorial should be complete because they thought teacher designers might need the tutorial part to learn how to use the program.

b3. Most of them mentioned that they liked the examples very much, because they could learn easily from them. However, they thought that an example button should be added to the explanation part of each keyword in the help file.

Page 98: computer support for multimedia curriculum design

86 Chapter 3

Interface In general, comments on interface design were positive. They commonly agreed that: + The interface was easy to use and to learn. + The interface was error-free and tolerant of mistakes. Specifically, the following comments and suggestions were collected: c1. The cover screen in the Main Frame part was not attractive. c2. The summary at the end of the Main Frame part was superfluous because

it was not very informative. c3. Some captions on buttons were confusing, such as the label of 'C' on

buttons of 'Close', 'Cancel' and 'Clear'. c4. The red color of keywords was not bright enough, and was confusing since

the color red is mostly used for attention or alert. c5. The 'Tutorial' button should be moved to the help menu rather than on

the first screen of Designer's Aid since it was not related to the content on the first screen.

c6. The table of contents organization illustrating the relationship among knowledge units, sections and chapters was difficult to understand.

c7. The Windows help program was not easy to use. Scenario Almost all participants agreed that the scenario was well structured and easy to read since the content was presented in tables. However, some other comments and suggestions are summarized as follows: d1. The flowchart should be elaborated. d2. One subject teacher said that computer programmers did not need

scenarios because they usually liked to work with teachers directly when they were programming.

Revision decisions

Based on the comments and suggestions, the following main revision decisions were made on: Content r1. The learning theories guiding multimedia curriculum design would become

neutral and flexible. The explanations of objectivism and constructivism would be moved to the help file [a1].

r2. The definition of multimedia curriculum and its modules would be more precise [a3, a5], since multimedia curriculum is a key term in this study.

Page 99: computer support for multimedia curriculum design

Prototype development 87

r3. More concrete templates with minor variations would be added to each style [a6]. Although the three interface styles were rather representative, it might give users an incorrect impression that the interface of a multimedia curriculum has to be exactly the same as one of these three types of interface.

r4. The term of 'content description' would be changed to 'content representation' [a8].

r5. If an appropriate concept mapping tool could be found, it would be added to the prototype [a7].

r6. Some comments/suggestions were disregarded. The new goal division proposed by the instructional design specialist seemed to be not very comprehensive because attitude was not included [a2]. The prototype would still focus on the two subjects at secondary school level [a4]. In the future, however, it might be extended to other subjects and grades as well.

Support r7. Some wizards would be improved or removed based on complexity of the

tasks [b1]. r8. The process of summoning explanations and examples would be changed.

Clicking a keyword on a screen would always get an explanation in the help file. An example hotspot could be added to the explanation part in the help file. Users could get examples from the explanations [b3].

r9. The Tutorial part would gradually be completed with the continued development of the prototype [b2].

Interface r10. The summary at the end of the Main Frame part would be removed since

it was not deemed informative [c2]. r11. Different buttons would use different captions [c3]. r12. The color of keywords would be changed from red to blue [c4]. r13. The 'Tutorial' button would be moved to the menu [c5]. r14. A 'Close' button would be added to the help program [c7]. r15. The content organization part would be simplified [c6]. Scenario r16. The flowchart would be described in more detail, and a proper flowchart

tool needed to be added [d1]. But the problem remained that teacher-designers might have difficulties making a more comprehensive flowchart by themselves. Collaboration between teacher-designers and computer programmers became necessary.

r17. The comment of [d2] was disregarded. Computer programmers may like

Page 100: computer support for multimedia curriculum design

88 Chapter 3

to work together with subject teachers when they are programming, but it might be hard for subject teachers to be at spot during the whole programming process. Even when it is possible, instructional scenarios can still facilitate discussions between members of the different groups.

3.5 The third prototype Based on the revision decisions above, the third round of prototyping started, focusing on the content, support and interface. In addition, the Tutorial part began to take shape in the third prototype, and the connection between the various parts was improved. In this section, the main characteristics of the third prototype will be described in more detail. 3.5.1. Content Concepts and relationships

In the second prototype, multimedia curriculum was defined as 'a plan for learning with multimedia formats'. During the second formative evaluation, it was found that the term 'multimedia format' needed elaboration, because people had various understandings of what it meant [r2]. The term 'multimedia' has been used for several decades, even before the emergence of computers. At that time, multimedia mainly referred to hardware such as an overhead projector, audio recorder and video recorder. Even nowadays, with the presence of the computer firmly established in daily life, the term 'multimedia' is not only confined to computer-related technology; it may also refer to radio, television, overhead projectors, etc. In order to differentiate with the former conception of 'multimedia', another word was introduced in the third prototype, that was 'presentation form' (cf. Verhagen, 1992), referring to text, pictures, audio, video, etc. The hardware part of multimedia in the third prototype would be called 'media sources', referring to equipment such as audio recorder/tapes, video recorder/tapes, and the computer. In this case, a conventional curriculum and a multimedia curriculum may use the same presentation forms, but they are from different media sources. For example, in a conventional curriculum, presentation forms are from traditional media sources such as textbooks, audio tapes, or video tapes. But in a multimedia curriculum, the presentation forms are through a computer, although they might originally be from the same media sources of audio tapes or video tapes. The relationship between a conventional

Page 101: computer support for multimedia curriculum design

Prototype development 89

curriculum and a multimedia curriculum has already been depicted in Figure 2.1 and Table 3.1. More description can be found in Section 2.3.1. Learning theories

In this prototype, the learning theories used to guide multimedia curriculum design became more neutral and flexible [r1]. A multimedia curriculum would not be designed completely based on either of these two learning theories: objectivism and constructivism, because most teacher-designers might hold views on curriculum design somewhere between these two extremes. However, these two learning theories might be instructive to interested teacher-designers. Therefore, starting with the third prototype, these two learning theories have been moved to the help file, not explicitly shown on the program. Modules

In order to investigate what modules a multimedia curriculum could cover, components of a conventional curriculum and instructional strategies in CBL applications were investigated [r2]. According to Klein (1991), a conventional curriculum usually includes the following elements: goals, content, materials and resources, activities, teaching strategies, evaluation, and others such as grouping, time, and space. This classification is similar to what Eash (1991) did. He identified five basic components of a curriculum as assumptions, goals, subject matter, modes of transaction, and evaluation. In CBL applications, instructional strategies usually include tutorial, drill-and-practice, testing, simulation, educational games, intelligent tutorial systems, virtual reality, learning resources, cognitive tools, micro-world, and collaborative learning (cf. Zhu, 1996; Fenrich, 1997). Based on the components of a conventional curriculum and instructional strategies in CBL applications, a multimedia curriculum is reclassified into the following modules (see Table 3.3): i) goals; ii) content elaboration; iii) resources; iv) experiment, practice, case studies/ problem solving; v) test; and vi) collaborative learning. Compared to the modules in the second prototype, the main difference was that three modules were added to the third prototype: goals, resources, and collaborative learning. It was believed that each of these modules could be supported by or represented with various presentation forms. For instance, goals could be explained in text or audio; resources could be presented in any form of pictures, video clips, or animation sequences; collaborative learning could take place through exchanging text and/or audio.

Page 102: computer support for multimedia curriculum design

90 Chapter 3

Table 3.3: Modules of a multimedia curriculum

Curriculum Components/Instructional Strategies

Klein Goals,

objectives, purposes

Content Materials, resources Activities Teaching

strategies Evalua-

tion

Grouping, time, space

Eash Assump-tion, aims objectives

Subject matter Modes of transaction Evalua-tion

CASCADE

- MUCH Goals

Content elabo-ration

Resources Experiment, practice, case studies/problem

solving Test

Collabo-rative

learning

Zhu &

Fenrich

Tutorial, ITS, si-

mulation, virtual reality

Learning resources

Drill-and-practice, case studies,

educational games, cognitive tools

Testing Collabo-

rative learning

Development elements

In the third prototype, the sequence of the aspects was adjusted to the following: content selection, content representation, content organization, and interface design. Compared to the second prototype, interface design was moved to the end of the sequence. Because some experts suggested that, from an instructional design point of view, multimedia curriculum design starting from the interface design seemed not to be reasonable. In addition, the term 'content description' was changed to 'content representation' [r4]. The detailed description of these four elements can be found in Section 4.2.3 to Section 4.2.6 3.5.2 Support Based on the revision decisions made during the second round of prototyping, the following improvements were made on support: 1. Leaving some wizards to be optional [r7]

Since most tasks were not complex enough to necessitate the use of wizards, most wizards (such as content representation and interface design) were changed to be optional. Users could choose whether or not to use wizards based on their needs or preferences. Some wizards for simple tasks (such as content selection) were removed entirely.

Page 103: computer support for multimedia curriculum design

Prototype development 91

2. Moving the examples to the explanation part in the help file [r8] In the previous two prototypes, examples and explanations used the same button. When the button was up, clicking a keyword would get an explanation, otherwise one would get an example. It was found that this design did not work efficiently because users had to click the button once if they wanted to shift between explanations and examples. After a period, if users forgot the status of the button, they had to check the button again, or even worse they got an example when they needed an explanation. In the third prototype, access to support in the form of an example was moved to its explanation part in the help file. When users clicked a keyword, they would always get an explanation in the help window. Within this window, there was an example hotspot. Clicking it would provide an example.

3. Preliminary design of the Tutorial part [r9] The Tutorial part was not completely designed, which included an introduction of the program, and an explanation of the screen elements, etc. However, this part still needed to be completed with the further development of the program.

In addition, based on the designer's ideas, the following two revisions were made: 4. Adding resource links to the Internet

In order to support the two subjects of Biology and Geography efficiently, some specific resources should be supplied, such as biological or geographical pictures or videos. On the Internet, there were many web sites available featuring related pictures, animation and videos. It was not necessary or possible to download those pictures to local hard disk because they needed a lot of extra space and could not update automatically. In the third prototype, a number of related resource links on the Internet were found and linked to the prototype. Users could follow the links to visit the web sites and download whatever they needed.

5. Adding tools for making concept maps and flowcharts In the content selection part, two approaches were selected to facilitate content selection. One was 'by concept mapping'. In order to help teacher-designers make concept maps, the prototype provided a concept mapping tool: MindMan (for more information, please visit the web site: http://www.mindjet.com). In addition, a flowchart-making tool was expected to be linked. But at that moment, no satisfying flowcharting tool was found.

Page 104: computer support for multimedia curriculum design

92 Chapter 3

3.5.3 Interface Compared to the second prototype, the interface of the Designer's Aid part had no big differences, but some changes were made in the third prototype based on the revision decisions. For example, the revision decisions 10 to 14 were easily implemented in the third prototype. The content organization performance in the Edit Panel part was also simplified [r15]. The complex table illustrating the relationship among all knowledge units, sections and chapters was removed. Instead, users specified generally the organizational relationship (linear, menu, hierarchical or integrated) between knowledge units, and among knowledge units, sections and chapters. In addition, some further improvements were made on the following aspects based on the designer's new ideas: Font

In the third prototype, the font in most places was changed to Comic Sans MS because the new font seemed to be more attractive. In addition, the font sizes were also changed to be bigger because some participants complained that the font sizes were too small. Tool buttons

There were seven tool buttons (translation, tip, Word, edit, help, suggestion, and prediction) on the toolbar in the third prototype. In the previous prototypes, these tool buttons were located on the toolbar randomly. In the third prototype, these tool buttons were grouped based on whether they were related to the current screen. Some tools are general ones such as translation, tips, Word, and edit. These tools were not closely related to the current screens, and were grouped together. Others which were related to the current screens such as help, suggestion and prediction, were grouped together. Logos

In the previous prototypes, there was a logo on each screen presenting the functionality or aim of the screen. But these logos were not of the same size or in the same location partly due to the fact that they were designed not at the same time. In the third prototype, these logos were reorganized to be of the same size, at the same position, and of the same style.

A screen dump of Edit Panel is shown in Figure 3.4. Compared to the second prototype, the interface of the Edit Panel part had many changes. First, in the third prototype, the content representation part and the content selection part were integrated. The main reason was that these two parts were closely related

Page 105: computer support for multimedia curriculum design

Prototype development 93

to each other since content representation was performed based on the results of content selection. In the third prototype, after users selected a knowledge unit from a curriculum standard, they could immediately describe it without moving to another page for content representation. Second, in order to be easy to use, the process of selecting knowledge units from the curriculum standard was made simpler. Clicking on a knowledge unit with the right mouse button would cause the selection status of that knowledge unit to change. Third, in order to make content representation more transparent and easy to perform, this part was organized in an object-oriented style. Each screen element including presentation form was specified with some attributes and actions. The process of content representation was to specify the attributes and actions for each screen element. For example, on Figure 3.4, the presentation form of picture has attributes of file name and a description of a picture, and actions of loading, clearing a picture and editing the description in the textbox. Users could use the attributes and actions to describe the presentation form.

Figure 3.4: A screen dump of Edit Panel

Page 106: computer support for multimedia curriculum design

94 Chapter 3

Fourth, more concrete templates/examples were added to each interface style. Three general categories of interface styles were included in the prototype. Within each category, further concrete templates/examples with minor variations were added. In addition, the connection between the various parts of the program was improved. Each time only one copy of the other parts would be invoked, no matter where the click was made: in Main Frame or in Edit Panel. The two problems mentioned at the end of Section 3.3.3 had thus been resolved. 3.5.4 Micro evaluation Two micro evaluation workshops were organized in Shanghai. The participants of the first workshop were intended target users (subject teachers), who used the prototype to produce scenarios. The participants of the second workshop were computer programmers, who used the produced scenarios to create multimedia programs. It was expected that the validity and practicality of the prototype could be examined through the first workshop, and the quality of the produced instructional scenarios could be probed through the second workshop. Participants

The selection of intended target users was based on the following criteria. First, participants should be from two subjects: Biology and Geography. Second, the participants should preferably be from both junior and senior secondary schools, because the two subjects were taught/learned in both junior and senior secondary schools. Third, both experienced and novice teacher-designers would be involved in the formative evaluation, since the prototype was designed for a variety of users. Fourth, all participants should have basic computer skills and should be interested in computer-based learning, otherwise they could not make instructional scenarios in a short time, or even worse they might be apprehensive about using computers to make scenarios. Based on these criteria, seven intended target users were selected, of whom four were biology teachers, and three were geography teachers. Two biology teachers and two geography teachers had much experience regarding multimedia instructional software design, and the other three had limited experience in this field. All of them had basic computer skills. Table 3.4 shows

Page 107: computer support for multimedia curriculum design

Prototype development 95

some characteristics of these participants. Table 3.4: Characteristics of the subject teachers (n=7)

Subject Biology Geography

Experience of multimedia design

Junior Senior Junior Senior Experienced designers Novice designers 2 2 2 1 4 3

Compared to the selection of subject matter teachers, criteria for selecting computer programmers had fewer restrictions. The basic criterion for the selection of computer programmers was that they should have multimedia programming experiences with a computer language. In addition, it was preferred that they have basic knowledge and skills of multimedia courseware design. Considering that the scenarios made by the seven subject teachers within two hours might be very superficial, it was not necessary to select seven computer programmers. In the end, four computer programmers were selected. They were junior university students of the Department of Educational Information Technology. All of them had more than a half year of programming experiences with Visual Basic. They were considered to be higher-skilled computer programmers in their class, because they did better programming work and got higher marks in their courses. Approach, instruments and procedures

The approach used for the formative evaluation was micro evaluation, which was organized in two workshops. The first workshop was with the intended target users, and the second workshop was with the computer programmers. The workshop with subject teachers At the beginning of the workshop, the evaluator gave a general introduction of the workshop. The introduction mainly included: i) the aims, procedure, and duration of the workshop; and ii) a general introduction of the prototype. The introduction and answering of questions took about ten minutes. After that, each of the participants was invited to pick out a knowledge unit from their textbooks and use the prototype to make a scenario.

Page 108: computer support for multimedia curriculum design

96 Chapter 3

While the participants were busy with making scenarios, the evaluator observed their actions, answered their questions, and wrote down their suggestions and comments. This process took about one and a half hours. After the participants finishing their scenarios, the evaluator discussed with them individually and helped them check whether they had correctly performed what they wanted to do. The participants revised the scenarios based on the short discussions. The discussion and revision took about twenty minutes. After all participants finished their scenarios, the scenarios were saved and they were invited to fill in a questionnaire (see Appendix D), followed by a common debriefing. The debriefing was mainly around the three open-ended questions in the questionnaire. During the debriefing, participants' suggestions and comments were probed in depth and in details. All their opinions were written down by the evaluator. This process took about twenty minutes for all participants. After the debriefing, the evaluator collected all produced scenarios. These scenarios would be used by computer programmers in the second workshop. The workshop with computer programmers At the beginning of the workshop, the evaluator used about ten minutes to introduce the aims, tasks, duration (around two hours) and procedure of the workshop. Then, the participants spent about twenty minutes browsing the seven scenarios in turn. After each of them had reviewed all scenarios, a discussion followed. The discussion was mainly around the first two questions in the open-ended questionnaire (see also Appendix D). The discussion took about twenty minutes. After the discussion, each of them was invited to select a scenario and to create a multimedia program based on it. The evaluator walked around the workshop, answered their questions, and wrote down their suggestions and comments while they were programming. In the end, a short debriefing followed. Each of them was asked to answer the last two questions on the questionnaire. Their answers, comments, and suggestions were written down on paper by the evaluator. This process took about twenty minutes for the whole group of computer programmers. Comments and suggestions

During these two micro evaluation workshops, many comments and suggestions were collected from the participants through observation,

Page 109: computer support for multimedia curriculum design

Prototype development 97

discussions and their answers to the questions in the questionnaire. In this section, these comments and suggestions will be summarized. The workshop with subject teachers Content Through observation and discussion during the workshop, it was found that subject teachers were rather satisfied that: + The content was detailed, and basically fit their actual situations. + The explanations of keywords were useful. + The representational forms of multimedia were comprehensive. However, some participants mentioned the prototype lacked specific supporting materials for these two subjects, and some terms were hard to understand such as 'case studies' and 'collaborative learning.' Their responses to the statement ('I learned some information…') in the questionnaire showed that the opinions of the subject teachers were rather neutral. Four participants slightly agreed with, and three slightly disagreed with it. A strong consensus was expressed that some subject specific materials needed to be added to the prototype. Support During the workshop, it was found that the participants were rather satisfied that: + The Edit Panel tool was useful for them to make scenarios. + The scenarios could be exported into Microsoft Word. + The examples were attractive and useful. However, some of them complained that it was difficult to describe an animation sequence with the Edit Panel tool. Their responses to the statement ('The tools were useful.') in the questionnaire showed that their opinions varied widely and were a little bit positive. One participant disagreed with it; two slightly disagreed with it; three slightly agreed with it; and one agreed with it. In addition, it was found from the observation as well as from the discussion that the support did not receive much attention from the participants. Interface It was found that every participant could immediately start using the prototype without any difficulties. After they walked deeply into the prototype, they had some comments upon the interface. They were satisfied

Page 110: computer support for multimedia curriculum design

98 Chapter 3

that: + The interface was easy to understand and to start. + The interface looked consistent with other external applications. + The interface was consistent internally. + The icons on buttons and/or in list boxes were meaningful. + Not many errors occurred. + The linear and non-linear browsers were flexible to use. However, they complained that the pictures could not be previewed and/or cancelled. In addition, they mentioned that it was not clear how to jump to other parts. Their responses to the statements (' The interface was easy to use'; 'Errors did not often occur') in the questionnaire showed that the subject teachers were rather satisfied with the ease of use and error-free execution of the interface. Two participants slightly disagreed that the prototype was easy to use; while the other five participants slightly agreed with it. One participant slightly disagreed that the prototype was error free; while four participants slightly agreed with it and one agreed with it. Scenario The Scenario is the final outcome of the prototype. During the workshop with the subject teachers, each participant made a scenario with the prototype. It is conceivable that the scenarios made within two hours were fairly premature. When looking at the scenarios, it was found that the text description and narration of a knowledge unit were relatively complete, but the other parts were not complete such as picture, animation, and video. However, the participants were satisfied that: + The produced scenarios were well organized. + The scenarios could be easily modified in either the prototype or Word. + Making scenarios with the prototype seemed to be more effective than the

normal method of making them on paper. However, they complained that the produced scenarios lacked connection between various presentation forms of multimedia, and users could not adjust the order of the presentation forms. Their responses to the statements ('The prototype was helpful for me to make scenarios'; 'I am satisfied with the scenario just made') in the questionnaire showed that the subject teachers rather agreed with the usefulness of the prototype. Four participants slightly agreed that the prototype was helpful for them to make scenarios, and three participants agreed with it. In addition, they

Page 111: computer support for multimedia curriculum design

Prototype development 99

were also basically satisfied with the scenarios they produced. One participant was slightly dissatisfied with the produced scenario; while six participants were slightly satisfied with it. In conclusion, this workshop showed that, to some extent, the prototype had the potential to be valid and practical for intended target users, because the subject teachers were basically satisfied with the four components --in particular with the components of interface and scenario. However, the collected data also indicated that the prototype needed further improvements since participants were not satisfied with some issues in the prototype, such as some terms being hard to understand, and that pictures could not be previewed or cancelled. The workshop with computer programmers The four computer programmers mentioned that the scenarios were well structured and easy to read. Some of them mentioned that the scenarios seemed to be simple and not very elaborated. For instance, one of them said "On this scenario, it is written that this picture will be from page 52 of the textbook. I don't have the textbook, so I don't know what the picture is." Most of the computer programmers mentioned that from a computer programmer's point of view, a scenario should include the real teaching or learning process. For example, they needed very detailed information from the scenarios including what (text, or picture) would be shown on the first screen, and what would be shown next after a user clicks a button (or keyword). The scenarios lacked such detailed information. Usually this information should be provided in a flowchart. Since there were seven produced scenarios and four computer programmers, each computer programmer was able to select one scenario that he/she preferred. When they were asked why they chose a particular scenario, two participants said because they saw that the scenarios were relatively complete and included many presentation forms. One participant said because he was familiar with that part of (Biology) content. Another participant said because she saw that the chosen scenario was clear and easy to understand. Almost all of them complained that they lacked some related materials such as pictures or video clips when they were programming. For example, one participant mentioned that the scenario indicated a background picture was needed, and the designer described what the background picture would look like. But he could not find a similar picture, and he did not have enough time/skills to draw the picture. In that case, what he could do was to use another picture instead. If some specific supporting materials such as pictures,

Page 112: computer support for multimedia curriculum design

100 Chapter 3

animation/video clips existed, making scenarios as well as making programs would be much easier. The multimedia programs built by the computer programmers looked good. On each screen there were some basic components such as text, pictures, and buttons. When clicking a button, follow-up text, picture, or narration would be displayed. However, the multimedia programs were still very superficial. Within two hours, they could not find or create proper materials they needed, and most of the pictures, narration, or video clips were substitutes. In conclusion, the produced instructional scenarios were perceived to be well structured and easy to read and understand by the computer programmers. However, the produced scenarios seemed to be tentative and did not include enough information such as a flowchart or what would happen after clicking a button. Also, the produced multimedia programs by the computer programmers were unsophisticated. Revisions

Based on the comments and suggestions collected from these two workshops, the following major revisions were made in about one week after the workshop. With regard to the content, some terms that were hard to understand were described in more detail, such as 'case studies' and 'collaborative learning'. With regard to the support, some subject related learning materials on the Internet were connected to the prototype. However, a proper flowchart-making tool was still not found. The difficulty of describing animation with text was expected to be solved with further use of audio, but this revision was not made considering that audio might need a large amount of disk space and most instructional scenarios would be printed on paper. With regard to the interface, a preview function was added to most presentation forms such as pictures, animation and videos. With regard to the scenario, no further revisions were made. The connection between various presentation forms was expected to be enhanced with textual description, and the order of presentation forms seemed to be hard to adjust automatically in Word due to technical limitations. However, it could be done manually in Word.

Page 113: computer support for multimedia curriculum design

Prototype development 101

3.5.5 Expert appraisal at the UT Research question and participants

So far all formative evaluation activities were carried out in Shanghai. All participants were also from the context of Shanghai. After the above revisions had been made on the prototype, an expert appraisal workshop was organized at the University of Twente (UT) in the Netherlands, aiming at checking how experts from another context would react on the validity of the prototype. Meanwhile, it was also expected that the possibility of extending the prototype to other contexts could also be probed. The research question of the expert appraisal at the UT was:

To what extent is the prototype valid and to what extent is the prototype perceived to be practical in the context of Shanghai?

Eight experts from the Faculty of Educational Science and Technology (TO), UT, were invited to make contributions to the workshop. Two were multimedia experts, three were instructional design experts, and the other three were curriculum development experts. All of them had been engaged in the research and had gathered much experience and professional knowledge in their domains. Procedure, activities and instrument

The workshop was carried out in the following steps: 1. Introduction and presentation

At the beginning of the workshop, the evaluator gave a brief introduction of the workshop, including the aims, procedure, and time schedule for the workshop. After the brief introduction, a presentation of the prototype was given using PowerPoint. The presentation aimed to give the participants a basic idea of the aims, intended target users, structure, support and interface of the prototype. In addition, the situation of curriculum development in Shanghai was briefly introduced to give them an impression of what the practical context was. During and after the presentation, the evaluator answered some questions. In total, the introduction and presentation parts took about twenty minutes.

2. Hands-on experience After the presentation, the participants were encouraged to gain hands-on experience with the prototype. Participants followed the predetermined routes to walk through the prototype, and wrote answers to the associated

Page 114: computer support for multimedia curriculum design

102 Chapter 3

questions on the work plan. During their hands-on experience, the evaluator walked among them, observing their actions, giving them guidance and answering their questions. The main instrument used for collecting their comments was the associated questions. This process took about one hour.

3. Discussion After hands-on experiences, all participants sat around one table and exchanged their opinions about the prototype. Very often when a participant was expressing his/her opinions, others made complementary remarks. The evaluator wrote down their comments on paper, and the speakers were also encouraged to write down their own comments. The discussion part took about half an hour.

4. Questions Due to time limitation, in the end some of them remained to finish answering the questions, while others finished answering the questions and returned them to the evaluator later.

The main instrument used for data collection was a questionnaire, which includes two parts. The first part was a predetermined walking route and some associated questions. These questions were specific and very closely related to the content, support, interface or scenario. The second part consisted of four reflective questions. These questions were rather generic and related to design goals of the prototype. The original questionnaire can be found in Appendix E. Summary of the results

The results mainly derived from two sources: the associated questions and the reflective questions. Next, these results will be presented. Data from the associated questions a1. Most participants agreed with the definition of multimedia curriculum, but

they did not completely agree with the relationship between a conventional curriculum and a multimedia curriculum. They thought that a conventional curriculum and a multimedia curriculum should cover the same content, with the same presentation forms, but from different media sources.

a2. They thought that the learner analysis part seemed to be insufficient. It should include analysis of not only subject-related knowledge/skills and computer skills, but also learning styles.

a3. Most of the participants did not completely agree with the list of modules; one participant even mentioned that it was a strange collection. They suggested re-classifying the modules.

Page 115: computer support for multimedia curriculum design

Prototype development 103

a4. With regard to the design elements, participants' comments mainly focused on why those practical guidelines were listed, and how they guided the design in practice.

a5. Most of the participants provided some comments on the use of suggestions. They agreed that suggestion would be a very important type of support for users. But in the prototype, they said that the suggestions did not give much useful information or advice. They recommended that more intelligent expertise should be added to the prototype.

Data from the reflective questions Question 1. Is the prototype useful for intended target users to make instructional scenarios? The answers to this question were rather positive. It was believed that the prototype had the potential for being useful in developing instructional scenarios. Two participants explicitly answered this question with 'yes'; four participants gave answers like 'it might be', 'it has potentials', 'in principle yes', or 'to some extent'. One participant wrote 'it is hard to judge because I don't know the target users.' Another participant did not give a very clear answer to the question. Around this question, the participants proposed some comments and suggestions. For example, some participants mentioned that the most serious drawback of the prototype was that it did not provide direct help or advice for making design decisions, such as 'based on …, you are advised to …". One participant suggested that improvements were needed in: i) consistency of terms and lists; and ii) specification of theoretical models and examples. In addition, he also suggested adding more 'educational expertise'. Question 2. Can the prototype improve users' professional knowledge? Most of the participants thought that the prototype had the potential to encourage growth in users' professional knowledge, but some improvements needed to be made. For example, one participant said that it potentially could, but more information was needed on why the practical guidelines were important, and how they affected the design. One participant mentioned that the background information was not sufficient, and therefore the prototype could improve users' professional knowledge of multimedia curriculum design only to a limited extent. Another participant suggested that more elaboration on taxonomies and arguments should be added to enhance professional knowledge.

Page 116: computer support for multimedia curriculum design

104 Chapter 3

Question 3. Can the instructional scenario provide an easy way for interactive discussions between designers and computer programmers? Answers to this question were very positive, and most of the participants agreed that the scenario produced by the prototype could form a good platform for discussions between teacher-designers and computer programmers. Some comments and suggestions were proposed. For example, one participant mentioned that more 'intelligence' was needed in the instructional scenario. Another participant suggested adding more space in the prototype where designers and computer programmers could fill in and share their ideas. Question 4. Will the instructional scenario be useful for computer programmers? The answers to this question varied. Three participants agreed that it would be helpful, four mentioned they were not sure because they did not know the programmer group very well, while one participant disagreed with it because he thought the scenario was not described in sufficient detail. In conclusion, the answers to the reflective questions indicated that two design goals (making scenarios and enhancing discussion between designers and computer programmers, see Section 1.3.2) were relatively better achieved by the prototype, but another goal of improving designers' professional knowledge for multimedia curriculum design was poorly achieved. In addition, most of the participants believed that the produced scenarios indeed had the potential to be practical for computer programmers to create multimedia curricula. The workshop also showed that the opinions of the multimedia experts were more positive than that of the curriculum or instructional design experts. The main reason was probably that the multimedia experts found the prototype more productive, because they could make instructional scenarios by using Edit Panel, while the curriculum or instructional experts walked through the prototype without producing anything. Revision decisions

Based on the comments and suggestions, the following revision decisions were made after the workshop: r1. The relationship between a conventional curriculum and a multimedia

curriculum would be adjusted [a1]. It would not say that a multimedia curriculum covers extra content which is not covered in a conventional curriculum. A multimedia curriculum and a conventional curriculum usually cover the same content, with the same presentation forms but from different media sources.

Page 117: computer support for multimedia curriculum design

Prototype development 105

r2. The learner analysis part would become more comprehensive [a2]. The learner analysis would include not only analysis of computer skills, subject-related knowledge and skills, but also learning styles. After the first workshop, it was decided not to analyze learning styles since a multimedia curriculum would meet various needs of learners with various learning styles. However, explicit analysis of learning styles might help teacher-designers know what types of learning materials and/or activities would be of most benefit to the learners.

r3. The modules of a multimedia curriculum would be revised [a3]. A multimedia curriculum and a (conventional) curriculum usually have the same components such as aims/goals/objectives, content, instructional strategies and assessment, but differences usually exist in the component of instructional strategies. The modules of a multimedia curriculum would be revised based on the components of a conventional curriculum and the instructional strategies often included in instructional software.

r4. More intelligent expertise would be added to the prototype [a4, a5]. The intelligent expertise mainly refers to the interrelationship or link between the analysis part and the design part. In the prototype, the way in which the analysis part supports the design part would be stated explicitly.

3.6 The fourth prototype Based on the revision decisions, the fourth round of prototyping started. It mainly focused on the content and the support. In this section, the revisions of these two components will be described. 3.6.1 Content and support The improvement of the content in the fourth prototype mainly focused on the following aspects: the relationship between a multimedia curriculum and a conventional

curriculum [r1]; learner analysis [r2]; and modules of a multimedia curriculum [r3].

The revised relationship between a multimedia curriculum and a conventional curriculum was that they usually cover the same content with the same presentation forms, but the presentation forms are usually from different media sources. The detailed description has been introduced in Section 2.3.1. The learner analysis part added learning style analysis. The description of

Page 118: computer support for multimedia curriculum design

106 Chapter 3

learner analysis can be found in Section 4.2.2. The revised modules of a multimedia curriculum included aims/goals/objectives, content preparation, content elaboration, practice, and assessment. The detailed description can be found in Section 2.3.2. The improvement of the support in the fourth prototype focused on the following aspects: making intelligent expertise explicit [r4]; and adding explanations to the suggestion and preview [r4].

In order to make the suggestions more useful, some intelligent expertise was added to the fourth prototype. The detailed description of the intelligent expertise can be found in Section 4.3.2. In addition, in order to make users understand why the suggestions or previews were given, an explanation function was added to the suggestion as well as to the preview window. Users can immediately know why the suggestion or preview was given from the explanation. In addition, based on the designer's idea, a supporting web site was developed. The prototype would be a CD-ROM based program. After it was delivered, update of the program and communication between the designer of the program and users might be hard to implement. During the fourth round of prototyping, a web-site (http://cascade-much.20m.com) was built to provide users with: update information of the prototype; a bulletin board for discussion; and easy communication.

More descriptions and discussions of these characteristics can be found in Section 6.4.1. Two sub-items were added to the item help in the menu: i) home page on the WWW; and ii) about…. The home page on the WWW provided a link to the web site of the program, and the 'About…' provided the version and copyright information about the program. 3.6.2 Expert appraisal at the ECNU Research question and participants

After the expert appraisal workshop at the UT, some content and support of the prototype were revised and improved. The expert appraisal at the East China Normal University (ECNU) in Shanghai aimed at confirming the validity of the revised prototype by the experts in the context of China. The

Page 119: computer support for multimedia curriculum design

Prototype development 107

research question was:

To what extent is the prototype valid and would the prototype be practical for the intended target users to make instructional scenarios in the context of Shanghai?

Nine experts from the Faculty of Education, ECNU, were invited to participate in the workshop in addition to the organizer, who was also a multimedia instructional design expert. Four were multimedia experts, and the other five were curriculum/instructional design experts. All of them had a great deal of professional knowledge and experience about their domains. Procedure, activities and instrument

The workshop was organized remotely in Shanghai without the designer present. A local organizer was invited to help organize the expert appraisal workshop. In order to make the discussion more effective, the workshop was organized in two groups at two different times. The first group involved the four multimedia experts, and second group involved the five curriculum/ instructional design experts. Each time, the participants first walked through the prototype together quickly (about twenty minutes) to get a general idea of the prototype. Then they restarted from the first page, and walked through the prototype again by following the predetermined route specified on the work plan. They stopped at each breakpoint mentioned on the work plan, and answered the associated question(s) after discussion. Their common answers to the questions were written down by the organizer of the workshop. After they finished walking through the prototype, they also worked together on the discussion theme and the reflective questions. Except for the Chinese language, the instrument used for data collection was the same as the one used during the expert appraisal at the UT. Summary of the results

Data from the associated questions for multimedia experts The multimedia experts walked through the Edit Panel part of the prototype. First, they thought that the definition of multimedia curriculum was not very accurate. In their mind, a curriculum used to be presented in a form of textbook or teaching outline, 'a plan for learning' seemed to be strange for them.

Page 120: computer support for multimedia curriculum design

108 Chapter 3

Second, they thought that the content representation part organized in an object-oriented style was easy to use. On the one hand, users could easily specify various presentation forms for a concrete knowledge unit; on the other hand, the same presentation forms of various knowledge units could also be easily compared. In addition, the multimedia experts mentioned that the prototype had the flexibility to load presentation forms from both the local computer and from the Internet. Third, They thought that the three forms of content organization were representative and would be useful for content organization. They said that the hyperlink organization style was mostly distinguishable from the regular organization style of conventional curricula. They believed that the menu organization style was mostly used for the overall structure of content in a course. The hyperlink organization style would be very flexible for individual learners to browse the content of a multimedia curriculum. Fourth, they believed that these three interface styles were representative, and interface design performed through selecting a proper interface style seemed to be practical for teacher-designers. Meanwhile, they thought that it would be better if more interface styles were available, or teacher-designers could add interface styles by themselves. Finally, they agreed that instructional scenario development with the prototype was easy to carry out, and the produced instructional scenarios could also be easily modified within the prototype or in Microsoft Word externally. Data from the associated questions for curriculum/instructional design experts The curriculum/instructional design experts walked through the Designer Aid part of the prototype. First, they thought that the support of suggestion and preview would be useful for intended target users, especially the explanations could make it clear to them why the suggestion/preview was given. They suggested that more detailed suggestion be added to the prototype. Second, they agreed with the division of goals. They thought that the second goal (extended knowledge and higher-level skills) was consistent with the aim of the curriculum innovation in Shanghai. One important aim of the on-going curriculum innovation in Shanghai is to improve learners' creative thinking skills --one of the higher level skills. They agreed with the division of usage, and thought it was comprehensive. They agreed that classroom teaching was still a dominant instructional approach, although multimedia learning materials

Page 121: computer support for multimedia curriculum design

Prototype development 109

could enhance individual learning. Third, they believed that the prototype would be helpful for intended target users --especially novice users-- to improve their professional knowledge, since some theoretical models and practical guidelines were included in the prototype. Fourth, they thought that the learner analysis was comprehensive and reasonable, but more explanation would be added to explain why the analysis of learning styles focused on those preferred learning materials and activities. They also agreed with the division of multimedia curriculum modules. Finally, they agreed with the design of the four elements. They thought that the two approaches to content selection were practical, especially the first approach (from curriculum standard) seemed to be more practical for teacher-designers. The presentation forms were deemed proper and comprehensive. By using these presentation forms, knowledge units could be elaborated in more detail. They also agreed with the three forms of content organization and the three interface styles. Data from the reflective questions The four open-ended questions were discussed and answered by the participants. Their common answers are summarized below. The first question was to ask whether the prototype would be useful for intended target users to make instructional scenarios. Their answers were definitely positive. They thought that the prototype would be very helpful for teacher-designers, especially for those teacher-designers who were not familiar with multimedia instructional design, to make instructional scenarios. The prototype could help them easily and quickly make preliminary instructional scenarios. The second question was to ask whether the prototype was useful for intended target users to improve their professional knowledge. They thought the prototype had the potential to improve professional development of intended target users because it included a lot of theoretical models, guidelines and advice. Users could learn how to design multimedia learning materials and get useful information from it.

Page 122: computer support for multimedia curriculum design

110 Chapter 3

The third question was to ask whether the produced instructional scenario could play a facilitating role in discussions between teacher-designers and computer programmers. They believed that the produced instructional scenarios would certainly promote discussions between teacher-designers and computer programmers. The fourth question was to ask whether the produced instructional scenarios would be helpful for computer programmers to create multimedia programs (curricula). They thought that the instructional scenarios would be helpful for computer programmers to create multimedia programs because the instructional scenarios included the detailed description of the multimedia curriculum. However, they thought that face-to-face discussion between teacher-designers and computer programmers would still be necessary, because it is impossible to include everything in an instructional scenario, and communication with oral speaking and even body language is easier. Further suggestions

In addition to the above comments and answers to the questions, some other suggestions were also provided during the workshop. First, the participants suggested that key concepts should be defined more precisely. For example, the experts of multimedia technology thought that the term of multimedia curriculum was not precisely defined. Second, they suggested that the interface should be more consistent with other programs. For example, they suggested that the tool bar could be moved beneath the main menu, since most computer programs have such an interface. Third, they suggested that the prototype should add a tool to collect information about teacher-designers, including their names, correspondence addresses and background information. With such information, computer programmers could easily contact the scenario developers (teacher-designers) in case of needing discussions or consultations. Fourth, they wondered whether the title of the program would be called 'a computer support system for multimedia instructional scenario design' rather than 'a computer support system for multimedia curriculum design', because they thought the first title more concisely described what the prototype did.

Page 123: computer support for multimedia curriculum design

Prototype development 111

Conclusions

The answers to the associated questions for the multimedia design experts as well as for the curriculum/instructional design experts showed that the prototype was valid. Their answers to the reflected questions also indicated that the prototype had the potential to be a practical tool for intended target users. Nevertheless, in order to be more valid and practical, further improvements were expected to be made as indicated by the further suggestions.

Page 124: computer support for multimedia curriculum design

112 Chapter 3

Page 125: computer support for multimedia curriculum design

113

Chapter 4 Description of the final version

I

n this chapter, the main characteristics of the final version of the program are presented. Section 4.1 presents an overview of the program. The main characteristics of the program are described concerning the four components: content, support, interface,

and scenario. The content is described in Section 4.2. The support is presented in Section 4.3. Section 4.4 and Section 4.5 present the characteristics of the interface and the scenario respectively. 4.1 Overview of the program The final version of the program mainly includes five parts (see Figure 4.1): Main Frame, Designer's Aid, Edit Panel, Word, and Tutorial. The main reasons for splitting the program up into these parts are: i) they are closely related to instructional scenario development; ii) they can provide users flexibility when developing instructional scenarios. For example, new users of the program can start with the Tutorial part to learn how to use the program; novice designers can use the Designer's Aid part to design an instructional scenario by following step-by-step guidance; and experienced designers can use the Edit Panel part to directly edit an instructional scenario or can use Microsoft Word to edit an existing scenario. Next, these five parts will be briefly explained.

Page 126: computer support for multimedia curriculum design

114 Chapter 4

Main Frame Tutorial Edit Panel

Word

Designer's Aid

Figure 4.1: Overall structure of the CASCADE-MUCH program

Main Frame is the starting point of the program. It provides an integrated environment in which the other parts are interconnected, and users can directly enter other parts based on their specific needs. The key functions of the Main Frame screens include: i) language (Chinese or English) selection; ii) a brief introduction to the program; and iii) access to other parts (Designer's Aid, Edit Panel, Word, Tutorial). A detailed description can be found in Section 4.4.2. Designer's Aid is a key part of the program. It aims at providing users, in particular novice designers, with step-by-step guidance for designing an instructional scenario. Also, it includes some models, which are related to multimedia curriculum design. On the one hand, these models are expected to help the designers understand the content on the screens. On the other hand, novice designers can also learn some information from them and improve their professional knowledge of multimedia curriculum design. Edit Panel is an editing tool, by which (experienced) designers can directly edit an instructional scenario without following the step-by-step guidance. It supports the detailed design of content selection, content representation, content organization, and interface design for a multimedia curriculum. Microsoft Word, being a powerful word processing tool, is linked to the program. The draft of an instructional scenario can be exported to Microsoft Word for further alteration and printing. In this program, one-way transfer can be done from Edit Panel to Microsoft Word.

Page 127: computer support for multimedia curriculum design

Description of the final version 115

Tutorial is expected to be an interactive training environment in which (novice) users can learn how to use the program. This part was added last to the program, and it is in a preliminary stage in the current version. Among these five parts, Designer's Aid is a key part. In addition to the step-by-step guidance for (novice) designers, who are the key intended target users of the program, Designer's Aid also provides access to other parts when they have gained greater experience in multimedia curriculum design (see Figure 4.2). In this chapter, the four components of content, support, interface and scenario mainly refer to Designer's Aid (see also Figure 4.2). The explanations of these four components will be given in the following sections. Unless otherwise stated, the program refers to the Designer's Aid part in this chapter.

Figure 4.2: Overview of Designer's Aid

4.2 Content The content of the program includes two aspects: analysis and design. Analysis includes goal and usage analysis, and learner analysis. Design refers to guidance for content selection, representation, organization, and interface design. In this section, these two aspects will be introduced in more detail.

Page 128: computer support for multimedia curriculum design

116 Chapter 4

4.2.1 Goal and usage analysis Goals

Many distinctions of educational aims/goals/objectives can be found in the literature. Bloom, Englehart, Thurst, Hill and Krathwohl (1956) classify educational objectives into three taxonomies: cognitive, psychomotor skills, and affective. Gagné, et al. (1988) classify five learning outcomes: intellectual skills, cognitive strategies, verbal information, motor skills, and attitudes. In the Component Display Theory (CDT), Merrill (1983) identifies four types of content: fact, concept, procedure, and principle; and classifies three levels of performance: remember, use, and find. Posner and Rudnitski (1986) distinguish two broad categories of Intended Learning Outcomes (ILOs): understandings and skills. In China, educational aims are usually divided into three types: morality, intelligence, and physical education (cf. You, 1998). Generally speaking, these divisions of aim/goals/objectives are classified into three broad categories in the literature: knowledge, skills, and attitudes. More discussion about the goals of a multimedia curriculum can be found in Section 6.2.1. In China, a goal is often described as what basic knowledge and skills should be learned after the learners have learned the knowledge units in a curriculum standard. Basic knowledge and skills are usually perceived as being strongly connected, and are often used as a basic goal by teachers and other educators (cf. You, 1998). Therefore, in this study knowledge and skills are integrated. Two levels of knowledge and skills are distinguished: i) basic knowledge and skills; and ii) extended knowledge and higher-level skills. Together with the attitudes, the goals of a multimedia curriculum are classified as follows: Basic knowledge and skills

Basic knowledge refers to the cognitive knowledge (Bloom, et al., 1956), verbal information (Gagné, et al., 1988), fact and concept (Merrill, 1983), and understanding (Posner & Rudnitski, 1986). Basic skills refer to the motor skills (Bloom, et al., 1956; Gagné, et al., 1988), parts of intellectual skills such as response chain and discrimination (Gagné, et al., 1988), and the two lower levels (remember and use) of performance (Merrill, 1983). Extended knowledge and higher level skills

Extended knowledge refers to any in-depth information related to the basic knowledge. Higher-level skills refer to cognitive strategies and parts of intellectual skills such as rules and higher level rules (Gagné, et al., 1988), including creative thinking skills, problem solving skills and self-learning skills (cf. Anderson, Anderson & Varanka-Martin, 1994).

Page 129: computer support for multimedia curriculum design

Description of the final version 117

Attitudes Attitudes, as used by Gagné, et al. (1988), refer to the affective (Bloom, et al., 1956), affective understanding and skills (Posner & Rudnitski, 1986), including positive beliefs to life, society and the world. Usage

Generally speaking, a multimedia program in instruction can be used to facilitate i) individual learning for learners; and ii) classroom teaching for teachers (cf. Forcier, 1996). In addition, a multimedia program may facilitate collaborative learning as well as group discussion (Frazee & Rudnitski, 1995). Moore (1989) distinguishes three types of interaction in distance education: learner-content interaction, learner-teacher interaction, and learner-learner interaction. Until the end of the 1980s, the latter interaction had hardly been addressed in distance education. However, with the development of technologies such as computer networking and the Internet, collaborative learning between learners (in a group), in particular those located in remote sites, is becoming possible (cf. Heeren, 1996). In this study, the usage of a multimedia curriculum is divided into the following three categories: individual learning for learners; collaborative learning for learners; and classroom teaching for teachers.

A multimedia curriculum can be used by both learners and teachers. Learners can use it individually to review and/or study the knowledge units, or they can work together collaboratively on the same topics by discussion or peer coaching. In addition, teachers can also use a multimedia curriculum in the classroom for elaborating or demonstrating knowledge units in order to improve learners' understanding. No matter with which usage, learners are the key beneficiaries of a multimedia curriculum because classroom teaching carried out by teachers is also for learners. Therefore, in this program the analysis of users will focus on learner analysis. 4.2.2 Learner analysis Learners are the most important users of a multimedia curriculum. Their characteristics inevitably have impact on the design of a multimedia curriculum. According to the ASSURE model (Heinich, et al., 1996), learner analysis can be carried out from the following three aspects:

Page 130: computer support for multimedia curriculum design

118 Chapter 4

General characteristics The general characteristics include broad identifying descriptors such as age and grade level. As explained in Section 3.4.1, the program took two subjects (Biology and Geography) as its experimental samples. These two subjects are taught/learned at different grades in Shanghai. For example, Geography has Natural Geography in Grade 6 at primary schools, World Geography in Grade 7 (junior secondary school Grade 1), Chinese Geography in Grade 8 (junior secondary school Grade 2), and an extended course of Geography in senior secondary school Grade 2. Biology has Life Science in Grade 8 and an extended course of Biology in senior secondary school Grade 2. After the subject and grade have been determined, the typical age of the learners in that grade can also be easily determined. Specific entry competencies

The entry competencies specify what knowledge and skills the learners should already possess in order to start or continue learning. Because a multimedia curriculum is a curriculum as well as a multimedia program, entry competencies of learners should include two aspects: subject-related knowledge/skills and computer-related skills. Learning styles

Undoubtedly, different students have their own preference as far as learning styles are concerned (cf. Butler, 1986; Fenrich, 1997). For example, some students prefer learning by doing, while others prefer learning by teaching. Some prefer visual learning materials, while others prefer text or audio oriented learning materials. Furthermore, even the same students may prefer different learning styles in different situations. For example, some students prefer to learn grammar from a teacher while they prefer learning by doing when learning how to use a tool. A multimedia curriculum is often designed for a number of learners with various learning styles. A practical way of meeting the needs of these learners is to include many different materials in one multimedia curriculum, from which learners can select whatever they like. Fenrich (1997) summarizes the following guidelines for dealing with learning styles in the design of a multimedia application: 1. One method of matching the characteristics of multimedia to learning

styles is to include learning materials in various presentation forms so that learners can find their preferred media materials.

2. Different learning resources (such as fact, concept, example) are recommended to be included in an instructional multimedia package, in which learners with different learning styles can find their preferred learning resources.

Page 131: computer support for multimedia curriculum design

Description of the final version 119

3. Occasional mismatches between learning styles and multimedia designs can help students learn to adapt to other instructional styles.

These guidelines were adopted to design a multimedia curriculum in this study. However, it does not mean that learning style analysis is not important or necessary. A brief analysis of learning styles can help designers understand what learning materials or activities are necessary to be included in a multimedia curriculum. Based on the ASSURE model, the learner analysis in this study focuses on the following three aspects: 1. subject knowledge/skills; 2. computer skills; and 3. preferred learning materials and activities. The subject knowledge/skills refer to what subject knowledge and/or skills the learners should have in order to start learning, and what should be supplemented in advance should the learners not yet meet these prerequisites. The computer skills in this study mainly refer to: i) basic skills of using computers; ii) basic skills of using Windows; and iii) basic skills of inputting Chinese characters. The learning materials and activities are classified into: i) visual or text-oriented learning materials; ii) concrete examples or abstract reasoning; iii) active learning or passive rote learning; iv) individual learning or collaborative learning (cf. Fenrich, 1997). 4.2.3 Content selection As explained in Chapter 2, a multimedia curriculum and a conventional curriculum usually cover the same content. But in practice, a multimedia curriculum often functions as a complement to a conventional curriculum. It does not seem to be necessary to include all content of a conventional curriculum in a multimedia curriculum. The content selection deals with how to select proper content (or knowledge units) for a multimedia curriculum. In this study, the program provides two approaches for content selection: from curriculum standards; and by concept mapping.

In Shanghai each course has a concrete curriculum standard, in which all knowledge units in that course are listed. One approach to content selection would be that teacher-designers pick up those necessary knowledge units for the multimedia curriculum from the curriculum standard. No literature for content selection was found during the study. Therefore, Tyler's model is borrowed to help teacher-designers select proper knowledge units theoretically. Although Tyler (1949) cautions that the learning experience in

Page 132: computer support for multimedia curriculum design

120 Chapter 4

his model is not exactly the same as content, the following five principles seem to be useful for content (knowledge unit) selection: 1. Learning experiences/knowledge units should help learners reach the

objectives. 2. Learning experiences/knowledge units should make learners feel

satisfactory. 3. Learning experience/knowledge unit selection should start from where

learners are. 4. One learning experience/knowledge unit may serve for several objectives. 5. One objective may be reached by several learning experiences/knowledge

units. Sometimes teacher-designers may need to include extra knowledge units, which are not listed in the curriculum standard, or they may want to organize the content in a different way. A concept-mapping tool ('Mindman') is linked to the program, which can facilitate teacher-designers in illustrating the relationship between various knowledge units and determine which knowledge units need to be included in the multimedia curriculum. Posner and Rudnitski's model is borrowed to help teacher-designers select proper knowledge units in practice when they want to use the concept mapping approach. In the model, Posner and Rudnitski (1986) summarize six practical guidelines for Intended Learning Outcomes (ILOs) selection. Within the six guidelines, the first four deal with what should be selected and the last two deal with how to select these ILOs. In addition, the last guideline indicates that the concept mapping approach would be proper for ILOs selection. The six guidelines are: 1. If only few ILOs/knowledge units can be reached, these few ILOs/

knowledge units should be higher priority ones. 2. The ILOs/knowledge units, which are closely related to the objectives,

should be selected. 3. If some ILOs/knowledge units are prerequisites of others, then those

ILOs/knowledge units might be learned in advance. 4. ILOs/knowledge units should include what a successful learner should

learn. 5. ILOs/knowledge units can be selected by central questions. 6. ILOs/knowledge units can be selected by concept mapping and/or

flowcharting. In a conceptual map or flowchart, those ILOs/knowledge units in central or important positions might have higher priorities.

Page 133: computer support for multimedia curriculum design

Description of the final version 121

4.2.4 Content representation The content representation aims at presenting the selected content (knowledge units) with proper presentation forms on a computer. It deals with two questions: i) What presentation forms can be used? ii) How to select proper presentation forms? There are many presentation forms in practice, such as text, pictures, audio, video, and animation. Furthermore, some presentation forms may be divided into further sub-forms. For instance, audio can be divided into sound, narration, and music. In this study, presentation forms are classified as follows: Text refers to regular words or sentences. In a conventional curriculum,

text is usually the major presentation form. However, in a multimedia curriculum, text is still used, but other presentation forms (such as pictures and audio) are popularly used since a computer can easily encode and integrate these presentation forms. Graphics refers to both theme pictures and background pictures. Each

selected knowledge unit may need a theme picture to illustrate its structure or relationship with other knowledge units. The proverb 'One picture says more than one thousand words' well describes the function of a (theme) picture. Of course, sometimes a picture may need a number of words to explain it. Besides theme pictures, a multimedia curriculum usually also needs background pictures to provide further information and/or beautify its interface. Very often, related knowledge units in one chapter or section use the same or a similar background picture. Audio refers to sound effects, narration and music. Sound effects usually

refers to bird-song, wind whistle, thunder, etc. Narration refers to voice interpretation such as speaking and reading aloud. Music refers to background music such as light or classical music. Motion refers to both animation and video. Animation includes 2-

dimension and 3-dimension animated pictures usually drawn manually, and video is usually transferred from video programs.

The selection of presentation forms is affected by several factors, such as learning objectives and practical constraints. According to Allen's media selection model (see Table 2.1), there might be some proper presentation forms for some specific learning objectives. For example, still pictures may be more proper for learning visual identification, while animation better fits

Page 134: computer support for multimedia curriculum design

122 Chapter 4

learning procedures. If Allen's model would be used for guiding how to represent content, it should be known for each knowledge unit which type of learning objectives are intended to be achieved. However, in the context of Shanghai, it seems not to be practical for teacher-designers to associate specific knowledge units with learning objectives as explained in Section 3.4.1. In addition, the Allen's model is controversial, and many educators do not completely agree with it. In the program, a general guideline for presentation forms selection is to represent a knowledge unit with as many presentation forms as possible, and leave most presentation forms optional. Users can choose their preferred presentation forms when they are using the multimedia curriculum. In addition, some practical guidelines are provided in the program to inform teacher-designers how to design specific presentation forms. For instance, in a classroom-teaching context, the text should (or could) be bigger with sharper contrast, and the pictures should have the option of zooming in and out so that all learners in the classroom can clearly read or see it. The user should be allowed to switch the audio off so that teachers can give their own interpretation. However, in an individual learning context, the text can be smaller; pictures do not need the ability to be zoomed in on, etc. 4.2.5 Content organization The content organization deals with how to sequence the selected and represented content in a helpful way. Generally speaking, content organization can take place on three different levels: macro, meso and micro. The macro level refers to the content organization among various courses. On this level, learning experiences (or content) can be organized in two directions: horizontal and vertical (Tyler, 1949). Horizontal organization engages the curriculum developers with the concepts of scope and integration, that is, the side-by-side arrangement of curriculum elements. For instance, arranging content from the separate subjects of history, anthropology, and sociology into a course. Vertical organization centers on the concepts of sequence and continuity and is concerned with the longitudinal placement of curriculum elements. Frequently, curricula are organized so that the same topics are introduced and treated in different grades, but at different levels of detail and difficulty. In this case, the content organization usually refers to the spiral organization (Ornstein & Hunkings, 1993). The meso level refers to the content organization within a specific course. On this level, a curriculum is usually organized in a menu structure, in a way of showing a table of contents at the beginning of a book and presenting the content in a structure of

Page 135: computer support for multimedia curriculum design

Description of the final version 123

chapter-section-unit. The micro level refers to the content organization among concrete knowledge units. On this level, knowledge units are usually organized in a linear way in a conventional curriculum. The content organization of a multimedia curriculum mostly takes place on the meso and micro levels. Obviously, the traditional organizational methods can also be used in a multimedia curriculum. In addition, at least the hyperlink approach popularly used in multimedia applications can be used at the micro level. Therefore in this program, the content organizational methods are classified into the following: Linear

The linear organization means that the knowledge units are organized in a predetermined chain. Users can move forward or backward, to the head or to the tail in the chain. A typical linear organizational structure is shown in Figure 4.3. In this figure, A, B and C refer to various knowledge units. … C B A Figure 4.3: Linear content organization

Non-linear

In this study, the non-linear organization includes two formats: menu and hyperlink. The menu format means that the knowledge units are organized in a hierarchical structure. A typical menu format is shown in Figure 4.4 (Menu). Usually the knowledge units at the top level of the hierarchical structure are more abstract, and the knowledge units at the bottom level are most concrete. Learners can go down to its subordinate level or go up to its parent level in the hierarchical structure. For instance, from the knowledge unit 1.1, users may go up to 1 or go down to 1.1.1 or 1.1.2. The hyperlink approach means that the knowledge units are organized into an interrelated network, in which the knowledge units are linked together. A typical hyperlink format is shown in Figure 4.4 (Hyperlink). Users can visit related knowledge units by following its embedded links. For instance, from the knowledge unit A, users can directly go to B or F.

Page 136: computer support for multimedia curriculum design

124 Chapter 4

1.2 1.1

… …

1.2.2 1.2.1 1.1.2 1.1.1

1 A

E

F

C

D B

(Hyperlink) (Menu)

Figure 4.4: Non-linear content organization

Integrated

The integrated organization means that the knowledge units are organized in a more comprehensive manner including both the linear and the non-linear organization. Very often, the knowledge units in a multimedia curriculum are organized in a linear sequence. In addition to going forward or backward to its neighbor units, learners can also go to other related units by following its embedded links. A typical integrated organization is illustrated in Figure 4.5. The knowledge units are organized in a hierarchical (menu) structure. From the top unit, learners can go down to its lower units. Some concrete knowledge units in this structure are organized in a linear way, such as A to D. Also, learners can jump to other related units from a specific knowledge unit. For instance, users can jump from C to knowledge unit 1.2.2; and from D to 1.1.2 and back.

D C B A

1.2.2 1.2.1 1.1.2 1.1.1

1.2 1.1

1 Figure 4.5: Integrated content organization

Page 137: computer support for multimedia curriculum design

Description of the final version 125

4.2.6 Interface design Teacher-designers usually do not have sufficient computer skills to design an interface for a multimedia curriculum. However, interface design is essential since a multimedia curriculum is a computer program. A more practical way for teacher-designers would be to select a proper interface style or template from a list of recommended styles or templates. Based on the selected style or template, computer programmers can create a concrete interface for the multimedia curriculum. The idea of interface styles is similar to the templates in PowerPoint. There are a lot of available templates for various purposes such as for presentations, presentation designs, or web pages in PowerPoint. Within each template there are many concrete examples. In the literature, no divisions of interface styles have been found. However, much instructional software --in particular the DOS-based instructional software in the past-- used to have an interface with several functional areas such as title, text, picture, and buttons (cf. Harrison, 1995). Recently, an increasing number of instructional software applications on Windows use a graphical interface, which usually integrates a background picture, icons for audio and video, and pictures and text. The background pictures are usually static and are mainly used for beautifying the interface. In order to be more interesting or vivid, sometimes a background picture might also be dynamic or active. For example, the 'Oceans' program produced by Microsoft has a vivid interface. On the screens, an animal (such as a sea horse) on the background sometimes moves around. In this program, the interface styles of a multimedia curriculum are tentatively divided into three categories. A classic and elegant interface usually includes some clear functional areas, such as areas of title, text, picture, buttons, and feedback. The advantage of a classic and elegant interface is that it is easy to locate specific information on a screen. Some disadvantages are that the screens usually lack variation and look rigid. A modern and popular interface usually has a graphical background, on which various icons of presentation forms are integrated. Clicking these icons will get corresponding presentation forms. The advantage of a modern and popular user interface is that the screens look more beautiful and attractive. The disadvantage is that it is sometimes difficult to find a specific icon (presentation form) on a screen.

Page 138: computer support for multimedia curriculum design

126 Chapter 4

A vivid and vigorous interface, compared to the modern and popular interface, adds more active objects to the interface, mostly on the background. The main advantage of this user interface type is that it looks more attractive and interesting. The main possible disadvantage is that it might be more difficult to create. In the program, each of these three interface styles has a number of concrete templates with minor variations. For instance, the screen elements on the classic and elegant interface may be at different positions; the background pictures or icons on the modern and popular interface may be different. An example of each interface style is provided in Appendix G. In addition to the interface styles and templates, the program also provides an option for teacher-designers to design their own interface if they want. Although interface design is more an art than a science (Galitz, 1993), some principles can still be applied. For example, Brown (1988) suggests that interface design should pay attention to consistency, metaphor, ease of learning, and ease of use. Galitz (1993) mentions that clarity, simplicity and attractiveness are important for interface design. Nielsen (1993) states that much attention should be given to learnability, satisfaction, memorability, error rate and efficiency. It is not necessary for teacher-designers to focus on all of these principles, because some principles are less important for them, such as error rate. However, the following two aspects seem to be more important for them in designing an interface: 1. The interface should be consistent. It should have a similar appearance to

other popularly used systems, and information should be located in expected places.

2. The interface should be attractive. An attractive interface usually needs to have active components like video clips, or animation on it; pretty pictures rather than purely text; and beautiful fonts, convenient colors, and good-looking buttons.

4.3 Support As an EPSS, the program involves many types of support. As explained in Section 2.4.1, these types of support can be divided into four broad categories: information, advice, tools, and training. In this section, these categories of support will be described in more detail.

Page 139: computer support for multimedia curriculum design

Description of the final version 127

4.3.1 Information The program provides some supporting information for the content on screens as well as for the navigation. Specifically, the information includes: explanation examples help tips hints.

The explanations, examples and help are content related; they provide extra information for the content on the screen. Some terms, which are related to multimedia curriculum design, were defined to be keywords, such as 'multimedia curriculum' and 'learner analysis'. Each keyword in the program has an explanation, which is always presented in the Windows Help. Also, each keyword has a concrete example. In order to be more consistent, all examples are related to the MCB project (see Section 1.2.1). Some examples (such as goals, usage) are presented in Windows Help, while the others (such as linear, menu, hyperlink content organization) are illustrated with external programs to give users a more intuitive image. Clicking a keyword will always get the explanation of it first. An example hotspot is embedded in the explanation window. Clicking it will get a corresponding example. The help refers to both theoretical models and practical guidelines in the Windows Help. The theoretical models aim at: i) helping users understand why the content is shown on screens; and ii) improving their professional knowledge. The practical guidelines combine the actual educational situation in Shanghai and provide users with more practical information. The tips and hints provide extra information for navigating through the program. Tips are pieces of information usually shown at the beginning of a program as a 'tip of the day'. Some pieces of information about navigation are presented as tips, such as how to navigate the program by using the browser and how to deal with acronyms. Each time when users click the 'Tips' button they will get a piece of such information in a popup window. Users can also browse to other tips on the same window by clicking the 'Up' or 'Down' button. Hints are designed for speed buttons, which have no (or not enough) textual information on them. When the mouse moves over it, an extra piece of information will pop up, usually under the button, to explain what the speed button is or what functionality it has. When the mouse moves away, the piece of information disappears automatically.

Page 140: computer support for multimedia curriculum design

128 Chapter 4

4.3.2 Advice Advice is also a kind of information. But compared to the support of information as explained in the former section, advice is usually a kind of heuristic information sometimes in a form of 'Based on…, you are advised to…' It is often given based on the users' profiles, histories, and the program's embedded intelligent expertise. In this program, in order to help users decide how to design an element and make explicit how the current settings will affect the following design, two types of advice are provided: suggestions previews.

Suggestions are given based on the previous settings and the embedded intelligent expertise, aiming at helping users decide how to deal with the current design. In comparison, previews are given based on the current settings, sometimes also with the former settings, to help users know how the current settings will affect the succeeding design. In addition to the suggestions and previews, an explanation is supplied on the suggestion/preview window to explain why the suggestions or previews are given. In the program, the intelligent expertise mainly refers to the heuristic knowledge about the interrelationships between the elements in the analysis part and the design part. Table 4.1 lists the interrelationships between the elements in these two parts. The left two columns list the main elements in the analysis part, and the other columns list the elements in the design part. The cell content in the table indicates how the analysis elements affect the design elements. More detailed description of the cells will be give below. The content selection is mostly carried out based on the goals of a multimedia curriculum. For example, if the goal of a multimedia curriculum is to stimulate learners' basic knowledge and skills learning, it might be more practical to select those basic but hard to understand knowledge units from the curriculum standard. If the goal is to improve learners' extended knowledge and higher level skills, then the content can include some in-depth explanations of the basic knowledge units, as well as some authentic problems and cases selected from a broad scope of the subject domain by concept mapping (cf. Heinich, et al., 1996). If the goal is to develop learners' attitudes, then it would seem appropriate to include some exemplary materials such as human (i.e. hero) models (Gagné, et al., 1988). In addition, different grades of learners may have different educational goals. For instance, the goals of junior secondary schools in Shanghai mainly focus on learning basic knowledge of

Page 141: computer support for multimedia curriculum design

Description of the final version 129

culture and science; and basic skills of reading, writing, and calculating. The goals of senior secondary schools focus on the acquisition of basic knowledge of natural and social science; and basic skills of using ICT and creative thinking (SSCRCO & SECIRO, 1999). Certainly, if learners lack some subject-related knowledge or skills, the multimedia curriculum can provide some supplementary materials. The general guideline for the content representation is to use as many presentation forms as possible to present each knowledge unit. However, some presentation forms might be more appropriate or important for some special purposes. For instance, if the goal is to develop learner's positive attitudes, then audio and motion (video) seem advisable (cf. Fenrich, 1997). If learners prefer visual materials, graph and motion are of course more suitable. The recommended presentation forms in the table do not mean that only those presentation forms can be used. They are seen as the more proper or useful ones, but other presentation forms can also be used. With regard to the content organization, a multimedia curriculum had better be organized in an integrated manner including the three organizational modes of linear, menu and hyperlink. In such a case, the multimedia curriculum provides more flexibility and learners can walk through it in a way they choose. However, some specific approaches might be more proper or necessary for some goals, usage and learning styles. For example, the hyperlink format might be more proper for the goal of (extended knowledge and) higher level skills improvement, because a hyper linked environment constantly requires learners to make decisions and evaluate progress, and forces learners to apply critical thinking skills (Marchionini, 1988; Roblyer, et al., 1997). If the learners do not have enough computer skills, then the linear approach might be more proper for them, because it is relatively easier to use. If the learners prefer active learning, then the hyperlink approach might be more proper since it provides more options for active information search and knowledge construction. Similarly, the learners who prefer passive learning may like to use the linear manner to walk through the multimedia curriculum screen-by-screen.

Page 142: computer support for multimedia curriculum design

130 Chapter 4

With regard to the interface design, a multimedia curriculum can use any of these three interface styles or any others they create since interface design can be seen as more of an art than a science and is highly personal. Of course, many principles or guidelines in literature are useful for interface design or interface style selection, such as Brown (1988), Nielsen (1993), and Shneiderman (1992). The fact is that the content of a computer program with a classic and elegant interface is often organized in a linear way, and the program with a modern and popular or vivid and vigorous interface is often organized in a hyperlink or integrated way. Therefore, if the content organization has been determined to be linear, the classic and elegant interface style will be recommended. Otherwise it will be advised to use the modern and popular or vivid and vigorous style.

Page 143: computer support for multimedia curriculum design

Description of the final version 131

Table 4.1: The interrelationship between the analysis and the design elements Content

Selection Content

representation Content

organization Interface design

– Basic know-ledge and skills learning

– Basic know-ledge units

– Text – Graphics

– Extended knowledge and higher level skills improvement

– In-depth explanation

– Authentic problems/ cases

– Graphics – Motions

– Hyperlink – Modern and popular

– Vivid and vigorous

G

O

A

L

S

– Attitudes development

– Exemplary materials

– Audio – Motions

– Individual learning

– Text, pic-tures, and motions can be smaller

– Collaborative learning

– Text – Audio

U

S

A

G

E – Classroom

teaching – Text, pic-

tures, and motions should be bigger.

– Junior grades – Basic know-ledge units

– Limited text

– Senior grades – Basic and extended knowledge units

– Text is acceptable.

– Lack subject knowledge/ skills

– Supplemen-tary materials

– Lack com-puter skills

– Linear – Classic and elegant

– Prefer visual learning materials

– Graphics – Motions

– Prefer text-oriented materials

– Text

– Prefer active learning

– Motions – Hyperlink – Modern and popular

– Vivid and vigorous

L

E

A

R

N

E

R

S

– Prefer passive learning

– Audio – Text

– Linear – Classic and elegant

Note: The blanks in the table mean no direct relation

Page 144: computer support for multimedia curriculum design

132 Chapter 4

4.3.3 Tools In addition to the information and advice as explained in the former sections, this program also provides some tools, aiming at helping teacher-designers easily carry out the tasks related to instructional scenario development. These tools are introduced in this section. Edit Panel

Edit Panel is not only an integral part of the program, but also an important tool. By using the Edit Panel tool, users can select knowledge units to be involved in a multimedia curriculum. They can describe the selected knowledge units with proper presentation forms. Also, they can organize the selected knowledge units in a way of linear, menu, hyperlink, or integrated. Furthermore, they can also select an existing interface template or make a new interface. Microsoft Word

The final outcome of the program is an instructional scenario, which can be saved in two formats. The first format is the Edit Panel one. Teacher-designers can use the Edit Panel tool to input, edit, modify and save an instructional scenario. Later on they can also load and modify it in the Edit Panel tool. The second format is as a Word document. After a tentative instructional scenario has been finished, it can be exported to a document in Microsoft Word. Teacher-designers can then use Microsoft Word to directly revise or print it. Mostly, computer programmers will use the printed paper-based scenario to discuss with the scenario developers and to program. Concept mapping

As mentioned in Section 4.2.3, content selection can be carried out in two ways: i) from the curriculum standard; and ii) by concept mapping. A concept-mapping tool can help teacher-designers specify which concepts or knowledge units are of importance or at the central position, and what the relationship between the concepts is. In this program, the concept-mapping tool of Mindman is linked to the program. The detailed information of Mindman can be found on the web site: http://www.mindjet.com. Flowchart

A Flowchart, as a visual descriptor of a computer program, is usually helpful for giving computer programmers an overall image of a multimedia program, and to construct the multimedia curriculum as well. Because teacher-designers in Shanghai usually do not have competent computer skills, they may have difficulties in using a professional flowcharting tool to make a flowchart. They need to work together with computer programmers to make a flowchart.

Page 145: computer support for multimedia curriculum design

Description of the final version 133

Further discussion about flowcharting can be found in Section 6.2.4. So far, no satisfactory flowcharting tool has been found. But the addition of a proper flowcharting tool to the program is planned. Web communication

A web site (http://cascade-much.20m.com), as a communication tool, has been created in a preliminary form. It is expected that the designer and users of the program can keep in touch through this web site. On the one hand, the latest or updated information about the program can be published on the web site. On the other hand, teacher-designers can also post their comments or feedback on the bulletin board, or have live chat sessions with others. In addition to the above tools, the program provides a general tool: translation. It is designed not for carrying out a task about instructional scenario development, but for presenting the program in other languages: currently Chinese or English. On the first screen of the Main Frame part, users can select their preferred language. On the tool bar of the Designer's Aid part, the translation button also allows users to change the language from one to another. 4.3.4 Training As explained in Section 2.4.1, training is a component of an EPSS, aiming at improving user's task performance. In this program, two types of training are included: Wizards

In this program, the wizards have two purposes. First, they can help teacher-designers carry out some complex tasks by breaking them down into a list of sub-tasks. Second, they can also help teacher-designers learn and gain skills in carrying out the tasks in an orderly way. Few wizards remain in the final version of the program, such as learner analysis and content representation, and all wizards are optional. They will execute only when users request them. Whether or not it is necessary to design a wizard will be discussed in Section 6.2.2. Tutorial

The Tutorial part is in a tentative stage in the current program. It includes an introduction to the program, and an explanation of the screen elements in the current version. It will be an interactive training environment in which users, especially novice users, can learn what the program is and how to use it. It is expected that users will gain enough skills to use the program after using the Tutorial part.

Page 146: computer support for multimedia curriculum design

134 Chapter 4

4.4 Interface Interface design is an important part of the program. In this section, the general characteristics and screen design of the program will be elaborated. 4.4.1 General characteristics In order to be easy to learn (for novice users) and easy to use (for experienced users; see Section 6.2.3), the interface design of the program focused on the following three characteristics. First, it is consistent. On the one hand, to some extent it is externally consistent with other computer applications. For example, it has a main menu on the top of the screen; the buttons and short-cut keys are similar to those in other applications. On the other hand, it is also internally consistent. For example, all fonts, colors, and buttons are the same on each screen; the same elements on different screens are at the same positions. Second, it is flexible. The program is split up into several parts. Different users can start with or work on different parts based on their specific needs or conditions. It provides two navigation tools: linear and browser. Users can walk through the program in any way they prefer. In addition, most actions in the program can be done with either the mouse or the keyboard. For example, users can get help via clicking the 'Help' button with mouse, or just press the F1 key on the keyboard. Third, it is visual. A lot of metaphors are used in the program to give users intuitive images. For example, in the browser, the items that have been visited are marked with a metaphor of foot, and the item being visited is marked with a metaphor of hand. On the toolbar, each button has an icon on it to represent its functionality. 4.4.2 Screen design In addition to the Microsoft Word and Tutorial parts, the final version of the program mainly includes three separate parts: Main Frame, Designer's Aid, and Edit Panel (see Figure 4.1). These three parts were created in this study; they were not existing applications. In this section, the screen design of these three parts will be explained.

Page 147: computer support for multimedia curriculum design

Description of the final version 135

Main Frame

Since the program includes several parts, it might be easier to use if these parts can be integrated. The Main Frame part was designed to provide an integrated environment in which the other parts were interconnected and users can easily move about among the various parts. The screen dumps of Main Frame can be found in Appendix A-1. The main screens include: Language selection

The program currently supports two languages: Chinese and English. If the English option is selected, everything in the program after the language selection screen will be shown in English, otherwise it will be shown in Chinese. A brief introduction

The brief introduction aims at giving users an overall image of the program by explaining, for instance, what CASCADE-MUCH is, what subjects it supports, and who the intended target users are. Access to other parts

As shown in Figure 4.1, from the Main Frame part, users can select next part(s) to continue. They may use Designer's Aid to develop an instructional scenario by following the step-by-step guidance; or use Edit Panel to directly edit an instructional scenario; or directly use Microsoft Word to modify an existing scenario; or use Tutorial to gain skills in using the program. Designer's Aid

Designer's Aid is a key part of the program. It provides guidance for teacher-designers to design an instructional scenario step-by-step. The screen dumps of Designer's Aid can be found in Appendix A-2. In order to be easy to use, Designer's Aid adopts the classic and elegant interface style (see Section 4.2.6). The interface is divided into the following functional areas: Menu

To be consistent with most Windows based applications, a menu was designed and located at the top of the screen as shown in Table 4.2.

Page 148: computer support for multimedia curriculum design

136 Chapter 4

Table 4.2: The explanation of the menu Items Sub-items Explanation

New Create a new instructional scenario Open Open an existing instructional scenario Save Save the current being edited instructional scenario Save as … Save the current instructional scenario into another file

File

Return Exit Designer's Aid and return to Main Frame First Go to the first screen Back Go back to the previous screen Next Go to the next screen

View

Last Go to the last screen Translation Change to another language Tips Show/hide the tips window Edit Go to the Edit Panel part Export Export the instructional scenario into Word Help Show the theoretical models and practical guidelines

associated with the current screen Suggestion Show the expert recommendation

Support

Preview Preview what will be affected by the current settings Options Working

directory Set the working directory

Contents Show the help contents Index Show the help index Home page on the WWW

Visit the home page on the Internet

Tutorial Use the Tutorial part

Help

About… Show the 'About…' information Browser

In order to be more flexible, a browser was designed. It is located at the left side of the screen. On the one hand, it functions as a map or an index of the program, from which users can easily know what topics are shown on each screen, which screens have been visited, and where the current position is. The visited items on the browser are marked with an icon of a foot, and the current item is marked with an icon of a hand. On the other hand, users can also easily jump to other screens by clicking the corresponding items on the browser.

Page 149: computer support for multimedia curriculum design

Description of the final version 137

Toolbar In addition to the provided support in the menu at the top of the screen, a toolbar was designed to provide users with fast access to the support. Each commonly used support component has a speed button on the toolbar. Users can simply utilize the support by clicking the speed button. The speed buttons are divided into two groups according to whether or not they are related to screens. The first group includes translation, tips, edit and export, which are not associated with screens. The second group includes models, suggestion and preview, which are associated with screens. Content, notes and buttons

The main part of the screen is the content area. In order to be more intuitive, each screen has a logo representing the content on that screen. In order to let users know how to deal with the content on the current screen, a note function was designed and the note is located beneath the content area. The buttons for linear browsing are located at the bottom of the screen, including first, back, next, last and return. Edit Panel

Edit Panel is an editing tool by which teacher-designers can directly edit an instructional scenario. In order to be consistent, a menu was also designed and located at the top of the screen. Many items are the same or similar as those in Designer's Aid. The main tasks performed with Edit Panel include the design of the elements: content selection, representation, organization, and interface design. These four elements are related to each other and some are overlapping. For instance, the content representation part uses proper presentation forms to describe knowledge units. These presentation forms might affect the selection of an interface style; and conversely, once the interface style has been decided, it might also affect what presentation forms should be used, e.g. whether the background picture or music need to be described. In order to be more flexible to use, these four elements were designed in three separate folders as explained below rather than in a linear way. The screen dumps of Edit Panel can be found in Appendix A-3. Content selection and representation

These two elements are located in one folder since the performance of the content selection is rather simple and the content representation is performed based on the results of the content selection. The content selection is to select proper knowledge units --usually from a curriculum standard-- and the

Page 150: computer support for multimedia curriculum design

138 Chapter 4

content representation is to describe the selected knowledge units with various presentation forms. In this folder, the subject and a list of knowledge units (such as a curriculum standard) is shown on the left side, from which users can select a subject and proper knowledge units for the multimedia curriculum. The right side has sub-folders of presentation forms and other screen elements as shown in Table 4.3. Table 4.3: Presentation forms and screen elements for content representation

Elements Explanation Title The title to be shown on the screen for the knowledge unit Text The textual information for elaborating the knowledge unit Picture The picture for showing the knowledge unit Background picture

The picture shown on the screen as a background for all knowledge units in one section or chapter

Sound/narration The sound or voice for explaining the knowledge unit Background music

The music to be played during showing the knowledge unit

Animation The animation for demonstrating the knowledge unit Video The video for showing the knowledge unit Buttons The buttons on the screen Keywords The keywords displayed on the screen for the knowledge unit See also The other related knowledge units linked to the knowledge unit Notes Other remarks The presentation forms and screen elements are organized in an object-oriented style. Each presentation form or screen element is expressed in both attributes and actions. The attributes describe what it is and what it has. The actions include how to deal with the attributes. For example, the attributes of video include a file name and complementary text. The actions include loading, playing and clearing the video file, and editing the text in the text window. Content organization

As mentioned in Section 4.3.2, the content of a multimedia curriculum can best be organized in an integrated manner including linear, menu and hyperlink styles. However, in some cases a specific organizational style might be more proper or necessary. In this folder, teacher-designers are provided with an opportunity to specify which organizational style (linear, menu,

Page 151: computer support for multimedia curriculum design

Description of the final version 139

hyperlink and integrated) they intend to use. In order to be simple and easy to use, Edit Panel does not intend to illustrate the detailed relationship between all specific knowledge units. However, it does provide a simple way of claiming generally what approaches will be used for organizing various knowledge units as well as for organizing knowledge units, sections and chapters. Interface design

As explained in Section 4.2.6, three interface styles (classic and elegant, modern and popular, vivid and vigorous) are provided in the program. Within each style, there are also some concrete templates. In this folder, users can select a proper interface style and a template for the multimedia curriculum. If they want they can also add new interface styles and/or templates to the program. Each concrete template has some default screen elements. Users can add or remove some screen elements as well. After the users select a screen template, they can specify attributes for some common elements on all screens such as a background picture, background music, and buttons. Since these common elements are mostly the same for all knowledge units in one section or chapter, it is not necessary to describe them for each knowledge unit. Therefore, they are commonly specified in this folder. Of course, users can still make adjustments to the content selection/representation folder if they wish. 4.5 Scenario An instructional scenario is the final outcome of the program, which is used for two main purposes: i) facilitating discussions between teacher-designers and computer programmers; and ii) guiding computer programmers. In this section, the general characteristics and components of the instructional scenario will be elaborated. An example of an instructional scenario is shown in Appendix B.

Page 152: computer support for multimedia curriculum design

140 Chapter 4

4.5.1 General characteristics The instructional scenario has the following two characteristics. First, it includes much information. Not only the design information of each knowledge unit, but also the analysis information about the subject, learners and designers are included in the instructional scenario. It is expected that this information can provide computer programmers with rich information for programming as well as for discussions with the designers of the scenario. Second, it is well structured and easy to read. The instructional scenario can be exported to Microsoft Word and become a Word document. All information is presented in tables with different formats. It is easy to read although much information is included in it. 4.5.2 Components An instructional scenario mainly includes two components: analysis information and design information. Analysis information

The analysis information is collected from the analysis part of Designer's Aid, including: Subject

i. the name of the subject the multimedia curriculum covers; ii. the grade the multimedia curriculum is opened in; iii. the goals the multimedia curriculum intends to achieve; iv. the usage the multimedia curriculum will be used for. Learners

i. the age of the learners; ii. the subject knowledge and skills they have or do not have; iii. the computer skills they have or do not have; iv. their preferred learning materials/activities. Designers

i. their contact information such as names, addresses and telephone numbers.

Page 153: computer support for multimedia curriculum design

Description of the final version 141

Although this information may seem unimportant for computer programmers, it can provide them with some background information about the design of the instructional scenario. In addition, it can also help computer programmers better understand and even argue the designed instructional scenario. Design information

The design information is collected from the Edit Panel, since the detailed design of each element (content selection, representation, organization, and interface design) is carried out in Edit Panel. The design information mainly includes the results of: Module selection

As introduced in Section 2.3.2, a multimedia curriculum usually has five modules: aims/goals/objectives, content preparation, content elaboration, practice, and assessment. The current version of the program focuses on the module of content elaboration, and the reasons for it can also be found in Section 2.3.2. In the future, other modules may also be supported. The module selection information lists what module(s) is (are) to be included in the instructional scenario. Interface template, the common screen element attributes, and methods of content

organization In order to be consistent, all knowledge units in the same module, sometimes even in the multimedia curriculum, often use the same interface template. The selected interface template for the module is shown on the instructional scenario, giving computer programmers an image of what the intended interface will look like. Some common screen elements (such as the background picture, the background music and the buttons) need not be described in each knowledge unit. It is commonly described beneath the interface template. In addition, the general approach to content organization among units and among chapters, sections and units is listed in this part. Description of each knowledge unit

Each knowledge unit can be described with many presentation forms. In addition, it also has some attributes such as names of the section and the chapter in which it is located. In the instructional scenario, each knowledge unit is described with the factors as shown in Table 4.4.

Page 154: computer support for multimedia curriculum design

142 Chapter 4

In addition, a flowchart should be included in an instructional scenario. However, because the program cannot produce a flowchart in this version, the flowchart is not included in the scenario. Nevertheless, a flowchart will be added to the instructional scenario after it is developed by the computer programmers and the teacher-designers. Table 4.4: The factors used for describing each knowledge unit

Factors Explanation Code A unique number for the current knowledge unit Unit The name or title of the knowledge unit Section/chapter The name or title of the section/chapter where the knowledge

unit is in Course The name of the course which the multimedia curriculum covers Interface template

The intended screen layout

Title The title to be shown on the screen for the knowledge unit Text The textual description for the knowledge unit Graphics The descriptions, file names and previews (if applicable) of the

theme picture and the background picture Audio The descriptions, file names of the narration, sound and

background music for the knowledge unit Motion The descriptions, file names of the animation and video for the

knowledge unit Organization The descriptions of buttons, keywords and related topics on the

screen Notes Other remarks

Page 155: computer support for multimedia curriculum design

143

Chapter 5 Assessing the practicality of the prototype

T

his chapter elaborates the assessment of the practicality of the prototype in Shanghai. Two studies were organized for this assessment. The first study focused on primary target group users, and the second study focused on other users who were

related to, or interested in the program. This chapter starts with an introduction in Section 5.1. The detailed design of the assessment studies is elaborated in Section 5.2. The results with primary target group users and other users are presented in Section 5.3 and 5.4 respectively. Finally, this chapter ends with the main conclusions. 5.1 Introduction Through four rounds of prototyping, as elaborated in Chapter 3, a satisfying prototype has been developed. Based on the expert appraisal at the ECNU (see Section 3.6.2), it was found that the prototype was valid and had potential to be practical for intended target users. In order to verify to what extent the prototype would be practical, the prototype was assessed in Shanghai. According to van den Akker (1999) and Nieveen (1997, 1999), practicality usually refers to the extent to which users consider a system as appealing and usable under normal conditions. These normal conditions usually refer to the authentic context where the intended target users work. Based on this guideline, two studies were organized in person in Shanghai to assess the practicality of the program. These two studies involved two different kinds of users respectively. The primary target group users, teacher-designers of the subjects of Biology or Geography, took part in the first study. Other users, including teacher-designers of other subjects, CBL designers from secondary schools and software developers from education-related computer companies, took part in the second study. The overall research question was:

Page 156: computer support for multimedia curriculum design

144 Chapter 5

To what extent is the CASCADE-MUCH program practical for both primary target group users and other users in the context of Shanghai?

With regard to the first study, the following sub-questions were posed: 1. Are the four components (content, support, interface and scenario)

perceived to be practical by the primary target group users? 2. Do novice designers and experienced designers use the program

differently? With the second study, the following sub-questions were posed: 1. Are the four components (content, support, interface and scenario)

perceived to be practical by the other users? 2. Is it worthwhile to extend the program to support other

subjects/applications? 5.2 Design of the assessment studies 5.2.1 Participants Study 1: with primary target group users

With the intent of triangulation (Miles & Huberman, 1994), the following criteria were used to select participants in the first study. First, the participants should be real intended target users, who were either Biology or Geography teachers (or teaching researchers at district educational colleges) because only these two subjects were supported by the program. Second, some participants should have experiences of multimedia curriculum design and some should not, because the program was designed to be usable for both novice designers and experienced designers. Third, the participants must have basic computer skills; otherwise they might have initial difficulties in using the program. Eventually, based on the above criteria six participants were invited and agreed to participate in the study. Their general characteristics are briefly summarized in Table 5.1.

Page 157: computer support for multimedia curriculum design

Assessing the practicality of the prototype 145

Table 5.1: General characteristics of the participants in study 1 (n=6)

Participants Experienced designer Computer skills A Biology teaching researcher × B Biology teacher × C Geography teaching

researcher

× D Geography teacher × E Biology teaching researcher × × F Biology teaching researcher × × Of the six participants, four (A, B, E and F) were Biology teachers or teaching researchers (who worked at district educational colleges), two (C and D) were Geography teachers or teaching researchers; two (B and D) were secondary school teachers, and four (A, C, E and F) were teaching researchers from district educational colleges. Two (E and F) of them were experienced designers, who gained experiences of multimedia curriculum design during the MCB project (see Section 1.2.1). Both of them were responsible for the development of the instructional scenario. Four (A, B, C and D) of them were novice designers, who had never developed instructional scenarios before, but had experience in developing slide shows with PowerPoint or simple instructional software. All of them had basic computer skills and were able to use computers. Study 2: with other users

Because the second study aimed at collecting general opinions of other users who were not intended target users but were interested in the CASCADE-MUCH program, participant selection was rather broad. In general, those people who were related to multimedia curriculum design or were interested in CASCADE-MUCH were considered to be proper candidates. Three kinds of people were invited to take part in the workshop. They were: 1. teacher-designers of other subjects; 2. computer-based learning (CBL) designers at secondary schools; 3. software developers at education-related computer companies. It was assumed that winners of a CBL instructional software design competition in the Pudong new area of Shanghai, who were teachers or CBL designers at primary or secondary schools, would be interested in multimedia instructional design. An invitation letter for this study was sent to each winner (about 30 in total). In addition, invitation letters were also sent to some education-related computer companies because they were making similar

Page 158: computer support for multimedia curriculum design

146 Chapter 5

educational software and might also be interested. Eventually, 13 participants expressed interest in joining the workshop after they received the invitation letters of the workshop. Their general characteristics are summarized in Table 5.2. Table 5.2: General characteristics of the participants in study 2 (n=13)

Participants Organization Computer skills Mathematics teachers (n=2) Primary schools × History teacher (n=1) Secondary school × Physics teachers (n=2) Secondary schools × Chemistry teacher (n=1) Secondary school × Biology teacher (n=1) Secondary school × CBL designers (n=3) Secondary schools × Software developers (n=3) Computer companies × The seven subject teachers had experience with presentation slides design with PowerPoint or simple multimedia instructional software design, but had no experience of instructional scenario development for multimedia curricula. The three CBL designers at secondary schools were usually computer literacy teachers, but also provided technical support for subject teachers who wanted to design instructional software. The three software developers at education-related computer companies were computer programmers or CBL designers. 5.2.2 Procedures and activities In general, the procedures, activities and instruments of these two studies were similar, but slight differences existed. In this section, the procedures and activities of these two studies will be described. Study 1: with primary target group users

The first study was organized in three small groups at two separate times. Each group consisted of two participants. The first group included one Biology teaching researcher and one Biology teacher (A and B), who were both novice designers. The second group included one Geography teaching researcher and one Geography teacher (C and D), who were novice designers as well. The research activities of these two groups were held successively at one institute (Huangpu District Educational College) in one afternoon. The third group included two Biology teaching researchers (E and F), who were experienced designers of instructional scenarios. The research activity of this group was held one week later at another institute (Putuo District Educational

Page 159: computer support for multimedia curriculum design

Assessing the practicality of the prototype 147

College). Although the first two groups and the third group were organized at two different times and places, they are described together in this section since the procedures, activities and instruments were similar. In total, the whole process took about two and a half hours. The procedure of each group consisted of the following steps: 1. Introduction (about 10 minutes) During the introduction, the aims of the program and the workshop, the procedure and time schedule of the workshop were briefly explained. 2. Assignment (about one and a half hours) After the introduction, each participant was invited to carry out an assignment. The assignment was to make an instructional scenario for any knowledge unit in their textbooks. The knowledge unit was selected by themselves. Each participant worked on one computer individually. During this process, participants were encouraged to ask questions, and could chat with their peers and/or with the evaluator. Meanwhile, the evaluator observed their working processes, and made notes of their questions, errors, comments and suggestions. The evaluator did not disturb their activities during this process unless the participants met with some errors or needed personal support. 3. Questionnaire (about 15 minutes) After the participants tentatively finished their assignments, the evaluator helped them check whether the instructional scenarios had been successfully produced in Microsoft Word. If it was done, they were reminded to answer questions on the questionnaire. The detailed description of the questionnaire will be given in Section 5.2.3. 4. Discussion (about 20 minutes) After the participants had finished their questionnaires, a further in-depth discussion followed. The discussion focused mainly on the advantages and disadvantages of the program, and how to improve the program in the future. Study 2: with other users

The second study was held as a workshop in a computer lab at a secondary school (Jinhua Senior Secondary School). The main process of this workshop was similar to that of the first study. Some slight differences are explained below.

Page 160: computer support for multimedia curriculum design

148 Chapter 5

First, two assistants were invited to help the evaluator organize the workshop. They installed the program on each computer before the workshop, made video recordings and took observational notes during the workshop, and took discussion minutes during the discussion. Second, one more session (a presentation) was added after the introduction. The main aim of the presentation was to give the participants a general idea of what CASCADE-MUCH is, and more specifically to demonstrate its aims, structure, functionality and design elements (content selection, representation, organization and interface design). The presentation took about half an hour. Third, the assignment was slightly changed, as the users were not intended target users. The teacher-designers of other subjects were encouraged to develop an instructional scenario based on a section or a knowledge unit in the textbooks from their specific subject areas. The CBL designers and software developers were advised to make instructional scenarios for any subject they liked. 5.2.3 Data collection and analysis Table 5.3 illustrates the instruments used for data collection during these two studies and data analysis methods applied. Table 5.3: Overview of instruments and data analysis

Instruments

Study 1 (n=6)

Study 2 (n=13)

Likert scale questions on questionnaire Mean Mean, S.d, t-score Open-ended questions on questionnaire S&D S&D Built-in log files Figures, tables S&D Observational notes S&D S&D Discussion minutes S&D S&D Video S&D Note: S.d = Standard deviation; S&D = Summary and discussion During the first study, the instruments used for data collection included: a questionnaire, built-in log files, observational notes and discussion minutes. The questionnaire included two parts (see Appendix F). The first part contained 30 Likert scale questions covering the four components (content, support, interface and scenario) of the program. The second part included five open-ended questions. In the first study, only mean scores were calculated on the Likert scale questions since the number of participants was small. In the second study, means, standard deviations and t-scores were calculated. The open-ended questions on the questionnaire were summarized.

Page 161: computer support for multimedia curriculum design

Assessing the practicality of the prototype 149

The built-in log files were used to record how users really used the program, and find out the differences (if any) between novice designers and experienced designers. The built-in log files mainly traced the following variables: The paths of walking through the program

This variable indicated whether novice designers and experienced designers walked through the program differently, and helped to find out reasons why some screens (if any) were visited several times or not at all. The time spent on each screen

On the one hand, this variable indicated whether some screens were harder to understand or more interesting to some users; on the other hand, it also showed whether novice and experienced designers spent different amounts of time on each screen. The support tools utilized

This variable indicated what kinds of support were utilized (very often) and might be useful for users.

The data included in each built-in log file were visualized with figures or tables. In this way, the similarities and differences between each participant could be easily found. The observational notes contained questions the users asked, the opinions they expressed, the comments/suggestions they proposed, and the errors they met. The discussion minutes provided insight in the comments and suggestions the participants proposed during discussion and reasons for making those comments and suggestions. In the second study, the instruments used for data collection were similar. One additional instrument (video) was used to make a live record of some episodes of the workshop. The video would help the evaluator and also other people who are interested in the workshop to review what was happening during the workshop. In addition, the built-in log files were mainly designed for comparing the differences between novice and experienced designers. The data collected with this instrument were not systematically analyzed since all participants were novice designers in this study. But the built-in log files still provided some additional information as summarized in Section 5.4.2.

Page 162: computer support for multimedia curriculum design

150 Chapter 5

5.3 Results with primary target group users 5.3.1 Perceived practicality of the four components Content

There were four statements on the questionnaire, which were related to the component of content. The participants' perceptions with regard to the practicality of the content, based on the answers to the four-point Likert scale questions/ statements, are summarized in Table 5.4. The original data of the answers to the questions are shown in Appendix H. Table 5.4: Perceived practicality of the content (n=6)

Content Mean* Max Min 1. I could easily understand the content on each

screen. 3.3 4 2

2. The content fits my practical needs for multimedia curriculum (or learning materials) design.

3.0 4 2

3. I could easily understand the explanations of keywords/models.

3.5 4 3

4. I learned some useful information from CASCADE-MUCH.

3.3 4 2

3.3 Note: *4 = agree; 3 = slightly agree; 2 = slightly disagree; 1 = disagree The overall mean of the answers to these four Likert scale questions was 3.3, which indicated that the participants basically perceived the content to be rather practical although some participants slightly disagreed with some specific statements. Support

The program provides four broad categories of support: information, advice, tools and training. The participants' perceptions with regard to practicality of the support, based on answers to the Likert scale questions on the questionnaire, are summarized in Table 5.5.

Page 163: computer support for multimedia curriculum design

Assessing the practicality of the prototype 151

Table 5.5: Perceived practicality of the support

Support n Mean* Max Min 5. The help function provided me with useful

models. 6 3.3 4 2

6. The suggestions gave me some valuable expert advice.

5 3.4 4 3

7. The previews indicated to me what would be affected by the current settings.

5 3.4 4 2

8. The additional explanations of suggestions/previews helped me understand why the suggestions/previews were given.

5 3.0 4 1

9. The tips provided me with useful information. 5 3.8 4 3 10. The Edit Panel helped me easily make

instructional scenarios. 6 4.0 4 4

11. I could easily export scenarios into Microsoft Word.

6 3.7 4 2

12. I think the explanations/examples of keywords are practical.

6 3.3 4 2

13. The concept-mapping tool is useful for me to make content selection.

5 3.4 4 3

3.5 Note: *4 = agree; 3 = slightly agree; 2 = slightly disagree; 1 = disagree Generally speaking, the results in Table 5.5 indicate that the participants perceived the component of support to be practical, since the overall mean of answers to all questions was 3.5 on a four-point scale and even the lowest mean score was still as high as 3.0 (Q8). However, some participants (slightly) disagreed with some specific statements such as Q8 that got the minimal scale. An encouraging point was that all participants agreed that Edit Panel could be helpful when making instructional scenarios (Q10). In addition, it was probably because some categories of support were not utilized, some questions such as Q6 and Q7 were not answered by all participants. Interface

The participants' perceptions with regard to practicality of the interface, based on the answers to the Likert scale questions on the questionnaire, are summarized in Table 5.6.

Page 164: computer support for multimedia curriculum design

152 Chapter 5

Table 5.6: Perceived practicality of the interface (n=6)

Interface Mean* Max Min

14. User tasks on each screen are clear for me. 3.7 4 3 15. The meanings of buttons on each screen are clear

for me. 3.5 4 3

16. I like the fonts and colors on each screen. 3.2 4 1 17. The navigation tools (linear and browser) are easy

for me to use. 3.8 4 3

18. I think the amount of information on each screen is proper.

3.7 4 3

19. I think each screen has a consistent design. 3.8 4 3 20. I believe that the interface is consistent with other

computer programs. 3.7 4 3

21. The interface is easy to learn for me. 3.5 4 3 22. The interface is easy to use for me. 3.7 4 3 23. I feel the program is error free. 3.2 4 2 3.6 Note: *4 = agree; 3 = slightly agree; 2 = slightly disagree; 1 = disagree In general, the participants agreed with the practicality of the interface, since the overall mean of the answers to all questions was 3.6 and the lowest means were 3.2 (Q16 and Q23) that is still higher than 'slightly agree'. However, one point that needs concern was that Q16 got the minimal scale and its mean was also relatively lower, which indicates that the fonts and colors need improvements. In addition, the relatively low mean of Q23 shows that some errors occurred during the time period that the participants were using the program. Scenario

The scenario is the major product of the program. The participants' perceptions with regard to the practicality of the scenario, based on the answers to the questionnaire, are presented in Table 5.7.

Page 165: computer support for multimedia curriculum design

Assessing the practicality of the prototype 153

Table 5.7: Perceived practicality of the scenario (n=6)

Scenario Mean* Max Min 24. The scenario includes what I intended to include. 3.3 4 2 25. The scenario is well structured. 3.5 4 2 26. I can easily modify the scenario within Microsoft

Word. 3.7 4 3

27. I am satisfied with the produced scenario. 3.8 4 3 28. The meanings of the elements in the scenario are

clear for me. 3.3 4 2

29. The produced scenario can help me easily discuss my wishes with computer programmers.

3.5 4 3

30. I believe that the scenario would be easy to understand for computer programmers.

3.7 4 3

3.5 Note: *4 = agree; 3 = slightly agree; 2 = slightly disagree; 1= disagree Although the scenarios that were produced within one and a half hours during the workshop were rather preliminary, the results in Table 5.7 show that the participants perceived their scenarios to be practical. The overall mean of the answers to all questions was 3.5, and even the minimal means were as high as 3.3. However, the minimal scores indicate that some statements such as Q24, Q25 and Q28 were slightly disagreed with by some specific participants. Comparison of mean scores between novice and experienced designers

Figure 5.1 provides the specific means of novice and experienced designers. This figure shows that the mean scores of the components reported by the experienced designers were generally higher than those of the novice designers, with an exception of the component of interface. The results indicated that the experienced designers were more satisfied with these three components (content, support and scenario) than the novice designers. However, the differences on these three components between the novice and experienced designers were not statistically significant. The t-score of Q8, which had the biggest difference, was 1.55, while for a sample size of 3 and probability of 10%, the cutoff value for the t-score is at least 2.92 in order to say that the difference is statistically significant with 90% certainty (cf. Bhattacharyya & Johnson, 1977; Krathwohl, 1998). With regard to the component of interface, the difference between the experienced and the novice designers were less and not significant at all.

Page 166: computer support for multimedia curriculum design

154 Chapter 5

Mean4

3.9 (Experienced designers)3.83.73.63.53.43.33.23.1 (Novice designers)

32.92.82.72.62.52.42.3

Content Support Interface Scenario1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 Question

Figure 5.1: Means of answers given by the novice and experienced designers

Conclusion

The answers to all Likert scale questions on the questionnaire showed that the four components (content, support, interface and scenario) were perceived to be rather practical by the primary target group users. In addition, no statistically significant difference was found between the novice and experienced designers with regard to the perceived practicality of the four components. 5.3.2 Actual use of the program The results in the built-in log files reflect how participants really used the program. The results were recorded by the variables walking routes, time spent on each screen and support tools utilized. Walking routes

Figure 5.2 shows examples of two participants (B and E) walking through the Designer's Aid part. By comparing walking routes of the six participants, it was found that these two walking routes were rather representative. Of the six participants, four walked through the Designer's Aid part in a way similar to Figure 5.2 (A), and two in a way similar to Figure 5.2 (B). Figure 5.2 (A) illustrates how participant B started from the screen 1 and gradually stepped forward to the last screen (screen 16). Then he returned to the first screen again and quickly stepped forward to screen 13. After that he returned to screen 1 again and stepped forward to screen 10. Figure 5.2 (B) illustrates

Page 167: computer support for multimedia curriculum design

Assessing the practicality of the prototype 155

participant E stepped back and forward in a more flexible way. It was found that no obvious differences existed between novice and experienced designers. Of the four novice designers, three adopted a similar walking route as in Figure 5.2 (A), and one adopted a similar walking route as in Figure 5.2 (B). Of the two experienced designers, one adopted a similar route as in Figure 5.2 (A), and another one (participant E) adopted the route of Figure 5.2 (B).

Notes: - The numbers on the curve show the time (in seconds) spent o

Step4342 641 140 039 138 037 136 035 134 033 132 1231 530 529 328 227 026 125 024 023 122 021 120 019 118 6017 716 015 2514 2913 812 1411 2410 29 78 67 66 75 494 183 262 171 100 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 Screen

Step39 938 137 236 5035 534 333 232 031 230 229 228 427 25 [Mindman, Edit Panel]26 325 1124 123 1322 521 320 219 518 1917 516 5 [Main, Edit Panel]15 1514 013 712 4211 11210 19 08 27 396 575 314 03 572 31 220 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 Screen

(A)

- The underlined screen numbers (such as 1 and 2) show that tthe screens.

- The items in brackets[] mean that the user visited those parts - The screen dumps of screen 1 to 16 can be found in Appendi Figure 5.2: Examples of walking routes

(B)

n the screens. he user clicked keyword(s) on

then. x A-2.

Page 168: computer support for multimedia curriculum design

156 Chapter 5

Time spent on each screen

It was assumed that novice and experienced designers might spend significantly different amounts of time on each screen since they had various levels of expertise and might be interested in various aspects as well. Figure 5.3 shows the time (in seconds) each participant spent on each screen. No significant differences were found between the novice and the experienced designers. All participants spent a similar amount of time on each screen with the exception of participant A, who spent relatively more time on screens 7 and 11. The other participants did not have obvious differences on time spent on each screen. The results also show that on screens 1, 5, 7 and 11 the participants spent relatively more time. Screen 1 is the starting screen of the program, introducing what will be analyzed in the analysis part. Screen 5 is about learner analysis, in which a wizard is included. The wizard might cost the users more time. Screen 7 is a summary of the analysis part, and screen 11 is about content selection. Relatively little time was spent on screens 6, 10 and 12. Screen 6 is to specify the designer's role, screen 10 is the starting screen of the content elaboration, and screen 12 is about content representation.

Page 169: computer support for multimedia curriculum design

Assessing the practicality of the prototype 157

Time250240230220210200190180170160150140130120110100 B90 A8070 E6050 C4030 D20 F100 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 Screen

Note: A, B, C, D = novice designers; E, F = experienced designers Figure 5.3: Time spent on each screen

Support tools utilized

In addition to the walking route and time spent on each screen, the utilized support tools were also recorded in the built-in log files. Table 5.8 summarizes how many times each support tool was used by each participant. It shows that although minor differences existed between individuals, no obvious differences existed between the novice (A, B, C and D) and the experienced (E and F) designers. From this table, it can be found that keywords were frequently clicked by all participants with the exception of participant D. The Edit (Panel) tool was utilized at least one time by each participant. Each

Page 170: computer support for multimedia curriculum design

158 Chapter 5

participant used the Microsoft Word tool at least one time, either in the Designer's Aid part or in the Edit Panel part, to produce instructional scenarios in Microsoft Word. Four participants utilized the browser tool to visit screens arbitrarily. Of the six participants, only participant E attempted to get help, suggestions and previews intentionally. However, this does not mean that the other participants never used these kinds of support. They could also get related help when they clicked keywords. In addition, suggestions and previews at specific times pop up automatically when available; this was not recorded by the built-in log file. Furthermore, five of the six participants used tips, and four of them attempted to use the concept-mapping tool 'Mindman' to produce concept maps. Table 5.8: Support tools utilized by the participants

Designer's Aid

Edit panel

Kw Edit Word Brws Help Sgt Prv Tips Mm Word A 13 3 2 8 – – – 1 y 9 B 6 8 – 1 – – – 3 y 7 C 3 1 – 1 – – – 2 – 2 D – 1 – – – – – – – 3 E 8 1 1 19 2 3 1 1 y – F 5 2 – – – – – 1 y 1 Notes: - A, B, C, D: novice designers; E, F: experienced designers - Kw/Edit/Word/Brws/Help/Sgt/Prv/Tips: the numbers of Keywords/Edit/Word/

Browser/Help/Suggestion/Preview/Tips clicked - Mm: whether the concept-mapping tool (Mindman) was clicked - The sign – means the support tool was never intentionally used Furthermore, this table indicates that participant E was an active user who tried each kind of support tools several times. Comparatively, participant D was a more inert user, who only utilized the Edit tool one time and the Word tool three times without trying any other support tools. Summary and conclusion

This section presented results of three variables recorded in the built-in log files. With regard to the variable of the walking routes, it was found that two generalized walking routes (linear and flexible) were frequently adopted by the participants. But it did not show that the novice designers commonly adopted one route and the experienced designers adopted another. No obvious differences were found on the walking routes between the novice and the experienced designers. With regard to the variable of time spent on each screen, it

Page 171: computer support for multimedia curriculum design

Assessing the practicality of the prototype 159

seemed that no obvious differences existed between the novice and the experienced designers in this aspect, either. Also, no distinguishable differences were found between the novice and the experienced designers with regard to the variable of support tools utilized. In conclusion, although various users had minor differences, no significant overall differences existed between the novice and the experienced designers with regard to these three variables. 5.3.3 Comments and suggestions Open-ended questions on the questionnaire during the discussion were used to collect general comments and suggestions. In this section, these comments and suggestions will be summarized. Perceived advantages

Participant E, one of the two experienced teacher-designers, agreed that the program indeed had more advantages than the traditional approach to instructional scenario development on paper. She said that the program was easy and simple to use; the content was easy to understand; users could easily produce instructional scenarios by selecting some options. She believed that with the program users could immediately get help, explanations of keywords and expert advice when they met difficulties. She further added "In practice, mostly it is very difficult to find an expert to help you solve a problem very soon. But with this program, problems can be easily solved." Participant F said that the program was easy to use; instructional scenarios were easier to produce and to modify. He believed that the resulting instructional scenarios would help teacher-designers and computer programmers to communicate, because the instructional scenarios were based on systematic multimedia instructional design. Although the four novice designers had no experience in instructional scenario development, they expressed similar opinions on advantages of the program. Participant A mentioned that the instructional scenario he produced was well structured and easy to read. With the program, it was not necessary to spend much time in organizing the structure during the development of an instructional scenario. Participants B and D agreed that it could be very fast to produce a provisional instructional scenario with the program. Participant C mentioned that she had never made an instructional scenario, and had never used such a computer support system before. She felt some difficulties in using it, but she believed that she could gradually get used to it if she spent

Page 172: computer support for multimedia curriculum design

160 Chapter 5

more time on it. Doubted usefulness

The four novice designers (A, B, C and D) mentioned that they never developed instructional scenarios before. In reality, they were familiar with making instructional presentation slides with Microsoft PowerPoint. On the slides, they could put text, pictures and even video clips together. Shifts between two slides could be accompanied with special effects. For classroom teaching, they thought that these kinds of presentation were usually sufficient and also worked well. In addition, they thought that this kind of presentation slide was easy to make. Sometimes they made instructional software with Authorware, but not very often. In the latter case, they used to work together with computer teachers in their schools, because they thought development with Authorware required programming skills. However, in both cases they did not first develop instructional scenarios, but just made the instructional materials directly. These four novice designers thought that the program was not useful for them to make presentation slides or instructional software in practice. However, they agreed that what they developed so far was very limited instructional software, mostly used for one lesson, or only for a part of one lesson. For more complex multimedia projects, for instance a multimedia curriculum covering a whole course, this just-do-it approach would not work efficiently. In such a case, this program would become useful. The two experienced teacher-designers did not doubt the usefulness of the CASCADE-MUCH program. They believed that it would be helpful for teachers to make instructional scenarios for multimedia curricula. Expected materials

Four (B, C, E, and F) of the six participants mentioned that they needed more learning materials/resources when they were making instructional scenarios. They suggested that more lesson materials be added to the program. Participants E and F said that the lesson materials might include any texts, pictures, animation and/or video clips which were relevant to the knowledge units of the subjects. With these ready-to-use lesson materials, they would not need to make it again if they perceived that it would be appropriate for a multimedia curriculum. Participant B thought that teachers particularly needed lesson materials in preparation for classroom teaching. He believed that this program would become more useful and more attractive if it could provide more lesson materials. Participant C mentioned that when she made

Page 173: computer support for multimedia curriculum design

Assessing the practicality of the prototype 161

presentation slides with PowerPoint, she particularly needed some related supporting materials. If some materials could be found in this program, teachers would be more willing to use it. Simplified version

Two participants (C and D) suggested that the program should have a simplified version in order that new users of the program or novice designers of multimedia curriculum could easily start with it. They thought that the current program was complicated. It might be not very easy to learn or to use for new users or novice designers. They believed that a simplified (or 'foolproof') version and a comprehensive version should be provided. The simplified version might include some basic functionality. New users could start it with fewer difficulties. The comprehensive version might include more tools and broader functionality. Experienced teacher-designers could use the comprehensive version to make more complicated instructional scenarios. More discussions on this topic can be found in Section 6.2.5. Replaced product

Participant E thought that it might be better if the program could directly produce a final product of multimedia curriculum rather than an intermediate product of instructional scenario. She said that if teacher-designers could directly construct multimedia curricula with the program, then the process of multimedia curriculum development would become simpler and more efficient, because in that case the teacher-designers would not need to discuss and negotiate with computer programmers. Elaborated discussions on this issue can be found in Section 6.2.4. 5.4 Results with other users 5.4.1 Perceived practicality of the four components As mentioned in Section 5.2.1, 13 other users including teachers of other subjects, CBL designers at secondary schools and software developers at computer companies took part in this workshop. Although not all of them were teacher-designers, they were advised to use the program as if they were intended target users. The questionnaire was the same as that used in the first study. The mean and standard deviation of each question are presented in Table 5.9. Furthermore, in order to make the results easier to read, they are also presented in Figure 5.4.

Page 174: computer support for multimedia curriculum design

162 Chapter 5

Table 5.9: Perceived practicality of the four components by other users

n Mean* S.d. 1. I could easily understand the content on each screen. 13 3.6 0.5 2. The content fits my practical needs for multimedia

curriculum (or learning materials) design. 13 3.2 0.9

3. I could easily understand the explanations of keywords/models.

13 3.3 0.8

4. I learned some useful information from CASCADE-MUCH.

13 3.3 0.9

C

O

N

T

E

N

T Sub-total 3.3 0.8 5. The help function provided me with useful models. 13 3.1 0.9 6. The suggestions gave me some valuable expert advice. 13 3.2 0.6 7. The previews indicated to me what would be affected

by the current settings. 13 3.2 0.6

8. The additional explanations of the suggestions/ previews helped me understand why the suggestions/previews were given.

11 3.2 0.6

9. The tips provided me with useful information. 12 3.5 0.5 10. The support tool of Edit Panel could help me easily

make instructional scenarios. 12 3.5 0.7

11. I could easily export scenarios into Microsoft Word. 13 3.5 0.8 12. I think the explanations/examples of keywords are

practical. 12 3.4 0.9

13. The concept-mapping tool is useful for me to make content selection.

12 3.5 0.7

S

U

P

P

O

R

T

Sub-total 3.4 0.7 14. User tasks on each screen are clear to me. 13 3.2 0.7 15. The meanings of buttons on each screen are clear to

me. 13 3.3 0.6

16. I like the fonts and colors on each screen. 13 2.7 0.9 17. The navigation tools (linear and browser) are easy for

me to use. 11 3.3 0.5

18. I think the amount of information on each screen is proper.

13 3.1 0.8

19. I think each screen has a consistent design. 12 3.1 0.9 20. I believe that the interface is consistent with other

computer programs. 12 2.8 0.6

21. The interface is easy to learn. 13 3.5 0.5 22. The interface is easy to use. 13 3.5 0.5 23. I feel the program is error free. 11 2.5 0.9

I

N

T

E

R

F

A

C

E

Sub-total 3.1 0.7 To be continued on the next page

Page 175: computer support for multimedia curriculum design

Assessing the practicality of the prototype 163

n Mean* S.d. 24. The scenario includes what I intended to include. 11 3.1 0.8 25. The scenario is well structured. 12 3.3 0.7 26. I can easily modify the scenario within Microsoft

Word. 12 3.4 0.5

27. I am satisfied with the produced scenario. 11 3.3 0.6 28. The meanings of the elements in the scenario are clear

for me. 12 3.4 0.5

29. The produced scenario can help me easily discuss my wishes with computer programmers.

12 3.2 0.4

30. I believe that the scenario would be easy to understand for computer programmers.

12 3.0 0.7

SC

E

N

A

R

I

O

Sub-total 3.2 0.6 Total 3.2 0.7

Note: *4 = agree; 3 = slightly agree; 2 = slight disagree; 1 = disagree Generally speaking, the four components of the program were perceived to be rather practical by the 13 other users (7 teacher-designers of other subjects, 3 CBL designers at secondary schools and 3 software developers at computer companies) based on the data collected from the questionnaire. The average mean of answers to all questions was 3.2, and the average standard deviation was 0.7. Specifically, of these four components (content, support, interface, and scenario) the overall means were 3.3, 3.4, 3.1 and 3.2; and the standard deviations were 0.8, 0.7, 0.7, and 0.6 respectively. In particular, the participants more agreed with the practicality of the component of support (Mean=3.4) than the component of interface (Mean=3.1). However, the t-score was 1.11, which shows that the difference between these two means was still not large enough (should be at least 1.78 when sample size is 13 and probability is 10%) to reach statistical significance (cf. Bhattacharyya & Johnson, 1977; Krathwohl, 1998).

Page 176: computer support for multimedia curriculum design

164 Chapter 5

Mean3.73.63.53.43.33.23.1

32.92.82.72.62.5

Content Support Interface Scenario1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 Question

Note: = average mean; = mean of each question Figure 5.4: Illustration of means

One fact was that some specific questions had rather low means as indicated in Figure 5.4. Three means in the 'interface' part were lower than 3.0, and Q23 got the lowest score (2.5). The t-score of Q23 was -2.19 (<-1.78), which indicates that the difference between the mean of Q23 and the average mean was statistically significant. Further improvements need to be made to remove the errors. In addition, the differences between other means and the average mean were not statistically significant since the t-score of Q16, which got the second lowest score (2.7), was -1.56 (>-1.78). However, some improvements --particularly related to the component of interface-- were expected to be made. 5.4.2 Actual use of the program The data collected in the built-in log files reflect how the other users really used the program. In this section, a summary of the data is given. Walking routes

It was found that the other users used the program in a way similar to that of the target users. Of the 13 users, six used the program in a linear way as shown in Figure 5.2 (A), and the other seven users used the program in a more flexible way as shown in Figure 5.2 (B). The data show that no way was used more frequently than another was.

Page 177: computer support for multimedia curriculum design

Assessing the practicality of the prototype 165

Time spent on each screen

Figure 5.5 shows the average time spent on each screen for the 13 users. This figure indicates that users spent relatively more time on screens 1, 2, 8, 11 and 12, and spent relative less time on screens 9, 10 and 13. These two groups of data demonstrate that the users spent more time on the starting screens of the prototype and spent less time on the following screens. However, they spent relatively more time on the screen (#11) of the first design element (content selection), which indicates that the users paid much attention when they started to actually develop an instructional scenario.

Time807570656055504540353025201510

51 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 Screen

Figure 5.5: Average time spent on each screen

Support tools utilized

Table 5.10 lists the support tools utilized by each user in the second study. This table shows results similar to those in Table 5.8 in the first study. Keywords were frequently clicked by most users, and each user utilized the Edit Panel tool at least one time to actually edit an instructional scenario. The instructional scenario was also exported at least one time to Microsoft Word, either from the Designer's Aid part or from the Edit Panel part. The support of help, suggestion and preview was relatively less used intentionally. However, it does not mean that these types of support were not used. Users could visit help after clicking keywords, and some available suggestions and previews could pop up automatically. The concept-mapping tool of Mindman was used by almost each user. In conclusion, the other users could make instructional scenarios with the prototype; explanations and examples of the keywords seemed to be helpful since the users often clicked the keywords;

Page 178: computer support for multimedia curriculum design

166 Chapter 5

and the browser and Mindman might be helpful for them as well. Table 5.10: Support tools utilized by other users

Designer's Aid

Edit panel

Kws Edit Word Brws Help Sgt Prv Tips Mm Word P1 5 2 1 5 – – 1 1 y 2 P2 9 1 – – 2 – 1 2 y 3 P3 1 1 – – – 1 – 2 y 2 P4 3 1 2 – – – – 1 y 1 P5 15 9 – 9 – 2 – 2 y 3 P6 7 1 – 15 1 – – 3 – 4 P7 17 3 1 65 – 1 2 1 y 1 P8 1 4 – – – – – – – 6 P9 7 13 – 4 – 2 – 3 y 1 P10 2 3 – 3 1 – – 1 – 4 P11 1 3 1 9 – – 1 1 y 1 P12 13 3 – 8 – 1 – 2 y 2 P13 2 1 – 2 3 – – 1 – 7

Notes: - P1..13 = Participant 1 to 13 - Kw/Edit/Word/Brws/Help/Sgt/Prv/Tips: the numbers of Keywords/Edit/Word/

Browser/Help/Suggestion/Preview/Tips clicked - Mm: whether the concept-mapping tool (Mindman) was clicked - The sign – means the support tool was never intentionally used 5.4.3 Expected extension to other subjects The current version of the program supports two subjects: Biology and Geography. During the second assessment study, the teacher-designers of other subjects commonly mentioned that the program should be extended to support other subjects as well. In this section, some reasons for extension to other subjects will be discussed. The need of multimedia support for other subjects

Many teacher-designers mentioned that not only these two subjects, but also some other subjects could benefit from the support of multimedia. For example, one of the Physics teachers said that there are a lot of natural phenomena to be explained and experiments to be done in the subject of Physics. Some phenomena are hard to explain, and some experiments are expensive or not safe to do. In such cases, learners as well as teachers usually expect that computers can provide a virtual learning environment in which

Page 179: computer support for multimedia curriculum design

Assessing the practicality of the prototype 167

they can observe phenomena or do experiments safely. The History teacher also mentioned that in the subject of History, there are also a lot of historical events that can be better presented with multimedia. Some of the events can even be presented with games or simulations. In addition, other teachers also expressed that they wanted to use multimedia to improve or enhance their classroom teaching. When they found that only two subjects were supported by the program, they commonly expected that more subjects would be added so that they could also use it. Possibilities to support other subjects

Meanwhile, the prototype has the possibilities to support other subjects as well. For example, a Physics teacher said "All subjects have commonalities. I think this program is also suitable for the subject of physics. Of course, the curriculum standard and the supporting resources/materials should be changed to Physics." The Chemistry teacher said that "This program could be used for the subject of Chemistry, but something should be supported, such as experiments, equations and some special signs." In addition, the program had the following two advantages, which were also available for other subjects: The produced instructional scenarios were well structured and easy to

modify since the scenarios could be exported to Microsoft Word. The program could help teacher-designers save time and energy because

the program could automatically produce instructional scenarios after teacher-designers have entered content. Furthermore, the available supporting materials provided more convenience for teacher-designers to make instructional scenarios.

In addition, some participants also gave suggestions on how to extend the program to support other subjects during the workshop. The Chemistry teacher suggested that several versions of the program should be developed separately to support various subjects. In this case, each subject version could closely integrate specific characteristics of that subject and serve well for that subject. However, the History teacher suggested that it might also be possible to integrate several subjects in the same program. He said that "The current version supports two subjects of Biology and Geography, other subjects can also be added to it in a similar way." He thought that in this way teachers of one subject might learn something useful from other subjects as well.

Page 180: computer support for multimedia curriculum design

168 Chapter 5

Because of these potential advantages and possibilities, all of the seven teacher designers clearly expressed that they would be willing to use the program in the future if they would develop instructional scenarios. In addition, two CBL designers and one software developer mentioned that they would be glad to use it as well. In conclusion, the prototype seems to be helpful for other subjects, and the prototype is also feasible to be extended to other subjects. 5.4.4 Other comments and suggestions The CBL designers and software developers were not intended or potential target users, but they were a group of related people who were interested in such a computer support program. During this workshop, they proposed many comments and suggestions from their particular points of view. In this section, these comments and suggestions will be summarized. CBL designers

One CBL designer mentioned that one of the most significant advantages of the program was that it could help teacher-designers learn what a systematic approach to multimedia instructional design is. He said that although some teachers had experience in using multimedia software in their classroom teaching, and even some of them had developed presentation slides, most of them did not know how to systematically design and develop a multimedia instructional program. Those teachers who had experience in simple software development mostly developed instructional software based on their own experience, not based on a systematic approach. He said that the program provided a systematic approach to multimedia instructional design, including detailed analysis and design. He believed that this approach would be helpful for those teachers who want to develop multimedia instructional programs because they can learn some information from the program. Related discussion can be found in Section 6.2.5. Another CBL designer made three comments on the program. First, he thought that the Chinese name of the program (a Computer Support System for Multimedia Curriculum Design) was too complex to understand. He suggested making it simple, such as 'a Computer Support System for Scenario Development'. Second, he thought that the program would be useful for more complex multimedia projects, but not useful for development of simple instructional software. He said that, in practice, teachers used to make instructional software or presentation packages by themselves. They did not need to make instructional scenarios in advance because usually the

Page 181: computer support for multimedia curriculum design

Assessing the practicality of the prototype 169

instructional software was simple or small. This opinion was similar to some teacher-designers in the first study with primary target group users, but not consistent with the earlier CBL designer. More discussions on this issue can be found in Section 6.2.5. Third, he thought that the most important thing for teachers was to provide them with some teaching or lesson materials --not a support system for scenario development. He suggested that the materials part of the program should be enhanced. More discussion on this issue will be provided in Section 6.2.2. Software developers

One software developer suggested that the concepts of workspace and project could be introduced into the program to manage the various presentation forms of multimedia. He said in some programming tools, such as Visual C++, a workspace can include any number of projects, which can be of the same or different categories. Projects of different categories can have corresponding development platforms. In addition, any project can be included in one or many workspaces. Although the concepts of workspace and project were not used in the program, some ideas in the program were similar to those concepts. For instance, an instructional scenario can be seen as a workspace, and a presentation form of multimedia can be treated as a project in the program. An instructional scenario can involve many presentation forms, and one presentation form can be included in many instructional scenarios as well, since most presentation forms can be stored in separate files. Another software developer suggested that the concept of collaboration should be integrated into the program. He said that with the development of computer networks, collaborative work at different sites is becoming feasible, and a lot of computer systems, such as Lotus Notes, support collaborative work. Teacher designers and computer programmers may work at different sites even within one multimedia project group. In this way collaborative work becomes possible. If the program can support collaborative work, teacher designers and computer programmers can work together remotely and effectively. However, since the current version of the program is a CD-ROM based application, this good suggestion is hard to implement. It might be interesting to support collaboration in the future when the program is a web-based application.

Page 182: computer support for multimedia curriculum design

170 Chapter 5

The third software developer said that the program should be error free, otherwise users may be embarrassed or dislike the program when they encounter errors. Indeed, errors should be removed from the program, and the errors would be not very hard to remove. 5.5 Conclusions These two studies aimed at assessing the practicality of the program. At the beginning of this chapter, some sub-questions were listed that were to be answered via these two studies. Next, the conclusions drawn from these two studies regarding to these sub-questions will be given, and some issues and/or discussion points will be covered in the next chapter. The four components of the program were perceived to be rather practical by both primary target group users and other users Although some specific questions --in particular related to the component of interface in the second assessment study-- got low scores, the mean score of answers to the Likert scale questions covering each component was rather positive. Therefore, the four components of the program were perceived to be practical by both primary target group users and other users. No significant differences were found between novice and experienced designers Although minor differences were found between specific designers, no statistically significant differences were found between the novice and the experienced designers on the perceived practicality of the four components. Also, no obvious differences were found between them with regard to the walking routes, time spent on each screen and support tools utilized. The program was expected to be able to support other subjects The teachers of other subjects in the second assessment study commonly mentioned that too few subjects were supported by the program, and expected that more subjects should be added. They all expressed that they would be willing to use the program in the future when they designed instructional scenarios. It seems to be interesting to extend the program to support other subjects.

Page 183: computer support for multimedia curriculum design

171

Chapter 6 Discussion

T

his chapter discusses the main findings of the CASCADE-MUCH study. It starts with an introduction in Section 6.1. Section 6.2 discusses the main findings of the study regarding the four components (content, support, interface and

scenario), and the functionality for various user contexts. Section 6.3 reflects on the development research approach utilized in this study. Section 6.4 gives some recommendations for further research in line of this study as well as for other related research. Finally, this chapter ends with some closing remarks in Section 6.5. 6.1 Introduction The CASCADE-MUCH study aims at developing a valid and practical computer program that supports the development of a multimedia curriculum in Shanghai. It is very often the teacher-designers' task to develop instructional scenarios for multimedia curricula in Shanghai. However, in reality teacher-designers usually do not have sufficient knowledge and experience in multimedia curriculum development. As a consequence, they often have difficulties in developing instructional scenarios, or the instructional scenarios they develop do not fully exploit the advantages of computer-based learning (CBL). The main aim of the program, that is central in the study, is to help teacher-designers (both novice and experienced designers) develop usable instructional scenarios, which can guide computer programmers in constructing the actual multimedia curricula. Also, it is expected that the instructional scenarios can stimulate discussions between teacher-designers and computer programmers. This study follows a development research approach as presented in Section 1.3.3. By following this approach, two main stages have been carried out: a prototyping stage and an assessment stage. During the prototyping stage, the main research question was:

Page 184: computer support for multimedia curriculum design

172 Chapter 6

What characteristics should a valid (and potentially practical) computer support system for multimedia curriculum design have in the context of Shanghai?

Four rounds of prototyping were successively carried out, focusing on the four components of the support program: content, support, interface, and scenario (see Chapter 3). At the end of the prototyping stage, an expert appraisal workshop was held at East China Normal University (ECNU) in Shanghai to judge its validity. The results showed that the prototype was rather valid and had potential to be practical for the intended target users. The full answer to this research question has been given in Chapter 4. During the assessment study, the main research question was:

To what extent is the CASCADE-MUCH program practical for both primary target group users and other users in the context of Shanghai?

Two studies were held in Shanghai again to assess its practicality. The first study focused on primary target group users, and the second study focused on other users. The results indicated that the four components of the program were perceived to be practical, and it was expected that the program would be extended to other subjects as well. The results of the assessment studies and the answer to this research question have been given in detail in Chapter 5. Based on the results of the prototyping and assessment stages (described in Chapter 3, 4, and 5), this chapter will discuss the main findings and the development research approach utilized in this study. 6.2 Discussion of the main findings The main findings of the study will be discussed on the following four components: content, support, interface, and scenario. Also, the functionality for various user contexts will be discussed in this section. 6.2.1 Content Content is a key component of the program. Based on the expert appraisal workshop at the ECNU and the assessment studies in Shanghai, it was decided that the content of the program is valid and rather practical. However, during the prototyping stage, many issues related to the content of

Page 185: computer support for multimedia curriculum design

Discussion 173

the program arose, were discussed and needed to be revised, such as the goals of a multimedia curriculum and the learning theories followed when designing a multimedia curriculum. In this section, these two issues will be discussed based on two questions: i) How were these two issues settled? and ii) What advantages and disadvantages do these settlements have? Goals of a multimedia curriculum

At the start of this study, it was preliminarily decided that the goal of a multimedia curriculum is to meet the needs of quality-driven education. However, in practice the shifting process from test-driven education to quality-driven education was gradually taking place in Shanghai. Therefore, it was decided that a multimedia curriculum should also take the current (test-driven) education into account. As a consequence, the goal of a multimedia curriculum was classified to meet the needs of quality-driven education and/or test-driven education in the first prototype (see Section 3.3.1). During the formative evaluation of the first round of prototyping, it was found that subject teachers in Shanghai met with some problems with this classification of goals. First, they did not like the label test-driven education. They thought that this goal deviated from the current trend of educational reform. They mentioned that the government, educational administrators, as well as public newspapers were talking about quality-driven education. According to them, designing a multimedia curriculum in order to meet the needs of test-driven education seems not to be suitable. Second, although the term quality-driven education has already been frequently used nowadays, there is still a lack of consensus on its definition. Some educators even doubt whether it is a proper term, because it sounds like the aim of education in the past years was only for passing tests, not for improving the quality of learners at all (cf. You, 1998). Also, they argue that even in a so-called quality-driven education context, testing is still necessary. In the search for an alternative, and after considering the practical situation in Shanghai, several goal divisions proposed by various authors such as Bloom, et al. (1956); Gagné, et al. (1988); Merrill (1983); and Posner and Rudnitski (1986) were compared. Based on this comparison, the goals of a multimedia curriculum have been revised to: i) basic knowledge and skills; ii) extended knowledge and higher-level skills; and iii) attitudes (see Section 4.2.1). Although this division of goals may have the disadvantage that it is based on 'old' sources, it is consistent with the plan of the second round of curriculum innovation in Shanghai. According to this plan, the new curricula in Shanghai

Page 186: computer support for multimedia curriculum design

174 Chapter 6

will include three modules: basic courses, extended courses, and research courses (see Section 1.1.2). The first goal (basic knowledge and skills) can be achieved by the basic courses; the second goal (extended knowledge and higher level skills) can be achieved by the extended and research courses; and the third goal (attitudes) may be gradually achieved when learners are involved in practical work or hands-on experience on some basic or extended knowledge units included in these courses. In addition, this division of goals can also be used to explain quality-driven education and test-driven education. Generally speaking, test-driven education puts more emphasis on knowledge acquisition; whereas quality-driven education places more emphasis on the full development of these three goals, particularly on the development of higher-level skills. Learning theories

Since the goals of a multimedia curriculum were initially determined to meet the needs of test-driven education and quality-driven education, objectivism and constructivism -- two extreme views on the world-- were considered to fit these two perspectives on learning. On the one side, as explained in Section 3.3.1, objectivism seems to be rather consistent with the rationale of test-driven education, because the objectivistic learning theory focuses on knowledge transfer from instructors to learners, and on assessment to check the degree to which learners are able to recall the knowledge. On the other side, constructivism has more in common with the rationale of quality-driven education, because the constructivistic learning theory highlights the knowledge construction process of learners (Jonassen, 1991a, 1991b). The formative evaluation results showed that subject teachers were not interested in these two learning theories. On the one hand, they were not fully familiar with these two theories. On the other hand, most subject teachers appeared to hold views somewhere between these two extremes (cf. Molenda, 1991). Therefore, it was decided that the learning theory followed when designing a multimedia curriculum using this program should be more neutral. It was expected that a neutral learning theory would have the advantage that it would fit most teacher-designers' needs. A possible disadvantage is that those teacher-designers who do have strong beliefs of objectivism or constructivism might feel the support program is less helpful.

Page 187: computer support for multimedia curriculum design

Discussion 175

6.2.2 Support The CASCADE-MUCH program is an Electronic Performance Support System (EPSS), in which many different types of support are integrated, such as explanations, examples, suggestions, and wizards. The expert appraisal workshop at the ECNU and the assessment studies showed that the component of support is valid and practical. Related to the support of the program, there are at least two issues that were frequently discussed and finally settled: the support of exemplary lesson materials and the usefulness of wizards in the program. These two issues will be discussed in this section. Exemplary lesson materials

During almost each formative evaluation activity, the subject teachers mentioned that more exemplary lesson materials should be added to the prototype. The lesson materials refer to learning or teaching materials that are used to demonstrate or explain the knowledge units in their subjects. The demand of the teachers can be understood when taking into account that various teachers may prefer various teaching methods or styles in practice. They usually do not use a complete instructional program for one entire lesson. Alternatively, they usually prefer to adapt pieces of lesson materials to support parts of their teaching processes. However, they often do not have sufficient computer skills to develop such lesson materials. Therefore, exemplary lesson materials that are suitable for their lessons would be welcome. In order to answer this call for more examples, during each round of prototyping, several exemplary lesson materials for the two subjects of Biology and Geography were added. From the third prototype on, some web sites --including related lesson materials on the Internet-- have been linked to the prototype, and easy access to those web sites was provided. The reason for requiring more lesson materials might be the fact that not every computer in Shanghai has access to the Internet. Teachers were not able to access a lot of available materials on the Internet during the formative evaluation. With the rapid growth of computer networking and the Internet, this problem is expected to be naturally solved in the near future.

Page 188: computer support for multimedia curriculum design

176 Chapter 6

Two solutions are considered to be feasible for adding more lesson materials to the prototype. The first solution is to collect some existing examples of multimedia lesson materials which cannot be found on the Internet, and to save them together with the prototype. The second solution is that the prototype can include a list of recommended links to related web sites in anticipations of the day when each computer has fast access to the Internet. The list can be initiated by the designer in cooperation with other experts, and can be expanded or reorganized by users themselves in practice. An advantage of providing such a recommended list is that users can directly follow these links to access those materials, and do not need to search related materials in cyberspace. A possible disadvantage is that users might depend on the recommended list too much in their search for lesson materials, and might not want to try any others. However, no matter with which solution, the lesson materials or web sites should be organized in a clear and structured way --for instance by topics-- so that users can easily find what they need. Wizard

A wizard usually helps users carry out a complex task by following a set of predetermined steps. Also, users can learn how to perform the task for themselves during this process. In the second prototype, some wizards were added to the prototype to help teacher-designers learn how to carry out the tasks of content selection, representation, organization and interface design (see Section 3.4.2). Each time when users started to perform one of these tasks, the prototype would ask them whether they needed a wizard. If they answered yes, the prototype would popup a window to guide them through the steps of the task. It was assumed that such wizards would be helpful assistants for the users. However, the formative evaluation results showed that this assumption was not always true. During the formative evaluation activities, it was found that the wizards were not frequently used by the participants. Some users even mentioned that they basically disliked using wizards because they thought that wizards are usually boring, and they preferred trial-and-error. In addition, they thought that it was not necessary to use wizards in the prototype because most tasks were not complex enough. Even for complex tasks, wizards should also remain optional, because the tasks may be challenging for some users, but not complex at all for others. Therefore, most wizards were removed or kept optional since the third prototype. Only when users click on the button to request a wizard does the wizard window pop up.

Page 189: computer support for multimedia curriculum design

Discussion 177

Another option to complete a complex task is to use templates instead of wizards. Compared to wizards, templates first show tentative final products (or tasks) with some default content. Users can adapt the tentative products by replacing the default content with their own information. The advantage of templates is that users can get an understanding of what the final products look like from the beginning, and they can work on the content of the templates in any sequence they like. The disadvantage is that templates might not be suitable in some situations. For instance, when the task is not to make a visible product, but to perform a procedure such as connecting to the Internet. 6.2.3 Interface From the first prototype, interface design has drawn much attention. The expert appraisal workshop at the ECNU and the final assessment studies showed that the component of interface was valid and practical. Although interface design is more an art than a science (cf. Galitz, 1993), some basic principles are still useful for interface design. In the literature, many principles for interface design can be easily found, such as consistency, satisfaction, attractiveness, metaphor, and efficiency, etc. (Brown, 1988; Nielsen, 1993). However, during the development process of the program, it was found that the following two principles seemed to be more comprehensive and important for interface design: easy to learn and easy to use. In this section, these two principles will be discussed. Easy to learn

The principle 'easy to learn' is particularly useful for novice users of a program, because they need to learn how to use the program at the beginning. In order to be easy to learn, it seems to be necessary for a computer program to have the following characteristics (cf. Brown, 1988; Nielsen, 1993). First, it should be consistent. On the one hand, it should be externally consistent with other frequently used computer applications such as Microsoft Office. In this case, novice users of the program who already have experience using other applications would not feel completely disoriented when they first see or use it. For instance, in order to be externally consistent, the CASCADE-MUCH program divides the interface into several functional areas; and a main menu is on the top of the interface. On the other hand, the program should be internally consistent. Each screen should have the same appearance; and the same buttons (or other screen elements) should be in the

Page 190: computer support for multimedia curriculum design

178 Chapter 6

same position on different screens. In this case, after users have experience in using the first screen, they will easily get familiar with other screens in the same program. For instance, in order to be internally consistent, all screens in the CASCADE-MUCH program use the same browser buttons and toolbar; and the same information is in the same position on different screens, etc. Second, it should be intuitive or visual. Some metaphors should be used in a program to help users quickly understand the functionality of buttons or screens. For instance, the CASCADE-MUCH program has a logo on each screen to indicate what the screen is about; each speed button has an icon to show the functionality of the button; and the used items on the browser were marked with an icon of a foot. Third, in order to increase the ease of learning, the program should have a proper amount of information on each screen. Overload of information on screens may make it hard for users to remember and/or to learn. For instance, the CASCADE-MUCH program shows brief information on each screen, and the explanations of keywords and related models are all placed in the Windows help file and will be shown only when users request them. Easy to use

For experienced users, 'easy to use' often becomes more important than 'easy to learn' since they do not need to learn it any more. In order to be easy to use, it seems to be important for a computer program to have the following two characteristics (cf. Brown, 1988; Nielsen, 1993). First, it should be flexible. Users should be allowed to walk through the program in a way (linear or non-linear) they like; operations that are frequently used should have short-cut keys; and the fonts, colors, and other screen elements should be allowed to be personalized, etc. For instance, the CASCADE-MUCH program provides two navigation tools: linear and browser. Users can use both or either one as they want to walk through the program; and most operations can be done using either the mouse or the keyboard. Second, it should be error free. Too many errors will inevitably make users dislike the program even though it has many positive characteristics. In order to become more error free, a computer program should undergo pilot testing to find and remove as many errors as possible before it is distributed.

Page 191: computer support for multimedia curriculum design

Discussion 179

6.2.4 Scenario During the assessment studies, each participant developed a preliminary instructional scenario, and perceived it to be rather valid and practical. However, during these studies it was not verified by computer programmers to what extent the developed instructional scenario was valid and practical. Nevertheless, during the micro evaluation activity in the third round of prototyping, four computer programmers did review similar instructional scenarios and made many comments. During that micro evaluation process as well as the final assessment studies, two issues arose and were often discussed, i.e. need for a flowchart, and feasibility to produce a multimedia curriculum as the final outcome of the program. In this section, these two issues will be discussed in more detail. Flowchart

A flowchart often helps computer programmers solve a complex programming problem by specifying what should be done in each step, and what will follow it. Although a flowchart may have some drawbacks in handling iteration and selection (Blacklock, 1993; Downing, Covington & Covington, 1998), the results of the micro evaluation in the third round of prototyping showed that computer programmers still expected the type of specific information often included in a flowchart. A practical problem is to determine who is responsible for developing a flowchart when taking into account that teacher-designers usually do not have enough computer skills to be able to develop it. In addition, it may also be hard for computer programmers to develop a flowchart without help of teacher-designers, because the computer programmers may have difficulties in understanding the instructional scenarios. A possible solution is that computer programmers and teacher-designers work together to make a flowchart. The computer programmers can firstly make a tentative flowchart based on the description in the instructional scenario, and then discuss with the teacher-designers (perhaps through several rounds) to revise and finalize it. In the end, the computer programmers can make a multimedia program based on the flowchart and other content in the instructional scenario. Alternatively, another solution is that computer programmers can translate the instructional scenario into a (paper-based) prototype rather than a flowchart. Since the instructional scenario includes an interface template and descriptions of screen elements (such as buttons, keywords in context, etc.), computer

Page 192: computer support for multimedia curriculum design

180 Chapter 6

programmers can develop a concrete prototype based on this information by working together with the teacher-designers. On the prototype, they can specify what fonts and colors will be used, and what screen will be jumped to after clicking a button or hotspot. The main advantages of developing a prototype are that a prototype looks more concrete and intuitive than a flowchart; and consequently teacher-designers can also more easily understand it. The disadvantage of developing a prototype is that a prototype usually has more limitations on illustrating the inner variables, algorithms and control structures of the program than has a flowchart. Final outcome

The final outcome of the program is an instructional scenario. During the entire development process of a multimedia curriculum, the instructional scenario is an intermediate product because it still needs computer programmers who translate it into a multimedia curriculum. During the final assessment studies, some participants requested the ability to directly design a multimedia curriculum without this intermediate product. The following difficulties and potentials are related to this request. Multimedia curriculum development is a complex task. It usually starts with a preliminary research, and is followed by analysis, design, construction, formative evaluation and summative evaluation activities as shown in Figure 2.2. Also, it usually needs several types of experts to work together, such as subject matter experts and curriculum or instructional design experts as shown in Figure 1.3. It is nearly impossible for subject teachers to develop a multimedia curriculum without the help of others, since they usually do not have sufficient knowledge or skills. Even when it would be possible for teacher-designers to directly create a multimedia curriculum without the help of others, the support program they would use might become very complex because it should include support for every aspect of multimedia curriculum development. Nevertheless, to some extent it would be possible to directly produce a multimedia curriculum by adding a translating tool to the program as shown in Figure 6.1. The current version of the program produces instructional scenarios, which will be used by computer programmers to create multimedia curricula. This process is shown as ➀ in this figure. A translating tool can (theoretically) be created to translate the instructional scenarios into multimedia curricula automatically. This process is shown as ➁ in the figure. Although this is an interesting alternative, it can be imagined that the

Page 193: computer support for multimedia curriculum design

Discussion 181

translation tool would be hard to create since the multimedia curricula as well as the instructional scenarios are complex products. However, some authoring systems have such support characteristics, such as WinPLATO (Preese & Foshay, 1999) and SIMQUEST (de Jong, Limbach, Gellevij, Kuyper, Pieters & van Joolingen, 1999). Another possibility is that the translating tool translates the instructional scenarios into a higher-level programming language (such as Delphi or Visual Basic). After that, computer programmers finalize the multimedia curricula by adjusting the source codes. This process would look similar to designing a macro in Microsoft Word, in which a macro is recorded in Visual Basic language. Another example is that Designer's Edge can output designed courses to authoring system QUEST, or directly export designed courses to HTML or Java templates for online delivery. The advantage of producing a multimedia curriculum automatically is that the process of developing a multimedia curriculum becomes simpler and more streamlined since no computer programmers are needed. However, the disadvantage is that once a multimedia curriculum has been produced, the multimedia curriculum --especially the control structure of it-- might be hard to adjust. The advantage of translating an instructional scenario into a programming language is that the multimedia curriculum can still be adapted by computer programmers before or even after it has been produced. However, the disadvantage is that the source codes automatically produced might be hard for computer programmers to understand and to modify.

Translating tool

Multimedia curricula

Computer programmers

Instructional scenarios

CASCADE - MUCH

Teacher-designers

Figure 6.1: Possible extension of the program

6.2.5 Functionality for various user contexts During the assessment stage, an important question was whether the program might be useful for other user contexts. In this section, some representative opinions and results related to this question are discussed, including: i) various versions and extension to other subjects; and ii) differences between novice and experienced designers.

Page 194: computer support for multimedia curriculum design

182 Chapter 6

Various versions and extension to other subjects

During the two assessment studies, some participants suggested that the program should have several versions such as a compact version and an extended version so that both novice and experienced users of the program could use it at top efficiency (see Section 5.3.3 & 5.4.3). Although the suggestion was valued, the question remained how to simplify the program. Should the simplified program only include the basic function without theoretical models, suggestions and previews? If so, the simplified version would be easier to use, but might not be very useful for novice designers to learn from it. It seems that version division should not be done simply based on the amount of information to be included because it is hard to judge how much information a compact or an extended version should have. However, it is possible to break the program into several versions based on the following two criteria: language and subject. The current version of the program supports two languages Chinese and English. Clicking the translation button on the toolbar will immediately change the language of the entire program. The practical situation is however that one language is usually enough for individual teacher-designers. Splitting the program into two versions based on language will have the following advantages. First, each version will become much easier to manage. Two languages in one program sometimes may lead to some problems. For example, one line in Chinese on a screen may become two lines or even more in English. Second, it can make the program file size smaller. Although the file size is not a critical element, the update of a big file may take a long time to download through the Internet. The program can be divided into a Biology version and a Geography version based on the criteria of subject. The Biology version will particularly support Biology teacher-designers by providing Biology-related lesson materials, and with more considerations of the characteristics of Biology. Similarly, the Geography version will specifically support Geography teacher-designers. Other subject versions can also be added in the future. The main advantages of splitting the program into several versions based on subject are that each version can: i) better take the subject characteristics into account; and ii) reduce the amount of supporting materials of another subject and make the program file size smaller. The disadvantage is that some common parts may be replicated in both versions.

Page 195: computer support for multimedia curriculum design

Discussion 183

In addition, the program has potentials to extend to other subjects. One solution to support other subjects is that all subjects be integrated into the same program similar to the current version. Another solution is that some other related subjects --such as natural or social science subjects-- be integrated into one version. Such a version may have the advantages of combining the commonalities of related subjects and reduce the file size; however, the main disadvantage is still that the subject-specific support, such as the exemplary lesson materials of other subjects, is not relevant and useful for a subject teacher of another subject. The third solution is that each subject have a separate version as discussed above. Differences between novice and experienced designers

It was assumed that novice and experienced designers might use the program differently, particularly with respect to their walking routes and utilized support tools. However, the data collected during the assessment studies indicated that no significant differences existed between the usage of the novice and experienced designers (see Chapter 5). One possible reason for this result might be that they were all novice users of this particular program, although they had various experience in multimedia curriculum design. It is assumed that with an increase in using the program, the experience of the novice users will accumulate, and they will become experienced users gradually. However, these experienced users of the program may still have limited design experience because learning to use the program is usually easier than learning to design a multimedia curriculum. Nevertheless, some users may be experienced designers because they have designed multimedia curricula before and gained design experience. It is still assumed that experienced users of the program with various design experiences will use the program differently. For instance, novice designers may utilize the help and suggestions in Designer's Aid to learn how to design a multimedia curriculum; experienced designers may directly use the Edit Panel part to create or edit an instructional scenario. Since no users had experience of using the program, the assumed differences were not verified during the assessment studies.

Page 196: computer support for multimedia curriculum design

184 Chapter 6

6.3 Discussion of the development research approach This study followed a development research approach. As introduced in Section 1.3.3, by following the development research approach, this study aimed at producing a valid and practical computer support program (product improvement) and generating methodological guidelines (knowledge growth) for such studies. In general, prototyping and formative evaluation are often applied during the product improvement process, and the knowledge is described in the form of design principles (van den Akker, 1999). In this section, the prototyping and formative evaluation activities applied in this study will be discussed, and the design principles generated from the study will be presented and discussed as well. 6.3.1 Prototyping The prototyping approach can enable the process of discovering specifications and their adequacy (Nieveen, 1999). During the development process of the study, some benefits have been gained from the prototyping approach such as gradually clarifying the design specifications, successively approximating an optimal solution, and promoting communication between the designer and the users (see Section 1.3.4). However, some limitations or difficulties of applying the prototyping approach were also experienced. In this section, some points of attention regarding the limitations, and an often-neglected type of prototype --a paper-based prototype-- will be discussed. Points of attention

The first point is that conducting a prototyping approach within a development research study seems to be quite time consuming. This study evolved through four rounds of prototyping, and each round took about six months to one year. This duration of time seems to be hard to reduce because the activities are performed in the framework of a development research process, in which much time is devoted to carrying out analysis activities, design activities, planning and performing a formative evaluation, and reporting the formative evaluation results. The second point is that conducting a development research study with use of the prototyping approach in a different context is rather complex (cf. Thijs, 1999). This study was mostly conducted at the UT, the Netherlands; and the final program will be used in another context: Shanghai, China. Therefore, the communication between the designer of the program and the target groups

Page 197: computer support for multimedia curriculum design

Discussion 185

became a challenge. The formative evaluation activities during each round of prototyping needed much communication between the designer and the target groups. However, because of the distance between them, in both kilometers and time zones, the communication was hard to implement. In addition, comments and suggestions were mostly collected during the formative evaluation activities, and could not be easily collected at other times. Nevertheless, with the rapid growth of ICT, this problem might be gradually solved. The third point is related to language, which was also caused by the problem of carrying out the study in different contexts. The program supports two languages: English and Chinese. Comments and suggestions were mostly given to the Chinese version because most formative evaluation activities were carried out in Shanghai. However, some comments and suggestions were given to the English version, for instance during the expert appraisal workshop at the UT. Consequently, something in one version had been changed but might not be changed in another version. In this case, inconsistency between these two language versions often developed. In conclusion, if participants of all formative evaluation are from one context using one language, this problem can be effectively overcome by developing the program in one language and translating it into another, for instance, at the end of the study. However, if the participants often come from different contexts, this problem seems to be hard to solve. Paper-based prototype

According to Rettig (1994), a paper-based prototype is a 'wonderfully simple and effective tool ' (p. 22); and it is usually made at the beginning of a study, focusing more on content and overall structure rather than on interface design. However, in this study a paper-based prototype was not directly developed at the beginning but was used at the end of the first round of prototyping. This procedure turned out to be useful as well (see Section 3.3.4). The lessons learned from this study are summarized in this section. A paper-based prototype does not have to be developed at the very beginning of a study. If a study focuses on content validation rather than on interface design, then it can start with a paper-based prototype, otherwise it can start directly with a computer-based prototype. According to Moonen (1996), at the beginning of a study, a prototype usually puts more emphasis on interface design rather than on content specification. Since much effort was put into the component of interface design in the beginning of this study, this study

Page 198: computer support for multimedia curriculum design

186 Chapter 6

started with the design of a computer-based prototype. In addition, a paper-based prototype does not have to be directly drawn on paper or drawn in Microsoft Word. Nowadays a lot of visual software developing platforms are available, such as Visual Basic, Visual C++ and Delphi, which are excellent tools for designing screens and paper-based prototypes. Developing a paper-based prototype with these platforms will have at least two benefits. First, the screens on the paper-based prototype will look more true to life because they are made within the same platforms which will create computer-based prototypes later on. Second, the screens made within these platforms can also be used directly in later computer-based prototypes. Therefore, it is preferable to make paper-based prototypes (if necessary) by using one of these visual developing platforms. 6.3.2 Formative evaluation Formative evaluation holds a prominent place in development research, because it can 'provide the information that feeds the cyclic learning process of developers during the subsequent loops of a design and development trajectory ' (van den Akker, 1999, p. 10). During this study, it was found that user participation and revision decisions held important roles in a formative evaluation activity. This section reflects on these two issues. User participation

In the processes of the prototyping design and the final assessment studies, several users participated. It was found that user participation is very useful and very important for a development research study, because it has at least the following three advantages (cf. Monk, Wright, Haber & Davenport, 1993; Nieveen, 1999). First, user participation can help designers clarify design specifications gradually. Since a system is designed for specific users, their needs and expectations should be taken into account and well integrated into the system during the prototyping process. However, in the beginning of a development research study, the needs and expectations are usually not very clear. The user participation in the formative evaluation activities can help designers gradually approximate an optimal design specification.

Page 199: computer support for multimedia curriculum design

Discussion 187

Second, user participation may effectively prevent a system from being designed for its designers or experts. During the prototyping design of a system, designers have some initial ideas or assumptions, related experts may also advocate some theoretical models or guidelines. But the initial ideas, models or guidelines may not always work well in practice because of the often-existing gap between theory and practice. User participation can avoid that the designer's specifications deviate too far away from the users' needs. Third, user participation helps to gain commitment from target users. According to Tessmer (1993), with the rapid growth of constructivist learning theory, 'formative evaluation with learners (users) becomes a way for the learners to help design the content and curriculum itself ' (p. 19). Although the program is not designed completely based on the constructivistic learning theory, the idea of user participation is consistent with it. In addition, the following aspects related to sampling need to be considered when selecting proper users. First, in view of triangulation (Miles & Huberman, 1994), representative users with different characteristics should be selected. Second, the selected users should be those who are interested in the prototype or highly motivated, otherwise they might be inert or inactive, and consequently fewer comments and suggestions would be collected. Third, critical friends should be invited, because they usually can make some frank or critical comments without causing the designer to lose face. In conclusion, user participation is useful for a formative evaluation; and users should include representative, highly motivated, and critical persons. Revision decisions

At the end of each prototyping cycle, some revision decisions were made based on formative evaluation results. It was found that revision decisions hold an important position in a formative evaluation activity as well as in the development research approach. In general, comments and suggestions are collected during a formative evaluation process, and revision decisions are made based on the collected comments, suggestions, and the designers' ideas. This is oftentimes an uphill battle, because some comments or suggestions may be not consistent, or even contradictory with each other or with the designers' ideas. Meanwhile, it is also a process of knowledge growth for the designers, because some comments and suggestions can also be integrated into the designers' knowledge base and become their new knowledge.

Page 200: computer support for multimedia curriculum design

188 Chapter 6

In addition, revision decisions usually function as a bridge between two neighboring prototypes. They are usually made at the end of a formative evaluation process, and become a starting point of the next round of prototyping. Through this continuously cyclic process connected with revision decisions, an optimal product can be gradually approximated. In conclusion, revision decisions are necessary and helpful in a development research process since a prototype can be continually improved in the next round of prototyping based on the revision decisions, and designers can also improve their own professional knowledge. 6.3.3 Design principles Van den Akker (1999) suggests that knowledge gained from development research can be presented in the form of 'design principles', which are usually heuristic statements of a format such as; 'If you want to design intervention X [for the purpose/function Y in context Z] then you are best advised to give that intervention the characteristics A, B, and C [substantive emphasis] and to do that via procedures K, L, and M [procedural emphasis], because of arguments P, Q, and R' (p. 9). This format of design principles is compact and concise in itself, and also very helpful for designers to present their gained knowledge in a structured way. However, during this study it was found that generating such design principles is not an easy task. It is hard to strike a balance between being concrete and being abstract. To be useful, it seems desirable to give very concrete descriptions of the characteristics, procedures, and arguments in the design principles. However, it is hard to integrate that much explicit information in one principle along that format. Nevertheless, it has been attempted to generate an overall design principle (shown on the next page) based on the results of the entire study. The design principle is still rather abstract; more concrete information remains in Chapter 3 and 4. The arguments are given to both the characteristics and procedures.

Page 201: computer support for multimedia curriculum design

Discussion 189

If you want to design a (valid and practical) computer support system forthe purpose of developing instructional scenarios for multimediacurriculum design (in the context of Shanghai), you are best advised to givethe support system the following characteristics: It should include content that is derived from both theory and practice. It should integrate many categories of support such as information,

tools, advice, and training. Its interface should be consistent, flexible, and visual. Its outcome --instructional scenarios-- should include much information

needed by computer programmers and should be well structured. The arguments for giving the support system these characteristics are that: Usually there is a gap between theory and practice. The support can help users perform tasks efficiently. The system should be easy to learn for new users and easy to use for

experienced users. The scenario will stimulate discussion between computer programmers

and teacher-designers and help computer programmers construct amultimedia curriculum.

You can carry out the development of the support system by following adevelopment research approach and via the following procedure: Breaking down the support system into some key components such as

content, support, interface, and scenario. Focusing on analysis and design of specific components during each

round of prototyping. Conducting a formative evaluation and making revision decisions at the

end of each round of prototyping. Starting subsequent rounds of prototyping until a satisfying support

system has been reached. The arguments for following this procedure are that: Not much experience may be available and can be learned from since it

is a challenging and highly innovative study. This procedure can help you clarify the design specification and

approximate an optimal result gradually.

Page 202: computer support for multimedia curriculum design

190 Chapter 6

6.4 Recommendations In this section, recommendations for further research of this study as well as for other related studies are presented. The recommendations are given on the following three topics: i) web support; ii) a follow-up study; and iii) implementation of the support program. 6.4.1 Web support The current version of the program is a CD-ROM-based package, in which everything is included in a CD-ROM except for some linked support materials on the Internet. This version usually works well since it integrates everything in a single CD-ROM, and it can be used in any Windows computer with a CD-ROM optical drive. But some limitations exist as well. First, it has limited distribution. It can only be used by the people who have the CD-ROM. Second, this version is hard to update. Fortunately, with the development of the Internet, the update opportunities have become much easier nowadays. It is recommended that such a computer support system is preferable to have two media versions: CD-ROM and web. In addition to overcoming the limitations of the CD-ROM version mentioned above, a web-based version may have further advantages. First, designers can easily collect comments and suggestions from users by email or a Bulletin Board System (BBS). Second, it has potential to support collaborative work between users at different sites. For instance, users can share the same web pages or databases; and they may support their communication through chat programs, BBS or other tools. Third, it is easier to integrate links to other lesson materials on the Internet. Nevertheless, a web-based version might have some limitations as well. For instance, although the infrastructure of the Internet is growing fast, not every computer has access to the Internet now, especially in developing countries such as China. This fact may prohibit broad use of a web-based version of the program. In addition, low-speed access to the Internet may also make the web-based version program work slowly and therefore be less usable. If only one version for a computer program is possible, it is strongly recommended that this version should take advantage of the other version. For example, a CD-ROM version may have a supporting web site. New information can be published on the web site; discussion can be done in a BBS; and new updates of the program can be easily downloaded. Similarly, a web-based version may keep specific information stored in a local computer

Page 203: computer support for multimedia curriculum design

Discussion 191

after the first use. This might be particularly useful if the web-based version of the program is big or the Internet access is slow. Once this part has been downloaded to a local computer, it can be used indefinitely without the need of downloading it again. This download-once-forever method may help to overcome the limitation of low speed access to the Internet. 6.4.2 A follow-up study As mentioned in Section 1.3.4, product quality usually has three criteria: validity, practicality and effectiveness (Nieveen, 1997, 1999). In a development research process, the emphasis in criteria for quality usually shifts from validity, to practicality, to effectiveness (van den Akker, 1999). Mostly the first two criteria, validity and practicality, must be dealt with before the effectiveness of a product can be assessed (cf. Visser, 1998). In this study, after four rounds of prototyping, the program has turned out to be valid via the expert appraisal workshop at the ECNU (see Section 3.6.2). After that, the program has also been shown to be rather practical for both primary target group users and other users via the two assessment studies --a summative evaluation activity (see Figure 2.2)-- in Shanghai (see Chapter 5). However the effectiveness has not been assessed yet. No longer as a part of this study, but rather as a part of further research of this study, a follow-up study can be undertaken to further investigate the effectiveness of the program. As displayed in Figure 6.2, the follow-up study can be carried out in the following steps. First, the program will be used by teacher-designers for real multimedia curriculum development projects. These projects would preferably be in Biology or Geography in Shanghai. During this process, the actual use of the program when teacher-designers are producing ready-to-use instructional scenarios can be summarized and analyzed, and the following questions can be answered: Do teacher-designers need to consult with other experts such as

curriculum or instructional design experts? Do teacher-designers need other tools not included in the program to

facilitate the instruction scenario development? How do teacher-designers work together with computer programmers to

revise and finalize the instructional scenarios?

Page 204: computer support for multimedia curriculum design

192 Chapter 6

Teacher designers

Teachers/ learners

Computer programmers

Learning outcomes

Multimedia curricula

Instructional scenarios

CASCADE - MUCH

?

Figure 6.2: Assessment of the effectiveness of the program

Second, some computer programmers will create multimedia curricula based on the ready-to-use instructional scenarios. During this process, the study will attempt to answer the following questions: How do computer programmers really use the instructional scenarios? What difficulties do they have with understanding and utilizing the

instructional scenarios? What additional information in the instructional scenarios do they need?

Third, the created multimedia curricula will be distributed to subject teachers for classroom teaching and/or to learners for individual learning. The study will continuously investigate how the teachers and/or learners perceive and use the multimedia curricula. Finally, the effectiveness of the program will be assessed by comparing the learning outcomes with the goals of the multimedia curriculum specified in the instructional scenarios. 6.4.3 Implementation Although the primary target group users and the other users in the two assessment studies developed instructional scenarios with the CASCADE-MUCH program and they perceived it to be rather practical as well (see Chapter 5), the produced instructional scenarios were still very preliminary. The program has not been really used by any teacher-designers to develop an authentic instructional scenario in practice. Although some computer programmers created multimedia programs based on the preliminary instructional scenarios in the second round of prototyping, no computer programmers have used any ready-to-use instructional scenarios to create

Page 205: computer support for multimedia curriculum design

Discussion 193

multimedia curricula. This section gives some recommendations on how to develop and use an instructional scenario efficiently in practice. As displayed in Figure 1.4, developing and using an instructional scenario with CASCADE-MUCH is not a linear process. After being familiar with the program (based on the Tutorial part or a short introductory workshop), teacher-designers will first use the program to make a tentative instructional scenario. Meanwhile, they may need to consult with other experts in order to increase the validity and practicality of the instructional scenario. When a tentative instructional scenario is ready, the teacher-designers, experts and even computer programmers may need to discuss it together. Based on the comments and suggestions, the teacher-designers revise the preliminary instructional scenario by using the program again and produce a revised instructional scenario. This process might take several rounds until a ready-to-use instructional scenario has been developed. During this process the teacher-designers may particularly need to work and discuss together with computer programmers in order to finalize the interface and create a flowchart (if necessary). After the ready-to-use instructional scenario has been developed, the computer programmers will create a multimedia curriculum based on the instructional scenario. Before the programming process starts, it is suggested that the teacher-designers present an explanation to the computer programmers about the structure and content of the instructional scenario. Also, during the programming process, the computer programmers may still need to discuss several issues with the teacher-designers. From this point of view, teacher-designers, computer programmers and curriculum design experts may need to work together at any time during the process of multimedia curriculum development. Therefore, an instructional scenario is not only a medium conveying information from teacher-designers to computer programmers, but also an important tool which helps to make the discussions between computer programmers and teacher-designers much easier and more effective. 6.5 Closing remarks The CASCADE-MUCH program was initiated to support teacher-designers in Shanghai to produce an instructional scenario for multimedia curriculum development. The instructional scenario --the final outcome of the support

Page 206: computer support for multimedia curriculum design

194 Chapter 6

program-- will be used by computer programmers to create a multimedia curriculum. This study shows that the CASCADE-MUCH program can help teacher-designers immediately develop preliminary instructional scenarios. In a long term, it can stimulate and facilitate the discussion between teacher-designers and computer programmers; and it can also improve professional knowledge of teacher-designers who have less experience in multimedia curriculum design (see also 1.3.2). Although nowadays not many curricula have been presented with multimedia by using the computer, and most subject teachers are developing simple instructional software packages for instance by using PowerPoint or Authorware, more and more curricula in Shanghai will be presented on the computer with the rapid development of ICT. This tendency can be found in the plan of the second round of curriculum innovation in Shanghai (cf. Zhang, 1999). In addition, the program has potential to be used in other contexts as well, because the basic content of the Shanghai textbooks and the nation-wide textbooks are similar. The CASCADE-MUCH program has turned out to be valid and rather practical for intended target users by the expert appraisal workshop and the assessment studies that were both carried out in Shanghai. However, in order to be more useful for the intended target users, as well as for other users such as teacher-designers of other subjects, some further improvements are still expected to be made. For instance, a complete manual needs to be added; the program should be error free; and the Tutorial part needs to be completed. In addition, this study (CASCADE-MUCH) is in line with other CASCADE studies, such as an existing study CASCADE (Nieveen, 1997), a concurrent study CASCADE-SEA (McKenney, 2001), and an on-going study CASCADE-IMEI (Zulkardi, 1999). These studies focus on specific aspects of curriculum development and contexts as explained in Chapter 2. Many positive findings of computer support for curriculum development have been summarized and discussed. In the future, more CASCADE studies may continue to explore potentials of computer support in other contexts or to make further investigation on other aspects --such as effectiveness of CASCADE-MUCH-- in the same contexts. The main findings in the current study can be used, validated and extended in the future studies.

Page 207: computer support for multimedia curriculum design

195

References Allen, W. (1967). Media stimulus and types of learning. Audio-visual Instruction,

12(1), 27-31. Ambrose, D.W. (1991). The effects of hypermedia on learning: A literature

review. Educational Technology, 31(12), 51-55. Anderson, R.D., Anderson, B.L., & Varanka-Martin, M.A. (1994). Issues of

curriculum reform in science, mathematics and high order thinking across the disciplines. Washington, DC: U.S. Government.

Bhattacharyya, G.K., & Johnson, R.A. (1977). Statistical concepts and methods. New York: John Willey & Sons.

Biocca, F. (1992). Communication within virtual reality: Creating a space for research. Journal of Communication, 42(4), 5-22.

Blacklock, P. (1993). Computer programming (3rd ed.). Manchester: NCC Blackwell.

Bloom, B.S., Englehart, M.D., Thurst, E.J., Hill, W.H., & Krathwohl, D.R. (1956). Taxonomy of educational objectives. Handbook I: Cognitive domain. New York: McKay.

Borsook, T.K. (1997). Hypermedia: Harbinger of a new instructional paradigm. In C.R. Dills & A.J. Romiszowski (Eds.), Instructional development paradigms (pp. 721-744). Englewood Cliffs, NJ: Educational Technology.

Briggs, L.J., & Wager, W.W. (1981). Handbook of procedures for the design of instruction (2nd ed.). Englewood Cliffs, NJ: Educational Technology.

Brock, P.A. (1994). Educational technology in the classroom. Englewood Cliffs, NJ: Educational Technology.

Brown, C.M. (1988). Human-computer interface design guidelines. Norwood, NJ: Ablex.

Burton, J.K., Moore, D.M., & Magliaro, S.G. (1996). Behaviorism and instructional technology. In D.H. Jonassen (Ed.), Handbook of research for educational communications and technology (pp. 46-73). New York: Macmillan.

Page 208: computer support for multimedia curriculum design

196 References

Butler, K.A. (1986). Learning and teaching styles: In theory and in practice (2nd ed.). Columbia, CT: The Learner's Dimension.

Carroll, J.M. (1994). Designing scenarios for human action. Performance Improvement Quarterly, 7(3), 64-75.

Chapman, B.L. (1998). Objective-oriented scripting for interactive multimedia. Retrieved March 10, 2001, from ftp://ftp.mentergy.com/pdfs/objstory.pdf

Chen, Z.L. (1999). 改革开放二十年的我国教育 [The education in our country in the last twenty years since the reform and opening doors]. Retrieved February 5, 2001, from http://www.moe.edu.cn/wenxian/ cedu20/china43.htm

Choi, W. (1997). Designing effective scenarios for computer-based instructional simulations: Classification of essential features. Educational Technology, 37(5), 13-21.

Clark, R. (1983). Reconsidering research on learning from media. Review of Educational Research, 53(4), 445-459.

Clark, R. (1994). Media will never influence learning. Educational Technology Research and Development, 42(2), 21-30.

Coleman, S.D., Perry, J.D., & Schwen, T.M. (1997). Constructivist instructional development: Reflecting on practice from an alternative paradigm. In C.R. Dills & A.J. Romiszowski (Eds.), Instructional development paradigms (pp. 269-282). Englewood Cliffs, NJ: Educational Technology.

De Hoog, R., de Jong, T., & de Vries, F. (1994). Constraint-driven software design: An escape from the waterfall model. Performance Improvement Quarterly, 7(3), 48-63.

De Jong, T., Limbach, R., Gellevij, M., Kuyper, M., Pieters, J., & van Joolingen, W. (1999). Cognitive tools to support the instructional design of simulation-based discovery learning environments: the SIMQUEST system. In J. van den Akker, R.M. Branch, K. Gustafson, N. Nieveen & Tj. Plomp (Eds.), Design approaches and tools in education and training (pp. 215-224). Dordrecht: Kluwer.

Downing, D.A., Covington, M.A., & Covington, M.M. (1998). Dictionary of computer and Internet terms (6th ed.). New York: Barron.

Eash, M.J. (1991). Curriculum components. In A. Lewy (Ed.), The international encyclopaedia of curriculum (pp. 67-69). Oxford: Pergamon.

Eisner, E.W. (1985). The educational imagination: On the design and evaluation of school programs (2nd ed.). New York: Macmillan.

Page 209: computer support for multimedia curriculum design

References 197

Ertmer, P.A., & Newby, T.J. (1993). Behaviorism, cognitivism, constructivism: Comparing critical features from an instructional design perspective. Performance Improvement Quarterly, 6(4), 50-72.

Fenrich, P. (1997). Practical guidelines for creating instructional multimedia applications. Orlando, FL: Dryden.

Forcier, R.C. (1996). The computer as a productivity tool in education. New Jersey, NJ: Prentice-Hall.

Frazee, B.M., & Rudnitski, R.A. (1995). Integrated teaching methods: Theory, classroom applications, and field-based connections. New York: Delmar.

Fullan, M. (1991). The new meanings of educational change. New York: Teacher College.

Gagné, R.M., Briggs, L.J., & Wager, W.W. (1988). Principles of instructional design (3rd ed.). New York: Holt, Rinehart and Winston.

Galitz, W.O. (1993). User-interface screen design. New York: Wiley-QED. Gery, G. (1991). Electronic performance support systems: How and why to remake the

workplace through the strategic application of technology. Boston, MA: Weingarten. Glasgow, N.A. (1997). New curriculum for new times: A guide to student-centered,

problem-based learning. Thousand Oaks, CA: Corwin. Goodlad, J.I., Klein, M.F., & Tye, K.A. (1979). The domains of curriculum

and their study. In J.I. Goodlad and Associates (Eds.), Curriculum inquiry: The study of curriculum practice (pp. 43-76). New York: McGraw-Hill.

Goodlad, J.I. (1994). Curriculum as a field of study. In T. Husén & T.N. Postlethwaite (Eds.), The international encyclopedia of education (pp. 1262-1267). Oxford: Pergamon.

Gray, D.E., & Black, T.R. (1994). Prototyping of computer-based training materials. Computers & Education, 22(3), 251-256.

Gustafson, K.L. (2000). Designing technology-based performance support. Educational Technology, 40(1), 38-44.

Harrison, N. (1995). Practical instructional design for open learning materials: A modular course covering open learning, computer-based training and multimedia (2nd ed.). London: McGraw-Hill.

Heinich, R., Molenda, M., Russell, J.D., & Smaldino, S.E. (1996). Instructional media and technologies for learning (5th ed.). Englewood Cliffs, NJ: Merrill.

Heeren, E. (1996). Technology support for collaborative distance learning. Doctoral dissertation. Enschede: University of Twente.

Page 210: computer support for multimedia curriculum design

198 References

Hudzina, M., Rowley, K., & Wager, W. (1997). Electronic performance support technology: Defining the domain. Performance Improvement Quarterly, 10(1), 199-211.

Jonassen, D.H. (1991a). Objectivism versus constructivism: Do we need a new philosophical paradigm? Educational Technology Research and Development, 39(3), 5-14.

Jonassen, D.H. (1991b). Evaluating constructivistic learning. Educational Technology, 31(9), 28-33.

Johnson, D.W., & Johnson, R.T. (1996). Cooperation and the use of technology. In D.H. Jonassen (Ed.), Handbook of research for educational communications and technology (pp.1017-1044). London: Macmillan.

Kemp, J.E., & Smellie, D.C. (1989). Planning, producing, and using instructional media (6th ed.). New York: Harper and Row.

Klein, M.F. (1991). A conceptual framework for curriculum decision making. In M.F. Klein (Ed.), The politics of curriculum decision making: Issues in centralizing the curriculum (pp. 1-41). Albany, NY: State University of New York Press.

Kozma, R.B. (1991). Learning with media. Review of Educational Research, 61(2), 179-211.

Kozma, R.B. (1994). Will media influence learning? Reframing the debate. Educational Technology Research and Development, 42(2), 7-20.

Krathwohl, D.R. (1998). Methods of educational and social science research: An integrated approach (2nd ed.). New York: Longman.

Legent Services. (1992). Why performance support? Columbus, OH: Legent Information Technology Division.

Lorch, E.P., Bellack, D.R., & Augsbach, L.H. (1987). Young children's memory for televised stories: Effects of importance. Child Development, 58, 453-462.

Marchionini, M. (1988). Hypermedia and learning: Freedom and chaos. Educational Technology, 28(11), 8-12.

Marsh, C. (1991). Curriculum approaches. In C. Marsh & P. Morris (Eds.), Curriculum development in East Asia (pp. 3-21). London: Falmer.

Marsh, C. (1997). Planning, management and ideology: Key concepts for understanding curriculum. London: Falmer.

Page 211: computer support for multimedia curriculum design

References 199

Mayer, R.E., & Anderson, R.B. (1991). Animations need narrations: An experimental test of a dual-coding hypothesis. Journal of Educational Psychology, 83(4), 484-490.

McKenney, S. (1999). CASCADE-SEA: Computer assisted curriculum analysis, design, & evaluation for science education in Africa. In J. van den Akker, R.M. Branch, K. Gustafson, N. Nieveen & Tj. Plomp (Eds.), Design approaches and tools in education and training (pp. 225-233). Dordrecht: Kluwer.

McKenney, S. (2001). Computer-based support for science education materials developers in Africa: Exploring potentials. Doctoral dissertation. Enschede: University of Twente.

Merrill, M.D. (1983). Component display theory. In C.M. Reigeluth (Ed.), Instructional design theories and models: An overview of the current status (pp. 279-333). Hillsdale, NJ: Lawrence Erlbaum.

Miles, M.B., & Huberman, A.M. (1994). Qualitative data analysis: An expanded source book. London: Sage.

Molenda, M. (1991). A philosophical critique of the claims of "constructivism". Educational Technology, 31(9), 44-48.

Monk, A., Wright, P., Haber, J., & Davenport, L. (1993). Improving your human-computer interface: A practical technique. New York: Prentice Hall.

Moonen, J. (1996). Prototyping as a design method. In Tj. Plomp & D.P. Ely (Eds.), International encyclopedia of educational technology (pp. 186-190). Oxford: Pergamon.

Moonen, J. (1999). The design and prototyping of digital learning materials: Some new perspectives. In J.J.H. van den Akker, R.M. Branch, K. Gustafson, N.M. Nieveen & Tj. Plomp (Eds.), Design approaches and tools in education and training (pp. 95-111). Dordrecht: Kluwer.

Moore, D.M., Burton, J.K., & Myers, R.J. (1996). Multiple-channel communication: The theoretical and research foundations of multimedia. In D.H. Jonassen (Ed.), Handbook of research for educational communication and technology (pp. 851-875). New York: Macmillan.

Moore, K.D. (1992). Classroom teaching skills (2nd ed.). New York: McGraw-Hill.

Moore, M. (1989). Three types of interaction. In M. Moore (Ed.), Readings in principles of distance education (pp. 100-105). University Park, PA: Pennsylvania State University.

Nielsen, J. (1993). Usability engineering. San Diego, CA: Academic Press.

Page 212: computer support for multimedia curriculum design

200 References

Nieveen, N.M. (1997). Computer support for curriculum developers: A study on the potential of computer support in the domain of formative curriculum evaluation. Doctoral dissertation. Enschede: University of Twente.

Nieveen, N.M. (1999). Prototyping to reach product quality. In J.J.H. van den Akker, R.M. Branch, K. Gustafson, N.M. Nieveen & Tj. Plomp (Eds.), Design approaches and tools in education and training (pp. 125-135). Dordrecht: Kluwer.

Nieveen, N.M., & Gustafson, K. (1999). Characteristics of computer-based tools for education and training development: An introduction. In J.J.H. van den Akker, R.M. Branch, K. Gustafson, N.M. Nieveen & Tj. Plomp (Eds.), Design approaches and tools in education and training (pp. 155-174). Dordrecht: Kluwer.

Nieveen, N.M., & Gustafson, K. (2000, October). Computer-based tools to support curriculum developers. Paper presented at the AECT conference, Denver, CO.

Nieveen, N.M., & van den Akker, J.J.H. (1999). Exploring the potential of a computer tool for instructional developers. Educational Technology Research and Development, 47(3), 77-98.

Ornstein, A.C., & Hunkins, F.P. (1993). Curriculum foundations, principles, and theory (2nd ed.). Boston, MA: Allyn and Bacon.

Paquette, G., Aubin, C., & Cervier, F. (1994). An intelligent support system for course design. Educational Technology, 34(9), 50-57.

Park, O. (1991). Hypermedia: Functional features and research issues. Educational Technology, 31(8), 24-31.

Posner, G.J., & Rudnitski, A.N. (1986). Course design: A guide to curriculum development for teachers. New York: Longman.

Preese, F., & Foshay, W. (1999). The PLATO courseware development environment. In J. van den Akker, R.M. Branch, K. Gustafson, N. Nieveen & Tj. Plomp (Eds.), Design approaches and tools in education and training (pp. 195-204). Dordrecht: Kluwer.

Ragan, T., Boyce, M., Redwine, D., Savenye, W.C., & McMichael, J. (1993). Is multimedia worth it? A review of the effectiveness of individualized multimedia instruction. Paper presented at the Association for Educational Communications and Technology Convention, New Orleans, LA.

Reiser, R.A., & Gagné, R.M. (1983). Selecting media for instruction. Englewood Cliffs, NJ: Education Technology.

Page 213: computer support for multimedia curriculum design

References 201

Rettig, M. (1994). Prototyping for tiny fingers. Communication of the ACM, 37(4), 21-27.

Reynolds, R.E., & Baker, D.R. (1987). The utility of graphical representations in text: Some theoretical and empirical issues. Journal of Research in Science Teaching, 24(2), 161-173.

Richey, R.C. (1997). Research on instructional development. Educational Technology Research and Development, 45(3), 91-100.

Richey, R.C., & Nelson, W.A. (1996). Developmental research. In D.H. Jonassen (Ed.), Handbook of research for educational communications and technology (pp. 1213-1245). New York: Macmillan.

Roblyer, M.D., Edwards, J., & Havriluk, M.A. (1997). Integrating educational technology into teaching. Upper Saddle River, NJ: Prentice-Hall.

Romiszowski, A.J. (1981). Designing instructional systems: Decision making in course planning and curriculum design. London: Kogan.

Romiszowski, A.J. (1988). The selection and use of instructional media. London: Kogan.

Seels, B.B., & Richey, R.C. (1994). Instructional technology: The definition and domains of the field. Washington, DC: Association for Educational Communications and Technology.

Shneiderman, B. (1992). Designing the user interface: Strategies for effective human-computer interaction (2nd ed.). New York: Addison-Wesley.

Smith, M.E., & Brandenburg, D.C. (1991). Summative evaluation. Performance Improvement Quarterly, 4(2), 35-58.

Smith, M.F. (1991). Software prototyping: Adoption, practice and management. London: McGraw-Hill.

Sommerville, I. (1996). Software engineering (5th ed.). Workingham: Addison-Wesley.

SSCRCO, & SECIRO. (1999).面向21世纪中小学新课程方案和各学科教育

改革行动纲领(研究报告) [A research report of the new plan and guidelines for the curriculum reform facing the 21st century]. Shanghai: Shanghai Educational Press.

Stevens, G.H., & Stevens, E.F. (1995). Designing electronic performance support tools: Improving workplace performance with hypertext, hypermedia and multimedia. Englewood Cliffs, NJ: Educational Technology.

Page 214: computer support for multimedia curriculum design

202 References

Taba, H. (1962). Curriculum development: Theory and practice. New York: Harcourt Brace Jovanovitch.

Tessmer, M. (1993). Planning and conducting formative evaluations: Improving the quality of education and training. London: Kogan Page.

Tessmer, M. (1998). Meeting with the SME to design multimedia exploration systems. Educational Technology Research and Development, 46(2), 79-95.

Thiagarajan, S. (1991). Formative evaluation in performance technology. Performance Improvement Quarterly, 4(2), 22-34.

Thijs, A. (1999). Supporting science curriculum reform in Botswana: The potential of peer coaching. Doctoral dissertation. Enschede: University of Twente.

Tyler, R.W. (1949). Basic principles of curriculum and instruction. Chicago: The University of Chicago Press.

Van den Akker, J.J.H., & Plomp, Tj. (1993, April). Development research in curriculum: Propositions and experiences. Paper presented at the annual meeting of the American Educational Research Association, Atlanta, GA.

Van den Akker, J.J.H. (1998). The science curriculum: Between ideals and outcomes. In B.J. Fraser & K.G. Tobin (Eds.), International handbook of science education (pp. 421-447). Dordrecht: Kluwer.

Van den Akker, J.J.H. (1999). Principles and methods of development research. In J.J.H. van den Akker, R.M. Branch, K. Gustafson, N.M. Nieveen & Tj. Plomp (Eds.), Design approaches and tools in education and training (pp. 1-14). Dordrecht: Kluwer.

Verhagen, P.W. (1992). Length of segments in interactive video programmes. Doctoral dissertation. Enschede: University of Twente.

Verwijs, C. (1998). A mix of core and complementary media: New perspectives in media-decision making. Doctoral dissertation. Enschede: University of Twente.

Visscher-Voerman, J.I.A. (1999). Design approaches in training and education: A reconstructive study. Doctoral dissertation. Enschede: University of Twente.

Visscher-Voerman, J.I.A., Gustafson, K., & Plomp, Tj. (1999). Educational design and development: An overview of paradigms. In J.J.H. van den Akker, R.M. Branch, K. Gustafson, N.M. Nieveen & Tj. Plomp (Eds.), Design approaches and tools in education and training (pp. 15-28). Dordrecht: Kluwer.

Visser, L. (1998). The development of motivational communication in distance education support. Doctoral dissertation. Enschede: University of Twente.

Page 215: computer support for multimedia curriculum design

References 203

Vockell, E.L., & Schwartz, E.M. (1992). The computer in the classroom (2nd ed.). Watsonville, CA: Mitchell McGraw-Hill.

Walker, D.F. (1990). Fundamentals of curriculum. San Diego, CA: Harcourt Brace Jovanovich.

Wang, Q.Y. (1996). Developing a computer support system for needs analysis in the context of curriculum development of Shanghai. Master's thesis. Enschede: University of Twente.

Wang, S.H. (1992a). 在上海中小学课程教材改革委员会成立大会上的讲话

(摘要) [Speech on the foundation conference of Shanghai Curriculum Innovation Committee (excerpt)]. In Shanghai Curriculum Innovation Committee Office (Ed.), 上海中小学课程教材改革:专集1 [Shanghai curriculum innovation: Issue 1] (2nd ed., pp.1-8). Shanghai: Shanghai Educational Press.

Wang, S.H. (1992b). 关于上海中小学课程改革方案的说明 [Explanations of the plan for Shanghai secondary and primary school curriculum innovation]. In Shanghai Curriculum Innovation Committee Office (Ed.), 上海中小学课程教材改革:专集1 [Shanghai curriculum innovation: Issue 1] (2nd ed., pp.42-54). Shanghai: Shanghai Educational Press.

Wilson, B.G. (1997). Reflections on constructivism and instructional design. In C.R. Dills & A.J. Romiszowski (Eds.), Instructional development paradigms (pp. 63-80). Englewood Cliffs, NJ: Educational Technology.

Witkin, R., & Altschuld, J.W. (1995). Planning and conducting needs assessments: A practical guide. Thousand Oaks, CA: Sage.

Yat-ming, J.L. (1991). Curriculum development in the People's Republic of China. In C. Marsh & P. Morris (Eds.), Curriculum development in East Asia (pp. 61-81). London: Falmer.

You, M.J. (1998). 素质教育与课程改革 [Quality-driven education and curriculum reform]. Hangzhou: Zhejiang Educational Press.

Yuan, C. (1990). 以提高学生素质为核心,改革中小学课程教材 [Improving learners' full qualities and changing secondary and primary school curricula]. In Shanghai Curriculum Innovation Committee Office (Ed.), 上海中小学课程教材改革:专集1 [Shanghai curriculum innovation: Issue 1] (pp.58-64). Shanghai: Shanghai Educational Press.

Zhang, M.S. (1999). 前言 [Preface]. In SSCRCO & SECIRO (Eds.), 面向21

世纪中小学新课程方案和各学科教育改革行动纲领 (研究报告) [A research report of the new plan and guidelines for the curriculum reform facing the 21st century]. Shanghai: Shanghai Educational Press.

Page 216: computer support for multimedia curriculum design

204 References

Zhu, Z.T. (1996). Cross-cultural portability of educational software: A communication-oriented approach. Doctoral dissertation. Enschede: University of Twente.

Zhu, Z.T. (1997). 多媒体 CAI [Multimedia CAI]. Shenyang: Liaoning Science Technology Press.

Zhu, Z.T. (1999). Cultural classification and integration of network-based learning models. Bulletin of the Inter-University Lifelong Learning Research Institute, No.4. Kyushu Kyoritsu University, Kyushu, Japan.

Zulkardi (1999). Computer assisted curriculum analysis, design and evaluation for mathematics education in Indonesia. Master's thesis. Enschede: University of Twente.

Page 217: computer support for multimedia curriculum design

205

English Summary Introduction Shanghai (China) is currently engaged in a comprehensive curriculum innovation. The main aims of the curriculum innovation are to: i) classify the new curricula into three modules (basic courses, extended courses, and research courses) and develop lesson materials for these modules; ii) promote curriculum development on three different levels (city level, district level, and school level); and iii) develop new multimedia lesson materials for each course. The third aim implies that more multimedia lesson materials will be developed in the future. In this study, the lesson materials covering a whole course is called multimedia curriculum, which is defined as a plan for learning where various presentation forms (such as text, pictures, audio, and video) are integrated, encoded, and presented on a computer. The development of a multimedia curriculum is a complex task, which often starts with the development of an instructional scenario --an intermediate product of a multimedia curriculum-- and is followed by program construction. The development of an instructional scenario in Shanghai is usually carried out by experienced subject teachers, in particular those teaching researchers at district educational colleges --they are called teacher-designers in this study; and the program construction is carried out by computer programmers. In reality, the teacher-designers in Shanghai usually do not have sufficient computer skills or experience in computer-based learning (CBL) design. Consequently, they often encounter difficulties when they are developing an instructional scenario for a multimedia curriculum. For instance, they may not know how to design such a scenario, or the designed instructional scenario does not fully exploit the advantages of CBL. For this reason, they can greatly benefit from support tools in designing instructional scenarios. A computer support system was considered to be useful for them since it can integrate many types of support and provide them support just in time.

Page 218: computer support for multimedia curriculum design

206 English Summary

The CASCADE-MUCH study aimed at producing a valid and practical computer support system that can help teacher-designers in Shanghai develop usable instructional scenarios. The instructional scenarios can stimulate and moderate discussions between computer programmers and teacher-designers, and can facilitate computer programmers in constructing multimedia curricula. Furthermore, it was expected that teacher-designers with less experience in multimedia curriculum design could improve their professional knowledge concerning multimedia curriculum design during their use of the support system. In addition, this study focused on two subjects --Biology and Geography--in secondary schools in Shanghai. More detailed information about the context and origins of the study, the intended target users and the aims of the program, and the definition, and modules of a multimedia curriculum can be found in Chapters 1 and 2. Research questions, research design and results Research question This study followed a development research approach, aiming at: i) producing a valid and practical computer support system; and ii) generating methodological guidelines for the design and evaluation of such products. The main research question of the study is:

What characteristics should a valid and practical computer support system for multimedia curriculum design have in the context of Shanghai?

By following the development research approach, this study progressed through two main stages: a prototyping stage and an assessment stage. The research questions and design of these two stages are summarized in the following two sections. Prototyping stage In order to gain more clarity concerning the characteristics of the CASCADE-MUCH program, it was decided in this study to follow a prototyping approach. The research question of the prototyping stage is:

What characteristics should a valid (and potentially practical) computer support system for multimedia curriculum design have in the context of Shanghai?

Page 219: computer support for multimedia curriculum design

English Summary 207

In order to be more transparent, the CASCADE-MUCH program was split up into four key components: content, support, interface, and scenario. The program progressed through four rounds of prototyping, focusing on these four components. Each round of prototyping included activities of analysis, design, formative evaluation and revision decisions. Most of these formative evaluation activities were carried out in the context of Shanghai, and the remaining was done in the Netherlands. The first round of prototyping focused on three components: content, support, and interface. In the end, five intended target users participated in micro-evaluations on the prototype; and two curriculum/instructional design experts and three computer-based learning (CBL) designers conducted expert appraisals on it. The second round of prototyping focused on the components of content, support, and scenario. In the end, 27 participants in Shanghai --including nine subject teachers, six curriculum/instructional specialists, five CBL designers, and seven computer programmers-- made contributions to the formative evaluation via expert appraisal or micro-evaluation. The third round of prototyping focused on the same components as those in the second round. Two formative evaluation activities were organized in Shanghai and the University of Twente (UT, the Netherlands) respectively. During the formative evaluation in Shanghai, seven subject teachers were involved in a micro-evaluation workshop, and produced preliminary scenarios by using the prototype. Based on these instructional scenarios, four computer programmers made comments and created multimedia programs. In order to check how experts from another context would react on the validity of the prototype, and to get information on the possibility of extending the prototype to other contexts, an expert appraisal workshop was organized at the UT. Eight experts --including two multimedia design experts, three instructional design experts, and three curriculum development experts-- took part in the workshop at the UT, and believed that the prototype had potential to be valid and practical. It was concluded that several improvements were needed, particularly on the components of content and support. The fourth round of prototyping focused on the components of content and support. In the end, another expert appraisal workshop was organized in Shanghai to check to what extent the revised prototype was valid. The participants were nine experts from the Faculty of Education, East China Normal University (ECNU). The final results showed that the prototype was

Page 220: computer support for multimedia curriculum design

208 English Summary

valid and had potential to be practical for intended target users in Shanghai. The detailed description of each prototype and its accompanying formative evaluation activities can be found in Chapter 3. Assessment stage In order to check to what extent the prototype was practical for intended target users, two assessment studies were organized in Shanghai. The main research question for the assessment studies was:

To what extent is the CASCADE-MUCH program practical for both primary target group users and other users in the context of Shanghai?

The first study focused on primary target group users. Four Biology and two Geography teaching researchers (or subject teachers) took part in the micro-evaluation workshop. In general, the final results of the study showed that the participants perceived the four components of the program to be rather practical. In addition, no statistically significant differences were found between the novice and experienced designers concerning the perceived practicality. Also, no obvious differences were found between these groups with regard to the walking routes, time spent on each screen, and support tools utilized. The second study focused on other users. Seven teachers of other subjects, three CBL designers from secondary schools, and three software developers from educational related computer companies participated in this assessment study in a micro-evaluation workshop. The final results of this assessment study showed that the program was perceived to be rather practical by them. In addition, all teachers of other subjects expressed a desire that more subjects should be supported. The detailed description of the assessment studies is presented in Chapter 5. The assessment studies led to the following three conclusions: a. The four components (content, support, interface, and scenario) were

perceived to be rather practical by both primary and other users. b. No significant differences were detected between novice and experienced

designers on the perceived practicality of the four components, walking routes, time spent on each screen, and support tools utilized.

c. The program was expected to be extendable to other subjects.

Page 221: computer support for multimedia curriculum design

English Summary 209

Main characteristics of CASCADE-MUCH Content, support, interface and scenario are four key components of the CASCADE-MUCH program. In this section, the main characteristics of these four components will be summarized. More detailed information about the characteristics of the program can be found in Chapter 4. Content The content of CASCADE-MUCH is related to multimedia design and curriculum development. In the program, the content includes two aspects: analysis and design. The analysis aspect mainly includes goal analysis, usage analysis, and learner analysis. The design aspect mainly includes guidance for content selection, representation, organization, and interface design. These issues will be briefly summarized here. The goal of a multimedia curriculum was decided to be: i) basic knowledge and skills; ii) extended knowledge and higher-level skills; and iii) attitudes. The usage of a multimedia curriculum includes: a) individual learning for learners; b) collaborative learning for learners; and c) classroom teaching for teachers. The learner analysis focuses on subject knowledge/skills, computer skills, and preferred learning materials and activities. The content selection aims at selecting proper knowledge units to be included in the multimedia curriculum. Two approaches are recommended in the program: selection based on curriculum standards and selection based on concept maps. The content representation aims at presenting the selected knowledge units with proper presentation forms on a computer. In this program, the presentation forms include text, graphics (including theme pictures and background pictures), audio (including sound effects, narration, and music), and motion (including animation and video). The content organization deals with how to sequence the selected and represented knowledge units in a helpful way. In this program, three broad categories of content organization styles are included: linear, non-linear (including menu style and hyperlink style), and an integrated style. The interface design attempts to help teacher-designers select a proper interface style for the multimedia curriculum. Three interface styles are included in the program: classic and elegant, modern and popular, and vivid and vigorous.

Page 222: computer support for multimedia curriculum design

210 English Summary

Support Being an Electronic Performance Support System (EPSS), CASCADE-MUCH involves four broad categories of support: information, advice, tools, and training. Each broad category is divided into some specific types of support. These specific types of support are summarized in this section. In this program, information includes: explanation and examples of keywords; help for the content on the current screen; tips and hints for navigating the program. Advice is a kind of heuristic information given based on the user's profile, histories, and the program's embedded expertise. CASCADE-MUCH provides two types of advice: suggestions and previews. Suggestions are given based on the previous settings and the embedded intelligent expertise, aiming at helping users decide how to deal with the current design. Previews are given based on the current settings to help users know how the current settings will affect the subsequent design. Tools aim at helping teacher-designers easily carry out the tasks related to instructional scenario development. In CASCADE-MUCH, the following tools are included: Edit Panel, Word, a concept mapping tool, web communication facilities, and translation facilities. Training aims at improving user's task performance. Two types of training are included in this program: wizards and tutorial. Interface Being a computer program, interface design is an important component of CASCADE-MUCH. The interface of CASCADE-MUCH was designed based on two guidelines: easy to learn and easy to use. Specifically, the interface of the program has three main characteristics. First, it is consistent. On the one hand, it is rather externally consistent with other computer applications; on the other hand, it is also internally consistent. Second, it is flexible. The program is split up into several parts. Different users can start with or work on different parts based on their specific needs or conditions. Other examples are that the program provides two navigation tools: linear and browser; users can walk through the program in any way they prefer; most actions in the program can be done with either the mouse or the keyboard. Third, it is intuitive (visual). A lot of metaphors --such as logos and icons-- are used in the program to give users intuitive images.

Page 223: computer support for multimedia curriculum design

English Summary 211

Scenario An instructional scenario is the final outcome of the CASCADE-MUCH program. It is mainly used for two purposes: i) facilitating discussions between teacher-designers and computer programmers; and ii) guiding computer programmers in their programming task. The scenario produced by the program has two general characteristics. First, it includes not only the design information of each knowledge unit, but also the analysis information about the subject, learners and designers are included in the instructional scenario. This information can provide computer programmers with rich information for programming as well as for discussions with the designers of the scenario. Second, it is well structured and easy to read. The instructional scenario can be exported to Microsoft Word and become a Word document. All information is presented in tables and is easy to read. Conclusions and recommendations The expert appraisal workshop at the ECNU and the final assessment studies showed that the CASCADE-MUCH program was valid and practical. Here some recommendations are given to further the investigation of this study as well as to other relevant studies in the future. More detailed information can be found in Chapter 6. The current version of the CASCADE-MUCH program is a CD-ROM-based support program. Although the assessment studies showed that it is rather practical for intended target users, it might be more practical if it can combine some characteristics of a web-based support program, such as links to related web-sites, chat facilities, publishing the latest information, and downloading updates, etc. In addition, an instructional scenario is the final outcome of the CASCADE-MUCH program. Although some preliminary instructional scenarios were developed during several formative evaluation activities and the final assessment studies, no in-depth follow-up studies have been performed to check the quality of the instructional scenario and the multimedia curriculum to be designed. It might be interesting if a follow-up study can be initiated to trace the use of the instructional scenario, the construction and implementation of the multimedia curriculum, and the learning outcomes of learners.

Page 224: computer support for multimedia curriculum design

212 English Summary

Furthermore, in line with other CASCADE studies such as CASCADE (Nieveen, 1997) and CASCADE-SEA (McKenney, 2001), this study focuses on specific aspects -- analysis and design-- for a specific type of curriculum --multimedia curriculum-- in a specific context --Shanghai. In the future, additional CASCADE studies may continue to explore potentials of computer support in other aspects of curriculum development and in other contexts, such as the on-going study CASCADE-IMEI (Zulkardi, 1999). In this case, the main findings in the current study can be used, validated and extended in the future.

Page 225: computer support for multimedia curriculum design

213

Nederlandse samenvatting Introductie Shanghai (China) is bezig met de tweede ronde van een grootscheepse curriculumvernieuwing. De belangrijkste doelen van deze vernieuwing zijn: i) het ontwikkelen en classificeren van drie typen nieuwe curricula (basiscursus, verrijkingscursus, en onderzoekscursus); ii) het bevorderen van curriculumontwikkeling op drie niveaus (stadsniveau, districtsniveau en schoolniveau); iii) het ontwikkelen van nieuwe multimediale lesmaterialen voor iedere cursus. In dit onderzoek verwijst de term multimedia curriculum naar een leerplan waarbij verschillende representatievormen (zoals tekst, afbeeldingen, audio en video) via de computer zijn geïntegreerd, geëncodeerd en gepresenteerd. Het ontwikkelen van een multimedia curriculum is een complexe aangelegenheid, die meestal start met de ontwikkeling van een instructie-scenario gevolgd door de constructie van het programma. In Shanghai wordt de ontwikkeling van een instructie-scenario over het algemeen uitgevoerd door ervaren vakdocenten, en in het bijzonder door de onderzoeksdocenten die op ‘district educational colleges’ werkzaam zijn. Deze docenten worden in dit onderzoek aangeduid met de term docent-ontwerpers. De uiteindelijke constructie van het multimediale curriculum wordt uitgevoerd door computer programmeurs. In de praktijk hebben docent-ontwerpers in Shanghai vaak onvoldoende computervaardigheden en weinig ervaring met het ontwerpen van computerondersteund onderwijs (COO). Als gevolg daarvan ervaren ze vaak moeilijkheden tijdens de ontwikkeling van een instructie-scenario voor een multimedia curriculum. Zo weten ze bijvoorbeeld niet goed hoe ze een scenario moeten ontwerpen of ze ontwikkelen een scenario dat onvoldoende de mogelijkheden van COO benut. In dit onderzoek gaan we ervan uit dat docent-ontwerpers veel baat kunnen hebben bij hulpmiddelen voor het ontwikkelen van een instructie-scenario. We nemen daarbij aan dat in deze situatie met name een computerondersteuningssysteem bruikbaar kan zijn, omdat

Page 226: computer support for multimedia curriculum design

214 Nederlandse samenvatting

dergelijke systemen diverse typen ondersteuning op geïntegreerde wijze en op het juiste moment kunnen aanbieden. Het CASCADE-MUCH onderzoek is erop gericht een valide en praktisch bruikbaar computersysteem te ontwikkelen dat docent-ontwerpers in Shanghai ondersteunt bij het ontwikkelen van bruikbare instructie-scenario’s. De instructie-scenario’s kunnen discussies stimuleren computerprogrammeurs en docent-ontwerpers en kunnen computerprogrammeurs assisteren bij het construeren van de multimedia curricula. Daarnaast is de verwachting uitgesproken dat het gebruik van het ondersteuningssysteem een bijdrage kan leveren aan de professionalisering van docent-ontwerpers op het terrein van het ontwikkelen van multimedia curricula. Uitgebreidere informatie over de context en de oorsprong van het onderzoek, de doelgroep en de doelen van het ondersteuningsprogramma, en de definitie en inhoud van een multimedia curriculum staat beschreven in hoofdstukken 1 en 2. Onderzoeksvragen, onderzoeksontwerp en resultaten

Onderzoeksvragen Binnen dit onderzoek is een ontwerpgerichte onderzoekbenadering gevolgd, met twee hoofddoelen: i) het ontwikkelen van een valide en praktisch bruikbaar computerondersteuningssysteem; en ii) het genereren van methodologische richtlijnen voor het ontwerpen en evalueren van dergelijke systemen. De hoofdonderzoeksvraag luidt: Welke kenmerken dient een valide en praktisch bruikbaar ondersteuningssysteem voor het ontwikkelen van een multimedia curriculum te hebben in de context van Shanghai? Het onderzoek bevatte twee hoofdfasen: een prototypingfase en een evaluatiefase. Hierna volgt een samenvatting van de onderzoeksvragen, het ontwerp en de belangrijkste resultaten van beide fasen.

Prototypingfase Om meer zicht te krijgen op de kenmerken van het CASCADE-MUCH programma, is besloten een prototypingbenadering te volgen. De onderzoeksvraag behorend bij deze fase luidt:

Page 227: computer support for multimedia curriculum design

Nederlandse samenvatting 215

Welke kenmerken dient een valide en (potentieel praktisch bruikbaar) ondersteuningssysteem voor het ontwikkelen van een multimedia curriculum te hebben in de context van Shanghai? Om een beter overzicht te krijgen werd het CASCADE-MUCH programma opgedeeld in vier belangrijke componenten: inhoud, ondersteuning, gebruikersinterface en scenario. Gedurende deze fase zijn vier prototypes ontwikkeld die elk gericht waren op een of meerdere van deze componenten. Iedere prototypingronde bevatte analyse-activiteiten, ontwerpactiviteiten, een formatieve evaluatie en het genereren van revisiebeslissingen. De meeste formatieve evaluatie-activiteiten vonden plaats in de context van Shanghai, terwijl de meeste andere taken in Nederland werden uitgevoerd. De eerste ronde richtte zich op drie componenten van het prototype: inhoud, ondersteuning en gebruikersinterface. Aan de micro-evaluatie van het prototype namen vijf leden uit de beoogde doelgroep deel en aan de deskundigenbevraging werd deelgenomen door twee deskundigen op het terrein van curriculum-/instructie-ontwerp en drie ontwerpers van computerondersteund onderwijs (COO). De tweede ronde was gericht op drie componenten: inhoud, ondersteuning en scenario. In totaal droegen 27 participanten (negen vakdocenten, zes curriculum-/instructie-specialisten, vijf COO ontwerpers en zeven computerprogrammeurs) uit Shanghai bij aan de formatieve evaluatie via deskundigenbevraging of micro-evaluaties. De derde ronde was opnieuw gericht op de componenten inhoud, ondersteuning en scenario. Twee formatieve evaluatie-activiteiten werden georganiseerd in Shanghai en in Nederland (Universiteit Twente). Tijdens de formatieve evaluatie in Shanghai namen zeven docent-ontwerpers deel aan een micro-evaluatie workshop. Tijdens deze bijeenkomst ontwierp ieder met behulp van het prototype een eerste versie van een scenario. Vervolgens werden vier computerprogrammeurs uitgenodigd om op grond van de scenario’s een multimedia curriculum te ontwikkelen. Om te achterhalen hoe deskundigen uit een andere context de validiteit van het prototype beoordeelden en om informatie te verkrijgen over de bruikbaarheid van het prototype in andere contexten werd een workshop met deskundigen georganiseerd op de Universiteit Twente. De acht deskundigen --twee deskundigen op het terrein van multimedia-ontwerp, drie deskundigen op het terrein van instructie-ontwerp, en drie deskundigen op het domein van curriculumontwikkeling-- oordeelden dat het prototype in potentie valide en

Page 228: computer support for multimedia curriculum design

216 Nederlandse samenvatting

praktisch bruikbaar was, maar dat verscheidene verbeteringen doorgevoerd moesten worden, met name wat betreft de inhouds- en ondersteuningscomponent. De vierde en laatste ronde was gericht op de inhouds- en ondersteuningscomponent van het prototype. Gedurende een workshop met deskundigen in Shanghai werd de validiteit van het prototype achterhaald. Aan de workshop namen negen deskundigen van de Faculty of Education van de East China Normal University (ECNU) deel. De deelnemers oordeelden dat het prototype valide was en zij verwachtten dat het prototype praktisch bruikbaar zou zijn voor de beoogde doelgroep in Shanghai. Een gedetailleerde beschrijving van ieder prototype en de bijbehorende formatieve evaluatie staan in hoofdstuk 3.

Evaluatiefase en kernresultaten Om te onderzoeken in welke mate het prototype praktisch bruikbaar is voor de beoogde doelgroep zijn twee afsluitende evaluatiestudies georganiseerd in Shanghai. De hoofdvraag van deze fase luidt: In welke mate is het CASCADE-MUCH programma praktisch bruikbaar voor zowel de beoogde doelgroep als voor andere gebruikers in de context van Shanghai? De eerste evaluatiestudie was gericht op de beoogde doelgroep. Zes docent-ontwerpers (vier voor biologie en twee voor aardrijkskunde) namen deel aan een micro-evaluatie workshop. Over het algemeen laten de resultaten van deze evaluatie zien dat de deelnemers de vier componenten van het programma als praktisch bruikbaar beschouwden. Er bleken geen significante verschillen te bestaan tussen onervaren en ervaren ontwerpers in hun meningen over de bruikbaarheid van het programma. Tevens bleken er geen duidelijke verschillen tussen de manier waarop onervaren en ervaren gebruikers het programma gebruikten, de tijd die ze aan ieder scherm besteedden en de ondersteuning die ze benutten. De tweede evaluatiestudie richtte zich op andere gebruikers. Zeven docenten (die andere vakken gaven dan biologie of aardrijkskunde), drie COO-ontwerpers werkzaam binnen het voortgezet onderwijs en drie educatieve software-ontwikkelaars uit het bedrijfsleven namen deel aan de micro-evaluatie workshop. De resultaten lieten zien dat de deelnemers het programma praktisch bruikbaar achtten. Daarnaast spraken de docenten de wens uit dat het programma ook de ontwikkeling van multimedia curricula

Page 229: computer support for multimedia curriculum design

Nederlandse samenvatting 217

zou ondersteunen voor de vakken die zij gaven. Een uitgebreide beschrijving van beide evaluatiestudies is opgenomen in hoofdstuk 5. Beide evaluatiestudies hebben geleid tot de volgende conclusies: a. De vier componenten (inhoud, ondersteuning, gebruikersinterface en

scenario) werden over het algemeen als praktisch bruikbaar beschouwd door zowel de deelnemers uit de beoogde doelgroep als door deelnemers die niet direct tot de doelgroep behoorden.

b. De manier waarop de deelnemers over de praktische bruikbaarheid oordeelden liet geen duidelijke verschillen zien. Hetzelfde gold voor de wijze waarop ze het programma gebruikten, de tijd die ze aan ieder onderdeel van het programma besteedden en de ondersteuning die werd benut.

c. Het programma lijkt geschikt te zijn voor andere vakken dan biologie en aardrijkskunde, mits het programma voor die vakken aangepast wordt.

Belangrijke kenmerken van CASCADE-MUCH Het programma CASCADE-MUCH bestaat uit vier componenten: inhoud, ondersteuning, gebruikersinterface en scenario. In deze paragraaf zal een samenvatting gegeven worden van de belangrijkste kenmerken van deze componenten. Meer details over deze componenten zijn opgenomen in hoofdstuk 4.

Inhoud De inhoud van CASCADE-MUCH is gerelateerd aan de domeinen van multimedia- en curriculumontwikkeling. Het programma omvat twee ontwikkelfasen: analyse en ontwerp. De analysecomponent gaat in op de doelanalyse, gebruiksanalyse en doelgroepanalyse. De ontwerpcomponent bevat de elementen: inhoudsselectie, -representatie en --organisatie en interface-ontwerp. De kenmerken van de verschillende elementen zoals ze in het programma zijn meegenomen zullen hierna kort worden samengevat. Het doel van een multimedia curriculum kan bestaan uit: i) basiskennis en -vaardigheden; ii) meer geavanceerde kennis en hogere orde vaardigheden; en iii) houding. Het gebruik van het multimedia curriculum omvat de volgende mogelijkheden: a) individueel leren door leerlingen; b) samenwerkend leren door leerlingen; en c) klassikaal lesgeven door docenten. De analyse van de

Page 230: computer support for multimedia curriculum design

218 Nederlandse samenvatting

doelgroep richt zich op de aanwezige vakinhoudelijke kennis en vaardigheden, computervaardigheden, en voorkeur voor leermaterialen en --activiteiten. De inhoudselectie is gericht op het selecteren van de benodigde kenniseenheden die in het multimedia curriculum opgenomen zullen worden. Wat betreft de inhoudselectie beveelt het programma twee werkwijzen aan: selectie op basis van bestaande curriculumstandaarden en selectie op grond van een conceptueel raamwerk. De representatie van de inhoud is gericht op het keuzeproces van relevante representatievormen voor de geselecteerde kenniseenheden. In dit programma kan een keuze gemaakt worden uit de volgende representatievormen: tekst, afbeeldingen (inclusief thematische afbeeldingen en achtergrondafbeeldingen), geluid (inclusief geluidseffecten, gesproken tekst en muziek), en bewegende beelden (inclusief animatie en video). De organisatie van de inhoud betreft het structureren van de geselecteerde en gerepresenteerde kenniseenheden. In dit programma zijn drie manieren waarop de inhoud georganiseerd kan worden opgenomen: lineair, niet-lineair (inclusief menu- en hyperlinkstructuur) en geïntegreerd. Ten slotte is het element interface-ontwerp erop gericht docent-ontwerpers te ondersteunen bij het selecteren van een bruikbare gebruikersinterface voor het te ontwerpen multimedia curriculum. Het programma biedt drie interfacestijlen aan: ‘klassiek & elegant’, ‘modern & populair’ en ‘levendig en krachtig’.

Ondersteuning CASCADE-MUCH biedt vier vormen van ondersteuning aan die overeenkomen met de ondersteuning van een Electronic Performance Support System (EPSS): informatie, advies, hulpmiddelen en training. Iedere categorie bestaat uit een aantal concrete ondersteuningsmogelijkheden die hierna kort staan samengevat. In dit programma bevat de informatie: uitleg en voorbeelden van sleutelbegrippen; hulp die behoort bij een specifiek scherm; tips en hints voor het navigeren door het programma. Het advies van dit programma bestaat uit heuristieken. Op grond van het gebruikersprofiel, de gebruiksgeschiedenis en de kennis die ligt ingebed in het programma geeft het programma advies over welke activiteiten de gebruiker zou moeten uitvoeren om mogelijkerwijs tot een goed eindresultaat te komen. CASCADE-MUCH bevat twee soorten van dergelijk advies: suggesties en ‘previews’. Suggesties zijn gericht op vervolgactiviteiten, previews bieden inzicht in de manier waarop de huidige keuzes het ontwerp zullen beïnvloeden. De hulpmiddelen in het programma ondersteunen de docent-ontwerpers bij het ontwikkelen van een instructie-

Page 231: computer support for multimedia curriculum design

Nederlandse samenvatting 219

scenario. Daartoe zijn de volgende hulpmiddelen opgenomen: Edit Panel, Word, een concept mapping tool, faciliteiten om via het web te communiceren, en vertaalfaciliteiten. De training in het programma is bedoeld om de taakuitvoering van de gebruiker te verbeteren. Het programma bevat twee soorten training: wizards en een tutorial.

Interface Bij het ontwerpen van de interface van CASCADE-MUCH is ervan uitgegaan dat het programma eenvoudig te leren en eenvoudig te gebruiken moet zijn. Hierna volgen de drie belangrijkste kenmerken van de interface. Ten eerste is de interface in hoge mate consistent. Enerzijds betekent dit dat het programma overeenkomsten vertoont met andere computerapplicaties (extern consistent). Anderzijds zijn de schermindelingen en de werking van het programma zelf zoveel mogelijk vergelijkbaar door het programma heen (intern consistent). Ten tweede is het programma flexibel. Doordat het programma is opgedeeld in verschillende onderdelen kunnen gebruikers, afhankelijk van hun behoeftes, op verschillende punten in het programma starten. Daarnaast kunnen gebruikers lineair, maar ook sprongsgewijs (met een ‘browser’) door het programma lopen. Ook kunnen de meeste acties geactiveerd worden met de muis of met het toetsenbord. Ten slotte werkt de interface ‘intuïtief’ doordat er veelvuldig gebruik is gemaakt van metaforen, zoals logo’s en iconen.

Scenario Het instructie-scenario is het eindresultaat van het gebruik van het programma CASCADE-MUCH. Het vervult twee doelen: i) vereenvoudigen van discussies tussen docent-ontwerpers en computerprogrammeurs; en ii) het ondersteunen van programmeurs bij hun programmeertaak. De scenario’s die met behulp van dit programma worden ontwikkeld hebben twee algemene kenmerken. Ten eerste bevat een scenario niet alleen de informatie uit de ontwerpfase, maar ook de verzamelde analyseresultaten betreffende het vak, de leerlingen en de ontwerpers. Dit biedt de programmeurs rijke informatie op grond waarvan ze kunnen discussiëren met de docent-ontwerpers en op basis waarvan ze hun programmeertaak kunnen oppakken. Ten tweede is het scenario helder gestructureerd en --door gebruikmaking van tabellen-- eenvoudig te lezen. Het scenario kan geëxporteerd worden naar Microsoft Word.

Page 232: computer support for multimedia curriculum design

220 Nederlandse samenvatting

Conclusies en aanbevelingen Op grond van de workshop met deskundigen op de ECNU en de beide afsluitende evaluatiestudies luidt de algemene conclusie dat CASCADE-MUCH valide en praktisch bruikbaar is. In deze paragraaf volgt een aantal aanbevelingen voor verdere ontwerp- en onderzoeksactiviteiten. Meer details zijn te vinden in hoofdstuk 6. De huidige versie van het programma CASCADE-MUCH bestaat uit een CD-ROM. Hoewel de evaluaties aantoonden dat dit redelijk praktisch was voor de beoogde gebruikersgroep, zou de praktische bruikbaarheid verder vergroot kunnen worden door een aantal kenmerken van internetapplicaties te integreren in het programma, zoals het aanbieden van links naar gerelateerde websites, communicatiefaciliteiten, mogelijkheden om de meest recente informatie en versies te verspreiden, etc. Het gebruik van CASCADE-MUCH is er uiteindelijk op gericht te komen tot een instructie-scenario. Hoewel gebruikers tijdens diverse evaluatierondes conceptversies van dergelijke scenario’s hebben ontwikkeld, is er tijdens dit onderzoek geen specifieke studie gemaakt van de kwaliteit van de scenario’s en de multimedia curricula die op grond daarvan zijn gerealiseerd. Het lijkt zinvol om in een vervolgonderzoek het gebruik van de instructie-scenario’s, de constructie en implementatie van de multimedia curricula en de resultaten die de leerlingen op grond van het gebruik behalen te evalueren. Ten slotte, in lijn met de andere CASCADE-onderzoeken zoals CASCADE (Nieveen, 1997) en CASCADE-SEA (McKenney, 2001), richtte dit onderzoek zich op specifieke aspecten van het ontwikkelproces --analyse en ontwerp-- van een specifiek soort curriculum --multimedia curriculum-- in een specifieke context --Shanghai (China). In de toekomst zullen er mogelijkerwijs ook aanvullende CASCADE-onderzoeken uitgevoerd worden die voortgaan op de zoektocht naar de mogelijkheden van computerondersteuning voor curriculumontwikkeling. Een goed voorbeeld daarvan is het in gang gezette onderzoek CASCADE-IMEI (Zulkardi, 1999). Door op deze manier verder te werken aan de CASCADE-onderzoekslijn kunnen de resultaten uit het CASCADE-MUCH-onderzoek nader toegepast, uitgewerkt en gevalideerd worden.

Page 233: computer support for multimedia curriculum design

221

Appendix A-1 Screen dumps of Main Frame (English and Chinese version)

Page 234: computer support for multimedia curriculum design

222 Appendix A-1

Screen #0: The cover

Screen #1: Language selection

Page 235: computer support for multimedia curriculum design

Screen dumps of Main Frame 223

屏幕 #0: 主封面

屏幕 #1: 语言选择

Page 236: computer support for multimedia curriculum design

224 Appendix A-1

Screen #2: The introduction to CASCADE-MUCH

Screen #3: The options for entering other parts

Page 237: computer support for multimedia curriculum design

Screen dumps of Main Frame 225

屏幕 #2: 系统介绍

屏幕 #3: 选择进入其他部分

Page 238: computer support for multimedia curriculum design

226 Appendix A-1

Page 239: computer support for multimedia curriculum design

227

Appendix A-2 Screen dumps of Designer's Aid (English and Chinese version)

Page 240: computer support for multimedia curriculum design

228 Appendix A-2

Screen #1: The analysis part

Screen #2: Subject matter analysis

Page 241: computer support for multimedia curriculum design

Screen dumps of Designer's Aid 229

屏幕 #1: 分析部分

屏幕 #2: 学科分析

Page 242: computer support for multimedia curriculum design

230 Appendix A-2

Screen #3: Grade and subject

Screen #4: Goals and usage

Page 243: computer support for multimedia curriculum design

Screen dumps of Designer's Aid 231

屏幕 #3: 年级与学科

屏幕 #4: 设计目标与使用方式

Page 244: computer support for multimedia curriculum design

232 Appendix A-2

Screen #5: Learner analysis

Screen #6: Designer analysis

Page 245: computer support for multimedia curriculum design

Screen dumps of Designer's Aid 233

屏幕 #5: 学习者分析

屏幕 #6: 设计者分析

Page 246: computer support for multimedia curriculum design

234 Appendix A-2

Screen #7: The analysis results

Screen #8: The design part

Page 247: computer support for multimedia curriculum design

Screen dumps of Designer's Aid 235

屏幕 #7: 分析结果

屏幕 #8: 设计部分

Page 248: computer support for multimedia curriculum design

236 Appendix A-2

Screen #9: Module selection

Screen #10: The module of content elaboration

Page 249: computer support for multimedia curriculum design

Screen dumps of Designer's Aid 237

屏幕 #9: 功能模块选取

屏幕 #10: 内容讲解模块

Page 250: computer support for multimedia curriculum design

238 Appendix A-2

Screen #11: Content selection

Screen #12: Content representation

Page 251: computer support for multimedia curriculum design

Screen dumps of Designer's Aid 239

屏幕 #11: 内容选取

屏幕 #12: 内容描述

Page 252: computer support for multimedia curriculum design

240 Appendix A-2

Screen #13: Content organization

Screen #14: Interface design

Page 253: computer support for multimedia curriculum design

Screen dumps of Designer's Aid 241

屏幕 #13: 内容组织

屏幕 #14: 界面设计

Page 254: computer support for multimedia curriculum design

242 Appendix A-2

Screen #15: Module organization

Screen #16: The end screen

Page 255: computer support for multimedia curriculum design

Screen dumps of Designer's Aid 243

屏幕 #15: 功能模块组织

屏幕 #16: 结束屏幕

Page 256: computer support for multimedia curriculum design

244 Appendix A-2

Page 257: computer support for multimedia curriculum design

245

Appendix A-3 Screen dumps of Edit Panel (English and Chinese version)

Page 258: computer support for multimedia curriculum design

246 Appendix A-3

Screen #1: Content selection and representation

Screen #2: Content organization

Page 259: computer support for multimedia curriculum design

Screen dumps of Edit Panel 247

屏幕 #1: 内容选取与内容描述

屏幕 #2: 内容组织

Page 260: computer support for multimedia curriculum design

248 Appendix A-3

Screen #3: The selection of interface style and template

屏幕 #3: 界面风格选取

Page 261: computer support for multimedia curriculum design

249

Appendix B Example of an instructional scenario

Page 262: computer support for multimedia curriculum design

250 Appendix B

Analysis Name: Biology Grade: Senior Grade 2 Goals: * Basic knowledge and skills learning * Higher-order skills improvement

Attitudes development

SU

B

J

E

C

T

Usage: * Individual learning for learners Collaborative learning for learners

* Collective teaching for teachers Age: 14 - 16 years Subject knowledge/skills: * They have related subject knowledge and skills Computer skills: * They can use computers very well * They can use windows * They can input Chinese by Pinyin

LE

A

R

N

E

R

S Preferred learning materials/activities: * Visual learning materials and active learning

Name: Huang yonghuang Name: Yang yongjian Address: North Zhongshan road 3663 200062, Shanghai CHINA

Address: North Zhongshan road 3663 200062, Shanghai CHINA

Tel.: 021-87654321 Tel.: 021-12345678

DE

S

I

G

N

E

R

S Others: Wang li, Yin lan Notes: 1. Those items marked by * are selected. 2. This page provides computer programmers with background information. 3. The scenario file name is: D:\cscd\scenario.doc

Page 263: computer support for multimedia curriculum design

Example of an instructional scenario 251

Design Modules Selection: The following module(s) with * is (are) to be included in the intended multimedia curriculum: Goal Content preparation * Content elaboration Practice Assessment

Notes: 1. Above lists all modules might be included in a multimedia curriculum, of which marked with * is (are) to be included in the multimedia curriculum.

2. The selected module(s) will be described in detail on the following pages.

Page 264: computer support for multimedia curriculum design

252 Appendix B

Module: Content elaboration

Interface template (Modern & Popular):

Attributes: Elements on the interface:

Title, text, picture, background picture and icons Common properties:

The background picture is: transparent tree leaf No background music No button(s)

Content organization: Among units: Hyperlink Among units, sections, and chapters: Linear and menu

Page 265: computer support for multimedia curriculum design

Example of an instructional scenario 253

Component: Content Elaboration

Code: 1.1.1 Unit: Function of breath Section: 1.1 Chapter: 1 Course: S2: Biology Title: Function of breath Text: The function of breath is a kind of decomposition reaction, Which

decomposes organic compounds and releases energy. It has two categories: oxidation breadth and non-oxidation breath.

Graphics (Including a theme picture and a background picture): Description (theme picture): The theme picture is a cluster of green trees. File (theme picture): The file name of the theme picture is d:\cscd\tree.bmp Description (B-Picture): The background picture is transparent leafs. File (B-Picture): The file name of the background picture is

d:\cscd\leafs.bmp Audio (Including narration, sound, and background music): Description (narration): Reading aloud the text on the screen … File (narration): The file name of the narration is d:\cscd\breath.wav. Description (Sound): <No sound> File (Sound): <No sound file> Description (B-Music): <No background music> File (B-Music): <No music file> Motion (Including animation and video): Description (Animation): Illustrate the process of breath… File (Animation): The file does not exist. So the animation needs to be made. Description (Video): < No video > File (Video): < No video file > Organization (Including buttons, keywords in context, and related topics): Button(s): <No buttons> Keyword(s): There are two keywords: oxidation breadth and non-

oxidation breadth Related topic(s): Photosynthesis

Notes: I don't have any other notes for this knowledge point.

Page 266: computer support for multimedia curriculum design

254 Appendix B

Page 267: computer support for multimedia curriculum design

255

Appendix C Interview topic list (Used during the second round of prototyping)

Page 268: computer support for multimedia curriculum design

256 Appendix C

Date: ______________ Name: _____________ Position: Subject teacher (ST)

The subject is: _____________ Curriculum or instructional specialist (C/I) Computer-based learning experts (CBL) Computer programmer (CP) Others: ______________

Note: The first part of the interview topic list involves walkthrough questions to be answered by the participants when they are walking through the prototype or watching a demonstration. The second part of the interview topic list includes a list of follow-up questions to be discussed after the participants have walked through the prototype or watched a demonstration.

Page 269: computer support for multimedia curriculum design

Interview topic list 257

Part I. Walkthrough questions 1. Do you agree with the definition of multimedia curriculum? (For C/I and

CBL) Multimedia curriculum is a plan for learning in multimedia formats, which covers and extends the most content of a conventional curriculum. Interaction and flexibility are two outstanding characteristics of a multimedia curriculum.

2. Do you agree with the goal options of a multimedia curriculum? If not,

what goals should be added/removed? (For C/I, CBL and STs) Three goal options: 1. Basic knowledge and skills learning 2. Higher level skills improvement 3. Attitudes development

3. Do you agree with the possible usage of a multimedia curriculum? If not,

what usage should be added/removed? (For C/I, CBL and STs) Three kinds of usage of a multimedia curriculum: 1. Individual learning for learners 2. Collaborative learning for learners 3. Collective teaching for teachers

4. Do you agree with the learner analysis? (For C/I, CBL and STs)

In the prototype, learner analysis focuses on computer skills of learners, which include: 1. Computer use skills 2. Windows use skills 3. Chinese processing skills

Page 270: computer support for multimedia curriculum design

258 Appendix C

5. Do you think the wizards are useful for performing tasks? (For CBL and STs) In the prototype, several wizards are included for content selection, description, organization and interface design.

6. Do you like the examples for keywords? (For C/I, CBL and STs)

In the prototype, each keyword has an example, displayed either in window’s help or in external application format.

7. Do you have any beliefs for multimedia curriculum design? (Only for STs)

Designer’s beliefs may include objectivist and constructivist. Objectivist believes that knowledge is objective, and it can be transferred from teachers to learners, whereas contructivist believes that knowledge is constructed by learners themselves.

8. Do you agree with the multimedia curriculum modules? If not, what

modules should be added/removed? (for C/I, CBL and STs) A multimedia curriculum may include any of the following modules: Content elaboration, Practice, Test, Resources/Materials, Experiment, Simulation/ Virtual reality, Case studies/Problem solving, Communication tools

9. Do you agree with the practical guidelines for content selection? If not, what should be added/removed? (For C/I, CBL and STs) Content selection has three practical guidelines: 1. Basic but difficult 2. Often included in tests 3. Learners interested

Page 271: computer support for multimedia curriculum design

Interview topic list 259

10. Do you agree with the theoretical models for content selection? If not, what should be added/removed? (Only for C/I) Two theoretical models are selected for content selection: Tyler’s learning experience selection model and Posner and Rudnitski's ILOs selection model.

11. Do you agree with the models for content description/media selection? If not, what should be added/removed? (Only for C/I) Two models are selected for content description/media selection: Romiszowski’s media selection model and Allen’s media selection model.

12. Do you agree with the three content organization manners? (For C/I, CBL

and STs) Content organization has three manners: 1. Linear 2. Hierarchical 3. Hyperlink

13. Do you agree with the three interface styles? If not, what styles should be added/removed? (For CBL and STs) Three interface styles: 1. Classic and elegant 2. Modern and popular 3. Vivid and vigorous

Page 272: computer support for multimedia curriculum design

260 Appendix C

Part II. Follow-up questions

For STs: 14. Do you have any unclearness, difficulties, and /or expectations?

15. Do you think that the prototype fits your subject and your learners?

16. Have you learned something from the prototype? If yes, what?

17. Do you want to use it when you design a multimedia curriculum? If not, why?

18. Others:

For C/I specialists: 19. Do you think the content is helpful for designing a multimedia curriculum

as well as for improving professional knowledge for multimedia curriculum design? If not, why?

20. Do you think what modifications should be made?

21. Others:

__________________________________________________________

Page 273: computer support for multimedia curriculum design

Interview topic list 261

For CBL experts: 22. Do you think the interface design is consistent and easy to use? If not,

what should be changed?

23. Do you think the support is useful? What should be added/removed?

24. Do you think the scenario is acceptable? If not, what should be changed?

25. Others:

For CPs: 26. Is the scenario clear for you? If not, what are the difficulties?

27. What do you expect the scenario to be added/removed?

28. Do you think the scenario can be implemented by programming? If not, why?

29. Others:

Page 274: computer support for multimedia curriculum design

262 Appendix C

Page 275: computer support for multimedia curriculum design

263

Appendix D Instruments for the micro evaluation (Used during the third round of prototyping)

Page 276: computer support for multimedia curriculum design

264 Appendix D

Questionnaire for subject teachers Name: _____________________________ Gender: _____________________________ Age: _____________________________ School: _____________________________ Subject: ❑ Biology ❑ Geography Experience: ❑ I have experience for multimedia curriculum design ❑ I have no experience for multimedia curriculum deign 1 2 3 4 1. I learned some information for multimedia

curriculum design from the prototype. ❑ ❑ ❑ ❑

2. The interface was easy to use. ❑ ❑ ❑ ❑ 3. Errors did not often occur. ❑ ❑ ❑ ❑ 4. The tools (advice, prediction, help, tips, etc.) on

the toolbar were useful. ❑ ❑ ❑ ❑

5. The prototype was helpful for me to make scenarios.

❑ ❑ ❑ ❑

6. I am satisfied with the scenario just made. ❑ ❑ ❑ ❑ 7. The functionality and structure of the prototype

were easy to understand. ❑ ❑ ❑ ❑

8. I like to make scenarios by using the prototype rather than on papers.

❑ ❑ ❑ ❑

Note: 1 = disagree 2 = slightly disagree 3 = slightly agree 4 = agree

Page 277: computer support for multimedia curriculum design

Instruments for the micro evaluation 265

9. What are the critical difficulties you met when you were using the

prototype to make a scenario? 10. What characteristics of the prototype do you like most? 11. Which aspects of the prototype need improving? Interview questions for computer programmers 1. Do you think the scenarios are easy to understand? Which parts are easier

to understand, and which parts are more difficult to understand? 2. Do you think the scenarios include all information you need? If not, what

information should be added? 3. Why did you select that scenario? 4. Which parts of the scenario do you think need improving? Note: - Question 1 and 2 are used after computer programmers preview the scenarios - Question 3 and 4 are used after computer programmers finish their programming

Page 278: computer support for multimedia curriculum design

266 Appendix D

Page 279: computer support for multimedia curriculum design

267

Appendix E Instruments for the expert appraisal at the UT (Used during the third round of prototyping)

Page 280: computer support for multimedia curriculum design

268 Appendix E

Time: April 13 (Thursday), 14:00-16:00 Place: Mac-room (L102), TO Procedure and activities: 1. Introduction of the workshop, including aims, procedure, and time

schedule (5 minutes); 2. Presentation of CASCADE-MUCH with PowerPoint (20 minutes); 3. Hands-on experiences. Participants will use the prototype by following a

pre-structured routine (60 minutes); 4. Discussion. Discussions will be carried out around the predetermined

discussion theme in Part C (20 minutes); 5. Answering the questions in Part D (15 minutes). Organizers: Qiyun Wang, Nienke Nieveen, and Jan van den Akker Participants: Eight experts participate in the workshop. They are: Two multimedia experts: Koos Winnips Marinka Sysling

Three instructional design experts: Ton de Jong Ard Lazonder Mark Gellevij

Three curriculum design experts: Wilmad Kuiper Ellen van den Berg Annette Thijs

(If you are a multimedia expert, please go on with Part A; otherwise go on with Part B)

Page 281: computer support for multimedia curriculum design

Instruments for the expert appraisal at UT 269

Part A: The route of hands-on experience for multimedia experts The main activity of multimedia experts is to walk through the "Edit Panel" part, during which they can assess validity and perceive practicality of the content, interface, and scenario of the prototype. You may follow the following steps on the left column, and give short answers to the questions on the right column. Steps Questions 1. Run the main program by double clicking the "Main"

icon on the desktop.

2. On the first screen of the main program, select "English" language.

3. On the second screen, click the "Help" button, and click the keyword "multimedia curriculum" on the follow-up help file (➔ answer question 3a).

3a. Is the definition of "multimedia curricu-lum" accurate?

4. On the third screen, select the second item ("I don't need too much help, I want to design directly"), and click "Action" button to enter "Edit Panel" part after a few seconds.

(Now it is in the Edit Panel part) 5. Content selection and representation

- Select a knowledge unit On the left-side curriculum standard, knowledge units with √ are selected. You can use the left mouse button to locate a (new) knowledge unit, and use the right mouse button to (un)select it (➔ answer question 5a).

- Specify screen element properties for the selected knowledge unit After selecting a knowledge unit, you can specify its screen element properties. Only the elements with green spots are enabled. If you want to add/remove screen elements, click the "Properties" button on the right side (➔ answer 5b).

5a. Do you think it is easy to (un)select knowledge units for intended target users?

5b. Do you think it is easy to specify screen element properties for users?

(To be continued at the next page)

Page 282: computer support for multimedia curriculum design

270 Appendix E

Steps Questions 6. Content organization

- Specify content organization manners for the selected knowledge unit(s) by choosing organiza-tion styles among units, and/or among chapters, sections, and units.

- Do not press the "Flowcharting" button since the flowchart tool is not installed here (It will be used by the intended target users and computer programmers together).

- Click the "Help" button to get explanations and examples of organization styles (➔ answer 6a).

6a. Do you think these three content organization manners are valid?

7. Interface design - Select which interface style you want to use for

the knowledge unit. - Select what screen elements will be included on

the interface. - Set attributes for some common elements (back-

ground picture, background music, and buttons. ➔ answer 7a).

- Click the "Help" button to get explanations and examples of the interface styles (➔ answer 7b).

7a. Do you perceive it is easy to select an interface and set attributes for intended target users?

7b. Do you think the interface styles are representative?

8. If you want to select more than one knowledge unit, repeat the step 5 (it is not necessary to repeat the steps 6 and 7 because content organization and inter-face serve for all knowledge units).

9. Save the scenario by clicking "File" and "Save". 10. Export the scenario into a WORD document format

by clicking "File" and "Export…" (➔ answer 10a). 10a. Is it easy to export a

scenario? 11. Check the WORD document. If you are not satisfied

with the scenario, go back to the "Edit Panel" part, modify and export again (Note: do not close the WORD application and WORD document before exiting "Edit Panel" part. ➔ answer 11a).

11a. Is it easy to modify a scenario?

12. Save the WORD document (Go on with Part C)

Page 283: computer support for multimedia curriculum design

Instruments for the expert appraisal at UT 271

Part B: The route of hands-on experience for curriculum and instructional (C/I) experts The main activity of curriculum or instructional experts is to walk through the Designer's Aid part, during which they can assess validity and perceive practicality of the content, support, and interface of the prototype. You may follow the following steps and give short answers to the questions. Screen number

Steps

Questions

1. Run the main program by double clicking the "Main" icon on the desktop.

#1 2. Select "English" language. #3 3. Select the first item ("I need help, please

guide me through the design step-by-step"); click the "Action" button to go to the Designer's Aid part after a few seconds.

1/16 (found in the left-bottom corner)

(Now it is in the Designer's Aid part) 4. Click the keyword "multimedia

curriculum" (➔ answer question 4a).

4a. Do you think the definition of "multi-media curriculum" is accurate?

5. Click the "Next" button twice to go to the third screen.

3/16 6. Click the "Suggestion" button (or press F2) and "Prediction" button (or press F3) on the toolbar (➔ answer question 6a).

6a. Do you think the 'suggestion' and 'prediction' tools are useful for intended target users?

4/16 7. Click the keywords of "Goals" and "Usage" (➔ answer 7a); Click the "Help" button on the toolbar (or Press F1, ➔ answer 7b).

7a. Do you agree with the goals and usage?

7b. Do you perceive the models will be useful for users' professio-nal development?

5/16 8. Look at the three aspects of learner analysis (➔ answer 8a)

8a. Do you think the three aspects are valid for learner analysis?

(To be continued at the next page)

Page 284: computer support for multimedia curriculum design

272 Appendix E

Screen number

Steps

Questions

6/16 9. Mouse move on the acronyms "SMTs" will get complete words.

9/16 10. Click some of the seven components (➔ answer 10a); Click the "Suggestion" button on the toolbar (or press F2) to get an explanation why those components are checked or selected now.

10a. Do you think the seven components are valid for a multimedia curriculum?

11/16 11. Click the two approaches and two criteria of content selection to get explanations (➔ answer 11a); (Don't click the "Mapping" button, because the mapping tool is not installed.) Click "Next" button, now a confirm box will pop up, press "Yes" to go on to the next screen.

11a. Do you think the approaches and criteria are practical for users?

12/16 12. Click the four representation forms and four criteria of content representation to get explanations (➔ answer 12a); When a confirm box appears after you click the "Next" button, press "Yes".

12a. Do you think the representation forms and criteria are practical for users?

13/16 13. Click at least one organization manner (i.e. hyperlink) to see its explanation and example (➔ answer 13a). When a confirm box appears after you click the "Next", press "Yes".

13a. Do you agree with the classification of organization manners?

14/16 14. Click at least one interface style to see its explanation and example (➔ answer 14a).

14a. Do you agree with the classification of interface styles?

Page 285: computer support for multimedia curriculum design

Instruments for the expert appraisal at UT 273

Part C: Discussion theme Y X Validity Practicality

Content ? ?

Interface ? ?

Designer's Aid

(C/I experts) Support ? ?

Edit Panel

(multi-media experts)

Scenario ? ?

Validity refers to the extent that the design of the prototype is based on the state-of-the-art knowledge ('content validity') and the various parts of it are consistently linked to each other ('construct validity'). Practicality refers to the extent that users (and other experts) consider the system as appealing and usable in Shanghai context. Do you think that X meets the criteria of Y? If not, what characteristics of X should have? Part D: Questions: 1. Do you think that the prototype is useful for intended target users (subject

matter teachers) to make scenarios? 2. Do you think that the prototype can improve users' professional

knowledge for multimedia curriculum design (Only for C/I experts)? 3. Do you think that the scenarios produced by the prototype can provide an

easy way for interactive discussions between teacher-designers (subject teachers) and computer programmers?

4. Do you think that scenarios produced by the prototype are helpful for

computer programmers to create multimedia curricula? 5. Comments, suggestions and conclusions:

Page 286: computer support for multimedia curriculum design

274 Appendix E

Page 287: computer support for multimedia curriculum design

275

Appendix F Questionnaire for the assessment studies

Page 288: computer support for multimedia curriculum design

276 Appendix F

The questionnaire includes two parts. The first part consists of some Likert scale questions, and the second part consists of some open-ended questions. Name: __________________ (Optional) Part I: Likert scale questions The numbers in the right columns of the following tables mean: 1: disagree 2: slightly disagree 3: slightly agree 4: agree

Practicality of content 1 2 3 4 1. I could easily understand the content on each screen. 2. The content fits my practical needs for multimedia

curriculum (or learning materials) design. 3. I could easily understand the explanations of

keywords/models. 4. I learned some useful information from CASCADE-

MUCH.

Explanatory notes:

Page 289: computer support for multimedia curriculum design

Questionnaire for the assessment studies 277

Practicality of support 1 2 3 4 5. The help function provided me with useful models. 6. The suggestions gave me some valuable expert advice. 7. The previews indicated to me what would be affected

by the current settings. 8. The additional explanations of suggestions/previews

helped me understand why the suggestions/previews were given.

9. The tips provided me with useful information. 10. The Edit Panel helped me easily make instructional

scenarios. 11. I could easily export scenarios into Microsoft Word. 12. I think the explanations/examples of keywords are

practical. 13. The concept-mapping tool is useful for me to make

content selection.

Explanatory notes:

Page 290: computer support for multimedia curriculum design

278 Appendix F

Practicality of interface 1 2 3 4 14. User tasks on each screen are clear to me. 15. The meanings of buttons on each screen are clear to

me. 16. I like the fonts and colors on each screen. 17. The navigation tools (linear and browser) are easy for

me to use. 18. I think the amount of information on each screen is

proper. 19. I think each screen has a consistent design. 20. I believe that the interface is consistent with other

computer programs. 21. The interface is easy to learn. 22. The interface is easy to use. 23. I feel the program is error free.

Explanatory notes:

Page 291: computer support for multimedia curriculum design

Questionnaire for the assessment studies 279

Practicality of scenarios 1 2 3 4 24. The scenario includes what I intended to include. 25. The scenario is well structured. 26. I can easily modify the scenario within Microsoft

Word. 27. I am satisfied with the produced scenario. 28. The meanings of the elements in the scenario are clear

for me. 29. The produced scenario can help me easily discuss my

wishes with computer programmers. 30. I believe that the scenario would be easy to understand

for computer programmers.

Explanatory notes:

Page 292: computer support for multimedia curriculum design

280 Appendix F

Part II: Open-ended questions 1. Compared to other approaches to instructional scenario making, such as

writing on papers, what advantages and disadvantages do you think CASCADE-MUCH has?

2. Do you think how CASCADE-MUCH should be used in practice in order

to be more effective considering these advantages/disadvantages? 3. Would you be interested in future use of CASCADE-MUCH? Why or why

not? 4. In your opinion, how could the program of CASCADE-MUCH be

improved? 5. Other comments or suggestions (use the reverse paper if necessary)

Page 293: computer support for multimedia curriculum design

281

Appendix G Example of interface styles

Page 294: computer support for multimedia curriculum design

282 Appendix G

Classic and elegant

Modern and popular Vivid and vigorous

Page 295: computer support for multimedia curriculum design

283

Appendix H Original data collected from the assessment studies

Page 296: computer support for multimedia curriculum design

284 Appendix H

Study 1 (n=6) Study 2 (n=13)

Novice Designers (n=4)

Experienced designers (n=2)

Other users

S1 S2 S3 S4 S1 S2 S3 S4 S1 S2 S3 S4

Q1 1 2 1 2 5 8

Q2 1 3 1 1 1 1 6 5

Q3 3 1 2 2 5 6

CO

N

T

E

N

T Q4 1 2 1 2 1 1 4 7

Q5 1 2 1 2 1 1 7 4 Q6 3 2 1 8 4 Q7 1 1 1 2 1 8 4 Q8 1 1 1 2 1 7 3 Q9 1 2 2 6 6 Q10 4 2 1 4 7 Q11 1 3 2 2 2 9 Q12 1 2 1 2 1 4 7

SU

P

P

O

R

T

Q13 3 2 1 4 7 Q14 2 2 2 2 7 4 Q15 3 1 2 1 7 5 Q16 1 1 2 1 1 1 4 6 2 Q17 1 3 2 8 3 Q18 1 3 1 1 1 9 3 Q19 1 3 2 1 1 6 4 Q20 1 3 1 1 3 8 1 Q21 3 1 2 7 6 Q22 2 2 2 7 6

IN

T

E

R

F

A

C

E

Q23 2 1 1 2 2 3 5 1 Q24 1 1 2 1 1 3 4 4 Q25 1 3 1 1 1 6 5 Q26 2 2 2 7 5 Q27 1 3 2 1 6 4 Q28 1 2 1 2 7 5 Q29 3 1 2 10 2

SC

E

N

A

R

I

O Q30 2 2 2 3 6 3 Note: Q1..30 = Likert scale questions 1 to 30; S1..S4 = Scales 1 to 4