From Expectation to Evaluation of E-Learning - Experience from the Water Education E-Learning
Project -
Hsin-Yu Shan
Dept. of Civil Engineering, National Chiao Tung University, Taiwan
Jen-Gaw Lee
National Center for High-Performance Computing, Taiwan
Frances Su
Microelectronics and Information Systems Research Center, National Chiao Tung University, Taiwan
Abstract
While the rapidly development of information and communication technology constantly provides innovative tools and methods of education, the real world has not been able to catch up. Traditional face to face teaching and training still prevails in schools and in the work place. Although many forms of e-learning courses have been experimented in schools and delivered by global enterprises for on-the-job training, analysis on the difference between expectation of the participants before the courses and satisfaction afterwards is scarce. Questions on what the participants of e-learning programs expect have seldom been asked. Furthermore, little information is available on comparing the evaluation of the e-learning program afterwards and the expectation before it began. This paper presents the results of a study accompanying a large-scale e-learning program for continued education of thousands of school teachers in Taiwan. The expectation before beginning the asynchronous on-line courses and evaluation of the program and their own performance were collected by compulsory questionnaires. The results showed that most of the participants think highly of the e-learning program and considered the program delivered the knowledge and skills as expected. However, the interaction among participants and between instructors and participants was less satisfactory as other aspect of the e-learning program also as the participants expected. Nevertheless, e-learning remains to be the most efficient way to deliver high-quality education programs to a large number of participants. In addition, observation of the participants’ behavior reveals that good management of e-learning programs is labor-intensive.
Keywords: Distance learning, web-based, asynchronous, water environment.
Introduction The rapid development of information and communication technology (ICT) has not only
facilitated classroom learning tremendously but is also increasingly enabling education to move outside of the classrooms and even schools. The most up-to-date “ubiquitous learning
environment” (Shankar et al., 2001) enables learning at anytime and anywhere such that the whole paradigm of learning could be revolutionized.
However, research to date suggests that traditional classroom education results in a more favorable learner response than e-learning. This issue represents a perplexing problem for proponents of e-learning. It also raises the question of whether evaluation of e-learning compared with traditional learning should be the real issue or is it evaluation of e-learning within itself?
On the other hand, the design of e-learning programs have benefited from the use of instructional system design models very early on as many e-learning programs involved educational research. Thus in many cases, e-learning programs are better structured and managed than traditional education programs, which were not designed and implemented with systematic approach.
Instructional design is defined by Berger and Kam (1996) as ‘the systematic development of instructional specifications using learning and instructional theory to ensure the quality of instruction. Instructional system design (ISD) evolved from post-World War II research in the United States military to find a more effective and manageable way to build training programs. Numerous ISD models have been developed over the years, and they are also known as Instructional Systems Design and Development (ISDD), the Systems Approach to Training (SAT), or just Instructional Design (ID). These include ADDIE, Dick and Carey (1978), Hannifen and Peck (1988), Knirk and Gustafson (1986), Kemp (1985), and Gerlach and Ely (1980) to name a few. These models provide a step-by-step system for the evaluation of participants’ needs, the design and development of training materials, and the evaluation of the effectiveness of the training program. Almost all of these models are based on the generic ADDIE model while making modifications to emphasize a more holistic, iterative approach to the development of training. The five generic phases of ADDIE are: 1. Analyze, 2. Design, 3. Develop, 4. Implement, and 5. Evaluate. These phases provide dynamic and flexible guidelines which are used for effective and efficient instruction.
The cycle (Fig. 1) starts with the analysis as the designer develops a clear understanding of the desired outcomes or behaviors, and the participants’ existing knowledge and skills. The design phase documents specific learning objectives, assessment instruments, exercises, and content. The creation of learning materials is completed in the development phase. During implementation, these materials are delivered or distributed to the participants. And finally the effectiveness of the training program is evaluated.
Figure 1 The ADDIE Model
Analyze
Design
Develop
Implement Evaluate
On the other hand, for implementing education programs, Gagne et al. (1992) introduced
the concept of nine events of instruction and their corresponding cognitive processes (Table 1). The nine events are excellent steps that the design of e-learning programs can easily follow.
Table 1: Gagne’s Nine Instructional Events
1. Gaining attention (reception) 2. Informing learners of the objectives (expectancy) 3. Stimulating recall of prior learning (retrieval) 4. Presenting the stimulus (selective perception) 5. Providing learning guidance (semantic encoding) 6. Eliciting performance (responding) 7. Providing feed back (reinforcement) 8. Assessing performance (retrieval) 9. Enhancing retention and transfer (generalization)
Evaluation approaches based on the program goals are fundamentally oriented towards the
verification and quantification of the execution of the program goals. For this purpose, experimental or quasi-experimental research models are used in a 4-step approach: (1) Identification of the program goals, (2) Translation of the program goals into quantitative data, (3) Data collection from individuals related to the program, and (4) Comparison of stated goals with attained goals.
As indicated by Figueira (2003), there are five major clusters of variables that affect the outcome of e-learning: individual learner variables, environmental variables, technology variables contextual variables and pedagogic variables. Each of these can be disaggregated into more precise groups (Table 2).
Table 2 Variables of E-Learning Individual learner variables include:
Physical characteristics (e.g. age, sex, physical abilities) Learning history, (negative/positive experience, level of attainment, duration, recentness etc)
Learner attitude (positive / negative) Learner motivation (high / low) Familiarity with the technology
Learning environment variables include: The immediate (physical) learning environment The organizational or institutional environment The subject environment
Contextual variables include: Socio-economic factors (e.g. Class, gender,) The political context (e.g. Who is funding /paying for the e-learning and for what reason?)
Cultural background (e.g. How highly is learning / e-learning valued?) Geographic location (e.g. Country, language, urban/rural)
Technology variables include: Hardware Software, Connectivity,
The media Mode of delivery
Pedagogic variables include: Level and nature of learner support systems Accessibility issues. Methodologies Flexibility Learner autonomy Selection and recruitment Assessment and examination Accreditation and certification
This paper describes the results of a study conducted on the participants of the Water
Environment Education Project. The project implanted a large-scale e-learning continued education program for school teachers in Taiwan. It is the largest e-learning projects in Taiwan on a single subject and specific group of participants. The design and evaluation of the e-learning program followed the ADDIE model and other principles outlined in previous paragraphs.
The Water Education Project The Water Environment Education Project is sponsored by the Ministry of Education, the
Environmental Protection Agency, and the Water Resources Agency. The objective of the project is to enhance the knowledge and skill of school teachers to integrate water environment education into day-to-day classroom teaching in all subject areas in elementary schools and high schools. As a matter of fact, Taiwan has been promoting environmental education in schools for over a decade. However, the transfer of knowledge and attitude on sustainable development and water environment had not been as successful due to the difficulty to train more than 200,000 school teachers in the entire nation with good instructors and high quality course contents.
Fortunately, Taiwan is not only a country that produces most of the notebook computers and internet networking equipment, but also has a high household broadband internet connection percentage of 65% (Taiwan Network Information Centre, 2006). As a result, Taiwan already has a mature environment for wide application of e-Learning. Thus, the delivery of high quality water education training courses to thousands of teachers all over the island in a short time can rely on no other method than distance learning.
The project was initiated under the leadership of Taiwan International Institute for Water Education (TIIWE) and the work team consists of Graduate Institute of Environmental Engineering of National Taiwan University, ITS EduMedia Studio of Microelectronics and Information Systems Research Center of National Chiao Tung University (NCTU), Chinatimes Cultural and Educational Foundation, National Center for High-Performance Computing (NCHC) and Environment Conservation Center of National Taiwan Normal University. The training program is co-sponsored by the Ministry of Education, the Environmental Protection Agency, and the Water Resources Agency. The target participants of the water education training program described in this paper are the 200,000 teachers from elementary schools, junior and senior high schools, and vocational schools.
The course topics not only covers science, ecology, engineering knowledge on water environment but also include social, political, and economical issues surrounding water resources development and water environment protection. After two years of carrying out the
project, four modules each with 10 1.5-hour asynchronous classes are on-line. During 2004, 2005, 2006, a total of 5,617 people registered to participate and 3,940 participants finished at least one of the modules. The total number of modules taken by the participants was more than 15,000. The participants were awarded certificate for the modules and hours they completed.
The Web-Site and the E-Learning Program
Course Content and Research Process
In the first year, the e-learning program provides 20 1.5-hour courses on fundamental water education. In the second year, 20 advanced courses were added to the program. In addition to the on-line courses, there is an accompanying one-day face-to-face learning session which includes half day of classroom discussion and half day of field trip available for the more active participants to attend. The overall course structure and classes are listed in Table 3.
Table 3 List of On-Line Courses on Water Education Program
Fundamental Water Education Courses Advanced Water Education Courses Module A Module C Theme: Humanistic Aspect of Water Environment Theme: Humanistic Aspect of Water Environment A01 The Earth Environment and Sustainable Development: The Continuation of Civilization
C01 Water Environment and Civilization
A02 The History of Water Resources Development C02 Global Environmental Crisis A03 The Evolution of Environmentalism, Civil Movement, and International Environmental Treaties
C03 Biodiversity and Preservation of Taiwan’s Ocean Environment
A04 Impact of Economical Development on the Environment
C04 Sustainable Development and International Cooperation
A05 Governance of Water Environment – Legal and Economic Control
C05 Environmental Economics
Theme: Ecology of Water Environment C06 Oceanic Heritage and Expeditionary Spirit A06 Ocean and Coastal Ecology C07 Aboriginal Heritage Culture and Environment
Preservation A07 Ecology of Lakes and Reservoirs Theme: Environmental Education A08 Investigation Methods of Water Environment C08 Environmental Organizations and Environmental
Movements A09 River Ecology C09 Objectives and Strategies of Environmental
Education A10 Evolution of Taiwan’s Environment C10 Environmental Education Teaching
Methodology –Water Environment Themes Module B Module D Theme: Water Environment and Earth Science Theme: Water Environment and Management B01 The Atmospheric System and Global Climate Change
D01 Characteristics of River Environment and Its Evolution
B02 Hydrologic Cycle (I) – Atmosphere and Ocean D02 Crisis of the River Environment B03 Hydrologic Cycle (II) – Surface Water and Ground Water
D03 River Engineering and Management
Theme: Water Environment and Hydraulic Engineering
D04 Stories of Successful River Restoration Projects
B04 River Engineering and Flood Protection D05 Geographical Characteristics and Change of Taiwan’s Coastal Zone
B05 Water Resources Development, Conservation and Water Supply
D06 Development and Management of Coastal Zones
Theme: Water Environment Pollution and Natural Hazards
D07 Taiwan’s Groundwater Resources and Quality
B06 Surface Water Pollution and Prevention D08 Soil and Groundwater Pollution and Protection B07 Ocean Pollution And Prevention Theme: Environmental Education B08 Soil and Ground Water Pollution and Prevention D09 Sustainable Campus (I) – Sustainable Campus
Initiative and Program B09 Natural Hazards of Water Environment and Mountainous Regions and Hazard Mitigation
D10 Sustainable Campus (II) – Concept of Sustainable Campus Architecture
B10 Environmental Course Design and Teaching Skills The instructors include university professors, senior water managers and engineers of
government agencies. The experts of different domains work together smoothly. Coordination of instructors is often the most difficult task of managing a training program; however, the group of instructors worked together in harmony and delivered a course program with a perfect mix of key elements of water education.
The e-learning program was designed to incorporate studies on e-learning relative issues with the participants as the subjects. Since the most of participants are school teachers who are required to finish 20 hours of continued education each year, this program becomes one of the best way of the teachers to fulfill this requirement. As a result, the participants are included as the subjects of research in this project. The participants are required to finish questionnaires and tests before and after the on-line courses in order to be cleared as completing the course program and be eligible to receive formal certificate. The questionnaires and tests are built as web pages such unless the participants finished answering the questions, they are not allow to go into next steps to view on-line courses or receive final proof of completion. The flow char of the process is shown in Fig 2. The process also included the nine instructional events to facilitate a successful e-learning program.
Figure 2 E-Learning and Investigation Process
Registration
On-Line Welcome Ceremony
On-Line Introduction of LMS and Course Program
Questionnaire on Background and Expectation
Pre-Test
On-Line Course
On-Line Course
Quiz Quiz
Post-Test
Questionnaire on Program and Self Expectation
Field TripF2F ClassDiscussionWorkshop
In order to enhance the achievement of learning, the following policies are designed and applied:
40 courses are organized into 4 modules, each module contains 10 courses. Each student has to enroll at least 1 category. The certificate is only awarded to the participants after completion of modules. The minimum hours on the certificate is thus 15 hours. For each category
Each week two new e-learning courses were put on-line. Each course is open for 2 weeks.
There were also optional one-day face to face course and field trip available to the more active participants. These face to face sessions provide an opportunity to enhance interaction between instructors and participants and among themselves.
The Website and the LMS
The on-line course is accessed through the water environment e-learning website hosted in NCHC with the URL http://water.emc.nctu.edu.tw. In addition to the on-line courses, the website also contains other crucial components for e-learning: announcement board, concise articles on various topics regarding water environment, discussion boards, assorted and annotated internet links to environmental education related websites, and participants’ email list. The learning management system (LMS) is developed by ITS EduMedia Studio and fully conforms to SCORM standards.
Figure 3 Sample Screen of the On-Line Course
The courses were produced using Microsoft® Producer to display video of lecturing and
Powerpoint presentation at the same time. Figure 3 is a screen shot from the on-line lecture on “Earth Environment and Sustainable Development.” According to the course content, each 1.5-hour course was divided into sections of about 15 to 20 minutes which is the time span the participants can maintain highly concentrated. This short time section also allowed the participants to fully utilize fragmented time between their classes and other activities in schools or at home. All videos are encrypted by Microsoft® DRM (Digital Rights Management) technology to protect intellectual property rights. Microsoft® media servers were deployed to provide video 150 Kbps course video streaming to participants’ computers which is connected through TANet (Taiwan Academic Network) or ADSL.
Results of the Training Program and the Study
Participants’ Background Analysis A total of 4 terms of e-learning were delivered from 2004 through 2006. The numbers of
participants registered and completed at least one module are listed in Table 4. The dropout rate significantly decreased over time. In order to evaluate the quality of on-line courses and the participants’ responses on the e-learning model, the participants were asked to complete a background survey before the courses and an evaluation survey after completion of the courses.
Table 4 Number of Participants of the Program 2004 2005 1 2005 2 2006
Registered 1305 1252 1392 1668 Completed 702 915 987 1336 Dropout Rate 46.2% 26.9% 29.1% 19.9%
From the age distribution of participants shown in Fig. 4, it is obvious that this program
attracts much higher percentage of younger teachers as compared with statistics on teachers nationwide (MOE, 2006). About 80% participants are between 20 and 40. It is also important to note that approximately 82% of the participants never participated in any education or training programs on water environment and 68% had no previous e-learning experience. However, most of the participants are already frequent computer users where more than 80% of the participants use computers at least 5 hours per week (Fig. 5).
0%
5%
10%
15%
20%
25%
30%
35%
40%
45%
<20 21-30 31-40 41-50 >50
ParticipantWhole nation
Figure 4 Comparison of Age Distribution of the Participants and All Teachers in Taiwan
0% 5% 10% 15% 20% 25% 30% 35% 40%
Seldom (< 1 hr@week)
Little (< 3 hr@week)
Often (5 - 10 hr@week)
Frequent (10 - 20hr@week)
Intensive (>20 hr@week)
Figure 5 Intensity of Computer Usage of Participants
Expectation on E-Learning Programs Although the design of the e-learning program was completed without prior analysis of
the potential participants’ background and expectation, the questionnaire done before the on-line courses started provided vital information for the project team to modify the course structure and implementation strategy for terms except the first one. A total of about 10 questions were asked and a part of the results are presented herein.
When participants were asked whether they are more interested in attending e-learning programs than traditional F2F programs, more than 80% of the participants responded positively (Fig. 6). The percentage increases with time moderately. The participants strongly expected that e-learning programs should be able to let them enjoy the flexibility of taking classes when compared to F2F programs (Fig. 7).
0%10%20%30%40%50%60%70%80%
Stronglydisagree
Disagree Neitheragree ordisagree
Agree Stronglyagree
20042005 12005 22006
Figure 6 Interest in E-Learning Program
0%
10%
20%
30%
40%
50%
60%
70%
80%
Stronglydisagree
Disagree Neitheragree ordisagree
Agree Stronglyagree
2004 2005 12005 2 2006
Figure 7 E-Learning Program Should Have Greater Flexibility
The participants also expected e-learning programs can provide more additional
information related to the course subjects than F2F learning (Fig. 8) and they also thought that there should be more discussion on water environment related topics among peer participants (Fig. 9).
0%
10%
20%
30%
40%
50%
60%
70%
80%
Stronglydisagree
Disagree Neitheragree ordisagree
Agree Stronglyagree
20042005 12005 22006
Figure 8 More Additional Information from E-Learning Program
0%
10%
20%
30%
40%
50%
60%
70%
80%
Stronglydisagree
Disagree Neitheragree ordisagree
Agree Stronglyagree
20042005 12005 22006
Figure 9 Expect to Have More Discussion on Water Environment with Peer
Nevertheless, the participants were significantly more conservative when considering
whether they would have more interaction with e-learning programs and whether they would have more fun from the e-learning programs when compared with F2F programs. Only about 60% of the participants expected that they might have more interaction with peer participants (Fig. 10). On the other hand, about 65% of the participants thought they could have more fun during e-learning sessions (Fig. 11). It has to be reminded that 32% of the participants have previous experience with e-learning. It might be interesting to do a cross analysis to find out whether these participants in particular are more realistic due to their personal experience.
0%
10%
20%
30%
40%
50%
60%
70%
80%
Stronglydisagree
Disagree Neitheragree ordisagree
Agree Stronglyagree
20042005 12005 22006
Figure 10 Expect to Have More Interaction with Peer Participants
0%
10%
20%
30%
40%
50%
60%
70%
80%
Stronglydisagree
Disagree Neitheragree ordisagree
Agree Stronglyagree
20042005 12005 22006
Figure 11 Expect to Have More Fun from E-Learning Program
A very important while also interesting expectation of the participants was that they wish
to have instructor available whenever they have questions or want to have a discussion (Fig. 12). Although these participants had known clearly that it would be an e-learning program, they still consider that they need the instructor to be presence all the time.
0%
10%
20%
30%
40%
50%
60%
70%
80%
Stronglydisagree
Disagree Neitheragree ordisagree
Agree Stronglyagree
20042005 12005 22006
Figure 12 Expectation on Instructors’ Availability
Results of Learning Effectiveness Assessment
The effective of learning was assessed by results of (1) pre-test and post-test and (2) after course quiz. The test was designed to evaluate both the knowledge of Taiwan school teachers and the effectiveness of the e-learning program. The same questions (Table 5) were asked in on-line test before and after the completion of course modules. The participants were not notified of the tests thus could not “prepare” for the tests. The results of the tests are shown in Table 6. The percentage of improvement of participants understanding of important concepts varies with the highest close to 20%.
Table 5 Questions for Quiz before and after the E-Learning Courses Single Choice Questions 1. What is the average annual rainfall in Taiwan? (Answer: C)
A)1000 B)1500 C)2500 D)500 E)5000 mm 2. Hydrological cycle is a very important material cycle of nature. Part of the rainwater that is not
evaporated nor infiltrated becomes which of the following? (Answer: B) A)Percolation B)Runoff C)Transpiration D)Condensation E)Water vapor
3. Which of the following is not one of the major functions of reservoirs? (Answer: D) A)Storage of water B)Recreation C)Electricity generation D)Climate stabilization E)Flood regulation
4. What proportion of the surface of the earth is covered by water? (Answer: D) A)20% B)40% C)50% D)70% E)90%
5. What is the proportion of fresh water in the water on earth? (Answer: E) A)50% B)35% C)15% D)7% E)3%
6. If the river is polluted by animal waste which of the following item will be present in the water? (Answer:B)
A)Heavy metals B)Nitrogen as in ammonium C)Hardness D)Strong acid E)Color 7. Which is the longest river in Taiwan? (Answer:C)
A)Tanshui River B)Kaoping River C)Chuoshui River D)Shewguluan River E)Tadu River 8. Which is the largest proportion of water use in Taiwan? (Answer: C) A)Domestic B)Industrial C)Agricultural D)Mining E)Fish farming 9. Which of the following country’s rate of sewer service is lower than Taiwan? (Answer:E)
A)USA B)Philippines C)Malaysia D)Mexico E)None of the above. 10. The average domestic water usage per person per day is around 100 – 150 liters. What is the water
usage per person per day for Taiwan? (Answer: E) A)100 B)150 C)200 D)250 E)350
11. When water contains a lot of nutrients, the water quality is very good. (Answer: B) A)Agree B)Disagree C)Don’t know answer/Neither agree nor disagree
12. Most of the stream flow in the river comes from precipitation and the amount of water from groundwater recharge is so small that it can be ignored. (Answer: B)
A)Agree B)Disagree C)Don’t know answer/Neither agree nor disagree 13. Intercepting and treatment of sewage in a big centralized treatment plant before discharging into the
sea is the most advanced and effective way to treat waste water. (Answer: B) A)Agree B)Disagree C)Don’t know answer/Neither agree nor disagree
14. When rocks falling or sliding down slopes during heavy rain, it becomes debris flow. (Answer: B) A)Agree B)Disagree C)Don’t know answer/Neither agree nor disagree
15. Reclamation land from sea is an environmentally sound way of development. (Answer: B) A)Agree B)Disagree C)Don’t know answer/Neither agree nor disagree
16. Utilizing available water and geography and turning a grassland into a wetland for landscaping can benefit the ecosystem. (Answer: B)
A)Agree B)Disagree C)Don’t know answer/Neither agree nor disagree 17. The enhanced greenhouse effect due to climate change will cause temperature to rise globally and
most of the world will be threatened by draught. (Answer: B) A)Agree B)Disagree C)Don’t know answer/Neither agree nor disagree
18. In order to meet the ecological principles, engineers should design a permeable layer over the concrete pavement. (Answer: B)
A)Agree B)Disagree C)Don’t know answer/Neither agree nor disagree
Table 6 Improvement of Test Results after the E-Learning Courses No. of
Problem Correct Rate of Pre-Course Test
Correct Rate of Post-Course Tests
Increase (%)
1 59.63% 76.67% 17.04% 2 73.65% 78.94% 5.29% 3 62.68% 63.99% 1.31% 4 88.96% 89.69% 0.73% 5 52.17% 62.85% 10.69% 6 85.38% 87.20% 1.82% 7 83.32% 82.45% -0.87% 8 36.79% 56.74% 19.95% 9 29.40% 37.94% 8.54% 10 9.44% 13.14% 3.7% 11 86.75% 90.49% 3.74% 12 64.89% 70.55% 5.66% 13 32.67% 46.21% 13.54% 14 51.26% 57.76% 6.5% 15 86.60% 89.69% 3.36% 16 55.06% 56.96% 1.9% 17 36.94% 47.23% 10.29%
18 51.87% 58.21% 6.34%
Results of Course Evaluation
The design, organization and management of the program were evaluated with compulsory participant survey. As shown in Fig. 13, the participants give the e-learning program and the LMS very good rating. However, currently the quality of video is as satisfactory due to the limitation of the bandwidth on the users’ end and can be easily enhanced in the future. The most critical issue is that the interaction among the participants and/or with instructor is significantly less satisfactory. The limitation comes from the nature of asynchronous on-line course and the browser. The participants interacted with the instructor and among each other through discussion board and file sharing, which is far from spontaneous as face to face discussion. This particular result points out a very specific aspect for future improvement.
0% 10% 20% 30% 40% 50% 60%
Very poor
Rather unsatisfactory
Fair
Good
Excellent
Interaction ModelAV QualityWeb Site Designe Learning model
Figure 13 Rating of the E-Learning Program and the LMS Platform
When participants were asked whether they obtain enough useful knowledge on water
environment, more than 95% of the participants agree. Many participants think they obtained more knowledge than they expected as shown by comparing the results from questionnaire before and after the e-learning courses (Fig. 14). Statistics comparing the expectation and the self assessment of the participants regarding whether the courses will be helpful for their teaching also showed similar results (Fig. 15). Furthermore, more than 60% of the participants will utilize the knowledge on water environment and skills in their classroom for teaching in the future with a 60% or higher chances (Fig. 16).
0%
10%
20%
30%
40%
50%
60%
70%
80%
Stronglydisagree
Disagree Neitheragree ordisagree
Agree Stronglyagree
ExpectationResult
Figure 14 Comparison between Results and Expectation on Knowledge Obtained
Finally, 97% of the participant expressed they would like to join on-line learning
program on other subjects. In general, although about 68% of participants had their first experience in e-learning by joining this program and become supporters of e-learning. Considering the difficulties of many of the participants had with the software, operating systems, hardware, and internet connections at the start of the courses, the quick acceptance and preference of e-learning programs are quite encouraging.
0%
10%
20%
30%
40%
50%
60%
70%
Stronglydisagree
Disagree Neitheragree ordisagree
Agree Stronglyagree
ExpectationResult
Figure 15 Comparison between Results and Expectation of the Usefulness of the Courses
0%5%
10%15%20%25%30%35%40%45%50%
0-20% 20-40% 40-60% 60-80% 80-100%
20042005 12005 22006
Figure 16 Likelihood of Utilizing the Knowledge
Lessons Learned
There are several important lessons the project team learned from the program:
(1) Service manpower loading can be heavy The on-line course had a rough start since most of the participants did not have any
experiences such that they run into problems with their computer operating systems, internet browsers, or internet connections. Two LMS engineers were responsible to solve all the technical problems. Almost all of the technical difficulties were resolved quickly and some of them become the content of the website’s FAQ. As time went on, technical problems such as internet, operating system, and browser decreased. However, the loading of two staff members who served as teaching assistants remained as heavy as the start. Participants have all sorts of problems regarding asking register after closing date, how the course hours was recorded, how the certificate can be awarded, requesting for make up classes, and questions of the content of the courses or quizzes. This requires the staff constantly monitored the website and answered questions. Surprisingly, there is a distinct shift of user behavior over time.
Those who join earliest and most interested in water environment are most active all through the term including Saturdays and Sundays (Fig. 17). Moreover, they logged in the system weeks after the end of the term for discussion (Fig. 18). The later participants were much less interested and much less active, which in a way relieved some pressure on the staff.
0
1000
2000
3000
4000
5000
6000
7000
8000
Logi
n C
ount
.
Mon Tue Wed Thu Fri Sat Sun
Days
Week 7
Week 6
Week 5
Week 4
Week 3
Week 2
Week 1
A = 1378B = 1302Total = 2680Number = 1426
(a) 2005 1 E-Learning Term
0
2000
4000
6000
8000
10000
12000
14000
16000
Logi
n C
ount
.
Mon Tue Wed Thu Fri Sat Sun
Day
Week 7
Week 6
Week 5
Week 4
Week 3
Week 2
Week 1
A = 1487B = 1382C = 1180D = 1187Total = 5236Number = 1782
(b) 2006 E-Learning Term
Figure 17 2006 E-Learning Term Login Record Week Analysis
0
200
400
600
800
1000
1200
1400
1600
1800
1 5 9 13 17 21 25 29 33 37 41 45 49 53 57 61 65 69 73 77 81
Day
Logi
n C
ount
.
A = 1378B = 1302Total = 2680Number = 1426
End
(a) 2004 E-Learning Term
0
500
1000
1500
2000
2500
3000
3500
1 4 7 10 13 16 19 22 25 28 31 34 37 40 43 46 49 52 55 58 61 64 67
Day
Logi
n C
ount
.
A = 1487B = 1382C = 1180D = 1187Total = 5236Number = 1782
End
(b) 2005 2 E-Learning Term
Figure 18 E-Learning Term Login Record On the other hand, since most of the participants were school teachers, the time they
watched the on-line courses and joined the discussion was quite predictable. They started to login the system around 9 a.m. in the mornings and logout around 5 or 6 p.m. A lot of participants login around 7 p.m. to watch the courses and logout around 11 p.m. The engineers and staffs were available from 8 a.m. until 9 p.m. each day to manage the program and handle all the requests and problems of the participants.
0 2000 4000 6000 8000 10000 12000 14000
0 - 1
2 - 3
4 - 5
6 - 7
8 - 9
10 - 11
12 - 13
14 - 15
16 - 17
18 - 19
20 - 21
22 - 23
Hou
r .
Login
2005 12005 22006
Figure 19 Statistics of Login Hours
(2) Facilitation of interaction/discussion is important
Although more than 50% participants considered themselves to have more frequent communication than in classroom learning, interaction remains the least satisfactory part of the e-learning experience (Fig. 20) just as they expected before the program began. The knowledge transfer and time management is significantly better than interaction.
0% 10% 20% 30% 40% 50% 60%
Very poor
Rather unsatisfactory
Fair
Good
Excellent
InteractionTime managementLearning
Figure 20 Self-Assessment on E-Learning Ability
0% 5% 10% 15% 20% 25% 30% 35% 40%
Course subjects
Course induced
Environmental issues
Program Administration
Others Total = 266
Figure 21 Distribution of Discussion Categories of Topics
The statistics shows even worse. There were only 266 discussion subjects posted by the participants on the discussion board. The project manager and teaching assistants spent a lot of effort in responding the messages and keeping the discussion alive. These messages only got 1,211 responding posted messages and were read 45,219 times. In other words, most participants never actively interact with any fellow participants. Figure 17 shows the categories of discussion subjects posted. Most of them are related to the courses or environmental issues (Fig. 21). In addition, there were actually only 130 participants that are actively interacting with peers (Fig. 22). Interaction among participants remains to be an important issue to be addressed even though this was an asynchronous e-learning program.
0 20 40 60 80 100 120
1
2
3
4
5
6 to 10
11 to 15
> 15
Number of posts
Total = 130 person
Figure 22 Statistics of Active Participants
(3) Expanded use of the program is limitless
A lot of participants login the LMS and play back the on-line course in the geography, earth science, and social science class they are teaching and directly let the students watch the on-line course. Moreover, the courses have been rearranged to become formal on-line courses delivered in the universities and community colleges. Government agencies also requested the
project team to reorganize the courses to establish training programs for public servants and volunteer workers.
Conclusions
The water education innovative project is the pioneering in Taiwan. Vital experiences regarding the design and management of the program, the LMS, the on-line courses, and behavior of the learners were obtained.
It is shown that e-learning program can easily attract potential participants because it provides great flexibility in terms of time and place of learning. However, the nature of e-learning and the limitation of ICT up to now still can not overcome the less satisfactory while very, if not the most, important aspect of learning, i.e., interaction. Nevertheless, e-learning is still the most efficient method for large scale training programs such as the water education program described in this paper. However, supplemental face to face courses should be incorporated to augment the effectiveness of learning and also designing of the programs.
The final phase of the projects will be carried out in 2008. Fifteen new on-line courses will be developed and delivered, including 5 delivered in English to establish a module on Asian water environment. In addition, experiment with co-life on-line conferencing system to facilitate interaction will be incorporated into the program. The system will be used for case history discussion host by the instructors for experimental purpose. This system developed by NCHC allows 60 people to joint a discussion session simultaneously with full video and file presentation/exchange capacity. Application of new technologies will continued to be explored with the support from NCHC and other partners.
Acknowledgement
The authors thank the Ministry of Education, the Water Resources Agency, and the Environmental Protection Agency of Taiwan for the financial and administrative support. The authors also liked to thank all the members of the project team and the invited instructors for contributing their time and knowledge to this project.
References Berger, C., and Kam, R. (1996). Definitions of Instructional Design.
http://www.umich.edu~ed626/defin/html Accessed 28/06/2002
Dick, W., and Carey, L. (1978). The Systematic Design of Instruction. Glenview, IL.: Scott, Foresman.
Figueira, E. (2003). Evaluating the Effectiveness of E-Learning Strategies for Small and Medium Enterprises. European Seminars - Exploring models and partnerships for eLearning in SMEs
Gagne, R. M., Briggs, L. J., and Wagner, W. W. (1992). Principles of Instructional Design (4th ed.). New York: Harcourt Brace Jovanovich College Publishers.
Gerlach, V. S., and Ely, D. P. (1980). Teaching and Media: A systematic Approach. Englewood Cliffs, NJ: Prentice-Hall, Inc.
Hannifin, M. and Peck, K. (1988). The Design, Development, and Evaluation of Instructional Software. New York, NY: Macmillan.
Kemp, J. E. (1985). The Instructional Design Process. New York: Harper & Row.
Knirk, F. G. and Gustafson, K.L. (1986). Instructional Technology: A Systematic Approach to Education. Fort Worth, TX: Holt, Rinehart & Winston.
Microsoft. Microsoft Producer.
Microsoft. Microsoft Digital Rights Management.
Microsoft. Microsoft Streaming Technology.
Ministry of Education MOE. (2006). 2006 Education in Taiwan
Shankar B. Ponnekanti, F. Armando, P. Hanrahan, and T. Winograd I. (2001). Crafter: A Service Framework for Ubiquitous Computing Environments. Proceedings of Ubicomp 2001, September 30-October 2, 2001
Taiwan Network Information Center. (2006). Taiwan Broadband Network Usage Survey