Upload
kelley-howard
View
213
Download
0
Tags:
Embed Size (px)
Citation preview
Quality Evaluation methodologies for
e-Learning systems (in the frame of the EC Project UNITE)
Tatiana RikureResearcher, Riga Technical University (RTU), Latvia
UNITE “Unified e-Learning Environment for the School”
• EC 6th FP IST, STREP
• started: February 2006, duration: 30 months
• 13 partners from 11 countries (coordinator: Fraunhofer Institute for Computer Graphics)
• The strategic goal:
is to contribute to the improvement of European wide education in schools based on common, innovative principles in technology, pedagogy, scenarios as well as validation of success
• The strategic objective:
is the development of a set of frameworks to support Europe-wide eLearning in schools and to adapt these frameworks to national needs by "instantiation".
• will be implemented in 14 secondary schools in 11 countries
UNITE “Unified e-Learning Environment for the School”
Results
• UNITE e-Learning concept (new e-Learning
scenarios based on innovative principles of e-
Learning and a sound pedagogical model)
• UNITE system (a flexible e-Learning ICT
platform based on an innovative e-Learning
platform & pedagogical methodology)
• Validation & evaluation methodology (the
quality of the UNITE system & the success of
the e-Learning scenarios)
• Socio-economic evaluation
• Europe-wide network of schools
• Europe-wide repository of the re-usable e-
Learning content
• etc.
UNITELMS
communication & collaboration
mobilelearning
Microcosmos
Building the basis of UNITE, offering:• Multilingual schooling environment.• Personalized space and resource areas for pupils, teachers, parents.• Personalised agenda for pupils, teachers, parents.• Process-oriented assignments and evaluations.• Asynchronous / synchronous communication tools.• Multilevel access management.• Parents’ monitoring tools.• Logging facilities.• Any other necessary tools to be identified by the schools.
m-learning
Integration of mobile learning facilities in UNITE:• Incorporation of different uses of mobile phone devices in enhancing
the learning experience and test their effectiveness.• Creation of pedagogically meaningful “chunks” of learning over mobile
phone devices (P900, multimedia mobile phone, XDAs, etc ).• Incorporation of the use of a ‘media board’ to combine “in-house”
and “outside” schooling activities.
MTS
Incorporation of the MTS Learning Management System (LMS):• Adaptation of MTS to match the specific school requirements.• Central repository for learning elements and experiences. • Development facilities for eLearning resources repositories in schools.• Reusable learning materials, able to be combined with structured
lessons following certain pedagogical roles.• Recording of learning histories.
UNITE “Unified e-Learning Environment for the School”
System
UNITE “Unified e-Learning Environment for the School”
Validation framework
• Test the impact and viability of the technology and the e-Learning activities in a range of school settings
• Show the appropriateness, meaningfulness and usefulness of the UNITE system and the e-Learning scenarios
Riga Technical University (RTU) in UNITE “Unified e-Learning Environment for the School”
RTU (Division of Applied Systems Software) main tasks:
Designing the quantitative assessment methodologies for the validations Leading the validation activities in Latvia
Quality evaluation will be developed and executed on 2 levels:• Quality Assessment of the system:
• at the design phase,• of the final product,
• Measuring Efficiency (success \ impact) of the system:• using qualitative evaluation,• using quantitative evaluation.
Quality approaches
• any methods, policies, procedures, rules, criteria, tools, checklists or any other verification instruments or measures that have
the purpose of enhancing the quality of e-Learning products or services, can be classified for the purposes of the UNITE
systems’ validation.
Design phaseEfficiency of the system
Software verification approaches:• Static:
- Consistency techniques - Measurement techniques
• Dynamic: - White-box (structural) testing - black-box (functional) testing - Random testing
Software testing
Formative evaluation• Critical commenting• Developmental testing• Revising• Reviews (of literature, learners, learning context, existing solutions, etc.)• Walk-through• Storyboards with potential users & experts• Observations, video-taping, user-tracking, interviews with users & experts• Peer reviews
Final product
• Quality assurance guidelines• Quality evaluation criteria• Quality marks
• Best Practices
Benchmarking
Bloom’s Taxonomy
Expert evaluation
Quality assurance frameworks
Quality approaches
• any methods, policies, procedures, rules, criteria, tools, checklists or any other verification instruments or measures that have
the purpose of enhancing the quality of e-Learning products or services, can be classified for the purposes of the UNITE
systems’ validation.
Design phaseEfficiency of the system
Software validation approaches:• Formal methods• Fault injections:
- Hardware - Software
• Dependability analysis: - Hazard analysis
- Risk analysis
Software testing
Final product
Summative evaluation
Expert evaluation
Evaluation of the learning content
• Quality assurance guidelines• Quality evaluation criteria• Quality marks
• Best Practices
Benchmarking
Usability testing • Feedback from users (questionnaires, interviews, pre and post tests, etc.)• Revising• Expert / Peer review
Quality approaches
• any methods, policies, procedures, rules, criteria, tools, checklists or any other verification instruments or measures that have
the purpose of enhancing the quality of e-Learning products or services, can be classified for the purposes of the UNITE
systems’ validation.
Design phase Efficiency of the system Final product
• Pedagogical experiments
• Quality assurance guidelines• Quality evaluation criteria• Quality marks
• Best Practices
• Benchmarking
• Expert evaluation
• Kirkpatrick's 4 Levels of Evaluation
• ROI
• Learners performance testing
• Feedback from learners and users • Interviews (oral guided, written, telephone, etc.)
• Questionnaires • Lists of criteria (checklists)• Concept-mapping exercise
• BLOGs
• Observations
• Case Studies
• Collaborative assessment techniques
• WATTS (web-assessment techniques and tools)
• Quality management activities
• Quality audits
• Knowledge surveys
Conclusions
• The development of e-Learning products and the provision of e-learning opportunities is one of the most rapidly expanding areas of education and training. E-Learning is also one of the areas that attracts the most research and development funding. If this investment is to be maximised, it is our challenge to generate robust models for the systematic evaluation of e-learning and produce tools which are flexible in use but consistent in results.
• Measuring the effectiveness of e-Learning is an important issue in the development and use of any e-Learning system or strategy, which not only shows the value of their use, but also allows improving in the future.
• Measuring the quality of the e-Learning systems should be done with the use of multiple-method evaluation strategy that considers both qualitative and quantitative criteria.
• Evaluation of e-Learning is fundamentally the same as the evaluation of any other learning but with particular groups of variables playing a more prominent role and the impact of others differs significantly from their impact in traditional learning.
Acknowledgements
• UNITE (Unified e-Learning Environment for the School)
• IST4Balt (Information Society Technologies Promotion in Baltic States)http://www.balticit.com/ist4balt/
• eLOGMAR-M (Web-based and Mobile Solutions for Collaborative Work Environment with Logistics and Maritime Applications) http://www.elogmar-m.org
Contacts
Tatiana RikureRiga Technical University (RTU)
Faculty of Computer Science &
Information Technology
Division of Applied Systems Software
Meza str. ¼, Riga, Latvia
+371 7089096; fax: +371 7089572