Quality Evaluation methodologies for

e-Learning systems (in the frame of the EC Project UNITE)

Tatiana RikureResearcher, Riga Technical University (RTU), Latvia

UNITE “Unified e-Learning Environment for the School”

• EC 6th FP IST, STREP

• started: February 2006, duration: 30 months

• 13 partners from 11 countries (coordinator: Fraunhofer Institute for Computer Graphics)

• The strategic goal:

is to contribute to the improvement of European wide education in schools based on common, innovative principles in technology, pedagogy, scenarios as well as validation of success

• The strategic objective:

is the development of a set of frameworks to support Europe-wide eLearning in schools and to adapt these frameworks to national needs by "instantiation".

• will be implemented in 14 secondary schools in 11 countries

UNITE “Unified e-Learning Environment for the School”

Results

• UNITE e-Learning concept (new e-Learning

scenarios based on innovative principles of e-

Learning and a sound pedagogical model)

• UNITE system (a flexible e-Learning ICT

platform based on an innovative e-Learning

platform & pedagogical methodology)

• Validation & evaluation methodology (the

quality of the UNITE system & the success of

the e-Learning scenarios)

• Socio-economic evaluation

• Europe-wide network of schools

• Europe-wide repository of the re-usable e-

Learning content

• etc.

UNITELMS

communication & collaboration

mobilelearning

Microcosmos

Building the basis of UNITE, offering:• Multilingual schooling environment.• Personalized space and resource areas for pupils, teachers, parents.• Personalised agenda for pupils, teachers, parents.• Process-oriented assignments and evaluations.• Asynchronous / synchronous communication tools.• Multilevel access management.• Parents’ monitoring tools.• Logging facilities.• Any other necessary tools to be identified by the schools.

m-learning

Integration of mobile learning facilities in UNITE:• Incorporation of different uses of mobile phone devices in enhancing

the learning experience and test their effectiveness.• Creation of pedagogically meaningful “chunks” of learning over mobile

phone devices (P900, multimedia mobile phone, XDAs, etc ).• Incorporation of the use of a ‘media board’ to combine “in-house”

and “outside” schooling activities.

MTS

Incorporation of the MTS Learning Management System (LMS):• Adaptation of MTS to match the specific school requirements.• Central repository for learning elements and experiences. • Development facilities for eLearning resources repositories in schools.• Reusable learning materials, able to be combined with structured

lessons following certain pedagogical roles.• Recording of learning histories.

UNITE “Unified e-Learning Environment for the School”

System

UNITE “Unified e-Learning Environment for the School”

Validation framework

• Test the impact and viability of the technology and the e-Learning activities in a range of school settings

• Show the appropriateness, meaningfulness and usefulness of the UNITE system and the e-Learning scenarios

Riga Technical University (RTU) in UNITE “Unified e-Learning Environment for the School”

RTU (Division of Applied Systems Software) main tasks:

Designing the quantitative assessment methodologies for the validations Leading the validation activities in Latvia

Quality evaluation will be developed and executed on 2 levels:• Quality Assessment of the system:

• at the design phase,• of the final product,

• Measuring Efficiency (success \ impact) of the system:• using qualitative evaluation,• using quantitative evaluation.

Quality approaches

• any methods, policies, procedures, rules, criteria, tools, checklists or any other verification instruments or measures that have

the purpose of enhancing the quality of e-Learning products or services, can be classified for the purposes of the UNITE

systems’ validation.

Design phaseEfficiency of the system

Software verification approaches:• Static:

- Consistency techniques - Measurement techniques

• Dynamic: - White-box (structural) testing - black-box (functional) testing - Random testing

Software testing

Formative evaluation• Critical commenting• Developmental testing• Revising• Reviews (of literature, learners, learning context, existing solutions, etc.)• Walk-through• Storyboards with potential users & experts• Observations, video-taping, user-tracking, interviews with users & experts• Peer reviews

Final product

• Quality assurance guidelines• Quality evaluation criteria• Quality marks

• Best Practices

Benchmarking

Bloom’s Taxonomy

Expert evaluation

Quality assurance frameworks

Quality approaches

• any methods, policies, procedures, rules, criteria, tools, checklists or any other verification instruments or measures that have

the purpose of enhancing the quality of e-Learning products or services, can be classified for the purposes of the UNITE

systems’ validation.

Design phaseEfficiency of the system

Software validation approaches:• Formal methods• Fault injections:

- Hardware - Software

• Dependability analysis: - Hazard analysis

- Risk analysis

Software testing

Final product

Summative evaluation

Expert evaluation

Evaluation of the learning content

• Quality assurance guidelines• Quality evaluation criteria• Quality marks

• Best Practices

Benchmarking

Usability testing • Feedback from users (questionnaires, interviews, pre and post tests, etc.)• Revising• Expert / Peer review

Quality approaches

• any methods, policies, procedures, rules, criteria, tools, checklists or any other verification instruments or measures that have

the purpose of enhancing the quality of e-Learning products or services, can be classified for the purposes of the UNITE

systems’ validation.

Design phase Efficiency of the system Final product

• Pedagogical experiments

• Quality assurance guidelines• Quality evaluation criteria• Quality marks

• Best Practices

• Benchmarking

• Expert evaluation

• Kirkpatrick's 4 Levels of Evaluation

• ROI

• Learners performance testing

• Feedback from learners and users • Interviews (oral guided, written, telephone, etc.)

• Questionnaires • Lists of criteria (checklists)• Concept-mapping exercise

• BLOGs

• Observations

• Case Studies

• Collaborative assessment techniques

• WATTS (web-assessment techniques and tools)

• Quality management activities

• Quality audits

• Knowledge surveys

Conclusions

• The development of e-Learning products and the provision of e-learning opportunities is one of the most rapidly expanding areas of education and training. E-Learning is also one of the areas that attracts the most research and development funding. If this investment is to be maximised, it is our challenge to generate robust models for the systematic evaluation of e-learning and produce tools which are flexible in use but consistent in results.

• Measuring the effectiveness of e-Learning is an important issue in the development and use of any e-Learning system or strategy, which not only shows the value of their use, but also allows improving in the future.

• Measuring the quality of the e-Learning systems should be done with the use of multiple-method evaluation strategy that considers both qualitative and quantitative criteria.

• Evaluation of e-Learning is fundamentally the same as the evaluation of any other learning but with particular groups of variables playing a more prominent role and the impact of others differs significantly from their impact in traditional learning.

Acknowledgements

• UNITE (Unified e-Learning Environment for the School)

• IST4Balt (Information Society Technologies Promotion in Baltic States)http://www.balticit.com/ist4balt/

• eLOGMAR-M (Web-based and Mobile Solutions for Collaborative Work Environment with Logistics and Maritime Applications) http://www.elogmar-m.org

Contacts

Tatiana RikureRiga Technical University (RTU)

Faculty of Computer Science &

Information Technology

Division of Applied Systems Software

Meza str. ¼, Riga, Latvia

+371 7089096; fax: +371 7089572

rikure@cs.rtu.lv