Evaluation of learning resources

Evaluation of Learning Resources

Damian Gordon

There is a plethora of learning materials of all sorts now available on-line.

The range of ways in with these materials are produced and consumed means that there cannot be one single way to evaluate these materials.

However there is a major issue with quality control with regard to this material.

The Quantity versus Quality issue

Continued:

A great deal of material is developed without any evidence of it ever having being tested on learners.

A logical way to evaluate the quality of learning materials is to try it out on learners and get their feedback (both formative and summative).

It perhaps make most sense to submit this learner feedback in conjunction with the learner materials to verify that the materials have been tested by learners.

The Quantity versus Quality issue

There are two main evaluation tools for learning materials◦ MERLOT◦ LORI

Two slightly less used tools are◦ MicroSoft Evaluators Guide◦ National Council for Educational Technology for

evaluating CD-DROM

Evaluation Tools

MERLOT (www.merlot.org) is a repository containing educational resources classified into seven broad subject categories: Arts; Business; Education; Humanities; Mathematics and Statistics; Science and Technology; Social Sciences.

MERLOT

There are three general categories of evaluation standards used within MERLOT: 1. Quality of Content 2. Potential Effectiveness as a

Teaching-Learning Tool 3. Ease of Use

MERLOT

Quality of Content There are two general elements to quality of

content:1. Does the software present valid (correct)

concepts, models, and skills? 2. Does the software present educationally

significant concepts, models, and skills for the discipline?

MERLOT

Continued:

Quality of Content To evaluate the educational significance of

the content, reviews use the following guidelines:

MERLOT

Continued:

Quality of Content ◦ Content is core curriculum within the discipline. ◦ Core curriculum topics are typically covered to

some degree in the introductory classes within the discipline and/or "Everyone teaches it" and/or it is identified as a core area by the discipline's professional organizations

◦ Content is difficult to teach and learn. ◦ Content is a pre-requisite for understanding more

advanced material in the discipline

MERLOT

Potential Effectiveness as a Teaching-Learning Tool

Evaluating POTENTIAL effectiveness is asking the Peer Reviewer to judge, based on his expertise as a teacher, whether the learning material is likely to improve teaching and learning given the ways the faculty and students could use the tool.

MERLOT

Continued:


What stage(s) in the learning process/cycle could the materials be used? ◦ Explanation or description of the topic/stating the

problem ◦ Demonstration of the curriculum/exploration of the

problem ◦ Practice using the curriculum/analysis of the outcomes

from solving the problem ◦ Applying the curriculum to "new" problems/application

of the outcomes to other problems

MERLOT

Continued:


What is(are) the learning objective(s)? What should students be able to do after successfully learning with the materials?

What are the characteristics of the target learner(s)

MERLOT

Continued:


There are other general elements to effectiveness as a teaching-learning tool that MERLOT asks reviewers to consider:◦ Does the interactive/media-rich presentation of material improve

faculty and students' abilities to teach and learn the materials? ◦ Can the use of the software be readily integrated into current

curriculum and pedagogy within the discipline? ◦ Can the software be used in a variety of ways to achieve

teaching and learning goals? ◦ Are the teaching-learning goals easy to identify? ◦ Can good learning assignments for using the software

application be written easily?

MERLOT

Ease of Use The basic question underlying the ease of use

standard is: how easy it is for teachers and students to use the software for the first time?

Elements that affect ease of use include:

MERLOT

Continued:

Ease of Use◦ Are the labels, buttons, menus, text, and general layout of the

computer interface consistent and visually distinct? ◦ Does the user get trapped in the material? ◦ Can the user get lost easily in the material? ◦ Does the module provide feedback about the system status and

the user's responses? ◦ Does the module provide appropriate flexibility in its use? ◦ Does the learning material require a lot of documentation,

technical support, and/or instruction for most students to successfully use the software?

◦ Does the material present information in ways that are familiar for students?

◦ Does the material present information in ways that would be attractive to students?

MERLOT

eLera is a distributed group that researches and evaluates e-learning. Their specific interests include learning objects, e-portfolios, and learning design specifications, and related topics.

eLera provides tools and information for learning object evaluation and research, maintains a database of learning object reviews, and supports communication and collaboration among researchers, evaluators and users of online learning resources.

LORI from eLera

The goals of eLera are to:◦ improve the quality of online learning resources

through better design and evaluation◦ develop effective pedagogical models that

incorporate learning objects◦ help students, teachers, professors, instructional

designers and others to select pedagogical models and digital resources that meet their requirements

LORI from eLera

The Learning Object Review Instrument (LORI) is used to evaluate the quality of e-learning resources. LORI is an online form consisting of rubrics, rating scales and comment fields. The current version of LORI available from eLera is version 1.5.

http://www.elera.net/eLera/Home/Articles/LORI%201.5.pdf

LORI from eLera



Nine Dimensions of LORI1. Content Quality2. Learning Goal Alignment3. Feedback and Adaptation4. Motivation5. Presentation Design6. Interaction Usability7. Accessibility8. Reusability9. Standards Compliance

LORI from eLera

Nine Dimensions of LORI

1. Content Quality: Veracity, accuracy, balanced presentation of ideas, and appropriate level of detail

LORI from eLera


2. Learning Goal Alignment: Alignment among learning goals, activities, assessments, and learner characteristics

LORI from eLera


3. Feedback and Adaptation: Adaptive content or feedback driven by differential learner input or learner modelling

LORI from eLera


4. Motivation: Ability to motivate and interest an identified population of learners

LORI from eLera


5. Presentation Design: Design of visual and auditory information for enhanced learning and efficient mental processing

LORI from eLera


6. Interaction Usability: Ease of navigation, predictability of the user interface, and quality of the interface help features

LORI from eLera


7. Accessibility: Design of controls and presentation formats to accommodate disabled and mobile learners

LORI from eLera


8. Reusability: Ability to use in varying learning contexts and with learners from differing backgrounds

LORI from eLera


9. Standards Compliance: Adherence to international standards and specifications

LORI from eLera

9 items rated on a 5-point scale

LORI Review process

diagram here is showing individual reviews merging to a panel review and then published on web.

MicroSift Evaluators Guide INSTRUCTIONAL QUALITY The purpose of the package is well

defined The package achieves its defined

purpose Presentation of content is clear and

logical The level of difficulty is appropriate to

the target audience Graphics/colour/sound are used for

appropriate instructional reasons Use of package is motivational The package effectively stimulates

student creativity Feedback on student responses is

effectively employed The learner controls the rate and

sequence of presentation and review Instruction is integrated with previous

student experience Learning is generalisable to an

appropriate range of situations

CONTENT The content is accurate. The content has educational value The content is free of race, ethnic,

sex, and other stereotypes

TECHNICAL QUALITY The user support materials are

comprehensive The user support materials are

effective Informative displays are effective Intended users can easily and

independently operate the program

Teachers can easily employ the package

The program appropriately uses relevant computer capabilities

The program is reliable in normal use

National Council for Educational Technology for evaluating CD-DROM

1. Which computer system will the disc run on? 2. Will your computer system do justice to the illustrations? 3. Is the operation by keyboard or mouse, or both? 4. Can we have the disc for a trial period? 5. Is the language and spelling on the disc Queen’s English or American

English? 6. How much bias is there in the content of the disc? 7. Is printing out easy and intuitive? 8. Can the selected material readily be down-loaded to disc? 9. Can subsections of the disc be searched? 10. Is the software to control the CD-ROM on the disc itself or is it supplied on a

separate floppy disc? 11. Does the software manage memory resources well? 12. What search procedures are available? 13. What is the language level on the disc? 14. Is the user interface tolerant of typing and spelling errors? 15. Can you select exactly what you want to print out or save to disc? 16. Are there any supporting features? 17. Can the illustrations be printed out? 18. Can images be readily transferred? 19. Is there a sound capability to accompany the pictures?