SYNERGY Induction to Pedagogy Programme - Evaluation of the Learning Resources (ENGLISH)

Induction to Pedagogy

Evaluation of the Learning Resources

Introduction

SYNERGY Exchange (http://synergyexchange.eu/)

Created for “SYNERGY”, an ERASMUS+ project

Module Overview & Main Objectives

PARTS OBJECTIVES

Introduction Module Review Main Objectives

Unit I:

Basic Definitions

Basic definitions Identification of main evaluation criteria Examples of their application

Unit II:

Methods of Evaluation

The 5 WH- questions model Presentation and implementation

Unit III:

Other Models and Tools

The MERLOT model The LORI model The EFQUEL network

Unit IV:

Practical Tips & Final Remarks Create your own evaluation criteria



Unit I: Basic Definitions



4

What is evaluation?

Evaluating is the process of determining merit, worth, or significance; an evaluation is a product of that process.

(Scriven 2007)

5

What is a learning resource?

A learning resource is information represented, accessible or stored in a variety of media and

formats (texts, videos, software etc.), which assists an individuals’ learning as defined by the learning outcomes and/or the overall learning objectives.

6

Why do we evaluate resources?

The main purpose of an evaluation is to determine the quality of a learning resource. The quality of a learning resource is dependent on: • its properties and features • the context of the resource such as time, place and

human relationships. Quality is also connected to how the resource is used, when it is used, and in relation to whom it is used.

7

Why do we evaluate resources?

• We undertake an evaluation to check and measure the quality of something.

• Quality depends on two different types of factors: • There are intrinsic factors, i.e. internal

features of the object we have to evaluate;

• There are external factors, i.e. external and contextual conditions.

• Quality, therefore, is not a fixed idea with a fixed value; quality is a relative concept, which varies depending on the intrinsic and extrinsic factors.

8

How do we evaluate?

Evaluation is not a single and isolated process. It is a systematic approach, based on a number of criteria such as:

• Potential Effectiveness as a Learning Tool; • Ease of Use (for Practitioners and Learners); • Accuracy; • Appropriateness; • Clarity; • Completeness, etc. See Unit II for an example of

evaluation process and set

of criteria

9

FOR/ BY WHOM (target groups)

TO DO WHAT (objectives)

IN WHICH CONDITIONS (constraints)

Example: • FOR 5 welders • BY 1 external

expert

Example: To be able to satisfy a new customer.

Example: • Very close deadline • Limited time by the trainer • Very busy period for

production

The 3 main aspects to evaluate?

We can start approaching evaluation by analysing three main aspects, namely the objective(s); the target group(s) and the context

10

Exercise 1

To complete this quiz, rate the importance of the factors which are provided in the table, based on the descriptions given. Rating is between 1 and 5, where 5 is most important and 1 is least important.

FACTOR DESCRIPTION RANKING

TARGET GROUPS • Training for7 administrative staff from 2

companies

• To be facilitated by an external tutor

2

1

OBJECTIVES • Prepare the administrative staff of 2 companies

to manage customers from the Russian market

5

CONSTRAINTS • 3 months to deliver the programme;

• A mininum of a grade A2 to required from each

learner to pass the programme;

• There is a low-medium budget allocated.

4

5

5

11





Unit II: Methods of Evaluating

12

The Scope of this unit is to perform a more in-depth analysis of the learning resources without getting into complex and time-consuming studies and preliminary preparation.

Adapt a well-known outline for structuring written texts to the evaluation process, as it is based on a small set of key points that can correspond to key aspects of resources to be evaluated.

Suggest possible improvement.

Scope & Objectives

Objectives

13

This methodology is an adaptation of the 5-W question model which can be used to help in planning, writing and revising text: It is based on 5 key factors (who, what, why, when,

where) to identify and analyse the 5 most important factors that:

(a) define the context of the learning resource by establishing how it was used;

(b) affect and determine its quality.

The 5 ‘W’ Question Model

14

• Who are the learners? (age; role; background, etc.)

• Who is their possible trainer? (internal, external etc.) WHO

• What kind of resources do they need most?

• Are online resources easy to access for them? WHAT

• Outline the training objectives.

• Is the training being delivered a new or existing curriculum? WHY

• When can the training be performed?

• When is the deadline for completing the assignments? WHEN

• Planning where the training will take place and ensuring all equipment (incl. technical) are in place for training. WHERE

Asking the 5 ‘W’ Questions

15

• The following slide shows an example of how the 5-WH grid can be completed.

• The grid guides you to focus your attention on different factors and elements that can determine the quality of the resources you have to evaluate. • WHO corresponds to the target group the resources

are addressed to; • WHAT and WHY correspond to the learning objectives

and content of the learning; • WHEN and WHERE correspond to the planning and

implementation of the learning.

Sample Answers for the 5 ‘W’ Questions

16

• 9 individual aged 25-47; administrative staff, no previous experience in project designing.

• No internal trainer available. WHO

• They need resources for project design.

• They are familiar to online resources and know how to access them.

WHAT

• This training is from an established curriculum, therefore the training outline has already been developed to adhere to external quality standards.

WHY

• The training will take place from 9.00am until 5.30pm on Wednesdays for six consecutive weeks. WHEN

• The training will take place in a local training centre.

• Wi-fi connection is available and there are facilities for PowerPoint.

WHERE

Answering the 5 ‘W’ Questions

17

Possible findings from the 5 ‘W’s

• On the basis of the previous assessment using the 5-WH method, one can deduce that the proposed learning resources should be: Intended to support and guide the learning

process of individuals involved in a training programme;

Available and accessible online; Relevant to the learner’s needs; Practical and should be applicable to real-world

working situations, or contain real-world case-studies.

18

Exercise using the 5 ‘W’s

The following slides contains another short exercise relating to the 5 ‘W’s. • In the table provided, one column contains 5

statements which form sample answers to the 5 ‘W’ Questions.

• In the next column, there is a list of learning resources. • The aim of this exercise is to match the data from the 5

statements and identify which learning resource is most suitable.

• You have 10 minutes to complete this exercise.

19

Sample 5 ‘W’ Answers Sample Resources

1. 10 low-skilled workers of a company working in

cleaning and gardening sector, all aged over 55

Online resources

including vdeos and

simulations could be

very useful but are not

suitable to the target

group;

Leaflets with images,

graphs and lists of

content & check-lists

could be better.

Mobile learning can

be used (where

applicable) to help to

organise the workers’

learning tasks

2. Health, safety and first-aid course; 2 of the learners

have to become “health & safety manager” of their

unit

3. To comply with safety regulations which are legally

required

4. Training will take place at a time that suits the

workers because it is urgently needed by this is the

company’s peak time for contract work

5. The company has access to training facilities and

computer labs where resources can be accessed

Exercise 2: Find the Right Resource

20





Unit III: Other models and tools

21

The scope of this unit is to provide a set of criteria for assessing the quality of learning resources. These criteria are used in conjunction with the following tools:

• MERLOT • LORI

This unit also aims to explore the opportunities and services available for e-learning resources via the EFQUEL network.

Analyse the quality of selected learning resources to decide which are more respondent to the identified needs.

Scope & Objectives

Objectives

22

Introducing MERLOT

• MERLOT stands for Multimedia Education Resource for Learning and Online Teaching.

• MERLOT is an online repository. It has been developed by an international consortium of institutions (and systems) of higher education, industry partners, professional organizations and individuals.

• MERLOT is a community of staff, volunteers, and members who work together in various ways to provide users of OER (Open Educational Resource) teaching and learning materials with a wealth of services and functions that can enhance their instructional experience.

23

Introducing MERLOT

MERLOT’s partners and members are devoted to identifying, peer reviewing, organising and making available existing online learning resources in a range of academic disciplines for use by higher education faculty and students. These disciplines include:

1. Arts

2. Business

3. Education

4. Humanities

5. Mathematics and Statistics

6. Science and Technology

7. Social Sciences.

These resources can be accessed by visiting www.merlot.org

24

MERLOT Quality Insurance Process

• All MERLOT materials are examined in various ways to ensure that they are useful for the MERLOT community.

• The Quality Assurance Process in based on Peer Reviewing. • The Peer Review process is led by an Editor, and includes

editorial board members and peer reviewers.

• The MERLOT editorial boards provide leadership, tools, and training in developing evaluation standards and processes for peer review which include: • Quality of content • Potential effectiveness as a teaching tool • Ease of use

25

Evaluation Standards Key Questions

Quality of Content

1. Does the resource present valid (correct) concepts, models and skills?

2. Does the resource present educationally significant concepts, models and skills for the discipline?

Potential Effectiveness as a

Teaching-Learning Tool

1. In what stage(s) in the learning process/cycle

could the materials be used?

2. What is(are) the learning objective(s)? What should students be able to do after successfully learning with these materials?

3. What are the characteristics of the target group(s)?

Ease of Use 1. How easy it is for teachers and students to use

the resource for the first time?

MERLOT Quality Criteria

26

The second model and method is LORI. • LORI stands for Learning Object Review Instrument. • It is available through e-LERA (www.eLera.net), and is a

collaboration between a community of teachers, students, university faculty, researchers and instructional designers who all share an interest in the use and evaluation of learning resources.

• LORI is an effective tool for evaluating learning resources. • The reviews, in turn, facilitate teachers’ searches for relevant

and high quality resources, and also provide a means of developing best practices in the development of learning resources.

Introducing LORI

27

eLera is a collaboration among a group that researches and

evaluates e-learning. Their specific interests include online learning resources, e-

portfolios and learning design specifications, and related topics.

eLera provides tools and information for learning resource evaluation and research, maintains a database of learning resource reviews, and supports communication and collaboration among researchers, evaluators and users of online learning resources.

Introducing eLera

28

• LORI provides an easy-to-follow list of broad, yet targeted categories that allow reviewers to quickly assess a learning resource’s strengths and weaknesses.

• Each item has an associated 5-point rubric, and reviewers are also encouraged to provide comments with their reviews.

• Version 1.5 of LORI includes the 9 items which are as follows:

1. Content Quality 2. Learning Goal Alignment 3. Feedback and Adaptation 4. Motivation 5. Presentation Design 6. Interaction Usability 7. Accessibility 8. Reusability 9. Standards Compliance

LORI Quality Dimensions

29

1. Content Quality Authentic, relevant, accuracy, balanced presentation of

ideas, and appropriate level of detail

2. Learning Goal

Alignment

Alignment among learning goals, activities, assessments,

and learner characteristics.

3. Feedback and

Adaptation

Adaptive content or feedback driven by differential

learner input or learner modelling

4. Motivation Ability to motivate and interest an identified population

of learners

5. Presentation Design Design of visual and auditory information for enhanced

learning and efficient mental processing


30

6. Interaction Usability Ease of navigation, predictability of the user interface,

and quality of the interface help features

7. Accessibility Design of controls and presentation formats to

accommodate disabled and mobile learners

8. Reusability Ability to use in varying learning contexts and with

learners from differing backgrounds

9. Standards

Compliance Adherence to international standards and specifications


31

• EFQUEL is the leading international network in the field of quality in education.

• EFQUEL’s services cover training, review services and conformity checks for higher education and other educational organisations and consulting, as well as certification through EFQUEL’s E-Quality Label.

Introducing EFQUEL

For more information, visit: www.efquel.org

32

• EFQUEL has developed a service portfolio reacting to its members’ needs.

• The services are divided into three categories: 1. Quality services: Quality Services are designed to help members to improve educational quality through using ICT 2. Knowledge services: These provide members with comprehensive information on current developments in several areas of educational innovation, such as quality for open education, MOOCs, or quality management for technology-enhanced learning. 3. Training services: EFQUEL organises workshops, expert seminars, short courses and summer schools in EU countries

Introducing EFQUEL

33

• The E-Quality Review Services are providing organisations with individual experts’ reviews and consultancy, as well as compliance checks for courses, programmes or departments.

• Often these are considered as a first step into certification.

e-Quality Label & Review Service

34

Delivering an official certificate

Being free

Meeting the needs of the target group

Validated by a recognized authority

Easiness of use

Designed to be re-used

Being adaptive and feedback driven

From the list below, select the statements which, according to the 2 models

analysed, are quality criteria for learning resources Answer True or False:

TRUE

FALSE

FALSE

Exercise 3:

TRUE

TRUE

TRUE

FALSE

35





Unit IV Practical Tips & Final Remarks

36

After completing this module, you are now more familiar with the models and tools you can use to evaluate learning resources. You are now equipped with the basic knowledge which allows you to contribute to resource evaluation and to provide useful feedback. From this module, you have learned that: 1. There are several models and methodologies to assess the quality of

learning resources, but the main aspects to take into considerations can be summarised into 3 categories: Objective(s); Target group(s); Constraints.

2. Every lesson has to be adapted to the fit the needs of the learners and the learning environment.

3. Remember that feedback is crucial. You should always be open to receiving feedback and using this feedback to revise and improve the resource.

Conclusion & Summary

37

Thank you for your interest and attention

www.projectsynergy.eu facebook.com/pages/Synergy-for-the-SME-Business-Community www.linkedin.com/company/erasmus-synergy-project

38

• www.merlot.org

• Scriven, M. (2007). The logic of evaluation. In H.V. Hansen, et. al.(Eds),

Dissensus and the Search for Common Ground, CD-ROM (pp. 1-16).

Windsor, ON: OSSA.

• Leacock, T. L., & Nesbit, J. C. (2007). A Framework for Evaluating the

Quality of Multimedia Learning Resources. Educational Technology &

Society, 10 (2), 44-59.

• Prince Edward Island Department of Education (2008). Evaluation and

Selection of Learning Resources: A Guide, Charlottetown, Prince

Edward Island - Canada - http://www.gov.pe.ca/educ

• Student Achievement and Supports Branch Saskatchewan Ministry of

Education (2013). Learning Resources Evaluation Guidelines,

Saskatchewan Ministry of Education – Canada,

http://www.wncp.ca/media/49329/wncptool_%20assessing_validating.p

df

References from this Module

http://www.merlot.org/

http://www.gov.pe.ca/educ

http://www.gov.pe.ca/educ

http://www.wncp.ca/media/49329/wncptool_ assessing_validating.pdf

http://www.wncp.ca/media/49329/wncptool_ assessing_validating.pdf