Quality on line: What and how should we be measuring for

quality assurance in learning?

Prof. dr. Betty Collis

EDEN Research Workshop, 26 October 2006, Castelldefels, Spain

Moonen & Collis Learning Technology Consultants

bettycollisjefmoonen@gmail.com

Emeritus professor, University of Twente

Betty.Collis@Utwente.nl

Quality?

From whose perspective? Against what criteria? What do you do with what you find?

An illustration from practice

From whose perspective?

Institutional perspective?

Institutional perspective

Benchmarking:Self-improvement tool for

organizations Compare with others and/or reflect

internally Review comparative strengths &

weaknesses Identify possible routes to

improvement Rather than statistical indicators

of results (completion rates, unit costs, etc), focus on processes by which results are achieved

Accreditation reviewsExternal review body

• For licensing

• For comparisons among institutions

• Implications for funding

Items are typically responded to on a scale, with labels corresponding to 0=Not applicable, 1= No information available, 2=Not adequate, 3=Partly adequate, 4=Largely adequate, 5=Fully adequate, 6=Best practices processes

http://www.insight.strath.ac.uk/projects/risk/index.htm

How might external funding policies affect your e-learning strategy?

What are the objectives of your investment in e-learning? (ie., widening access?)

How are working practices and staff roles being affected? What are students’ expectations?

What structures are in place to support students? Staff?

Should you make, buy, update, or change?

How do your students access e-learning technologies and resources?

Is your curriculum development being affected by e-learning?

How is the quality of (e)-learning being monitored?

http://www.utdc.vuw.ac.nz/research/emm/index.shtml

How can Institutions help teaching staff to be more successful in their use of technology to support student learning?

Source: S. Marchall, (2005). New Zealand E-Learning Capability Determination. Available at

http://www.utdc.vuw.ac.nz/research/emm/Presentations.shtml

Learner perspective?

Example:

Student perceptions (5-pt scale) of: Flexibility Responsiveness and support Learning Participation/interaction Usefulness and ease of use of technology Overall satisfaction

McGorry, S. Y. (2003). Measuring quality in online programs. The Internet & Higher Education, 6, 159-177

http://www1.tqi.ac.uk/sites/tqi/home/index.cfm

Content perspective?

Content, assembly, metadata, standards

Output oriented

One set of stakeholders may have one (implicit) view on quality

Output oriented

Infrastructure (to achieve output goal)

Another set of stakeholders may have a different view on quality

Learning-process oriented

And some perspectives may not get represented in these clusters…

(Personal) processes?

Theory oriented?

Or may not be fully taken up in the quality priorities of other clusters

Content, assembly, metadata, standards

Thus quality reflects the perspective

The challenge: Try to combine the perspectives

Example of an integrated evaluation approach in a corporate learning context

Collaborative learning

Shell EP Learning Centre

Collis, B., Bianco, M., Margaryan, A., & Waring, B. (2005). Putting blended learning to work: A case study from a multinational oil company. Education, Communication and

Information, 5(3), 233-250.

Overall company impact & alignment

Multinational knowledge sharing

Supervisor engagement

Workplace learning, learning- environment design

Merrill’s Principles of instruction

Capture, reuse, from the business and from participant contributions

Competence framework

Participant assessment

LMS, VLE for strategic/pedagogic goals

Shell: overall company

Work-based activities

As part of an integrated approach: Generic principles of learning

“Learning is promoted when:1. Learners are engaged in solving real-world [i.e., business-

relevant] problems.2. Existing knowledge [in the learner or in his or her workplace] is

activated as a foundation for new knowledge.3. New knowledge is demonstrated to the learner [through seeing

it in action in the workplace].4. New knowledge is applied by the learner [to his or her current

workplace situation].5. New knowledge in integrated into the learner’s world

[workplace].” (Merrill, 2002, pp. 44-45, augmented with linkages relating to

the corporate context) Collis, B., & Margaryan, A. (2005). Design criteria for work-based learning: Merrill's First Principles of Instruction expanded. British

Journal of Educational Technology, 36(5), 725-738.

Merrill, D. (2003). First principles of instruction. Educational Technology Research and Development, 50(3), 43-59.

Regardless of the context, an approach for quality should include Merrill’s “first principles of instruction”

Merrill’s five first principles of instruction

And finally, what to do with the results?

Use an action research approach to integrate strategic planning, stimulating, implementing, monitoring and interpreting quality

Action research

For further ideas:

Prof. dr. Betty Collis & Prof. dr. Jef Moonen

Moonen & Collis Learning Technology Consultants, BV

http://bettycollisjefmoonen.nl

Footnote: This talk is not related to the report: “Quality on the Line” which is a benchmarking study

Quality On the Line: Benchmarks for Success in Internet-Based Distance EducationBy: Ronald Phipps, Jamie Merisotis, 2000; Institute for Higher Education Policy, Washington DChttp://www.ihep.com/organizations.php3?action=printContentItem&orgid=104&typeID=906&itemID=9239&templateID=1422