Peer Reviewer Training(f2f)

Quality MattersQuality Matters: : Inter-Institutional Quality Assurance Inter-Institutional Quality Assurance

in Online Learningin Online Learning

Updated January 2007

Proposed for use by

all QM trainers.

After this Peer Reviewer Training…

You should be able to:

• Describe the critical materials, processes, and administrative elements of the Quality Matters online course quality assurance program.

• Apply the Peer Course Review rubric to review online courses.

• Write useful recommendations for course improvement.

• Explain the QM scoring system.

• Describe the Peer Course Review process and your role in it.

QM Materials Overview

• Today’s Agenda

• Binder Overview

• QM Website

• QM Contact Information

• QM Resources

Introductions

• Pair up

• Share name, institution, job, and best distance learning practice.

• Introduce your partner to the rest of us. Briefly (in one sentence) describe your partner’s best practice.

About About Quality Matters Quality Matters

Peer Course Peer Course ReviewReview

FeedbackFeedback

CourseCourse

Course Meets Course Meets Quality ExpectationsQuality Expectations

Course Course RevisionRevision

Instructional Designers

InstitutionsFaculty Course Developers National Standards &

Research Literature

Rubric

Faculty ReviewersTraining

Quality Matters:Quality Matters: Peer Course Peer Course

Review ProcessReview Process

For Our Purposes, Quality Is…

• More than average; more than “good enough”

• An attempt to capture what’s expected in an effective online course at about an 85% level

• Based on research and widely accepted standards

85 %

What this process is NOT

• Not about an individual instructor

(it’s about the course)

• Not about faculty evaluation

(it’s about course quality)

• Not about “win/lose” or “pass/fail”

(it’s about continuous improvement in a supportive environment)

QM Collegial Review vs. Faculty Evaluation

A QM Review is• Ongoing

• Focus: course design

• Outcome: course improvement

• Non-threatening• Team approach that

includes faculty• Full disclosure to faculty

A Faculty Evaluation is• Single point in time

• Focus: delivery

• Outcome: decision on performance for promotion/tenure

• Win/lose situation• Confidential/secretive

Design vs. Delivery

The faculty member is integral to both design and delivery.

Course Design … is the forethought and planning that a faculty member puts into the course.

Course Delivery …is the actual teaching of the course, the implementation of the design.

QM is about DESIGN - not delivery or faculty performance

Distinguish Between Design vs. Delivery

Example: Discussion Board

Design: A discussion board has been planned into the course; students have been told how they should participate and how they can expect the faculty to participate.

Delivery:How often the faculty member actually participates in the discussion; what the faculty member actually says to students.

The Peer Review Team

• 3 faculty peer reviewers:– must be experienced online instructors– must attend QM training– one MUST be external to the course’s originating

institution– there must be a subject matter expert (SME)

on the team. NOTE: The SME could also be the external reviewer.

AND • Faculty course developer:

– access to rubric prior to review– involved in pre-review discussions– consulted during review

Your Point of View as a QM Peer Course Reviewer

• Take the students’ point of view• Advocate for the student• If you can’t find evidence that the

standard is met, don’t assume it is or isn’t there….. ask the faculty member.

Factors Affecting Course Quality

• Course design QM REVIEWS THIS

• Course delivery (i.e. teaching, faculty performance)

• Course content• Course management system• Institutional infrastructure• Faculty training and readiness• Student engagement and readiness

Underlying Principles of QM• The QM toolset and process are:

– based in national standards of best practice, the research literature and instructional design principles

– designed to promote student learning

– integral to a continuous quality improvement process

– part of a faculty-driven, peer review process

• Course does not have to be “perfect” but better than just “good enough.” (Standards met at about 85% level or better.)

Underlying Principles of QM• Process designed to ensure all reviewed

courses will eventually meet expectations

• Collegial review process, not an evaluation process

• Review team must include an external peer reviewer

• Course faculty or instructor considered part of the review team

What’s In It For Institutions …

• External validation

• Strengthen accreditation package

• Raise QA as a priority activity

• Access to a sustainable, replicable, scalable QA process

• Inform online course training & practices

• Provide professional development activities

QM Process Provides

• Institutional toolset and process to meet quality expectations:– Online course design– Student learning– Improved instruction– Assessment and feedback loops– Professional development

What’s In It For Faculty …• Improve online courses

• External quality assurance

• Expand professional community

• Review other courses and gain new ideas for own course

• Participation useful for professional development plan and portfolio

• Receive $150 for each completed peer course review

QM in Transition

• 2003 – August 2006– QM project funded by FIPSE grant money– materials and some services freely available

• August 2006 and beyond– QM project funded by MarylandOnline– Some limited materials will be freely available– Other materials available to individuals and

institutions at nominal fees– Institutional membership affords full access to

materials and services

The InstructorThe Instructor Worksheet Worksheet

Instructor Worksheet

• Important part of review

• Includes info such as:– Institutionally mandated objectives, materials,

practices, policies– Materials outside course pages– Types of interaction used & instructor’s

statement on the appropriateness of interaction in the course

– Additional items that may require review

Instructor Worksheet

• Read it first

• Refer to it during the review

• Use in team discussions

The Rubric The Rubric

The Rubric

• Eight standards: – Course Overview and Introduction– Learning Objectives – Assessment and Measurement– Resources and Materials– Learner Interaction– Course Technology– Learner Support– Accessibility

Key components must align.

What is Alignment?

Critical course elements

work together to ensure

that students achieve

the desired learning outcomes.

Course Learning Objectives (II)

Resources, Materials (IV) & Technology (VI)

Assessment and Measurement (III)

Learner Interactions & Activities (V)

Key Sections that Must Align

Key Standards that Must Align• Objectives

– Standard II.1: Measurable outcomes– Standard II.2: The module/unit learning objectives describe

outcomes that are measurable and consistent with the course-level objectives.

• Assessment and Measurement– Standard III.1: Measures objectives; consistent with learning

activities

• Learner Interactions and Activities– Standard V.1: Help students achieve the objectives

• Course Materials– Standard IV.1: Deep and comprehensive enough for students to

achieve the objective

• Course Technology– Standard VI.1: Tools and media support the objectives

Rubric ScoringStandards Points Relative Value

14 3 Essential

12 2 Very Important

14 1 Important

TOTALS

40 80

• Team of three (3) reviewers

• One score per standard based on team majority

• Assigned point value; not sliding scale

Assigned Point Values

If the standard

is met …

If the standard

is not met …

Essential

Standards

3 points 0 points

Very Important

Standards

2 points 0 points

Important

Standards

1 point 0 points

Points are NOT assigned on a sliding scale:

Awarding Points

Points are awarded for each standard based on

1. the team majority, and

2. the pre-assigned weighting of each standard

If 2-3 Reviewers believe that a standard is:• met, then the full pre-assigned points are awarded• not met, then zero points are awarded

How to decide….

For EACH standard:1. Read the specific review standard and the

annotation; review the examples, if needed.2. Look for evidence that the standard is met in this

course.3. Ask yourself: Does this course meet the standard

at an 85% or better level?4. Decide Yes or No and enter your answer in the

web-based rubric form.5. Include comments/suggestions as documentation.

To Meet Expectations…

A course must achieve:

• “Yes” on all 14 of the 3-point “essential” standards.

• A minimum of 68 out of 80 points

68/80 = 85%

Online & Hybrid Courses

• Rubric designed for application to fully online and hybrid/blended courses

• Same set of standards apply to both • How we achieve the standards may differ• For hybrids, focus on pedagogical integration

of online and F2F components• Refer to Course Format Definitions document

RecommendationsRecommendations

Writing a Useful Recommendation

Constructive Try to offer solutions, not just identify problems.

Specific Include a specific example of what is being recommended.

Measurable How will the ID/instructor know when the recommendation has been implemented?

Sensitive Avoid negative language. Keep recommendations and comments on a positive note.

Balanced Point out strengths as well as weaknesses.

When MUST you write a recommendation?

You MUST write a recommendation if you decide that the course does not meet the specific standard.

Improve these Recommendations• Assignment instructions weren’t clear.

– I wasn’t sure if assignment #1 required a written paper.

• You didn't tell the students how to find the additional resources.– I had trouble locating the resources; you may

want to put a link on the home page for easy access.

• The text on the page was too hard to read. – The green text on a blue background was

difficult to read. Suggest black on white.

Improve these Recommendations

• The learning objectives aren’t measurable.– Rephrase the learning objectives to include an

active verb (explain, distinguish, compare, etc.)

• The assessments are weak.– The assessments are unrelated to the learning

objectives. Review your learning objectives and make sure that they are reflected in your exams.

Hands-OnHands-OnPracticePractice

Hands-On Practice• Work in Pairs• Review the Anthropology Course

– Go to http://pgcconline.blackboard.com– Username is fipse– Password is fipse

• Focus on 14 essential (3 point) standards• Write 1 recommendation per standard

Steps1. Get your training pair assignment2. Find your computer in lab (2 per pair)3. One person logs into the course 4. Other person logs into QM rubric5. Read Instructor Worksheet for course

Then… follow facilitator’s instructions for first standard

General Standard I:Course Overview and Introduction

1.1: Navigational instructions make the

organization of the course easy to

understand.

1.2: A statement introduces the student to the

course and to the structure of the student

learning, and, in the case of a hybrid

course, clarifies the relationship between

the face-to-face and online components.

General Standard II:Learning Objectives (Competencies)

II.I: The course learning objectives

describe outcomes that are

measurable.

II.2: The module/unit learning objectives

describe outcomes that are measurable and consistent with the

course-level objectives.

General Standard III: Assessment and Measurement

III.1: The types of assessments selected

measure the stated learning objectives

and are consistent with course activities

and resources.

III.2: The course grading policy is stated clearly.

III.3: Specific and descriptive criteria are

provided for the evaluation of students’

work and participation.

General Standard IV: Resources and Materials

IV.I: The instructional materials support

the stated learning objectives.

IV.2: The instructional materials have

sufficient breadth, depth, and

currency for the student to learn the

subject.

General Standard V:Learner Interaction

V.1: The learning activities promote theachievement of stated learning objectives.

V.2: Learning activities foster instructor-student, content-student, and if

appropriate to this course, student-studentinteraction.

V.3: Clear standards are set for instructorresponse and availability (turn-aroundtime for email, grade posting, etc.)

General Standard VI:Course Technology

VI.I: The tools and media support the

learning objectives and are

appropriately chosen to deliver the

content of the course.

General Standard VII: Learner Support

• No 3-point elements

This standard has no “essential” 3-point elements because it’s primarily concerned with academic support services, student support services and technical support services….usually thought to be the primary responsibility of the institution and not the individual instructor.

General Standard VIII:Accessibility

VIII.1: The course acknowledges the importance ofADA requirements.

To meet this standard, the course must include BOTHof these elements:

• The course must be offered using software that is accepted as “ADA compliant.”.

AND• The course should include a brief statement that

clearly tells students how to access ADA services at the institution.

Showcase Your CoursesShowcase Your Courses

Course Showcase

• Think about your own course:– What do you do that meets QM

expectations?– What improvements might you make?

• Share your thoughts with your partner

• Summarize for the group

ScenariosScenarios

Evaluating Scenarios

• Divide into groups.• Discuss the scenarios that focus on

the 14 essential standards. • Take the “quiz” with your group.

Use the material in your books and each other as references.

• Jot down your reasoning and be prepared to discuss your decision.

Peer Course Review Peer Course Review ProcessProcess

About the Course

• QM is designed to review “mature” courses (taught at least two semesters)

• QM logo indicates year course met expectations

• Triggers for subsequent reviews:• Faculty request• More than 3 years since original review• New textbook or instructor• Professional or accreditation review pending

About the Review

• On average, a course review takes 7-10 hours

• Factors affecting review time include• Reviewer familiarity with the discipline• Reviewer familiarity with the CMS• Reviewer familiarity with the QM review process• Organization of the course

• Suggested review methodology:• Read Instructor worksheet• Familiarize yourself with CMS and course • Pre-review team discussion• Proceed through standards, but save Standard III

(Assessment and Measurement) for last• Post-review team discussion• Submit your review

Your Point of View as a QM Peer Course Reviewer

• Take the students’ point of view• Advocate for the student• If you can’t find evidence that the

standard is met, don’t assume it is or isn’t there….. ask the faculty member.

Timeline

• Active review period approximately 3 weeks

• Compiled reports due in approximately 6 weeks

• Teams:– Have pre-review discussion.– Set Team calendar.– Commit to 2-3 week review period.– Have post-review discussion.

Post-Review

• Reviewers – complete exit interview– receive stipends

• Faculty – receives Final Review Report– completes Faculty Response Form

Review Outcome• If meets expectations:

– Recognized by Quality Matters– Notifications distributed– ID support provided if requested

• If does not yet meet expectations:– ID support provided if requested– Team Chair and ID approve revisions– Course meets expectations

Roles and Responsibilities

Faculty Developer

1. Part of the review team.

2. Provides access to the course.

3. Completes Instructor Worksheet

4. Part of the initial team discussion

5. Receives compiled report

6. Returns Faculty Response Form

Peer Reviewers

1. Establish Team Calendar

2. Review the course individually

3. Complete the online web review form

4. Discuss review with Team as needed

5. Complete an Exit Interview

Receive… Recognition as Certified Peer Reviewer. Compensation

Team ChairReviewer Roles & Responsibilities plus….

1. Organizes Team calendar

2. Confirm Instructor Worksheet is used

3. Creates draft report from compiled reviews

4. Convenes Team discussions

5. Reviews, edits and submits Team Report

Receives… Recognition as Certified Peer Reviewer and Chair. Compensation

QM To DateQM To Date

QM to Date

• Overall Participation:– Individuals & programs from 130

institutions across 28 states

• Course Reviews:– 103+ courses reviewed– 18 MD schools; 10 non-MD schools

• Peer Reviewer Rubric Training:– 600+ trained

Multiple Uses of QM

Reported Uses of QM System:• Guidelines for initial online course development

• Quality assurance of existing courses

• Ongoing faculty professional development

• Institutional reaccredidation packages

• Formation of distance learning policies & steering committees

Awards - 2005

• WCET Outstanding Work (WOW) Award, November 2005.

• USDLA 21st Century Best Practice Award,October 2005.

• Maryland Distance Learning Association (MDLA) Best Program Award,March 2005.

QM in Transition

• 2003 – August 2006– QM project funded by FIPSE grant money– materials and some services freely available

• August 2006 and beyond– QM project funded by MarylandOnline– Some limited materials will be freely available– Other materials available to individuals and

institutions at nominal fees– Institutional membership affords full access to

materials and services

Research FindingsResearch Findings

Overall Course Review Results

Upon initial review:

• 53% meet expectations

• 22% do not meet expectations - missing at least one essential 3-point element(s)

• 25% do not meet expectations - missing at least one essential 3 point element(s) and a minimum of 68 points

Summary of Course Reviewsvariety of course management systems

– 50% Bb, 35% WebCT, 16% WebTycho, 2% other

– 65% of respondent Reviewers felt unfamiliarity with CMS did not impede review; 34% somewhat

• variety of disciplines– general studies, information technology,

teacher education, engineering, allied health– 75% of respondent Reviewers felt unfamiliarity

with discipline did not impede review; 25% somewhat

Analysis of Infrastructure

Whether a course meets/does not meet QM

expectations is NOT correlated with:

• Course’s institution of origin

• Course management system– 53% of Bb courses– 54% of WebCT courses– 50% of WebTycho courses

Analysis of Academic Area

Met Expectations Academic Area # Courses

78% information technology 9

60% math & statistics 10

55% allied health, health, nutrition, nursing 11

50% natural sciences 11

50% education 10

50% history, government, politics 6

50% behavioral & social sciences 12

50% english, writing, communications 10

33% arts & humanities 9

Post-Course Review

• Met expectations– Most faculty made suggested

improvements even though their course met expectations!

• Did not meet expectations– Improvements made or in progress – Most improvements made by faculty– Some ask for ID support

Common Themes

• Course reviews revealed 11 common areas for course improvement

• Elements that are missing in 20% or more of the courses reviewed

• These are potential targets for – faculty training– special attention in the initial course

development phase:

Common Areas for Improvement• Instructor self-introduction (I.4) 22%

• Activities that foster interaction (V.2) 22%

• Technology/skills/pre-req knowledge stated (I.6) 24%

• Links to academic support, student services, tutorials/resources (VII.2-VII.4) 24-27%

• Learning objectives at module/unit level (II.5) 27%

• Netiquette expectations (I.3) 32%

• Self-check/practice with quick feedback (III.5) 38%

• B/W alternatives to color content (VIII.4) 54%

• Alternatives to auditory/visual content (VIII.2) 59%

Serving as a Serving as a

QM Peer ReviewerQM Peer Reviewer

Quality Matters Peer Course ReviewerCertification Process

Training Course Review Experience

Quality Matters Certification

Attend QM Training

Trained

Assigned to Peer Review

Team

Submit Report

Certified

+ =

• Name on QM website

• Use of QM Logo

• Eligible for Peer Reviewer PoolDemonstrate

Competencies

Kane 1/15/05

Quality MattersPeer Course Review Process

1. Course Selected

2. Trained Review Team Assigned

3. MOUs and Instructor Worksheet

4. Pre-Review Discussion

5. Individuals complete reviews

7. Final Review Report

6. Team Discussion(s)

Next StepsNext Steps

Next Steps

• Within the next week– Return to “Foundations” Blackboard site– Complete Scenarios Quiz– Grade of 85% or greater required to

pass training and be a Peer Reviewer

Thanks to YOU…Thanks to YOU…Quality Matters! Quality Matters!