Qm rubric for training july 9

About Quality Matters

Underlying Principles of QM

QM is a faculty-driven, peer review process that is…

CollaborativeCollegialContinuousCentered - in academic foundation

- around student learning

For Our Purposes, Quality Is…

• More than average; more than “good enough”

• Attempt to capture what’s expected in an effective online course at about an 85% level

• Based on research and widely accepted standards

Factors Affecting Course Quality

QM Reviews Course Design ONLY

Design vs. Delivery

The faculty member is integral to both design and delivery.

QM is about DESIGN - not delivery or faculty performance

Course Design … is the forethought and planning that a faculty member puts into the course.

Course Delivery …is the actual teaching of the course, the implementation of the design.

Distinguish between design and delivery…

Example: Discussion Board

Design:

Discussion board planned in course; students told how they should participate and how they can expect the faculty to participate.

Delivery:

How often the faculty member actually participates in the discussion; what the faculty member actually says to students.

About The QM Rubric 2011-2013

Eight General Standards:

1. Course Overview and Introduction2. Learning Objectives (Competencies)3. Assessment and Measurement4. Instructional Materials5. Learner Interaction & Engagement6. Course Technology7. Learner Support8. Accessibility

Key components must align.

Alignment: Critical course elements work together to ensure that students achieve the desired learning outcomes.

Key Sections that Must Align

Course Learning Objectives (2)

Resources, Materials (4) & Technology (6)

Assessment and Measurement (3)

Learner Interactions & Activities (5)

What QM is NOT About…

• Not about an individual instructor (it’s about the course)

• Not about faculty evaluation (it’s about course quality)

• Not about judgment(it’s about diagnosis and improvement)

• Not about “win/lose” or “pass/fail” (it’s about continuous improvement in a supportive environment)

The Process

Peer Course Review Process

About the Course

• QM is designed to review “mature” courses (taught at least two semesters)

• QM logo indicates year course met expectations

• Triggers for subsequent reviews:– Faculty request– More than 3 years since original review– New textbook or instructor– Professional or accreditation review pending

The Peer Review Team

• 3 faculty peer reviewers:• must be experienced online instructors• must attend QM training• one MUST be external to the course’s originating institution• there must be a subject matter expert (SME)

on the team. NOTE: The SME could also be the external reviewer.

AND • Faculty course developer:

• access to rubric prior to review• involved in pre-review discussions• consulted during review

About the Review

• On average, a course review takes 7-10 hours

• Factors affecting review time include• Reviewer familiarity with the discipline• Reviewer familiarity with the LMS• Reviewer familiarity with the QM review process• Organization of the course

Review Outcome

• If meets expectations and is recognized:– Recognized by Quality Matters– Notifications distributed– ID support provided if requested and available– Recognized courses on QM website

• If does not yet meet expectations:– Instructor (and/or ID) make changes– Team Chair/Master Reviewer approve revisions– Course meets expectations and is recognized

The Rubric

Rubric Scoring

# Standards Points Relative Value Total Points

21 3 Essential 63

12 2 Very Important 24

8 1 Important 8

41 95

In an Official QM Review

• Team of 3 reviewers initially score individually• One score per standard based on team majority• Pre-assigned point value• Met/Not Met decision; All/None points• Consensus is NOT required

Points are NOT on assigned on a sliding scale…

Standards If the standard is met …

If the standard is not met …

Essential 3 points 0 points

Very Important

2 points 0 points

Important 1 point 0 points

To Meet Expectations…Two Thresholds

A course must achieve:

1.“Met” on all 21 of the 3-point “essential” standards.

2.A minimum of 81 out of 95 points (81/95 = 85%)

The 85% Rule

Two 85% Uses:

A minimum of 81 out of 95 points (81/95 = 85%)

Reviewers use their own gauge of 85% as to if a standard is met or not. Standards do not have to be 100% to be marked “Met.”

More About Alignment

Key Sections That Must Align

Course Learning Objectives (2)

Resources, Materials (4) & Technology (6)

Assessment and Measurement (3)

Learner Interactions & Activities (5)

Think About “Alignment”

Objectives From Intro Psych

Course : Summarize the relationships that exist between biology and human behavior

Module 1.Identify and define 7 major biological areas.2.Recognize examples of how each area affects behavior.

Assignment: Read Chapter 4 (assume it deals with relationships between biology and human behavior) in text; review study guide, objective 2, page 4.

Assignment: View Chapter 4 PPT; listen to Chapter 4 podcast

Graded Assignment:Prepare table that lists the 7 major biological areas; ask students to define/describe each area and list 3 specific ways this area affects human behavior.

Graded Discussion Forum Question: Psychology is the study of behavior and mental processes. Why, then, are psychologists interested in biological matters such as the nervous system, the endocrine system and heredity?Read and respond to classmates’ postings.

Exam: 50-item multiple choice exam taken in campus Testing Center.

The General Standards

General Standard 1

About the General Standard

Brief Description from the Rubric

To meet the General Standard

General Standard 1,Course Overview and Introduction:

The overall design of the course is made clear to the student at the beginning of the course.

The course introduction sets the tone for the course, tells students what to expect, and provides guidance to ensure they get off to a good start.

You must share your deep knowledge of course1.Organization2.Navigation3.Expectations (Policies)

Standard 1

Answer the following questions:

1.Do you know what to do first?2.Do you know how to start the course? 3.Note your initial reaction to the course: 4.Do you feel comfortable with the navigation? 5.What are your first impressions about how this course works?

General Standard 2




General Standard 2,Learning Objectives:

Learning objectives are measurable and are stated clearly.

The learning objectives establish a foundation upon which the rest of the course is based.

You must provide measurable, precise learning objectives at the course- and module-level objectives so your students will know what is expected of them.

Standard 2 as a Sentence

Quality is measurable objectives (2.1) and consistent module-level objectives (2.2), that are written from the student perspective (2.3), with instructions on how to meet them (2.4), that are appropriate for the level of the course (2.5).

Submitted by Sasha Thackaberry, Cuyahoga Community College

Standards 2.1 & 2.2

Write measurable course and module learning objectives that•precisely describe what students will gain from instruction •guide instructors to accurately assess student accomplishment.•are consistent

Measurable Learning Objectives

• Begin with an action verb• Precisely describe behavior that

can be observed or evaluated

Test of “Measurability”

A Measurable Learning Objective should complete this sentence:

Upon completion of this course/module, students will be able to (DO SOMETHING).

Helpful Hundred….

Remember Bloom?

http://projects.coe.uga.edu/epltt/index.php?title=Image:Bloom_1.jpg

Standard 2: Learning Objectives

Bloom In Practice

University of Victoria Counseling Services http://www.coun.uvic.ca/learning/exams/blooms-

taxonomy.html

Analyze Learning Objectives

• Complete the table (handout): Look at ANT 103 learning objectives, Apply Standards 2.1 through 2.5

Quality is

measurable objectives (2.1) and consistent module-level objectives (2.2), that are written from the student perspective (2.3), with instructions on how to meet them (2.4), that are appropriate for the level of the course (2.5).

Measurable? Precise?

General Standard 4




General Standard 4,Instructional Materials:

Instructional materials are sufficiently comprehensive to achieve stated course objectives and learning outcomes.

The instructional materials form the core of the course, and these standards respect the instructor’s prerogative in selecting them. The focus of this standard is on supporting the course objectives and competencies, rather than on qualitative judgments about the materials.

You must provide rich, robust and appropriate resources and materials that support your stated learning objectives.

Apply Standard 4.1

Standard 4.1 Met Not Met Helpful Recommendation

The instructional materials contribute to the achievement of the stated course and module/unit learning objectives.

General Standard 5




General Standard 5,Learner Interaction & Engagement:

Forms of interaction incorporated in the course motivate students and promote learning.

Engaging students to become active learners contributes to the learning process and to student persistence.

You must provide meaningful and productive activities that prompt your students to actively practice their learning.

Standard 5.1

Learning Objective Possible Aligned Activities

Analyze the causes of the Civil War and how they persist in our modern society.

1.

2.

3.

Standard 5.2

Type of Interaction

Student-Instructor

Student-Content

Student-Student

General Standard 3




General Standard 3,Assessment and Measurement:

Assessment strategies are designed to evaluate student progress by reference to stated learning objectives; to measure the effectiveness of student learning; and to be integral to the learning process.

Assessment is implemented in a manner that not only allows the instructor a broad perspective on the students’ mastery of the content, but also allows students to measure their own learning throughout the course.

Your assessments must align with your stated learning objectives and guide students to measure their own learning progress.

Apply Standard 3.1 using Learning Objective

Standard 3.1 Met Not Met Helpful Recommendation

The types of assessments selected measure the stated learning objectives and are consistent with course activities and resources.

General Standard 6




General Standard 6,Course Tools and Technology:

Course navigation and technology support student engagement and ensure access to course components.

The technology enabling the various course components facilitates the student’s learning experience and is easy to use, rather than impeding the student’s progress.

You must select the tools and media that best support your learning objectives and provide opportunities for your students to actively practice their learning.

Standard 6 Includes…

Course Tools –•functional software that provide areas for interaction in the course •Examples include forums, quizzes, surveys, blogs, social media, wikis, etc.

Media – •one-way delivery modalities that enhance learning•Examples include YouTube videos, podcasts, audio recordings, pictures, graphics, etc.

Standard 6.1

Technology Used

Course Tools

Media

General Standard 7




General Standard 7,Learner Support:

The course facilitates student access to institutional support services essential to student success.

In the learner support standard, four different kinds of support services are addressed: technical support, accessibility support, academic services support, and student services support.

You must direct your online students to the support services they need to be successful and would be available to them if they attended campus-based courses.

General Standard 8




General Standard 8,Accessibility:

The course demonstrates a commitment to accessibility for all students.

The accessibility standard incorporates the principles of Universal Design for Learning (UDL) and is consistent with Web Content Accessibility Guidelines (WCAG).

You must design your course so students with special needs can access course materials and be successful in your course.

Sites to Explore

Universal Design for LearningSite provides “a framework for •designing educational environments that enable all learners to gain knowledge, skills, and enthusiasm for learning•reducing barriers to the curriculum •providing rich supports for learning.”

WCAG•Web Content Accessibility Guidelines (WCAG)•Site provides documents that “explain how to make Web content more accessible to people with disabilities.”

Standard 8.1

Consider the following scenario:

An online Nursing course is offered through Desire2Learn and requires that students watch six videos that illustrate key concepts in critical care. These videos are provided by the textbook publisher and are not closed-captioned.

If you were reviewing this course, how you would apply Standard 8.1 and what helpful comments would you give the instructor?

Advice to a Colleague

Professor Dancealot Needs Help!

• Watch this brief You Tube video (You Tube: http://www.youtube.com/watch?v=1k8aeDUC9XQ)

• Apply your understanding of the General Standards and alignment.

• Convince the Professor that he needs to “practice alignment” if he wants to help his students meet the stated learning objectives of this course.

Spur Your Thinking

Course ObjectivesUpon completion of this course, you will be able to :

1. Perform the basic steps of foxtrot, waltz, swing, chacha, rumba and tango in proper dance position.

2. Demonstrate confidence on the dance floor.

3. Find your eternal dance partner.

Standard 3.1

Standard 4.1:

Standard 5.1:

Standard 6.1:

Serving as a QM Peer Reviewer

Peer Reviewer Eligibility

You must be ELIGIBLE:•Must complete this F2F Applying the QM Rubric workshop.•Must complete all assessments in the online Peer Reviewer Course

– Two weeks online– Includes review material– Objective and written assessments

•Must have recent (within last 18 months) online teaching experience.

How Do I Become a QM Peer Reviewer?

Register for Peer Reviewer Course

Sign up for a Peer Review Course athttp://qmprogram.org

Print your Certificate of Completion at MyQM!

Underlying Principles of QM

1. Based on national standards of best practice, the research literature and instructional design principles

2. The course under review does not have to be “perfect” but better than just “good enough” (Standards met at about 85% level or better.)

3. Integral to a continuous quality improvement process

4. Valued as part of a faculty-driven, peer review process

5. Designed to promote student learning

Underlying Principles of QM cont.

6. Designed to ensure all reviewed courses will eventually meet expectations

7. QM is a collegial review process, not an evaluation process

8. A review team must include an external peer reviewer

9. Set up so that the course faculty or instructor considered part of the review team

Who’s On the Team?

3 faculty peer reviewers– Must attend QM Training– Must be experienced online instructors– One MUST be external to the course’s originating

institution– Must include one subject matter expert (SME) on the

teamNOTE: The SME could also be the external reviewer– Chaired by Master Reviewer

ANDFaculty course developer (Instructor)

– Access to rubric prior to review– Involved in pre-review discussions– Consulted during review

FormalPeer Review Team

To Meet Expectations: Two Thresholds

A course must achieve:

1.“Yes” on all 21 of the 3-point “essential” standards.

2.A minimum of 81 out of 95 points

Online & Hybrid Courses

• Rubric designed for both• Same set of standards apply to both• How standards achieved may differ• For blended courses, focus on pedagogical

integration of online and onsite components

Your Point of View …

As a QM Course Reviewer, you should:

• Take the students’ point of view • Advocate for the student• Support your decisions with

– Citations from the standards and annotations– Evidence from the course

Thanks to YOU… Quality Matters!

More Information at: www.qmprogram.org

Education

Qm rubric for training july 9