Applying the QM Rubric 2011-13 Rubric ©2014 MarylandOnline

Applying the QM Rubric

2011-13 Rubric

©2014 MarylandOnline

Workshop Overview

• Today’s Agenda• Folder Overview• QM Website

www.qmprogram.org• QM Contact Information


http://www.qmprogram.org/

Login to MyQM

1. Go to MyQMhttp://www.qmprogram.org/myQM/

2. Click on Need an Account? link (lower part of page) or login if you already have an account

3. Review the MyQM site: My Account, My Activity, My Tools, and Workshops

4. Certificates


http://www.qmprogram.org/myQM/

After this workshop, you will be able to:

1. Identify the underlying principles of QM. (Recognize key QM underlying principles and concepts.)

2. Identify the critical elements of the QM quality assurance program, including the QM Rubric, materials, processes, and administrative components.

3. Apply the 2011-2013 QM Rubric to review online courses. 4. Decide if the SPCH 1113 course meets selected QM Rubric

standards. 5. Apply the concept of alignment. 6. Write helpful recommendations for course improvement by

citing annotations from the QM Rubric and evidence from the course.


And in addition, we expect that you will:

1. Collaborate and network with colleagues through substantive and timely interactions.

2. Reflect on key QM concepts and whether you wish to serve on a peer review team.

3. Explore the challenges of online teaching by familiarizing yourself with the QM rubric and the SPCH 1113 course.

4. Relate the QM rubric (standards and annotations) to your own course and consider changes that might benefit your online students.

5. Evaluate the QM APPQMR workshop.


Activity 1: Introductions

• Share name, institution, job, and best distance learning practice.

• Briefly (in one sentence) describe your best distance learning practice.


Activity 2: Trying it out with Standard 1.2

Standard 1.2: Students are introduced to the purpose and structure of the course.


Review Scenario

This course reflects a shift in the importance that the world outside the schools increasingly places on thinking and problem solving. Procedural skills alone do not prepare students for that world. Therefore, students deserve a curriculum that develops their mathematical power and an assessment system that enables them to show it. Assessments that match the current vision of school mathematics involve activities that are based on significant and correct mathematics. These activities provide all students with opportunities to formulate problems, reason mathematically, and make connections among mathematical ideas. Students engage in solving realistic problems using information and the technological tools available in real life. Moreover, skills, procedural knowledge, and factual knowledge are assessed as part of the doing of real life mathematics.

1. 5-Tests (Must be taken on campus) 4002. 15-Quizzes 753. Project (Optional) 50Total number of points possible 475 (no project)

525 (if you do the project)Grading Scale:90% to 100% = A 80% to 89% = B 70% to 79% = C 60% to 69% = D 0% to 59% = F

A student who earns a final average of 70% or more on all assessments of objectives and intended learning outcomes for the College Algebra course has successfully fulfilled the general education and other essential core skill goals. Any extra credit problems are given to the entire class. There are no extra credit problems for any one individual during any part of the semester.

You are reviewing a College Algebra course and find the following statement:


About Quality Matters


Underlying Principles of QM

QM is a faculty-driven, peer review process that is…

CollaborativeCollegialContinuousCentered - in academic foundation

- around student learning


Peer Course Review Process


For Our Purposes, Quality Is…

• More than average; more than “good enough”

• Attempt to capture what’s expected in an effective online course at about an 85% level

• Based on research and widely accepted standards


Factors Affecting Course Quality

QM Reviews Course Design ONLY©2014 MarylandOnline

Design vs. Delivery

The faculty member is integral to both design and delivery.

QM is about DESIGN - not delivery or faculty performance

Course Design … is the forethought and planning that a faculty member puts into the course.

Course Delivery …is the actual teaching of the course, the implementation of the design.


Distinguish between design and delivery…

Example: Discussion Board

Design:

Discussion board planned in course; students told how they should participate and how they can expect the faculty to participate.

Delivery:

How often the faculty member actually participates in the discussion; what the faculty member actually says to students.


Activity 3: Design vs. Delivery

• Find a partner (or two)• Share stories about your own

experiences with design vs. delivery (professional or personal)

• Decide on an experience that you’d like to share with the group.


What QM is NOT About…

• Not about an individual instructor (it’s about the course)

• Not about faculty evaluation (it’s about course quality)

• Not about judgment(it’s about diagnosis and improvement)

• Not about “win/lose” or “pass/fail” (it’s about continuous improvement in a supportive environment)


The QM Rubric


About The QM Rubric 2011-2013

Eight General Standards:

1. Course Overview and Introduction2. Learning Objectives (Competencies)3. Assessment and Measurement4. Resources and Materials5. Learner Engagement6. Course Technology7. Learner Support8. Accessibility

Key components must align.

Alignment: Critical course elements work together to ensure that students achieve the desired learning outcomes.


Key Sections that Must Align


Submitted by James Fowlkes & Brenda Boyd

QM Rubric – Standard Point Values

8 General Standards– 21 Standards are “Essential”– 12 Standards are “Very Important”– 8 Standards are “Important”

# Standards Points Relative Value

21 3 Essential

12 2 Very Important

8 1 Important


Activity 4: QM Rubric Puzzle

• Divide into groups of 3• You will have 10 minutes to:

– Decide which general standard each specific review standard supports

– Decide whether your standard is essential, very important or important.

– Note: your group does not have come to agreement on where to place the specific standard or its relative value; majority rules


The Faculty Developer (Instructor)Worksheet


Faculty Developer (Instructor) Worksheet

• The Voice of the Instructor• Key Piece in the review• Includes information about

– Institutionally mandated objectives, materials, practices, and policies

– Materials outside the course site– Types of interaction used and instructor’s statement on

the appropriateness of interaction in the course

• Read it before you begin the review• Refer to it during the review and in team discussions


Activity 5: Faculty Developer Worksheet

• Locate the Faculty Developer (Instructor) Worksheet in your packet

• Read the Worksheet, be sure to pay close attention to the instructor’s answers to:– Course- and Module-Level Objectives– Student Engagement– Technology– Accessibility– Assessment


Hands-On Practice


Goals for Hands-on Practice

1. Introduce the General Standards2. Use the Faculty Developer Worksheet3. Take an in-depth look at Alignment4. Make decisions on specific review standards

by looking at an online course5. Practice writing useful recommendations6. See how the recommendations affect a

course


Your Point of View …

As a QM Peer Reviewer, you should:• Take the students’ point of view • Advocate for the student• Support your decisions with

– Citations from the standards and annotations– Evidence from the course


Accessing the SPCH 1113 Course

1. Find a Partner

2. Read Faculty Developer (Instructor) Worksheet for course

3. Login to Principles of Speech (SPCH 1113)

• URL: https://sautech.blackboard.com/• Username: qmstudent• Password: qmstudent


https://sautech.blackboard.edu/



Strategies for Applying the QM Rubric

General Standard 1: Course Overview and Introduction


General Standard 1

About the General Standard

Brief Description from the Rubric

To meet the General Standard

General Standard 1,Course Overview and Introduction:

The overall design of the course is made clear to the student at the beginning of the course.

The course introduction sets the tone for the course, tells students what to expect, and provides guidance to ensure they get off to a good start.

You must share your deep knowledge of course1. Organization2. Navigation3. Expectations (Policies)


How To Decide… For EACH Standard

1. Read the specific review standard and the annotation; review the examples.

a. Identify key components: ideas, directions, examples, etc.b. Ask relevant questions

2. Look for evidence that the standard is met in this course.

3. Ask yourself: Does this course meet the standard at an 85% or better level?

4. Decide Met or Not Met.

5. Write Helpful Recommendations.


Look at a Helpful Recommendation


Review Standard 1.1

1.1 Instructions make clear how to get started and where to find various course components.

As a Reviewer:What do you do first? Where do you start?

(Refer to the “How to Decide” handout)


Read the Annotation

Instructions provide a general course overview, present the schedule of activities, guide the new student to explore the course website, and indicate what to do first, in addition to listing detailed navigational instructions for the whole course.

Instructors may choose to incorporate some of this information in the course syllabus. In this case, students should be directed to the syllabus at the beginning of the course. A useful feature is a “Read Me First” or “Start Here” button or icon on the course home page, linking students to start-up information.

Examples: 1. A course “tour” 2. Clear statements about how to get started in the course 3. A “scavenger hunt” assignment that leads students through an exploration of the different areas of the

course 4. A graphical table or diagram that depicts the relationship between the online and face-to-face portions of

a blended course

Blended Courses: Instructions in the online classroom make it clear to students that the course is a blended course, with both online and face-to-face components and activities. Instructions specify the requirements for participation in both the online and face-to-face portions of the course. The introductory information clearly states when and where students should participate each week, and a structured set of topics and a schedule are provided for each face-to-face meeting.


Questions to ask:1. Does the instructor

provide instructions about key course components?

2. Do the students know what to do first and how to start the course?

Annotations specify: 1. What to look for:

• Instructions: a general course overview, schedule, website

• Indication of what to do first• Detailed navigation instructions

2. Where to look: • Course syllabus • “Read Me First” or “Start Here” button

or icon

3. Examples

Standard 1.1 – How to use the Annotations


Apply Standard 1.1

• Apply Standard 1.1 to the SPCH 1113 course– Imagine that you are a new student (new to the course, the

discipline, and/or the Learning Management System)

• Consider the following questions: – Do you know what to do first?– Do you know how to start the course? – Can you easily find clear instructions?– Note your initial reaction to the course:

• Do you feel comfortable with the navigation?• What are your first impressions about how this course works?


Activity 6: Your Decision

MET NOT MET

Standard 1.1


Consequences of Your Decision?

In a formal QM review:

— What does a MET decision mean?

— What does a NOT MET decision mean?


Importance of Learning Objectives

General Standard 2:Learning Objectives (Competencies)


General Standard 2




General Standard 2,Learning Objectives:

Learning objectives are measurable and are stated clearly.

The learning objectives establish a foundation upon which the rest of the course is based.

You must provide measurable, precise learning objectives at the course- and module-level objectives so your students will know what is expected of them.


Standard 2 as a Sentence

Quality is measurable objectives (2.1) and consistent module-level objectives (2.2), that are written from the student perspective (2.3), with instructions on how to meet them (2.4), that are appropriate for the level of the course (2.5).

Submitted by Sasha Thackaberry, Cuyahoga Community College©2014 MarylandOnline

Review Standard 2.1

2.1 The course learning objectives describe outcomes that are measurable.




Review Standard 2.2

2.2 The module/unit learning objectives describe outcomes that are measurable and consistent with the course-level objectives (alignment).




Standard 2.1 and 2.2How to apply the Annotations

Questions to ask:1. Are there course level

objectives?2. Are there module level

objectives?3. Are they measurable?4. Are they precise?5. Do they provide criteria for

assessment?6. Do they describe a behavior

that can be observed or evaluated?

Annotations specify: 1. What to look for:

– Measurable course and module learning objectives precisely describe what students are to gain from instruction and provide the criteria instructors need to accurately assess student accomplishment.

– Objectives describe student performance in specific, observable terms.

2. Examples3. Special situations4. Alignment


Standards 2.1 & 2.2 – Your Job

Write measurable course and module learning objectives that• precisely describe what students will gain from

instruction • guide instructors to accurately assess student

accomplishment.• are consistent


Measurable Learning Objective

Component Example

Completes this sentence: Upon completion of this course/module, students will be able to (DO SOMETHING).

At the end of this course, you will be able to:

Begins with an action verb:See: Clemson’s Bloom’s Taxonomy Action Verbs

deliver

Precisely describes behavior that can be observed or evaluated

a carefully constructed persuasive speech to an audience


http://www.clemson.edu/assessment/assessmentpractices/referencematerials/documents/Blooms%20Taxonomy%20Action%20Verbs.pdf

Activity 7: Measurable? Precise?

1. Realize the significance of ethical behavior in the business environment.

2. Document the critical events leading to the beginning of World War I.

3. Identify the seven most serious risks associated with cigarette smoking.

4. Understand the continuing impact of World War II on the European Union.

5. Demonstrate the effects of improper blood handling in a critical care environment.

6. Describe how to create a financial statement using MS Excel.


Cassinelli, C. (2008). Bloom's Taxonomy: Original and revised. In edTEch Vision, Updated Bloom’s Taxonomy. Retrieved September 21, 2012, from http://edtechvision.org/?p=123

Bloom’s Taxonomy


http://edtechvision.org/?p=123

Action Verbs

generate, plan, produce

judge, check, critique

differentiate, organize, attribute

execute, implement

interpret, exemplify, classify, summarize, infer, compare,

and explain.

retrieve, recognize, recall


Cassinelli, C. (2008). Bloom's Taxonomy: Original and revised. In edTEch Vision, Updated Bloom’s Taxonomy. Retrieved September 21, 2012, from http://edtechvision.org/?p=123

http://edtechvision.org/?p=123

Mnemonic Device: Bloom’s Taxonomy

R U At An Excellent College?


Bloom’s For Students

University of Victoria Counseling Services http://www.coun.uvic.ca/learning/exams/blooms-taxonomy.html


http://www.coun.uvic.ca/learning/exams/blooms-taxonomy.html

http://www.coun.uvic.ca/learning/exams/blooms-taxonomy.html

In search of … a “precise” verb

Verbs from Bloom’s TaxonomyOld and New Version

A list of verbs that focus on performance (what the students will do)


http://dox.utdallas.edu/chart1057

Activity 8: Learning Objectives Worksheet

For this activity:• Find a partner• Locate the Learning Objectives Worksheet in your

packet• Review the SPCH 1113 course site• Complete the worksheet by applying Standards 2.1

through 2.5 to the SPCH 1113 course site


Practice Alignment

General Standards 2 – 6Objectives, Assessments, Materials,

Engagement, and Tools/Media


Key Sections That Must Align


Think About “Alignment”

Objectives From Intro Psychology:

Course : Summarize the relationships that exist between biology and human behavior

Module 1. Identify and define

7 major biological areas.

2. Recognize examples of how each area affect behavior.

Assignment: Read Chapter 4 (assume it deals with relationships between biology and human behavior) in text; review study guide, objective 2, page 4.

Assignment: View Chapter 4 PPT; listen to Chapter 4 podcast

Graded Assignment:Prepare table that lists the 7 major biological areas; ask students to define/describe each area and list 3 specific ways this area affects human behavior.

Graded Discussion Forum Question: Psychology is the study of behavior and mental processes. Why, then, are psychologists interested in biological matters such as the nervous system, the endocrine system and heredity?Read and respond to classmates’ postings.

Exam: 50-item multiple choice exam taken in campus Testing Center.


General Standard 3




General Standard 3,Assessment and Measurement:

Assessment strategies are designed to evaluate student progress by reference to stated learning objectives; to measure the effectiveness of student learning; and to be integral to the learning process.

Assessment is implemented in a manner that not only allows the instructor a broad perspective on the students’ mastery of the content, but also allows students to measure their own learning throughout the course.

Your assessments must align with your stated learning objectives and guide students to measure their own learning progress.


Review Standard 3.1

3.1 The types of assessments selected measure the stated learning objectives and are consistent with course activities and resources.




Questions to ask:1. Do the assessments

flow from the course and module/unit objectives?

2. Are they consistent with these objectives?

3. Do you (and will the students) recognize alignment?

Annotations specify: 1. Key Ideas (What to look for):

• Consistent with course and module level objectives.

• Alignment: Easy to see relationship between learning objectives and assessments. Clear that learning objectives guide students to mastery

2. Examples• Alignment • Lack of Alignment

3. Special situations: Course objectives are mandated and do not align with assessments

Standard 3.1: How to use the Annotations


Activity 9: Apply Standard 3.1

Standard 3.1 Evidence from the Course

Met or Not Met

Aligned?

The types of assessments selected measure the stated learning objectives and are consistent with course activities and resources.


General Standard 4




General Standard 4,Instructional Materials:

Instructional materials are sufficiently comprehensive to achieve stated course objectives and learning outcomes.

The instructional materials form the core of the course, and these standards respect the instructor’s prerogative in selecting them. The focus of this standard is on supporting the course objectives and competencies, rather than on qualitative judgments about the materials.

You must provide rich, robust and appropriate resources and materials that support your stated learning objectives.


Review Standard 4.1

4.1 The instructional materials contribute to the achievement of the stated course and module/unit learning objectives.




Questions to ask:1. What instructional

materials are used in this course?

2. Do these materials enable students to meet learning objectives?

3. Do I need to consult with the SME?


The course materials and resources enable students to achieve the stated learning objectives. Consult with the team SME (subject matter expert) and use common sense. Focus only on the alignment of the instructional materials with the learning objectives rather than attempt to evaluate the content.

2. Special situations: Use module level objectives if course-level objectives are not measurable

3. Alignment: Part of alignment concept



Activity 10: Apply Standard 4.1


Met or Not Met

Aligned?

The instructional materials contribute to the achievement of the stated course and module/unit learning objectives.


General Standard 5




General Standard 5,Learner Interaction & Engagement:

Forms of interaction incorporated in the course motivate students and promote learning.

Engaging students to become active learners contributes to the learning process and to student persistence.

You must provide meaningful and productive activities that prompt your students to actively practice their learning.


Review Standard 5.1

5.1 The learning activities promote the achievement of the stated learning objectives.




Questions to ask:1. What learning activities are

used in this course?2. Do these activities enable

students to meet learning objectives?

3. Are the learning activities engaging and varied?


Engage students in activities that directly contribute to the achievement of learning objectives.

2. Examples: readings, student presentations, science labs, class discussions, case studies, role playing, simulations, practice quizzes, tests, etc.

3. Special situations: blended courses4. Alignment: Part of Alignment Concept

Standard 5.1: Analyze the Standard


Activity 11: Standard 5.1



Met or Not Met

Aligned?

The learning activities promote the achievement of the stated learning objectives.

General Standard 6




General Standard 6,Course Tools and Technology:

Course navigation and technology support student engagement and ensure access to course components.

The technology enabling the various course components facilitates the student’s learning experience and is easy to use, rather than impeding the student’s progress.

You must select the tools and media that best support your learning objectives and provide opportunities for your students to actively practice their learning.


Review Standard 6.1

6.1 The tools and media support the course learning objectives.




StandardQuestions to ask:1. What tools and media are

used in this course (if any)?2. Do the tools and media

selected enable students to meet objectives?

3. Is the purpose of the tools and media clear?

Annotations specify: 1. Key Ideas (What to look for): The

tools and media align with objectives of the course by supporting assessments, instructional materials, activities.

2. Examples: tools and media; not required.

3. Special situations: use module objectives if course-level not measurable

4. Alignment: Part of Alignment Concept




Technology Used

Evidence From the Course

Met or Not Met

Aligned?

Course Tools

Media


Activity 13: Alignment Worksheet

• Complete the Alignment Worksheet if you haven’t already done so.

• Share and compare your alignment worksheets and any “epiphanies” you had with the group.


Write Helpful Recommendations

General Standards 7 – 8Learner Support and Accessibility


No Choice!

You MUST write a Helpful Recommendation

if you decide that the course does not meet

the specific standard.


Look at a Helpful Recommendation


Write a Useful Recommendation

Constructive Try to offer solutions, not just identify problems.

Specific Include a specific example of what is being recommended.

Measurable How will you or the ID/instructor know when the recommendation has been implemented?

Sensitive Avoid negative language. Keep recommendations and comments on a positive note.

Balanced Point out strengths as well as weaknesses.


Mnemonic Device

Can Someone Make Some Brownies?


Analyze A Recommendation

The Start Here button was a great idea, but when I read the information it contained, I still couldn’t tell exactly how to begin the course. It might be very helpful to include a prominent link and directions about what students should do next to actually begin the first lesson of the course.

Constructive

Specific

Measurable

Sensitive

Balanced


Activity 14: Improve Recommendations

For this activity you will work in groups. Each group will be assigned one of the recommendations listed below and will rewrite it using the “components” of an effective recommendation. Each group will share their recommendation.

1. Assignment instructions weren’t clear.2. You didn't tell the students how to find the

additional resources.3. The text on the page was too hard to read. 4. The learning objectives aren’t measurable.5. Your assessments are weak.©2014 MarylandOnline

General Standard 7




General Standard 7,Learner Support:

The course facilitates student access to institutional support services essential to student success.

In the learner support standard, four different kinds of support services are addressed: technical support, accessibility support, academic services support, and student services support.

You must direct your online students to the support services they need to be successful and would be available to them if they attended campus-based courses.


Review Standard 7.2

7.2 Course instructions articulate or link to the institution’s accessibility policies and services.


Refer to the “How to Decide” handout


Questions to ask:1. Does the institution

have a stated accessibility or accommodation policy?

2. Where are they? Can a student find them?

3. What if there is no stated policy?

Annotations specify: 1. Key Ideas (What to look

for): accessibility policies, accommodations; types of and how to access services

2. Examples: link to institution’s policy; access statement

3. Special situations: no institutional policy




Standard Link to the Institution’s formal policy included (evidence from the course)?

Statement to the students included?

Your Decision

Recommendations

7.2


Activity 16: Write a Recommendation

• Use the evidence you discovered in the SPCH 1113 course to write a recommendation for improving Standard 7.2

• Each team will share their recommendation and point out each component of an effective recommendation.


General Standard 8




General Standard 8,Accessibility:

The course demonstrates a commitment to accessibility for all students.

The accessibility standard incorporates the principles of Universal Design for Learning (UDL) and is consistent with Web Content Accessibility Guidelines (WCAG).

You must design your course so students with special needs can access course materials and be successful in your course.


Sites to Explore

Universal Design for Learning (UDL)Site provides “a framework for:

• designing educational environments that enable all learners to gain knowledge, skills, and enthusiasm for learning

• reducing barriers to the curriculum • providing rich supports for learning.”

Web Content Accessibility Guidelines (WCAG)Site provides documents that

• “explain how to make Web content more accessible to people with disabilities.”


http://www.advocacyinstitute.org/UDL/

http://www.advocacyinstitute.org/UDL/

http://www.w3.org/WAI/intro/wcag

Review Standard 8.1

8.1 The course employs accessible technologies and provides guidance on how to obtain accommodation.




Standard

Questions to ask:1. Does the course contain

a link to the LMS statement?

2. Are the other tools, content, or software accessible?

3. Do the students know how to obtain accommodation if needed?


LMS accessibility statement, documentation on any content, tools, and software, information on how to obtain accommodation

2. Examples: link to the LMS accessibility statement, link to other tools, content, or software accessibility, statement from instructor




Standard 8.1 - What to Look For

• Link to the Learning Management System Accessibility Statement• Students will be using Microsoft PowerPoint to create a

presentation:‒ Microsoft PowerPoint® will be used to create your e-portfolio. The

Microsoft Corporation has developed this software product with accessibility in mind. Product accessibility information is available on their website: http://www.microsoft.com/enable/default.aspx

• Students are asked to watch videos created by their classmates that are not closed captioned.– If you are unable to view the videos created by your classmates please

contact the Disability Services Center. The Disability Services personnel will work with you to ensure you have access to this content. The center can be reached by calling (555-555-1234) or email ([email protected]). In addition, you can review their website at: http://www.somecollege.edu/disabilityservices.

http://www.microsoft.com/enable/default.aspx

http://www.microsoft.com/enable/default.aspx

mailto:[email protected]


Standard Evidence from the Course

Decision Recommendations

8.1


Activity 18: Write a Recommendation

• Use the evidence you discovered in the SPCH 1113 course to write a recommendation for improving Standard 8.1

• Each team will share their recommendation and point out each component of an effective recommendation.


Effects of Helpful Recommendations

SPCH 1113 courseStandards 7.2 and 8.1


Activity 19: Course Improvements

• Read the decision and the recommendations from the review team for standards 7.2 and 8.1 (located in your packet)– Did you make the same decision?– What additional information did you include in your recommendations?

• Log in to the revised version of the SPCH 1113 course (login information located in your packet)– Did the recommendations from the review team help the instructor

improve the course?– What impact will the revisions have on the student’s experience?– Do you feel the revised course meets these standards?


The Peer Review Process


QM = Process and Rubric

Process Rubric

OFFICIAL Outcome: Earn QM recognitionMust follow official QM guidelines and procedures

Outcome: Improve courses, meet institutional goals, demonstrate commitment to quality

Outcome: Earn QM recognitionTool to assess online courses during formal QM review

Outcome: Improve courses, meet institutional goals

INFORMAL Outcome: Improve courses, meet institutional goals, demonstrate commitment to quality

Institutions determine use and procedures

Outcome: Improve courses, meet institutional goals, demonstrate commitment to quality

Guide to develop new online courses AND review and update online courses


Peer Course Review Process


About the Course

• QM is designed to review “mature” courses (taught at least two semesters)

• QM logo indicates year course met expectations

• Triggers for subsequent reviews:– Faculty request– More than 5 years since original review– New textbook or instructor– Professional or accreditation review pending


The Peer Review Team

• 3 faculty peer reviewers:• must be experienced online instructors• must attend QM training• one MUST be external to the course’s originating institution• there must be a subject matter expert (SME)

on the team. Note: The SME could also be the external reviewer.

AND • Faculty developer:

• access to rubric prior to review• involved in pre-review discussions• consulted during review


In an Official QM Review

• Team of 3 reviewers initially score individually

– One score per standard based on team majority

– Pre-assigned point value

– Met/Not Met decision; All/None points

– Consensus is NOT required

• To Meet Expectations…Two Thresholds– “Met” on all 21 of the 3-point “essential” standards.

– A minimum of 81 out of 95 points (81/95 = 85%)©2014 MarylandOnline

Points are NOT assigned on a sliding scale…

Standards If the standard is met …

If the standard is not met …

Essential 3 points 0 points

Very Important

2 points 0 points

Important 1 point 0 points


The 85% Rule

Two 85% Uses:

1. A minimum of 81 out of 95 points (81/95 = 85%)

2. Reviewers use their own gauge of 85% as to if a standard is met or not. Standards do not have to be 100% to be marked “Met.”


About the Review

• On average, a course review takes 7-10 hours

• Factors affecting review time include• Reviewer familiarity with the discipline• Reviewer familiarity with the LMS• Reviewer familiarity with the QM review

process• Organization of the course


Timeline for a QM Review

• Pre-Review: (1-2 weeks)– Instructor Worksheet– Conference Call– Set Team Calendar– Faculty Developer

• Active Review: (3 weeks)– Reviewers actively review course

• Post-Review: (1 week)– Have post-review discussion, if applicable

• Revisions: (14 weeks)– Course improvements made to meet standards


Formal Review Outcome

• If meets expectations:– Recognized by Quality Matters– Notifications distributed– Recognized on QM website

• If does not yet meet expectations:– Instructor (and/or ID) make changes– Team Chair/Master Reviewer approve revisions– Course meets expectations and is recognized


http://www.qmprogram.org/qmresources/courses/index.cfm

Self-Review Tool

• Used to do a self-review of online or blended courses• Available through MyQM >> HE tab >> My Tools >>

Course Review System• Course Review System

– Start a self-review• Lists each specific standard with links to the annotations• Allows you to make a decision and add a recommendation• Can save and return• Once all the standards are completed and saved you can “View” or

“Email” the results• Can do as many self-reviews as you want (one for every course you teach)• No one else has access to the self-review


Serving as a QM Peer Reviewer


Peer Reviewer Eligibility

You must be ELIGIBLE:• Must complete the Applying the QM

Rubric workshop.• Must complete all assessments in the

online Peer Reviewer Course– Two weeks online– Objective and written assessments– Practice Review

• Must have recent (within last 18 months) online teaching experience in a for-credit course.

How Do I Become a QM Peer Reviewer?


Reflections


Reflections

• What are the three most significant (or surprising) things you learned by participating in our APP course?

• What do you intend to do next in regards to beginning to actively use QM and the QM Rubric with your course and/or at your institution?

• Do you have plans to become a QM peer reviewer? Explain, please.

• Do you have any general questions about the course review process, the rubric, or Quality Matters?


Wrap Up and EvaluationLink to the online evaluation:


Thanks to you… Quality Matters!

For more informationvisit the QM website at:

www.qmprogram.org


http://www.qmprogram.org/

Documents

Applying the QM Rubric 2011-13 Rubric ©2014 MarylandOnline