– QM Higher Education Rubric, Fifth Edition, 2014 – Course Reviews and Course Certification – Professional Development and QM Certification
What’s In It for Faculty What’s In It for Institutions The QM Organization
Presenter
Presentation Notes
The agenda for this presentation includes an introduction to Quality Matters (QM), an explanation of the basic elements of QM, an overview of what QM has to offer faculty and institutions, and will close with a perspective on the QM organization.
About Us
Quality Matters
Quality Assurance through Faculty Development and Course Design
Academic & Non-Profit
• Not-for-profit subscription service
• Initiated in 2003 by MarylandOnline with funding from FIPSE
• Adopted by a large and broad user base – HE, K12, Publishers, Corporations, Government
U.S. Department of Education
Presenter
Presentation Notes
The Quality Matters project was initiated by the MarylandOnline (MOL) consortium, a voluntary, non-profit, association of post-secondary two and four-year institutions in the state of Maryland, U.S. MOL was established in 1999 to leverage the efforts of individual campuses whose academic leaders were committed to the expansion of online educational opportunities in Maryland through collaborative activities. MOL and its members cooperate to support and maintain a portal for online programs and courses in Maryland, engage in joint faculty training initiatives, develop joint online programs, share online courses through a seat bank arrangement, and pursue federal, state, and foundation support for a variety of distance learning initiatives. One of these initiatives was the Quality Matters project. In the spring of 2003, MOL submitted a proposal to the U.S. Department of Education’s Fund for the Improvement of Postsecondary Education (FIPSE) for the creation of a rubric for the design of online courses and a faculty-centered, peer review process for certifying the quality of online courses and of online components. The title of the proposal was “Quality Matters: Inter-Institutional Quality Assurance in Online Learning.” FIPSE awarded MOL $509,177 over three years (September 2003 – August 2006) to carry out the project. The agency was interested in this proposal among many that involved quality assurance in online education because the proposed standards would be inter-institutional and inter-segmental and because of the unique peer-to-peer structure of the proposed course review process. The project had the potential to yield a quality assurance tool that was both scalable and replicable, criteria that are fundamental to the FIPSE grant program. In July 2014, Quality Matters became its own stand-alone non-profit organization.
What is Quality Matters?
•A set of Standards (Rubric) for the design of online and blended courses • A peer review process (faculty to faculty) for reviewing and
improving online and blended courses • A course development tool for faculty and instructional
development staff •A professional development opportunity • An opportunity to participate in a community dedicated to quality in online courses
Presenter
Presentation Notes
In practice, QM is a set of quality Standards for online course design, a faculty-centered peer review process for online courses, a tool used by instructional design staff in their work with faculty, and a professional development opportunity for both faculty and staff.
The QM Approach
QM is a faculty-driven, peer review process that is…
Collaborative Collegial Continuous Centered
- in academic foundation - around student learning
Presenter
Presentation Notes
About The Content: The underlying principles of QM are a primary reason for its wide-spread adoption. Quality Matters provides a faculty-driven, peer review process that is…. Collaborative: QM was designed by and for faculty to share expertise and experience relative to the design of a course. Collegial: The course review process is a collegial discussion between faculty peers committed to Continuous quality improvement. It is not an evaluation. Centered in national standards of best practice, the research literature and instructional design principles designed to promote student learning.
Factors Affecting Course Quality
Presenter
Presentation Notes
There are many factors that affect the quality of an online course. Among these factors are:� the course design (the forethought and planning that goes into an online course,) the course delivery (i.e. the way the course is taught, also known as faculty performance), the course content, the learning management system and its functionality; technical support the institutional infrastructure (help desk, online library access, online tutoring access, etc), a faculty member’s training and readiness for online teaching, and the students’ role with respect to engagement and readiness for an online course. � QM reviews just one aspect of online course quality – Course Design.
QM’s Role in Quality Assurance
• The QM Rubric looks at course design • The harnessing of technology to deliver instruction and promote student learning
• QM provides a process for peer-to-peer feedback for faculty in the continuous improvement of their course and certifies courses as meeting shared Standards of best practice
• Quality Matters is not the complete answer to quality assurance for online education, but it can be a critical component
Presenter
Presentation Notes
QM is focused on the design of the course and it is not meant to resolve all quality assurance issues. For example, although reference is made in the Rubric Standards to technical support or student support services, QM is not designed to evaluate those aspects of quality assurance, rather QM strictly focuses on course design.
What QM is NOT About…
•Not about an individual instructor (it’s about the course) •Not about faculty evaluation (it’s about course quality) •Not about judgment (it’s about diagnosis and improvement) •Not about “win/lose” or “pass/fail” (it’s about continuous improvement in a supportive environment)
Presenter
Presentation Notes
It’s important to compare what QM is and is not: about course design, not about the instructor about course quality, not about faculty evaluation score that comes out of a QM course review is diagnostic rather than judgmental. It tells us how much revision a course may need. about continuous improvement, not about pass or fail
QM as an (Inter) National Standard
More than 800 subscribing institutions 48 US states; 6+ countries QM has trained more than 33,000+ faculty and
instructional design staff 5000+ national peer reviewers Broad application
- Higher Education, K12, Educational Publishing, Continuing and Professional Education
Presenter
Presentation Notes
Quality Matters has become a widely adopted program of quality assurance. It currently has more than 870 institutional subscribers across 47 states + District of Columbia, Guam and Puerto Rico, as well as subscribers in Canada, Saudi Arabia, Singapore, Australia, Ireland, and Greece. Statewide systems also subscribe to provide coordination and more economical access to QM tools for their member institutions. Please visit our website for a list of current subscribers. https://www.qmprogram.org/qmresources/subscriptions/subscribers.cfm?program=2 Quality Matters has trained well over 33,000 faculty and instructional design staff and has been recently recognized by both the Sloan Consortium and USDLA for excellence. Data about QM as of July 2014 QM Coordinators (formally IRs) – 1,462 Unique Individuals who have taken a QM professional development workshop – 32,991 National and International Facility serving as Peer Reviewers – 5,120 Master Reviewers – 681 Combined F2F and Online Training Facilitators – Over 500 F2F Training Facilitators – 271 QM Subscribers– Over 950 Implementation Plans Recognized – 31 Course Reviews – 7,202 (1,823 QM Managed)
MONTANA
WYOMING
IDAHO
WASHINGTON
OREGON
NEVADA
CALIFORNIA
ARIZONA
NORTH DAKOTA
SOUTH DAKOTA
NEBRASKA
UTAH
NEW MEXICO
TEXAS
OKLAHOMA
KANSAS
ARKANSAS
LOUISIANA
MISSOURI
IOWA
MINNESOTA
WISCONSIN
ILLINOIS INDIANA
KENTUCKY
TENNESSEE
MISS ALABAMA GEORGIA
FLORIDA
SOUTH CAROLINA
NORTH CAROLINA
VIRGINIA
WV
OHIO
MICHIGAN NEW YORK
PENN
MARYLAND
DELAWARE
NEW JERSEY
CONN RI
MASS
MAINE
VT NH
ALASKA
HAWAII
PUERTO RICO
VIRGIN ISLANDS
Current Subscribers
Consortium Subscribers Statewide Subscribers
QM: A National Standard
COLORADO
Presenter
Presentation Notes
The map provides another view of the wide-spread adoption of QM. The color red/rust indicates states where there are QM subscribers, blue shows the presence of large subscribing consortia, and yellow indicates statewide subscriptions. Even though this slide doesn’t indicate, we have subscribers in Canada, Greece, Guam, Singapore, Australia, and Saudi Arabia with other countries pending. Check out our global subscriber community map https://www.qualitymatters.org/
QM Basics
More than the sum of its three parts
Presenter
Presentation Notes
This section of the presentation will introduce the basic elements of the Quality Matters: The QM Rubric, professional development workshops and certifications, and course reviews.
Community and technical colleges, colleges and universities, K-12 schools and systems, government agencies, corporations, and other education-related organizations subscribe to QM and use specific Rubrics created for their own particular use.
The QM HE Rubric Eight General Standards:
1. Course Overview and Introduction 2. Learning Objectives
(Competencies) 3. Assessment and Measurement 4. Instructional Materials 5. Course Activities and Learner
Interaction 6. Course Technology 7. Learner Support 8. Accessibility and Usability
Key components must align.
Alignment: Critical course elements work together to ensure that students achieve the desired learning outcomes.
Presenter
Presentation Notes
Of the 8 general Standards, 5 contain Standards that address key components that must be in alignment. QM tries to take a holistic view of the course and that’s why it’s so important that the Learning Objectives ALIGN with the Assessments, Resources and Materials, Interaction, and Technology.
The Rubric is the core of QM
Consisting of: • 8 key areas (general Standards) of course quality • 43 specific review Standards
• 21 essential Standards • Detailed Annotations and examples of good
practice for all 43 Standards
Presenter
Presentation Notes
Note that the detailed explanations of the 43 review Standards is contained in the Annotations to each Standard. The Annotations are quite extensive and deal with the nuances in each Standard, issues of interpretation and examples of good and poor practice. QM has a pdf on our website of the Rubric Standards for public view. The fully Annotated version of the Rubric cannot be distributed outside your subscribing institution.
Peer reviews, internal reviews, self reviews and custom reviews
Quality Matters is a continuous improvement model for assuring the quality of online and online components of blended courses through a faculty review process. This circle graphic represents the continuous nature of a Quality Matters course review, which starts with a course developer submitting a course worksheet which is shared and discussed with the review team before the start of the course review. Reviewers work independently of each other, but communicate throughout the review process as necessary. Once the reviewers submit their forms, the online tool aggregates the points as well as the feedback and provides a report for the course developer. If the course meets standards at this point, the course will be recognized by QM. If it does not, the course developer will have an opportunity take all the feedback, make amendments to the course, and resubmit all as part of the single review process. There are only two possible states for a course submitted for a QM review – it is either under review or it has met Standards.
Official Reviews – follows QM policies/protocols • QM-Managed review - QM manages and pays review
team • Subscriber-Managed review - Subscriber manages and
pays review team
Unofficial Reviews • Internal/Informal subscriber reviews not required to
follow QM process • Self-Reviews – for review of own course • Custom Reviews
Types of Course Reviews
Presenter
Presentation Notes
There are two general types of course reviews – distinguished by whether or not they are recognized by QM. Unofficial reviews are what QM calls internal reviews and these are not required to follow the QM process. Official reviews, recognized by QM, can be done in one of two ways. Either QM, for a fee, manages the review and compensates the review team, or Subscribers (with the appropriate subscription) takes the management role. Most subscribers do both types of reviews. None of the subscription types come with “free” official course reviews. Subscriber-managed reviews are enabled under license with only a Full or Statewide Higher Education Subscription. The previous slide with the circle graphic refers to an Official Review. All subscribers have access to the Course Review Management System to initiate, manage, and maintain their unofficial reviews QM has a self-review tool, available to anyone with a QM subscription and an individual MyQM account, that guides and archives the review of one’s own course. The review report is saved in the individual MyQM account and can be emailed as desired.
QM = Process and Rubric Process Rubric
OFFICIAL
Outcome: Earn QM recognition in national registry
Must follow official QM guidelines and procedures
Outcome: Improve courses, meet institutional goals, demonstrate commitment to quality, benchmark on national standard
Outcome: Earn QM recognition in national registry
Tool to assess online courses during formal QM review
Guide to develop new online courses AND review and update online courses
Unofficial
Outcome: Improve quality assurance to meet institutional goals, demonstrate commitment to quality
Institutions determine use and procedures of CRMS tools
Outcome: Improve courses, meet institutional goals, demonstrate commitment to quality
Guide to develop new online courses AND review and update online courses
Presenter
Presentation Notes
It’s important to note that QM is both a Process and a Rubric (tool) for course reviews. There are significant benefits of using QM informally, but official QM recognition comes only if the “official” process is followed.
The Peer Review Team
On an official review, review teams consist of: Three QM-Certified Peer Reviewers
• Pre-requisite: Current online teaching experience & training Master Reviewer as Team Chair
• Peer Reviewer with additional experience and training One reviewer must be a subject matter expert At least one reviewer must be external to the institution sponsoring the course
AND Course Representative:
• access to Rubric prior to review • involved in pre-review discussions • consulted during review
Presenter
Presentation Notes
Three QM-certified faculty peer reviewers use the Course Review Management System (the fully automated Rubric tool) to complete the review. One of these reviewers must be a Subject Matter Expert (SME) and at least one must be external to the course’s home institution. The team chair must be a Master Reviewer. A single person could play multiple roles. They could be both the SME and the master reviewer, for example. The Faculty Course Developer (the Instructor) should be consulted by the team as questions or issues arise.
What to Expect from Course Reviews
• Consistency and Rigor
• Professionalism and Commitment
• Useful and Constructive Feedback
• Positive Outcomes for Faculty
• 61% of QM-managed course reviews meet standards upon initial review (but all can meet after amendment)
Presenter
Presentation Notes
A QM course review is a rigorous process, but there are many positive outcomes for faculty as course developers and as reviewers. In QM-Managed 61% of QM course reviews meet standards upon initial review. Subscriber-Managed 76%.
Peer Feedback
The peer feedback provided in a course review is collegial in tone and language; however, the reviews are rigorous and even courses that meet Standards will benefit from the detailed, specific, and relevant feedback provided.
QM Certification of Courses
Course reviews take 4-6 weeks; 20-week window for reviews, including any amendments, to be completed. Courses that successfully meet QM Rubric Standards in an official course review are eligible to carry the QM Certification Mark. QM recognition is provided by year recognized. QM logo on course and catalog; Inclusion in national registry on QM website
Presenter
Presentation Notes
QM Certification is provided by the year the course met Standards and is good for 5 years, unless the course undergoes significant change. It’s important to note that while QM Certification is the benchmark of quality online course design, a course that meets QM Standards is not considered “perfect.” Continuous improvement, a basic principle of QM, is encouraged and expected for all courses that carry the QM Certification Mark.
• Prepare faculty and staff to hold QM roles • Faculty Development Workshops
• Prepare faculty to design and improve courses
Presenter
Presentation Notes
QM workshops and certification courses are offered online, live, and onsite. The focus for some are on certification of roles necessary in the implementation of QM and others are more about faculty development.
Faculty Development Workshops
Course-focused Workshops • Applying the QM Rubric* (APPQMR) • Teaching Online: An Introduction to
Online Delivery • Designing Your Online Course (DYOC) • Improving Your Online Course (IYOC) • Designing Your Blended Course (DYBC)
*Note: Flagship training is also required for QM Implementation
Standards-Focused Workshops • Design that Welcomes Your
Students • Connecting Learning Objectives
and Assessments • Using Instructional Materials and
Faculty development workshops differ by those focused on working with the whole course, those that drill down to provide more time to work on specific QM Standards, and those that focus on helping faculty achieve alignment in their courses.
Workshops • Continuing and Professional Education Accelerated Peer Reviewer
Course (CPE) • K-12 Accelerated Reviewer Course for Higher Ed (K-12 RC
ACCELLFY14) • QM Coordinator Training – QM Subscribers only
Presenter
Presentation Notes
The certification courses for QM implementation include Peer Reviewer Certification, Master Reviewer Certification, and Certified Trainer courses. The QM Coordinator workshop is also required for implementation. The Applying the QM Rubric workshop is a prerequisite to these certification courses.
Peer Reviewers
Current Online Instructor
Applying the QM Rubric Workshop
(online or F2F)
Peer Reviewer Course (online)
Submission of application & MOU
(for certification)
Independent Facilitators
Applying the QM Rubric Workshop
Peer Reviewer Course
(certification not required)
Certified Facilitator Workshop
Institution on Full or Statewide Subscription
Master Reviewers
Certified QM Peer Reviewer
Experience on 2+ course reviews
Master Reviewer Workshop
Annual Re-certification
QM Coordinators (QMC)
Applying the QM Rubric Workshop
QM Coordinator Training
Institution may designate
additional QMCs
Capacity Building: QM Roles & Criteria
Presenter
Presentation Notes
For a cost effective, robust QM implementation, you will want to develop your internal capacity. To manage your own reviews, you will want to have your own certified Peer Reviewers and Master Reviewers. (You may always use reviewers and Master Reviewers from QM’s database, but you would be responsible for compensating them.) You will also need a QM Coordinators (QMC). All QMCs are STRONGLY encourage to take at least the APPQMR training and the PRC is also suggested. To conduct the training, you will need a certified QM-facilitator. And you would need the licenses that come with the Full Subscription option.
By Faculty for Faculty
Quality Matters: By Faculty for Faculty
What’s In It for Faculty?
Roles for Faculty in QM
Peer Reviewers
Applying the QM Rubric Workshop
Peer Reviewer Certification
(online)
Current online instructor
Master Reviewers
Certified QM Peer Reviewer
Experience on 2+ course reviews
Master Reviewer
Certification
Presenter
Presentation Notes
The primary roles for faculty in QM are as Peer Reviewers and Master Reviewers.
Tools in the Spirit of Collaboration
• Standards grounded in the research literature and best practices
• Tools and process developed by faculty and instructional design experts for faculty support
• Peer review and other opportunities to collaborate and mentor around quality in online learning
• Research opportunities on teaching and learning
Presenter
Presentation Notes
The tools and processes in QM were developed collaboratively by faculty for faculty. Peer review using research-based tools and collegial interactions are hallmarks of QM.
What’s In It For Faculty?
Review other courses and gain new ideas for own course; expand professional community • Improve student learning • External quality assurance and course
validation based on research and best practices
• Professional development plan and portfolio • Faculty development workshops • Become part of a community dedicated to
quality online learning
Presenter
Presentation Notes
For faculty, in particular, Quality Matters provides a number of benefits. The direct and primary benefit of an improved course is obvious. But undergoing and participating in a peer review process is an enriching activity that provides external quality assurance in the spirit of the collaboration. Quality Matters has an extensive pool of peer reviewers that provide a professional network connected by the association with Quality Matters and its emphasis on collaboration to improve student learning outcomes. Participation in Quality Matters is a meaningful professional development tool.
Research Opportunities and Support QM encourages research through
– Consultation with QM Director of Research – Collaboration on research and data resources – Participation in faculty surveys of all QM
subscribers – Sponsorship of research - see RFP at
www.qualitymatters.org/research-grants-0 – Online research library and tool kit
Presenter
Presentation Notes
QM also provides research opportunities, to include issuing annual RFPs and funding a small number of research projects on QM-related topics.
This slide is a screen shot of the searchable research library available on the QM website. It connects specific review standards to the existing literature.
Standards Supported by Research
Presenter
Presentation Notes
The Rubric is research-based and was developed in conjunction with best practices in the industry. (QM Standards come from a detailed review of existing research literature, updated every few years, from existing best practice sets, and from our community of practitioners’, that provide feedback on every new Rubric version.) QM focuses on the Standards needed in course design and to promote student learning. This slide is an excerpt from the research library available on the QM website, that shows how the specific review standards are supported by existing literature. A literature review is conducted for each new Rubric edition that builds upon this core matrix.
What Faculty are Saying
Sample comments from course representatives review exit survey
• “I was too close to see what could be improved.”
• “Provides a great way to get an objective view of your course.”
• “It made all of my online courses better.”
• “It provides a view from a more student oriented perspective.”
• “Many elements that might contribute to a student withdrawing can be eliminated.”
What Faculty are Saying
Sample comments from Peer Reviewers on review exit survey
• “I always learn new wonderful approaches from viewing other courses and working with expert peer reviewers.”
• “Whenever I read the annotations, I make changes to my courses.”
• “I always find reviews to be a learning experience.”
• “This experience was very rewarding for me as both an educator and course developer.”
• “It’s a great tool for honing your own course.”
QM Resources
Quality Matters: A Multi-faceted Resource
What’s In It for the Institution
Multiple Uses of QM
Reported Uses of QM Tools: • Guidelines for initial online course development • Quality assurance of existing courses • Ongoing faculty professional development • Institutional re-accreditation • Examination of distance learning policies • Create a campus climate and structure that
promotes teaching and learning
Presenter
Presentation Notes
QM is used by schools in a number of ways and for different purposes. Many schools use the Rubric as a guideline for their initial development of courses. Many use if for quality assurance of existing courses. Often, QM professional development serve as an important piece of a schools faculty development efforts. Schools often specifically reference their use of Quality Matters in their re-accreditation packages. Finally, QM can be a useful tool in a school’s institution building efforts as it relates to their distance learning program. Early research on this topic is suggesting that QM can be a very useful tool to create and sustain a campus climate more focused on teaching and learning.
Reported uses: • Educate faculty about quality in online courses • Reduce time and labor in designing courses. • Provide professional development opportunities for
faculty and staff • Threshold requirement for adjunct faculty • Inform teaching practices related to design and
structure for BOTH online and F2F courses • Quality process indicators for accreditation self-studies
QM Professional Development
Presenter
Presentation Notes
In terms of the professional development benefits, the education of faculty on quality in course design provides professional growth opportunities for them in ways that directly benefit the institution.
Subscription Benefits
• Course Review Management System • MyQM, a personalized and secure portal & User’s Group • Training License* • Subscriber-Managed Course Reviews • Current QM Rubric and related Rubrics • Fully automated QM Rubric Tool • Subscriber Discounts • QMC Training • QM community dedicated to quality in online learning.
* Depends on subscription type
Presenter
Presentation Notes
Subscription benefits include access to the QM Rubric, the online Rubric tool (CRMS), subscriber discounts for trainings and services, QM Coordinator training, and participation in the online users group. The user’s group, a password-protected area of the QM website, serves as a resource for communication and information sharing, collaboration with other QM Subscribers, access to important documents and web links. The QM Coordinator training documents may be accessed through the QM Institute site. In addition, the Full Option includes the site license for training and the right to conduct your own official course reviews.
• MyQM Data, communications, reporting • CRMS Official Reviews, Internal Reviews, Self-Review • MyCR Custom Rubric Review System
QM Tools
MyQM is a personalized, secure portal for your QM data and tools. It is a real-time data repository for your institution’s complete QM Program history and involvement. • Provides access to some key tools in the implementation
of the QM program. • Offers at-a-glance access to your institution’s
The QM Course Review Management System (CRMS): • The CRMS is a secure, automated, online tool
exclusive to QM subscribers to manage course reviews.
• CRMS users manage official and internal course reviews.
• The CRMS is based on the Quality Matters Rubric, a set of Standards by which to evaluate or certify the design of online and blended courses.
Presenter
Presentation Notes
The Quality Matters' Course Review Management System (CRMS) is secure, automated, online tool used by the QM community to manage official and internal course reviews. The CRMS is based on the Quality Matters Rubric, a set of Standards by which to evaluate or certify the design of online and blended courses. Subscribers use the CRMS throughout the course review process to collect, store and share course information with both the review team and the review's sponsor. The CRMS tracks each review's progress from initiation through completion. Automated features guide each participant as they move through the process, notifying individuals by e-mail of actions required as the review progresses. The CRMS gathers reviewer feedback and recommendations and produces a final report of the review's outcome. If a course needs amendments to meet standards, the CRMS guides the Chair and Course Representative through the amendment process. Review sponsors use the CRMS to track the progress of open reviews, report on recognized courses and obtain course certificates and certification marks for display on recognized courses.
My Custom Reviews (MyCR)
• My Custom Reviews (MyCR) is an online tool to
manage your own quality reviews using Rubrics of your design.
The impact QM is having on quality in online education has been achieved only through the cooperative efforts of many and through collaboration with the groups shown on this graphic. The small QM staff provides support but it is the faculty and staff throughout the country, and now beyond, serving as QM Coordinators, trainers, and reviewers, that are implementing the program. QM’s many volunteer committees bring community-based governance to the process. The many institutions supporting QM, from subscribers to our partners and sponsors, help to disseminate the best practices and standards as broadly as possible. And students, the primary beneficiaries of a quality course, are indicating that a well-designed course truly matters to them.
Community Impact
• Research – Grants, Library, Research Colleagues • National and regional Conferences • Hosted webinars and events • Reach and scale – participation across the
spectrum of Higher education institutions • Associations – QM Instructional Design Association
•Improved courses •Engaged faculty •Reduced course development time •Ongoing faculty development •Quality benchmarking •Awareness & support for online learning •Institutional improvement
Presenter
Presentation Notes
The point of Quality Matters is continuous improvement. The purpose is to provide tool and processes that are adaptable to the different goals institutions want to achieve in pursuit of quality.
Contact QM
Your Questions?
Please contact the appropriate QM representative – available to you via email,
Skype or phone.
We are here for you!
Deb Adair, Managing Director and Chief Planning Officer [email protected] Grace Hall, Senior Media Relations and Publications Specialist [email protected] Nancy Ragias, QM Registrar, Higher Education [email protected]
