Cyclical Program Review2014-2015

John ShepherdVice-Provost and Associate Vice-President (Academic)

Office of Quality Assurance

The Context – QA in Ontario

Ontario Universities’ Council on Quality Assurance (Quality Council) was established July 1, 2010

No more OCGS appraisals; No more UPR

The new regime covers the approval and review of all new and existing undergraduate and graduate programs.

The Context – QA in Ontario

Each university in Ontario has been required to develop its own Institutional Quality Assurance Process (IQAP) for ratification by the Quality Council.

Carleton’s Institutional Quality Assurance Process (IQAP):│ Approved by Senate June, 2010│ Ratified by Quality Council March, 2011│ Revised to include DUC – Approved by Senate February, 2012│ Ratified by Quality Council May, 2012│ Joint Carleton-University of Ottawa IQAP approved by Senate January, 2012│ Ratified by Quality Council May, 2012

QA@Carleton – What’s the point?

• To constantly improve programs that are already of good quality

• To have units feel the exercise was worthwhile and beneficial

• To constantly strengthen Carleton’s academic planning processes

• To help position Carleton advantageously in the changing Provincial context

The Carleton Process:Who Administers It?

Carleton’s Office of Quality Assurance (Academic Programs) (OQA):| Vice-Provost| Manager| Quality Assurance Officer| Program Review Co-ordinator

The Carleton University Committee on Quality Assurance (CUCQA):| Vice-Provost (Chair)| Provost (ex officio)| Associate Dean (Programs and Awards), FGPA (ex officio)| Associate University Librarian| 7 senior faculty members| 1 Dean

OQA and CUCQA – What’s Our Objective?

Quality assurance can have two purposes:| Program improvement| Accountability

Our objective is program improvement We work with programs to support them We are not the QA police!!

Not just an exam to be passed and forgotten!

Cyclical Program Review

Cyclical Program Review should not be approached as a hurdle to be overcome.

Rather, it should be seen as an opportunity to strengthen programs that are currently of good quality or can achieve good quality as part of an

overall process of continuous improvement.

Cyclical Program Review

Occurs on an 8 year cycle

Simultaneous review of graduate and undergraduate programs

Self-assessment of a program’s strengths and areas for improvement

Consultative – informed by input from faculty, staff, students, associate dean(s), dean(s), and external reviewers

20-month process

Cyclical Program Review - Steps

Self-Study (3 volumes)

Site Visit

Reviewers’ Report

Final Assessment Report

Action Plan

OQA – Here to Help

Our objective is program improvement – we are here to help

Customized template for program reviewCoordination of data, reports and other materials to inform program reviewCompilation of data into the required tables. Customized data and reports may be available upon request. Electronic document sharing site - cuCollabMeetings with Review Team membersReview full and partial drafts of the self-study

Require other assistance? Just ask!

Volume I: The Self-Study

Critical, self-reflective, and program-centricProgram history – response from the last reviewProgram structure and deliveryDegree Level Expectations and Learning Outcomes and Learning ObjectivesStudents: Enrolment, retention, satisfactionFaculty, staff resourcesProgram improvements

Volume I: Template

Each review team will have an electronic workspace on cuCollab for document sharing

| OQA will post the template, data, reports, etc. on this site.| Review teams will also be able to post documents

Each Review Team will be provided with a customized template for Cyclical Program Review.

Model tables are included in the template| Data from OIRP and CURO will be provided. Tables will be populated by

OQA. | Customized tables can also be generated.

How to Write a Bad Self-Study

Ineffective self-studies are:| Descriptive rather than reflective, analytical, self-critical, and evaluative| Defensive or aimed at justifying the status quo| Focused on the academic unit rather than the program(s)| Does not address or only superficially addresses Degree Level Expectations

and Learning Outcomes| Raw data are attached as appendices, or only used in a descriptive manner| Written by a single faculty member without evidence of buy-in of faculty

and students

Adapted from: Quality Assurance Framework: Creating an Effective Self-Study for Program Reviews

Writing an Effective Self-Study

Effective Self-Studies are:| Reflective, analytical, self-critical, and evaluative| Aimed at quality improvement. Self-appraisal includes an analysis of strengths and

weaknesses, and outlines how improvements can be made| Focused on the programs under review| Curriculum is fully examined, with an eye to Degree Level Expectations, Learning

Outcomes, and to change and improvement.| Expresses Degree Level Expectations and learning objectives that operationally

drive admission requirements, curriculum content, modes of delivery, bases of evaluation of student performance and commitment of resources.

| Data are analyzed and contribute to the evaluation of strengths and weaknesses of the program.

Adapted from: Quality Assurance Framework: Creating an Effective Self-Study for Program Reviews

Participant Experiences

Dr. Stephen Godfrey

Department of Physics

Degree Level Expectations and Learning Outcomes

Degree Level Expectations

➊ Depth and breadth of knowledge ➋ Undergraduate: Knowledge of methodologies

Graduate: Research and scholarship ➌ Application of knowledge ➍ Communications skills ➎ Awareness of the limits of knowledge ➏ Autonomy and professional capacity

COU Report: Ensuring the Value of University Degrees in Ontario - November 2011

Learning Objectives and Outcomes

Learning Objectives| Learning Objectives are the instructional goals of the program.

They outline the program inputs or learning opportunities students will have over the course of the program

Learning Outcomes│ Learning Outcomes are the measurable skills, knowledge,

competencies and/or behaviours that students will have as a result of successfully completing the program

Learning Objectives and Outcomes

For QA, program-level learning outcomes apply to programs not to individual courses.

We need to look at the individual courses to get a “picture” of how the learning outcomes are addressed within the program.

Model Tables enable an account of how program level learning outcomes are achieved and Degree Level Expectations are met.

A summative narrative is required of how courses as a collective achieve the intended learning outcomes.

Support will be provided in evaluating learning outcomes.

Volume II: Faculty CVs

The CVs of all faculty associated with the program need to be included.

Flexibility in format: No more OCGS template!

All CVs must be in same format.| Format to be approved by OQA.

Volume III: External Reviewers

10 external academic reviewers.

(4 external professional reviewers.)

An internal reviewer will be nominated by OQA, in consultation with the unit and Dean(s)

Timeline & Milestones

September 2013Review team is established

October 2013Attend OQA workshopReview team begins developing student survey/focus group questions (template provided)Review team confirms faculty/instructor information with OQAReview team should begin work on Sections A-D of self-study, with particular focus on the development of program learning objectives/outcomesOQA conducts student surveys/focus groups as applicable

November/December 2013Review team receives library reports, research funding tables, space management report, and survey/focus group reports

Timeline & Milestones

January 2014Review team receives data tables from OIRP

February/March 2014Review team should be finalizing the self-studyFaculty, students, associate dean(s), dean(s), should be consulted on drafts

It is highly recommended that OQA review drafts of the brief and provide feedback before it is submitted.

April 1, 2014Volumes I, II, and III submitted to OQA

Timeline & Milestones

Once OQA confirms that the brief is ready, it is submitted to the Carleton University Committee on Quality Assurance (CUCQA):| Assigned a discussant, who prepares a report to CUCQA| Program Review Lead is invited to attend the CUCQA meeting at

which the program is discussed| Supplementary questions (dean consulted)| Review Committee selected (dean consulted)

Timeline & Milestones

Summer 2014In consultation with the unit, OQA will arrange the site visit for fall 2014.

Fall 2014Two day site visit: external reviewers meet with dean(s), faculty, staff, and studentsWithin one month of the site visit, the external reviewers submit a report to OQAThe Review team prepares a response to the report, in consultation with the dean(s)CUCQA receives the external reviewers report and the Review team ’s response.

Timeline & Milestones

CUCQA considers the brief, report and response:| Discussant recommendation report| CUCQA recommends a categorization:

• Good Quality with international or national presence• Good Quality• Good Quality with report• Conditional approval• Not approved to continue

│ Categorization sent to unit and dean(s)

Action plan requested

Timeline & Milestones

Winter 2015Action plan submitted to CUCQACUCQA: Final Assessment Report and Executive Summary

│ Submitted to the ProvostProvost considers Report and SummaryDocumentation to SAPC/Senate for approval

academic unit; dean(s); Board of Governors; Quality Council; Carleton website

Participant Experiences

Dr. Mitchell Frank

Institute of Comparative Studies in Art and Culture

Help!?

Office of Quality Assurance| Office of Institutional Research and Planning| Carleton University Research Office| Office of Space Management and Capital Planning| University Library

Undergraduate Programs: Faculty undergraduate associate deans

Graduate Programs: Associate Dean (Programs and Awards), FGPA

Contact

Office of Quality Assurance421 Tory Building

www.carleton.ca/oqa

Jessica DeVries, Manager (Interim) & Program Review Coordinatorjessica.devries@carleton.ca x 3231

Ann Clarke-Okah (OQA Consultant)ann.clarke-okah@carleton.ca