Embed Size (px)
Citation preview
Program Quality Assurance atAlgonquin College of Applied Arts and Technology
Program Quality Review
Ontario College Credentials
2015-2016
A Guide for Team Leaders
PREFACEThis Program Quality Review (PQR) Team Leader guide is written to aid Program Quality Reviews for Ontario College Certificate, Diploma, Advanced Diploma and Graduate Certificate programs offered at Algonquin College. It serves as the master Team Leader Guide for Program Quality Reviews. Adaptations are provided separately for Apprenticeship programs, Algonquin College Certificates and Bachelor’s degrees going through Quality Review.
2
REVISION HISTORYThe PQR Team Leader guide is updated at least annually, Revisions since 2010 are noted below.
Team Leader Guide Major revisions / Notes2015-2016 Data review section and templates updated
Minor rewordings to Quality Criteria Process flowchart updated
2014-2015 Issue 02 (for Winter/Summer 2015)
Quality criteria re-ordered and modified to align with OCQAS Accreditation Quality Standards specific to Program Quality Review. Minor enhancements made to the Quality Criteria and elements.
2014-2015 Issue 01 (for Fall 2014)
Note regarding upcoming change to Accreditation process, and guidelines for Fall Team Leaders
Updates to data access through BI data reporting portal Introduction of Curriculum Services
2013-2014 Updates to quality criteria elements to include assessment of AODA, and recognition of the deployment of the College-wide Sustainability and Internationalization VLO
2012-2013 Notice that documents are retained and distributed electronically Addition of process for assigning an external team leader
2011-2012 Update to use six quality criteria, based on changes to PQAPA Introduction of Data Reporting Portal for data access Introduction of new format for Implementation Plan Formal inclusion of Sign-Off sheet Additional information on data (KPI)
2010-2011 Re-formatting of guide Introduction of Guide specific to review of Bachelor’s degrees
3
ContentsPREFACE......................................................................................................................................2
REVISION HISTORY.....................................................................................................................3
PROGRAM QUALITY ASSURANCE – AN OVERVIEW...............................................................9
Program Quality Assurance Process Audit (PQAPA); Accreditation Process...........................9
Program Quality Review Quality Criteria.................................................................................10
PROGRAM QUALITY ASSURANCE AT ALGONQUIN COLLEGE............................................12
Annual Curriculum Review.......................................................................................................12
Program Mix Review................................................................................................................12
Program Quality Review..........................................................................................................12
Cyclical Review of Online Elective General Education courses..............................................12
PROGRAM QUALITY REVIEW..................................................................................................13
PQR Goals...............................................................................................................................13
PQR Process...........................................................................................................................13
Schedule..................................................................................................................................13
Program Quality Review Process Flowchart...................................................................14
Roles........................................................................................................................................15
Program Quality Assurance Administrator.....................................................................15
Program Quality Review Team Leader.............................................................................15
Dean.....................................................................................................................................16
Chair....................................................................................................................................16
Orientation................................................................................................................................17
Release Time...........................................................................................................................17
Fall 2015 or Winter 2016:...................................................................................................17
Timeline....................................................................................................................................18
Need Assistance?....................................................................................................................18
THE FINAL REPORT – AN OVERVIEW.....................................................................................19
The Main Document.................................................................................................................19
Executive Summary...........................................................................................................19
Introduction.........................................................................................................................19
Evaluation of the Program against the Criteria...............................................................19
Conclusions........................................................................................................................19
4
Recommendations and Implementation Plan..................................................................19
The Appendices.......................................................................................................................20
Appendix A..........................................................................................................................20
Appendix B..........................................................................................................................20
Appendix C..........................................................................................................................20
Appendix D..........................................................................................................................20
Appendix E..........................................................................................................................20
DOCUMENT REVIEW.................................................................................................................22
Program Monograph................................................................................................................22
Program Promotional Material.................................................................................................23
Program Council Meeting Minutes...........................................................................................23
Program Advisory Committee Meeting Minutes.......................................................................23
Further Study Pathways...........................................................................................................24
Reports from External Accreditation or Licensing Bodies........................................................24
Program (Vocational) Learning Outcomes...............................................................................24
Program of Study.....................................................................................................................24
Course Outlines and Course Section Information....................................................................24
DATA REVIEW............................................................................................................................26
Program Performance Tab......................................................................................................26
Key Performance Indicators..............................................................................................28
Program Costing Tab...............................................................................................................29
CURRICULUM MAPPING...........................................................................................................30
Program Standards..................................................................................................................30
Vocational Standard...........................................................................................................30
Essential Employability Skills...........................................................................................30
General Education Requirement.......................................................................................31
Curriculum Services..........................................................................................................32
Purposes of Curriculum Mapping.............................................................................................34
Performing Curriculum Mapping..............................................................................................34
Course Outline Review......................................................................................................36
Vocational Learning Outcomes – Reviewing the Map....................................................36
Essential Employability Skills – Reviewing the Map......................................................37
General Education Requirement – Reviewing the Map..................................................37
5
EVALUATION AGAINST THE QUALITY CRITERIA...................................................................38
Evaluation Criteria Listing........................................................................................................38
Criterion 1............................................................................................................................38
Criterion 2............................................................................................................................38
Criterion 4............................................................................................................................39
Criterion 5............................................................................................................................40
Criterion 6 –.........................................................................................................................40
Responding to the Evaluation Criteria.....................................................................................41
Evaluation Criteria Listing – with help text...............................................................................42
CONDUCTING FOCUS GROUPS..............................................................................................62
Memos from the Chair and Team Leader................................................................................62
Faculty Education, Experience, and Professional Development Survey......................62
Program Self-Audit.............................................................................................................63
Course Outlines..................................................................................................................63
Setting the Focus Group Agendas...........................................................................................63
Tips for Planning and Facilitating Focus Groups.....................................................................63
Preparing and Planning for the Meeting..........................................................................63
Facilitating the Meeting.....................................................................................................63
Faculty Focus Group................................................................................................................65
Facilitating the focus group..............................................................................................65
Preparing the focus group report.....................................................................................66
External Stakeholder Focus Group..........................................................................................67
Planning the External Stakeholder Focus Group............................................................67
Time Frame for the Meeting..............................................................................................67
Leadership of the Meeting.................................................................................................67
Ground Rules for the Meeting...........................................................................................67
Writing the external stakeholder focus group report.....................................................68
Possible Items for Discussion..........................................................................................68
Student Focus Group...............................................................................................................71
Planning the Student Focus Group..................................................................................71
Time Frame for the Meeting..............................................................................................71
Leadership of the Meeting.................................................................................................71
Ground Rules for the Meeting...........................................................................................71
6
Writing the Student Focus Group Report........................................................................72
Possible Items for Discussion..........................................................................................72
COMPLETING THE FINAL REPORT – AN OVERVIEW............................................................77
ANNUAL FOLLOW UP................................................................................................................78
GLOSSARY AND LIST OF ACRONYMS....................................................................................79
APPENDIX A – THE FINAL REPORT.........................................................................................80
APPENDIX B – THE CREDENTIALS FRAMEWORK.................................................................87
APPENDIX C – TEMPLATES.....................................................................................................95
Introductory Email from the Chair....................................................................................96
Introductory Email from the Team Leader.......................................................................97
Faculty Education, Experience, and Professional Development Survey......................98
Program Quality Review - Program Self-Audit..............................................................102
Course Outline Check-Up Review Tool for PQR...........................................................104
Course Outline Review Summary Sheet........................................................................105
Curriculum Mapping - Terms and Concepts..................................................................106
Analysis and Review of VLO Maps for PQR..................................................................107
Analysis and Review of EES Maps for PQR..................................................................109
Analysis and Review of General Education Maps and Courses for PQR...................111
Faculty Focus Group – Sample Agenda........................................................................113
Faculty Focus Group – Sample Invitation Email...........................................................114
Sample Invitation to External Stakeholders..................................................................115
External Stakeholder Focus Group - Sample Agenda..................................................116
Student Focus Group – Sample Invitation.....................................................................117
Student Focus Group - Sample Agenda........................................................................118
Thank You Note to Focus Group Participants - Sample...............................................119
PQR Submission Sign Off Form.....................................................................................120
APPENDIX D – Program Data..................................................................................................122
Key Performance Indicators (KPIs)........................................................................................122
Student Satisfaction Survey............................................................................................122
Graduate Outcomes/Graduate Satisfaction Survey......................................................122
Employer Satisfaction Survey.........................................................................................122
Algonquin College Performance Measures...........................................................................123
Application/Registration Ratio:.......................................................................................123
7
Intake/Projected Enrolment:............................................................................................123
KPI Graduation Satisfaction:...........................................................................................123
KPI Total Employment:....................................................................................................123
KPI Related Employment Rank:......................................................................................123
KPI Employer Satisfaction:.............................................................................................123
Algonquin College Quality Measures.....................................................................................124
KPI Student Satisfaction..................................................................................................124
Instructional Quality.........................................................................................................124
Quality Average:...............................................................................................................124
School Average:...............................................................................................................124
Benchmark:.......................................................................................................................124
8
PROGRAM QUALITY ASSURANCE – AN OVERVIEWProgram Quality Assurance Process Audit (PQAPA ); Accreditation Process One of the prerequisites for granting greater autonomy to the Colleges of Applied Arts and Technology in Ontario to govern their own programs was the implementation of a process to assure program quality. The Ontario Colleges of Applied Arts and Technology Act, 2002 outlined two elements of this expectation:
quality assurance at the individual college level, and a self-regulating process at the system level.
The Program Quality Review process forms a major part of adherence to providing quality assurance at the College level, and is therefore essential to College operations.
At the system level, that is, at the level which encompasses all Colleges of Applied Arts and Technology in Ontario, a separate quality assurance process takes place, the Program Quality Assurance Process Audit (PQAPA). This is run through the Ontario College Quality Assurance Service (OCQAS1). OCQAS was established to provide effective and efficient mechanisms that ensure specific program quality and consistency standards are met by the Colleges of Applied Arts and Technology (CAATs) in Ontario.
Building on the PQAPA, Ontario’s Colleges are moving to an accreditation process effective September 2015. In preparation for this transition, six Accreditation Quality Standards have been developed from current PQAPA quality criteria, and approved in March 2015 by the Management Board of OCQAS. The essence of the PQAPA criteria has been retained, although the ordering/number has changed.
Quality Standard 1 (Program Quality Management System)Quality assurance processes ensure program development, program review, and on-going program quality assessment involve faculty, students, industry representatives, and others as appropriate for the purpose of continual improvement.
Quality Standard 2 (Existence and Communication of Policies and Practices) Quality assurance processes ensure the existence and communication of effective policies and practices related to academic issues that support program development and student achievement of program learning outcomes. This would include, for example, admission, credit for prior learning, promotion, graduation, and other related academic policies.
Quality Standard 3 (Program Design) Quality assurance processes ensure that: program-level learning outcomes are established for all program of instruction; they are consistent with the programs’ intended purposes; and, they are appropriate for the credentials offered upon successful completion of the programs.
1 http://www.ocqas.org/index-en.html
9
Quality Standard 4 (Program Delivery and Student Assessment) Quality assurance processes ensure that methods of program delivery, including the design, development and execution of teaching and learning activities and student evaluation strategies, are consistent with the program learning outcomes.
Quality Standard 5 (Conformity with Government Requirements) Quality assurance processes ensure that all programs of instruction conform to current government policy2 related to the design and delivery of programs of instruction, are consistent with accepted college system nomenclature / titling principles, and maintain relevance.
Quality Standard 6 (Availability and Allocation of College-wide Resources) Quality assurance processes ensure the existence and availability of human resources, physical resources, financial resources, student support resources, and technological infrastructure to support student achievement of program learning outcomes wherever and however they are delivered.
For current information on the transition to the Accreditation model, please visit the OCQAS web page at http://www.ocqas.org/en/?p=5014.
Program Quality Review Quality Criteria Based on the above Accreditation Quality Standards, PQR Quality Criteria have been updated and are noted below. These PQR Quality Criteria are effective Winter 2015.
PQR Quality Criterion 1 Regular program quality assessment that involves faculty, students, industry representatives, and others as appropriate for the purpose of continual improvement is in place and happens.
PQR Quality Criterion 2 Admission, credit for prior learning, promotion, graduation, and other related academic policies support student achievement of program learning outcomes.
PQR Quality Criterion 3 Program-level learning outcomes are established for all programs of instruction, are consistent with the programs’ intended purposes, and are appropriate for the credential offered upon successful completion of the program.
PQR Quality Criterion 4 Methods of program delivery, including the design, development and execution of teaching and learning activities and student evaluation strategies, are consistent with the program learning outcomes.
2 Previously “Programs conform to the Framework for Programs of Instruction”. This new wording extends the scope to allow for changes in government policy, and extend to requirements of other bodies, such as the Postsecondary Education Quality Assessment Board (PEQAB). Regarding the ‘Framework’, the Minister’s Binding Policy Directive Framework for Programs of Instruction is one of the policy documents issued by the Ministry of Training, Colleges and Universities in April 2003 to guide the development and delivery of programs of instruction offered by the Colleges of Applied Arts and Technology in Ontario.
10
PQR Quality Criterion 5 Programs conform to current government policy related to the design and delivery of programs of instruction, are consistent with accepted college system nomenclature / program titling principles, and maintain relevance.
PQR Quality Criterion 6 Human resources, physical resources, financial resources, student support resources, and technological infrastructure to support student achievement of program learning outcomes wherever and however they are delivered exist and are available.
11
PROGRAM QUALITY ASSURANCE AT ALGONQUIN COLLEGEProgram review is annual event at Algonquin College, and currently consists of four processes:
Annual Curriculum Review Annual Program Mix Review Cyclical Program Quality Review Cyclical Review of Online Elective General Education courses
Annual Curriculum Review Each year program faculty members revise and update curriculum to maintain its currency and to ensure that graduates will have the skills necessary to be successful in the workplace.Feedback from faculty, students, and advisory committee members is taken into consideration before updates are made to keep the program current. Revisions to the curriculum and the program narrative information are made in GeneSIS, the student information system, and are subsequently reviewed in the Office of the Vice President Academic. Once the changes are approved, a new program version is created. The information in GeneSIS becomes the foundation for course loading, fees loading, scheduling, registration and College publications.
Program Mix Review Staff in the Academic Operations and Planning department compile data that reflect the Quality Measures for a program. Admissions and registration data, employment rates, capstone questions from Key Performance Indicators (KPI) and course assessment surveys, and program costing data are reviewed. Programs not meeting college benchmarks are reviewed and strategies are developed by the Chair and Dean to improve the quality of the program and/or the financial contribution.
Program Quality Review Program Quality Review (PQR) ensures that each program goes through a comprehensive review every five years. The PQR process builds on annual review activities by including analysis of three to five years of program data, and soliciting direct feedback from stakeholders and students into the program review. Each PQR results in an implementation plan containing resourced actions to maintain and improve the program over the next five years.
This manual guides Team Leaders when conducting a Program Quality Review.
Cyclical Review of Online Elective General Education courses Online General Education courses form an integral part of many programs offered at the College. In 2009, the review process for these courses was formalized. Each of the online General Elective courses in the College pool is reviewed according to a published cycle, first by the General Education committee for overall adherence of the course to General Education objectives, and then in detail by individual faculty and student reviewers.
12
PROGRAM QUALITY REVIEWPQR Goals To assess the program against established quality criteria in order to determine whether the
program meets or exceeds College and Ontario College Quality Assurance Service (OCQAS) standards (for Ontario College Credentials) or Postsecondary Education Quality Assessment Board (PEQAB) standards (for Bachelors’ Degrees).
To establish and implement a quality improvement plan3 for the program, in response to the findings of the review.
PQR Process Program Quality Review follows a systematic process, and is outlined in Figure 1. After the Team Leader has been identified to conduct the review, information pertinent to the program is collected and analyzed, allowing an initial evaluation of the program against established quality criteria. The program information reviewed includes published and internal documentation, program data, current curriculum maps, and input collected from key players in the program – faculty, students, and external stakeholders. Once the review is complete, the evaluation against the criteria is finalized, and all findings and recommendations for improvement are captured in a report. An implementation plan for the recommendations is drawn up, with an update on the actions provided to the PQAA at least annually until completion.
The PQR Team Leader conducts the PQR, with assistance from the Chair.
Note that this guide is written to provide assistance with each of the main components of PQR: Document Review Data Review Curriculum Mapping Evaluation Against the Quality Criteria Conducting Focus Groups Completing the Final PQR Report
Schedule Programs are reviewed according to a defined schedule, normally every five years. The schedule is revised annually, in consultation with the Academic Departments, and aligned with other Quality Assurance requirements, for example, accreditations or consent renewals for bachelor’s degrees.
The PQR schedule is published at http://www.algonquincollege.com/academic-development/our-services/program-quality-assurance/
3 Also referred to as “the implementation plan”
13
Program Quality Review Process Flowchart
14
15
Roles Program Quality Assurance AdministratorProgram Quality Reviews are coordinated by the Program Quality Assurance Administrator (PQAA), who reports to the Dean, Academic Development, who in turn reports to the Vice President Academic.The responsibilities of the Program Quality Assurance Administrator include
ensuring the College is compliant with Ministry requirements regarding Program Quality Assurance.
consulting with major academic constituencies College-wide regarding Program Quality Review.
preparing documents related to Program Quality Review. planning and organizing orientation/training workshops for Team Leaders. supporting the Program Quality Review process in programs. meeting regularly with PQR Team Leaders during the review process to monitor
progress and provide guidance. facilitating evaluation of the Program Quality Review process. ensuring that the final report is signed off fully by the PQAA, Chair and Dean arranging archiving of the final report; forward e-copies to the TL, Chair, Dean, and
providing access to all reports to the VPA.
Program Quality Review Team LeaderThe PQR Team Leader is normally a faculty member who is assigned to conduct Program Quality Review for the program in which s/he teaches.The responsibilities of the Team Leader include
attending meetings scheduled by the Program Quality Assurance Administrator. attending orientation/training workshops related to the Team Leader role. reporting to the Chair weekly to monthly on the progress of the Program Quality Review. analyzing data and incorporating the results of the analysis in the final PQR report. convening meetings of the program faculty team. convening a Student Focus Group. convening an External Stakeholder / Advisory Committee Focus Group. facilitating Student and External Stakeholder Focus Groups for a program from another
School and submitting reports for these meetings. evaluating the Program Quality Review process. submitting the final report to the Chair and PQAA, who will review and sign-off. (The
Chair then forwards the report to the Dean for final review/sign-off, and notifies the PQAA who will advise the Vice President Academic of its completion.)
16
DeanThe responsibilities of the Dean include
identifying programs for Program Quality Review. supporting the Program Quality Review process. reviewing and signing-off on the Program Quality Review report. following up regarding results and recommendations included in the report. reporting to the Vice President Academic on the progress of Program Quality Review
and regarding the follow-up on the recommendations.
ChairThe responsibilities of the Chair include
identifying Team Leaders (TL) each spring for programs identified to be in the next cycle. It is preferred that a full-time professor or coordinator is assigned.
acting as a facilitator for Focus Groups in the event of illness of the TL. meeting with each TL in the Spring to discuss the upcoming process, and to encourage
and facilitate attendance at the Aligning and Building Curriculum Conference if the TL has not already attended.
attending information meetings for Chairs. providing release time for the TL as outlined in the PQR process meeting with the TL early in the process to review the program data and initial
assessment of the quality criteria. Assisting with agenda setting for focus groups and helping to create a list of those to invite.
providing the relevant budget code for all expenses. sending an email to all program faculty inviting them to participate in the PQR process
and to fill out the credentials and experience sheet. providing a summary report of faculty credentials and experience to the Team Leader, to
be included in the final report. arranging secretarial support as outlined in the PQR process meeting regularly with the TL and monitoring progress. attending focus group meetings and introducing the group members to the process. ensuring feedback is provided to all those who participated in the focus groups. providing feedback regarding the PQR process. collaborating in the development of the final recommendations and implementation plan,
ensuring adequate budgetary support for those recommendations that are to be implemented immediately (i.e. within one year) and consideration of those recommendations for which funds are not currently available.
including PQR recommendations in their Performance Contract. facilitating the implementation of the final recommendations. tracking progress on the implementation plan, and preparing the submission of the PQR
Annual Follow Up Report until all recommendations have been implemented.
17
Orientation Orientation for Team Leaders is scheduled where possible outside of classroom contact weeks to minimize the negative impact of training within teaching weeks.
The orientation process for Team Leaders involves a series of workshops. Upon completing the Team Leader orientation process, Team Leaders are able to
evaluate a program against identified criteria; identify quality in a program when measured against identified standards and determine how to reinforce/maintain this quality.
lead focus groups for internal and external stakeholders. map curriculum to validate that the program of study meets current Ministry
requirements. analyze program data and make inferences based on trends. analyze curriculum maps to identify gaps and redundancies. formulate realistic and attainable recommendations to improve quality where needed.
Release Time Team Leaders are entitled to receive the following release time for the duties and responsibilities associated with Program Quality Review:
Fall 2015 or Winter 2016: Release from teaching hours for the equivalent of one three-hour course including evaluation and preparation time which, for consistency purposes, has been equated to six hours of assigned workload. Faculty members who are reviewing a three-year program, a four-year program, or the equivalent of three or more years of curriculum, will be assigned a nine hour workload. All authorized travel expenses incurred by Team Leaders to facilitate Focus Groups at another campus will be reimbursed.
Where an internal Team Leader is not available to conduct the PQR, an external Team Leader may be assigned to the role. Before the PQR commences, the Chair, in consultation with the PQAA, identifies a Team Leader and negotiates a contract with remuneration within the PQR budget allocation for the program under review. Note that the funds allotted may be less for smaller programs, for example for short College Approved programs. The PQAA provides details on the funds available for review of each program.
In addition to the Team Leaders’ release time, additional financial support is provided to departments as follows:
Support Staff: up to 48 hours per PQR at $15 per hour Part-time Faculty: up to 16 hours per PQR at $30 per hour Hospitality4: up to $100 per PQR to contribute towards light refreshments at focus
groups.
PQR expenses are charged to the program budget code with the exception of travel:4 Expenditure must follow guidelines as per College Policy AD12 “Travel, Meals and Hospitality”
18
Departments are compensated annually by Academic Development for the specified release time.
Travel expense forms are submitted directly to the PQAA.
Timeline Detailed individual work plans are maintained by the PQR Team Leader with assistance from the Program Quality Assurance Administrator (PQAA) as required.
Fall Course ReleaseTiming Event(s)May 2014 Confirmation of Programs to undergo PQR, Team
Leaders assigned:June or August 2014 Team Leader orientationSeptember 2014, or as required Refresher orientation (for those who have been TLs
before)September to November, 2014 Touch base meetings with PQAA, as requiredNovember 27, 2014 Draft report due
(submit to Chair and PQAA)December 18, 2014 Submit final report to PQAA for review/approval by
PQAA/Chair and Dean.
Winter Course Release Timing Event(s)May 2014 Confirmation of Programs to undergo PQR, Team
Leaders assigned:June or August 2014 Team Leader orientationJanuary 2015, or as required Refresher orientation (for those who have been TLs
before)January to March, 2015 Touch base meetings with PQAA, as requiredMarch 26, 2015 Draft report due
(submit to Chair and PQAA)April 16, 2015 Submit final report to PQAA for review/approval by
PQAA/Chair and Dean.
Once approved, the PQAA forwards a template for annual reporting on the Implementation Plan to the Chair with a due date. Typically, follow up reports are due early May every year until actions are closed.
Need Assistance? Contact the PQAA at any time with questions about Program Quality Review:
Wilma McCormack, Room C-532, 613-727-4723 ext. 7681 [email protected]
19
THE FINAL REPORT – AN OVERVIEWFor more details on the contents of each section in the main document, see Appendix A.
The final output of the PQR is a report which is submitted to the Office of the Vice President Academic. This report follows a standard structure:
The Main Document Executive SummaryWritten last, the Executive Summary prefaces the entire report. It summarizes the highlights of the quality review of the program, and includes the overall status of the program, its strengths, weaknesses, as well as challenges and recommendations.
IntroductionWritten first, the Introduction paints a picture of the program at the outset of the review – it does not contain any information gained as part of the review process.
Evaluation of the Program against the CriteriaThis section is the working part of the document. Information is gathered as each stage of PQR - document review, data analysis, and the outcomes of the focus groups – is carried out, and notes are captured against each of the elements of the quality criteria as the review continues. By the end of the review process, each element of the quality criteria will have been responded to (yes/no) and a justification of the response will have been provided.
ConclusionsThe conclusions wrap up the findings of the review. The outcomes of the document and data review are provided, along with the highlights from each of the focus groups. The recommendations are stated, and any observations or non-implementable recommendations are noted.
Recommendations and Implementation PlanThe recommendations that are to be implemented are captured in a table, and are linked, as far as possible, to the pillars of the College Strategic Plan. Each recommendation will have an associated list of action items that must be completed in order to meet the goal of the recommendation. Each action item must be resourced, and have a realistic completion date.
The implementation plan is a living document – it is carried forward year over year, and updated at least annually until action items are completed or closed.
The Appendices The appendices carry the information collected during the review process. This information will be referenced within the main document.
Appendix AThis contains program and curriculum information. It consists of the
Program Monograph and Review of Promotional Materials Program Vocational Learning Outcomes5
Program Delivery Information Curriculum Mapping Information:
Vocational Learning Outcomes curriculum map (Overall and by Level) Analysis of the Vocational Learning Outcomes curriculum map Essential Employability Skills curriculum map (Overall and by Level) Analysis of the Essential Employability Skills curriculum map General Education curriculum map Analysis of the General Education offering Course Outline Review Summary – this is a living document and can be
used to update course outlines prior to inputting into COMMS for the subsequent academic year.
Appendix BThis contains the information collected from the program faculty, namely
Faculty focus group report(s) Summary of the Faculty Program Self-Audit Summary of Faculty Credentials and Experience
Appendix CThis contains the information collected from the External Stakeholders, normally the
External Stakeholder focus group report
Appendix DThis contains the information collected from the current students, normally the
Students’ focus group report
Appendix EThis contains summary information only from the review of the statistical and survey data for the program. Typically, observations from analysis of registration, graduation, OCAS program choice, program withdrawal information and KPI surveys are included. There is no need to enclose all program data available through the Enterprise Reporting Portal. However, if department/program specific surveys have been utilized in the data analysis, these should be included in Appendix E. Please contact the PQAA for guidance if required.
5 In the case of Bachelor’s Degrees, Program Outcomes. The curriculum mapping section is adjusted accordingly.
21
22
DOCUMENT REVIEWThe following documents must be reviewed by the Team Leader, in conjunction with the Chair and department faculty, to ensure that the information contained therein is complete, accurate, unambiguous, current, that it conforms to Ministry and College Standards, and that pertinent information is readily available for applicants and students to allow them to make informed decisions about program choice.
Program Monograph Program Promotional Materials Program Council Meeting Minutes Program Advisory Committee Meeting Minutes Further Study Pathways Reports from External Accreditation or Licensing bodies Program Learning Outcomes Program of Study Course Outlines and Course Section Information
These documents are referred to when evaluating the program against the quality criteria.
Program Monograph The program monograph contains several key pieces of information used by applicants when determining which program to choose. It is reviewed annually as part of Annual Curriculum Review. Although the whole monograph should be reviewed during PQR, an in-depth review of the following sections may be necessary in order to respond to some elements of the quality criteria.
Admission Requirements should ensure that students are adequately prepared to enter the program (i.e. that they have the foundational knowledge needed on which to base their learning) without unnecessarily limiting access to a program.
College Eligibility is Ministry mandated and is usually standard for postsecondary programs - Ontario Secondary School Diploma, or at least 19 years of age.
Program Eligibility is determined by the department and should reflect any necessary knowledge and skills that the students must have prior to entering the program, for example a specific level of mathematics or science. Mature students may be required to meet the program eligibility even though they are 19 or older and do not have a high school diploma. Proven competency in English is a College-wide requirement.
Note that the current monograph publishes admission requirements for the subsequent academic year.
23
Fees and Expenses should list all relevant program expenses to allow students to plan their financial commitment. Fees and Application Information are supplied by the Registrar’s Office.
The Program of Study is published in the monograph and is the foundation of the College’s contract with an applicant. It outlines the learning opportunities that student can expect in return for paying tuition.
The full program monograph (not just a weblink) is included in the final PQR report within Appendix A.
Program Promotional Material Other promotional materials used by the program (in addition to the Program Monograph) are also to be reviewed. The list of promotional materials should be provided in the final PQR report within Appendix A, with URLs if appropriate. Full documents are not required to be included.
Program Council Meeting Minutes Minutes of Program Council meetings for the past 5 years are reviewed to ensure that
Program Council meetings occur on a regular basis, and in accordance with Policy AA03 – Program Council. (Program Council meetings must happen once a term at a minimum.)
issues raised at Program Council meetings are captured, and have an action plan to address.
there is a feedback mechanism in place to ensure that the students are informed of what action, if any, will take place and the proposed timeline for any resolution.
The review of the minutes may also help identify agenda items for the upcoming focus groups.
The Program Council meeting minutes are not included in the final PQR report.
Program Advisory Committee Meeting Minutes Minutes of Advisory Committee meetings for the past 5 years are reviewed to ensure that
Advisory Committee meetings occur on a regular basis, and in accordance with Policy AA01 – Advisory Committees. (Advisory Committees must meet at least once a term)
issues raised at Advisory Committee meetings are captured, and have an action plan to address these issues.
there is a feedback mechanism in place to ensure that students are informed of what action, if any, will take place and the proposed timeline for any resolution.
The review of the minutes may also help identify agenda items for the upcoming focus groups.
The Advisory Committee meeting minutes are not included in the final PQR report.
24
Further Study Pathways Established agreements for further study currently in place for the program, for example, articulation agreements, are reviewed to ensure that curriculum changes made since the last issue have been reflected within the agreement.
The final PQR report must include a list and brief description of any such agreements that exist with other postsecondary institutions. If no such agreements currently exist and graduates from the program regularly seek further education, it may be prudent to recommend in the PQR final report that an articulation agreement be considered.
Also review any pre-College laddering opportunities, for example School to College to Work Initiative (SCWI) projects occurring in partnership with the program, (e.g. program specific Specialist High Skills Majors (SHSM) or Dual Credit initiatives).
Information on pathways is found at: http://www.algonquincollege.com/prospective/programs_degrees.htm .
Reports from External Accreditation or Licensing Bodies If applicable to the program, information on the success or otherwise of graduates obtaining licenses or external accreditation should be collected and noted against quality criteria element #6.8. This information may also be summarized in the Executive Summary and Conclusions sections within the PQR final report.
Program (Vocational) Learning Outcomes The current ministry approved program Vocational Learning Outcomes (VLOs) are stored in GeneSIS, with a copy available in the Course Outline Mapping and Management System (COMMS). VLOs are reviewed during Curriculum Mapping.
A listing of the program Vocational Learning Outcomes is included in the final PQR report within Appendix A. Note here if there are Provincial Program Standards for the program, and include any pertinent differences between the Program Standard and the VLOs used at Algonquin (e.g., inclusion of College-specific VLOs).
Program of Study The Program of Study is stored in GeneSIS, with copies available on the Program Monograph and in COMMS. The Program of Study is reviewed during Curriculum Mapping. The Program of Study forms part of the Program Monograph, which is included in the final PQR report within Appendix A.
Course Outlines and Course Section Information Course outlines and Course Section Information are reviewed by the course owners each year as part of Annual Curriculum Review, with the output of this review utilized within PQR. Course outline review will be discussed further in the Curriculum Mapping section of this document. To note:
25
a course outline must be developed for each and every course offered in the program. all sections of the course outline must be reviewed. Policy AA26 – Course Outlines and Course Section Information provides additional
information regarding the expected content of course outlines and course section information.
Course outlines and course section information for full-time Ontario College Credentials and Bachelor’s degrees are maintained in COMMS. Course outlines and course section information for programs delivered through CCOL are being integrated into the COMMS system.
26
DATA REVIEWProgram specific data must be reviewed by the Team Leader, with support of the Chair, to gain an understanding of the program trends before focus group meetings are held. This aids with evaluation against the criteria and setting the agenda for focus group meetings.
The following data are provided, when available, for Program Quality Reviews, and are currently accessed through the Data Reporting Portal at https://reporting.algonquincollege.com. Note that data repository is continuously being updated and improved. Information below pertains to layout and data availability as of Fall 2015. The PQAA will inform Team Leaders and Chairs to changes in access and content.
Program Performance Tab Up to five years of registration related data are provided, where available, in order to review trends in admissions and enrolment. These data should be reviewed in the context of specific constraints of the individual program, for example, the maximum registration allowed, or any physical (space) constraints.
PQR1000 – Registration Milestones by ProgramActual registrations are provided for Day 1, Day 10 and the audit date. Items of concern for a program include a high drop-out rate in level 01, significant withdrawals either from Day 1 to Day 10 or between Day 10 and the audit date, and an academic level that appears to have a low returning rate.
Application / Registration Ratio (pending, to be deployed in PQR1000): This ratio is the number of applications vs. number of registered students (both full-time and part-time) on Day 1 in the intake level. It is important to look at the trends in this ratio: although each program is unique, established programs tend to have a fairly constant ratio over the years. If the ratio is trending upwards, one thing to consider is whether additional spaces should be opened in the program. Conversely, if the ratio is decreasing year by year, it may be time to consolidate. If no trend is evident – the ratio is erratic – consider why the applications and registrations vary, and whether modifications should be made to the program to introduce stability.
Intake / Projected Enrolment The actual enrolment (full-time only) over the projected enrolment (full-time only) on the audit date in the intake level expressed as a percentage.
Projected enrolment is used for planning financial, human and space resources. Ideally, the intake / projected enrolment should run close to 100% - significant deviation either way results in recalculation of the budget and related items during the academic year.
27
PQR1020 – Program WithdrawalsThis report may help identify why students are not continuing in the program. It is important to review the reasons given to see if any patterns arise. Note that students do not always fill out the standard withdrawal form and for those that do, the reason given may not be the real reason for withdrawing.
PQR1040 – OCAS Program ChoiceThe OCAS program choice report shows the distribution of registered students by their program choice. It may assist with identifying why students have withdrawn from level 01. For example, a program with a significant number of 4th and 5th choice applicants may have higher withdrawal rates. Occasionally, values of 0 or 6 appear in the program choice report. Where a 0 is present, this means that the student did not apply via OCAS. Where a 6 is present, this means the student is enrolled in a program that was not on their OCAS submission.
PQR1030 – Course Failure RateThis report is produced when 20% or more of the students with a registered status (on the audit date) in the course receive an F grade. This report is only produced when five or more students are registered in the course for the term.
The Failure Rate report helps identify issues and raise questions for consideration regarding student success, for example:
are admission requirements adequate to allow the students to be successful (especially for level 01 courses)?
are the courses sequenced properly to ensure the students are taught foundational information on which to build their learning?
are students allowed to progress in the program after failing a course when the failed course should be considered a prerequisite for the next level?
are the learning materials, the learning activities, and the evaluation methodologies appropriate to allow students with a variety of learning styles to learn the material and demonstrate the course learning requirements?
is the content or the assessment more difficult than it need be?
Many teachers have noted a link between attendance and student success on exams. If this is the experience in the program under review, consider
how to communicate this link to students how students with involuntary absences are assisted to be successful
while respecting that, with a few exceptions, attendance is not mandatory and that adult learners are accountable for their own learning. Formally monitoring attendance in class on an ongoing basis has been shown to have a positive influence on attendance and success.
28
In some cases, the report will indicate high failure rates as a result of a course being offered off-cycle for a few students who had previously dropped or failed a course, and did not successfully complete the course in the off-cycle term.
Key Performance Indicators
KPI0001 – KPI Student Satisfaction – Capstone Questions Percentage of students who rated Very Satisfied or Satisfied for the overall program/college experience. This measure is the average of responses from the four capstone questions of the Student Satisfaction survey:
Overall, your program is giving you the knowledge and skills that will be useful in your future career
The overall quality of the learning experiences in this program The overall quality of the facilities/resources in the college The overall quality of the services in the college.
Reviewing the trend of the individual questions provides insight into the (1) program content, (2) its delivery, (3) quality of services in the College and (4) quality of facilities/resources in the College. It is important to review the trends for each question as well as the aggregate value to determine where the strengths and challenges of the program lie.
The four KPI Student Satisfaction Survey capstone questions have remained constant for the past 6+ years; the numbering changed in 2014:
KPI Question Numbering 2009-2013 Numbering 2014Overall, your program is giving you the knowledge and skills that will be useful in your future career
Q14S Q13S
The overall quality of the learning experiences in this program
Q26S Q24S
The overall quality of the services in the college
Q45S Q39S
The overall quality of the facilities/resources in the college
Q44S Q49S
KPI0004 – KPI Student Satisfaction – All QuestionsStudent satisfaction responses for all KPI Student Satisfaction survey questions.
29
KPI0002 – KPI Graduate & Employer Satisfaction – Capstone Questions This report tabulates the graduate satisfaction and employer satisfaction rates, and also presents key items such as employment rate. Review the past five years data to garner trends. Sample sizes are indicated.
Graduate Satisfaction Percentage: Percentage of students who rated Very Satisfied or Satisfied for the usefulness of the college education in achieving goals after graduation.
Total Employment: Percentage of students who are employed six months after graduation from their college education.
Related Employment: Percentage of students who responded Yes or Yes, partially to related employment after graduation. This question is only asked of students who are currently employed.
Employer Satisfaction: Percentage of employers who answered Very Satisfied or Satisfied for the graduate’s overall college preparation for current employment.When considering the information from Employer and Graduate surveys, do take into account that the trends may not be as valid as those from the Student Survey - there are lower numbers of respondents and students who are working successfully are more willing to have employers contacted than students who may not be performing well. For detailed information on graduates, and comparable statistics for programs at other colleges, see the MTCU Employment Profile report at http://www.tcu.gov.on.ca/eng/labourmarket/employmentprofiles/ .
Program Costing Tab
ACA1000 – Costing Year Detail by Program This report provides a multi-year program contributions trend chart, along with costing figures for the program for a given fiscal year. The report allows drill-through to further data, including space cost, depreciation and enrolment. Details for other fiscal years is accessed via drop-down menu.
30
CURRICULUM MAPPINGAnalyzing the mapping of a program’s curriculum to its Program Standards is an essential part of PQR. Building on the course outline reviews conducted as part of annual curriculum review, it provides the opportunity to review cumulative effects of changes made to the program’s curriculum over the last five years, determine whether it meets the requirements of the Program Standards, and plan for updates to the program. The exercise of performing this analysis is called Curriculum Mapping.
Program Standards Since 1993 the Government of Ontario has been developing system-wide program standards for postsecondary programs in colleges across Ontario. Program standards apply to all similar programs of instruction offered by colleges across the province. Each program standard contains the following elements:
Vocational Standard (Vocational Learning Outcomes) – VLOs Essential Employability Skills - EES General Education requirement
Learning outcomes represent culminating demonstrations of learning and achievement. They describe performances that demonstrate significant integrated learning by graduates of the program has been achieved and verified. Learning outcomes are interrelated and cannot be viewed in isolation of one another.
Vocational StandardThe Vocational Standard captures the vocationally specific learning outcomes which apply to each diploma and certificate program. Vocational standards apply to all similar programs offered by colleges across the province.
Not all programs have a vocational standard. In this case, the College will have established internal Vocational Learning Outcomes, and these are reviewed and used for mapping purposes.
Note that, effective Fall 2013, all Ontario College Credentials offered at Algonquin College were to formally embed the Sustainability and Internationalization VLO “The graduate has reliably demonstrated the ability to identify and apply discipline-specific practices that contribute to the local and global community through social responsibility, economic commitment and environmental stewardship” within the program. Curriculum mapping will help assess the extent to which this has occurred.
Essential Employability Skills Essential Employability Skills relate to the essential employability skills (previously known as Generic Skills) needed for both career and personal success (communication skills, numeracy skills, thinking and problem-solving skills, information management skills, interpersonal skills, and personal skills). They are expressed as learning
31
outcomes. There are eleven Essential Employability Skills learning outcomes which apply to each program:
1. communicate clearly, concisely and correctly in the written, spoken, and visual form that fulfills the purpose and meets the needs of the audience.
2. respond to written, spoken, or visual messages in a manner that ensures effective communication.
3. execute mathematical operations accurately.4. apply a systematic approach to solve problems.5. use a variety of thinking skills to anticipate and solve problems.6. locate, select, organize, and document information using appropriate technology
and information systems.7. analyze, evaluate, and apply relevant information from a variety of sources.8. show respect for the diverse opinions, values, belief systems, and contributions
of others.9. interact with others in groups or teams in ways that contribute to effective
working relationships and the achievement of goals.10. manage the use of time and other resources to complete projects.11. take responsibility for one’s own actions, decisions, and consequences.
See the EES web page http://www.algonquincollege.com/ees/ or contact the EES committee or PQAA for more information.
General Education RequirementThe General Education Requirement relates to the requirements for general education courses that provide all learners with choice and breadth of experience beyond the vocational areas. General education courses cover areas of general interest and are divided into five themes:
Theme 1 - Arts in Society Theme 2 - Civic Life Theme 3 - Social and Cultural Understanding Theme 4 - Personal Understanding Theme 5 - Science and Technology
See the General Education web page for more information http://www.algonquincollege.com/gened/ .
The Essential Employability Skills and General Education Requirement apply only to Ontario College Certificate, Diploma and Advanced Diploma programs. Apprenticeship and Graduate Certificate programs are not required to meet these standards. Degree programs also have different standards to meet with respect to breadth courses. For details on each credential, refer to the Credentials Framework in Appendix B of this guide.
Figure 2 illustrates how Ontario College Credentials fit within the Credentials Framework, and the interdependency between the Program Standards and the entirety of the courses within that
32
program. A short explanatory video on Provincial Program Standards is also available at http://www.youtube.com/watch?v=awWQhUeAgdc
Curriculum ServicesIn addition to the EES and General Education committee representatives, Algonquin College has a dedicated team, Curriculum Services, which supports curriculum development and review. A Curriculum Consultant is assigned to each area of the College, and may assist in Curriculum Review and Remapping activities for PQR.
33
Figure 2: Aligning Curriculum - A Roadmap
34
Purposes of Curriculum Mapping A curriculum map can be used to
provide an overview of the curriculum for the total program document curriculum and the inclusion of standards for accrediting/credentialing groups identify opportunities in the program for learners to demonstrate learning outcomes at
the required level serve as a planning tool to ensure that all program standards are developed within the
program6 identify areas of redundancy where content is more than adequately covered identify paths that learners can follow to meet graduation requirements help faculty, learners and others “situate” courses and learning experiences within the
program curriculum
Performing Curriculum Mapping At Algonquin College, course outlines for courses which form part of an Ontario College Credential are housed within COMMS, the Course Outline Mapping and Management System. This system facilitates the review of various curriculum maps by generating reports at the program and course level. Currently, available maps/reports include
Course Essential Employability Skills by Level and Program Course Vocational Learning Outcomes by Level and Program EES Program Summary Program Overview Vocational Learning Outcome Program Summary
When course owners enter the course outline into COMMS, they provide the information on which the maps are based. They also identify whether the course is core or elective, and vocational or general education. Additionally, whether each learning outcome is taught, assessed and if there is opportunity to demonstrate a culminating performance of the outcome is captured in the outline prior to approval.
Table 1 provides the guidelines used to categorize teach (T), assess (A) and culminating performance (CP) for each activity within the course outline.
6 Some programs were established and approved prior to Program Standards being introduced into the College system. If this is noted as part of PQR, a plan needs to be initiated to develop the VLOs.
35
Teach Do you provide instruction/learning opportunities in this skill in your course?
Is this skill identified in one or more course learning requirements?
Do you devote a significant amount of time to facilitating student development of the skills and knowledge embedded in the outcome?
Assess Are there assignments, tests or projects which are designed to allow you to evaluate or assess student performance of this outcome or some of its elements?
Do you, in your evaluation of student performance, verify that this particular outcome (or a significant component of it) has been achieved?
Is this outcome reflected in your course outline in the course learning requirements and/or embedded knowledge and skills, and in the evaluation of the course outline?
Culminating Performance Culminating performances are tasks or activities designed to assess a learner’s ability to demonstrate one or more learning outcomes in their totality. While they do not necessarily occur at the end of a program of study, they do evaluate whether a learner is able to integrate and apply their learning to demonstrate the performance described in the learning outcome(s) at the exit level.
Is there an opportunity for you to evaluate the outcome in its totality?
Does the evaluation result in a final product or performance which allows you to determine whether the learner has integrated the knowledge and skills identified in the elements of performance?
Can you determine from this performance if the learner has demonstrated the outcome?
Program culminating performances should require learners to demonstrate learning similar to what would be expected of new graduates in as close to a “real world” context as possible.
If a culminating performance is assigned to a course, the professor teaching the course will be asked to evaluate whether or not the students successfully demonstrated the outcome(s) the culminating performance addresses
Table 1: Guidelines for Teach, Assess, Culminating Performance
36
Course Outline ReviewEach year, as part of Annual Curriculum Review, course owners are expected to formally review their course outlines and course section information and update courses as necessary to reflect changes made in the course content. Course outline updates may include changes to course learning requirements, learning activities, or assessment methodologies.
In the PQR year, each course teacher submits their course outline check-up worksheets or equivalent for each course taught to the Team Leader ahead of the faculty focus group meeting. The course teacher uses this review to reflect on the course outline, identify areas which may require strengthening, and to determine whether the course is meeting College requirements for course documentation (see for example, the Data Entry Guidelines and Policy AA26 Course Outlines and Course Section Information). The Team Leader builds on the course teachers’ observations, using the check-up sheets to formulate discussion points for the faculty focus group, and prepare a Course Outline Review summary sheet for the final PQR report.
For the summary sheet, the course teacher and Team Leader need to assess each course against the five evaluation elements pertaining to course learning requirements, program outcomes, learning activities and resources, evaluation methods, and prior learning assessment and recognition (PLAR) to determine whether:
There is congruency between the course learning requirements and the program learning outcomes (Quality Criteria Element #3.8). (Note this is for VLOs and EES outcomes.)
There is a match between course learning requirements, course learning activities, and learning resources (Quality Criteria Element #3.9).
Learning methods are published and are matched to the learning requirements (Quality Criteria Element #4.3).
Evaluation methods allow students to demonstrate the course learning requirements (Quality Criteria Element #4.7).
PLAR opportunities exist and are based on course learning requirements (Quality Criteria Element #1.4).
The program overview report, available via COMMS, is particularly helpful when reviewing the Curriculum Map, as it provides in one report, a side-by-side view of all courses per level. For any curriculum gaps noted, an action plan is developed and included in the final recommendations.
Vocational Learning Outcomes – Reviewing the MapAt the faculty focus group, the Team Leader facilitates the review of the Vocational Learning Outcomes maps (by Program and by Level) using the checklist Analysis and Review of Curriculum Maps for PQR . This checklist is included in the final PQR report.In addition to analyzing the VLO maps for coverage of each VLO and flow of the T, A and CP for each outcome, the Program of Study is reviewed. Due consideration must be given to the sequencing of the courses – is it appropriate? – and to the prerequisites and corequisites – are they appropriate to ensure students have an adequate foundation on which to build their learning without unnecessarily impeding progression in the program?
37
Any actions arising are noted, and included in the recommendations in the final PQR report.
Essential Employability Skills – Reviewing the MapAlgonquin College has an Essential Employability Skills committee whose membership includes faculty members and the PQAA. This committee reviews the current approved EES map for all mandated courses in the program under review, and submits a report to the PQR Team Leader. Items that are considered in the review include whether
all EES learning outcomes are included at a program level there are at least two culminating performances for each skill there is an opportunity for learners to be taught and assessed in each skill prior to
demonstrating the culminating performance there is congruency between what is reflected on the map and in the course outlines there are any gaps or redundancies within the map.
The EES report is used by the Team Leader to aid in Curriculum Mapping. It is also included in Appendix A of the final PQR report.
Details regarding EES at Algonquin College are found at: http://www.algonquincollege.com/ees and within Policy AA28 -- Essential Employability Skills. Both sites contain information regarding the leveling expected from EES outcomes for each Ontario College Credential.
General Education Requirement – Reviewing the MapAlgonquin College has a General Education committee, whose membership includes faculty members and the PQAA. This committee reviews the current approved General Education map for all mandated courses in the program, and submits a report back to the PQR Team Leader. In addition, the committee reviews the mandated General Education courses to determine whether they meet General Education requirements. This information is also forwarded to the Team Leader.
The General Education report is used by the Team Leader to aid in Curriculum Mapping. It is also included in Appendix A of the final PQR report.
Details regarding General Education at Algonquin College are found at: http://www.algonquincollege.com/gened and within Policy AA27 – General Education.
38
EVALUATION AGAINST THE QUALITY CRITERIAThe criteria against which the program is evaluated are provided below. First, a listing of the evaluation criteria and its corresponding elements is provided. This is followed by a listing with help text to assist in the evaluation of each element of the quality criteria. Note that these elements are based on the PQAPA quality criteria, and at times there is necessary repetition.
Evaluation Criteria Listing Criterion 1 – Regular program quality assessment that involves faculty, students, industry representatives, and others as appropriate for the purpose of continual improvement is in place and happens.
1.1 Students indicate that they are satisfied with the program.1.2 Program Council meetings are held regularly in accordance with college policy. Issues
raised are captured and feedback provided to students in a timely fashion. 1.3 Learners progress through the program, achieve program outcomes and graduate in a
timely fashion. 1.4 Learners with a wide range of abilities demonstrate the expected learning outcomes. 1.5 Program Advisory Committee meetings are held regularly in accordance with college
policy. Issues raised are captured and feedback provided to the committee in a timely fashion.
1.6 Graduates are satisfied with the overall program experience. 1.7 Graduates are obtaining employment in their fields. 1.8 Graduates are successful in obtaining external licenses or credentials where relevant. 1.9 Employers are satisfied with graduate performance.
Criterion 2 – Admission, credit for prior learning, promotion, graduation, and other related academic policies support student achievement of program learning outcomes.
2.1 The qualifications and prerequisites required of the applicant are published and are appropriate to allow the student to be successful without limiting access to the program.
2.2 Students have information to allow them to make informed choices about: selecting the correct program to meet their career goals; the financial commitment; the workload commitment; and the available study options.
2.3 Students know how to get internal and external transfer of academic credits and recognition for prior learning.
2.4 Students know what is needed to ensure they will be able to demonstrate program outcomes and complete the program.
2.5 Students know how they will be evaluated. 2.6 Students indicate the learning requirements are relevant and meaningful. 2.7 Students indicate that assessment methods relate to the learning requirements.
39
Criterion 3 – Program-level learning outcomes are established for all programs of instruction, consistent with the programs’ intended purposes, and appropriate for the credential offered upon successful completion of the program.
3.1 Program learning outcomes are appropriate to the level at which the qualification is offered, consistent with the requirements of the Credentials Framework, and appropriate to the occupational requirements of the program graduates.
3.2 Program learning outcomes are consistent with MTCU Provincial Program Standards where they exist.
3.3 Program learning outcomes are reflected in course outlines. 3.4 Program learning outcomes are used in the design and development of teaching and
learning activities and student learning assessments. 3.5 Program learning outcomes are used in prior learning and assessment (PLAR). 3.6 Changes to courses and program learning outcomes are introduced on a timely basis
and are designed to maintain the relevance of the program. 3.7 The capabilities of program graduates, including knowledge, understanding, skills, and
attitudes are consistent with the intended program learning outcomes.
Criterion 4 – Methods of program delivery including the design, development and execution of teaching and learning activities and student evaluation strategies are consistent with the program learning outcomes.
4.1 Program delivery (including that which takes place off-site) is consistent with the nature of the program, the learning outcomes, and the needs of the students.
4.2 There is a range of instruction methods consistent with a variety of learning styles and learner needs and abilities.
4.3 Learning activities are published and are matched to the learning outcomes. 4.4 There is congruency between the course learning requirements and the program
learning outcomes. 4.5 There is a match between course learning requirements, course learning activities,
and learning resources. 4.6 Learners are provided the skills necessary to be successful with the learning strategies
selected. 4.7 Evaluation criteria are published and students are aware of how and when they are
going to be evaluated. 4.8 There is a match between course learning requirements and evaluation methods, i.e.,
evaluation methods allow students to demonstrate the course learning requirements. 4.9 There is a range of evaluation methods used consistent with a variety of learning
styles. 4.10 Evaluation methods are valid and reliable. 4.11 Students indicate that feedback is timely and allows them to build on their learning. 4.12 Students perceive evaluation to be fair. 4.13 Practices for resubmissions, supplementals, and appeals are published, appropriate,
fair, valid, and implemented consistently.
40
4.14 Student workload and assessment is balanced across the term at both the course and program level.
4.15 Learners can earn credit for up to 75% of the program hours using the PLAR process. 4.16 Academic policies and practices that provide for the development and continuous
improvement of teaching and learning methods are valued, documented, and supported.
Criterion 5 – Programs conform to current government policy related to the design and delivery of programs of instruction, are consistent with accepted college system nomenclature / program titling principles, and maintain relevance.
5.1 The duration and structure of the program are consistent with the program learning outcomes and the credential offered.
5.2 Appropriate credit is allocated for each component of the program, and transfer and laddering options are stated.
5.3 Prerequisites do not unnecessarily hinder progress in the program. 5.4 The program title is consistent with college system nomenclature / titling principles. 5.5 The program has established articulation agreements. 5.6 The program conforms to the College requirement for the number of English courses. 5.7 College designated targets regarding hybrid courses are met. 5.8 All curriculum documentation is up-to-date including course outlines and the program
monograph information. 5.9 Concepts of social, economic and environmental sustainability are embedded in the
program curriculum. 5.10 Work Integrated Learning, such as co-operative work placement, clinical/field
placement, service learning and/or participating in applied research projects, is embedded in the program curriculum.
5.11 Students have opportunities to develop the skills, knowledge and attitudes necessary to succeed in a global economy.
Criterion 6 – Human resources, physical resources, financial resources, student support resources, and technological infrastructure to support student achievement of program learning outcomes wherever and however they are delivered exist and are available.
6.1 Program faculty members, as a whole have adequate academic preparation and workplace experience to deliver a quality program.
6.2 Program faculty members are evaluated every three years. 6.3 Program faculty members engage in professional development activities that ensure
they are current in their field and developing teaching expertise. 6.4 Program faculty members work within clear and well-structured instructional plans. 6.5 Program faculty members participate in reflective practice. 6.6 Students consider faculty to be available. 6.7 Students consider faculty to be adequately prepared for class. 6.8 Students consider faculty to promote a positive attitude to learning. 6.9 All students are assigned an academic advisor
41
6.10 Academic Advisors contact their students early in the term with an invitation to meet and to ensure that students know who their advisor is.
6.11 Labs, clinical facilities and placement facilities are complementary to and integrated into the program and allow the learner to demonstrate the learning outcomes.
6.12 Students indicate that there are adequate and accessible learning resource materials to allow them to be successful, including: textbooks in the bookstore; online materials; print resources; equipment and student support services.
6.13 The program is financially viable. 6.14 The demand for the program has been sustained for the last five years. 6.15 There is a future demand for graduates of this program. 6.16 The learning environment is supportive and conducive to learning. 6.17 The students are provided with the information they need to know to function safely in
both the College and workplace learning environments. 6.18 Program materials, including the program monograph, course outlines and course
materials, are provided in an accessible ready format as per AODA regulation.
Responding to the Evaluation Criteria Each quality criterion examines a specific area of quality in the program. The responses to the quality criteria take stock of the information arising from the review – data analysis; student, faculty and external stakeholder input; curriculum and document review; as well as informal feedback.
When you respond to the criteria, you are looking for areas of agreement, areas where there is not alignment, and areas where there is no information available, based on the information compiled during the review. You then tick Yes, No, or Other, and support the response by providing the reason for the assessment. Every criterion needs a response. A basic approach is to write two sentences - the first responding directly to the criterion, the second elaborating by providing the reason or citing evidence from the report appendices.
Here is a sample response:
4.11 Students indicate that feedback is timely and allows them to build on their learning.
YES
KPI student satisfaction data indicates that students are satisfied or neutral regarding the feedback on their progress (range is 75-85% of students over past 5 years).
This was confirmed by current students at the student focus group, and alumni at the External Stakeholder focus group. Excepting one ‘outlier’ course, feedback is provided with enough detail, and in enough time, to allow students to know where to concentrate their study/practice time.
42
Evaluation Criteria Listing – with help text Evaluation criteria should be considered on an iterative basis during the program review. In some situations there will be adequate survey data and additional information to state that an element is met without further inquiry. In other areas, there may be residual questions after reviewing the available program data and documentation. These questions then need to be explored with any or all of the Chair, the faculty, the students or external stakeholders.
The following help text provides guidance to evaluate the program against the quality criteria.
Criterion 1. Regular program quality assessment that involves faculty, students, industry representatives, and others as appropriate for the purpose of continual improvement is in place and happens.1.1 Students indicate that they are satisfied with the program.Main sources of input: Program Council Minutes, Student Surveys, Student Focus Group
Several questions on the KPI Student Satisfaction Survey speak to student satisfaction. Capstone questions are:
overall quality of knowledge and skills given by the program overall quality of the learning experiences in the program overall college facilities overall quality of services
In addition to detailed responses on every KPI question, the five year trend for the average student satisfaction, as well as for each individual capstone, is in the program data. Outcomes of these questions may be discussed at the student focus group.
Analysis of student survey data student may identify issues that need to be discussed at the student focus group meeting.
1.2 Program Council meetings are held regularly in accordance with college policy. Issues raised are captured and feedback provided to students in a timely fashion.
Main sources of input: Program Council Minutes
Discuss Program Council activities here, and how that process works in the program. A review of Program Council minutes can provide supporting data that indicates that issues from one term are followed up until resolved or identified as no longer needing action. The feedback mechanism that is in place for the academic staff to communicate to students across the program can also be discussed.
1.3 Learners progress through the program, achieve program outcomes and graduate in a timely fashion.
Main sources of input: Program Data, Faculty Focus Group
When considering this element, examine retention, progression and graduation rate for the program.
43
Registration reports for the last five years help identify any terms in particular that appear to inhibit a student’s progress through the program.
Identification of courses which have high failure rates may need to be reviewed and strategies recommended that allow the students to be successful while maintaining the academic integrity of the program. Faculty and students can provide valuable feedback for this element.
Concerns here also include core courses that are only offered once a year. Consider whether there are strategies available to allow students to continue without having to lose a whole academic year if they fail the course.
When courses with high failure rates are offered off-cycle to help students remain on cycle in the program, this should be discussed together with strategies developed to help the students be successful, for example specialized learning tools for courses that require a lot of rote learning, such as anatomy or geography.
In cases where students are not successfully progressing through the program, endeavor to establish the cause – curriculum, financial, other inhibitor – in order that consideration may be made to provide additional support and/or adapt the curriculum.
1.4 Learners with a wide range of abilities demonstrate the expected learning outcomes.Main source of input: Faculty Focus Group
While no statistics are available for the number of students with differing abilities and their success in the program, faculty and students may provide some anecdotal information relevant to this element. Highlight special accommodation that program staff members have made to assist students with special needs to meet the program outcomes and be successful in graduating from the program.
1.5 Program Advisory Committee meetings are held regularly in accordance with college policy. Issues raised are captured and feedback provided to the committee in a timely fashion.
Main source of input: Program Advisory Committee Minutes
Discuss Program Advisory Committee activities here, and include specifically how the relevance of the program to the field of practice it serves is assessed, and the program modified if required. A review of Program Advisory Committee minutes can provide supporting data to indicate whether program items are followed up until resolved or identified as no longer needing action.
1.6 Graduates are satisfied with the overall program experience.Main sources of input: Program Data, Alumni Surveys, External Stakeholder Focus Group
Five years of Graduate Satisfaction data are provided within the program data (KPI graduate satisfaction report). Low satisfaction and/or declining trends in this area can be further
44
investigated at the external stakeholder focus group meeting. It is critical to ensure that several graduates attend this meeting.
1.7 Graduates are obtaining employment in their fields. Main sources of input: Program Data, Alumni Surveys, External Stakeholder Focus Group
The program data provides five years of data about employment rates, which is calculated directly from the KPI Employer Satisfaction Survey. Trends within these data can be discussed at the external stakeholder focus group meeting.
It is important to identify factors affecting employment rates at the time of the KPI Employer Satisfaction Survey. For example, graduates of a building trade program who finish their program in August may not find employment until the following Spring, however the survey may be carried out in February.
1.8 Graduates are successful in obtaining external licenses or credentials where relevant.Main sources of input: Program Chair/Coordinator
Include this information and its relevance as applicable. For example, if the students write a licensing examination to obtain provincial or national certification and the results of the success of your graduates on those examinations are available, include them here.
Note any formal arrangements with external bodies, and how these are communicated to students.
1.9 Employers are satisfied with graduate performance.Main sources of input: Program Data, Employer Surveys, External Stakeholder Focus Group
Review the five year trend for employer satisfaction within the program data. Note that the response rate for the employer survey is usually low, as prior to contacting the employer, the graduate’s permission must be obtained (along with employer contact details). Due to the lower response rates, the data may not be as meaningful as data from other surveys.Any issues that arise from the survey data can be added to the agenda of the external stakeholder focus group meeting.
Criterion 2. Admission, credit for prior learning, promotion, graduation, and other related academic policies support student achievement of program learning outcomes.
2.1 The qualifications and prerequisites required of the applicant are published and are appropriate to allow the student to be successful without limiting access to the program.
Main sources of input: Faculty Focus Group/Curriculum Review; Monograph Review
In order to assess this element, each Team Leader needs to review the current admission requirements with faculty, students, and the external stakeholder groups. Faculty teaching in level 01 of the program will be able to provide valuable input about the degree of preparedness
45
of the students entering the program. In addition, review the failure rate report. If there are courses in level 01 with consistently high failure rates, admission criteria may need to be reviewed or the course content may need to be spread out across more levels.
Students in the student focus group will give their perspective about the admission criteria and how well they feel they were prepared to be successful in level 01 without encountering difficult barriers when applying to the program.
Some programs offer remedial opportunities to assist students with specific deficiencies when the department does not want to inhibit access to the program, but recognizes that students need assistance to be successful. These opportunities should be highlighted in the report indicating the contribution they make to student success.
The response to this element may start with the wording:
Application information is published approximately 15 months in advance of the student starting in the program. This information is entered into the student information system (GeneSIS) and approved during the curriculum review period each Winter. The admissions information is published on the OCAS website, in the Monograph, on the College website, and the College Calendar each year in September for the next year.
and be supported by additional evidence as outlined above.
2.2 Students have adequate information to allow them to make informed choices about: selecting the correct program to meet their career aspirations; the financial commitment needed; the workload commitment needed; and the study options available to them.
Main sources of input: Student Focus Group; Faculty Focus Group; Monograph Review
This information is published in the Program Narrative section of the Program Monograph. Also, one of the four capstone questions on the KPI Student Satisfaction Survey asks students to comment on whether the program provides the skills and abilities specific to their chosen career. It is essential that the information given to applicants allows them to understand what skills they will learn, and how the program will prepare them for the workplace, as well as for which jobs in the workplace.
The items within this element should form key discussion points for the Student Focus Group. The program narrative can be reviewed with students. For example, ask Did students find the program description useful? How useful are the success factors and the employment information in helping the student with decisions regarding selecting the right program?
In addition, review the Reason for Withdrawal report to identify further issues, for example, if a large number of students withdraw citing Program unrelated to career goals, as the reason for leaving the program, the published materials may need to be revised.
The members of the external stakeholder focus group will be able to provide feedback regarding the Employment Opportunities section of the Program Monograph.
46
2.3 Students know how to get internal and external transfer of academic credits and recognition for prior learning.
Main sources of input: Student Focus Group; Faculty Focus Group; Monograph Review
Several College publications identify the process for students to follow to obtain credit for previous education and experience. A key item for discussion at student focus group meetings is whether students knew where to find this information and if they found this information useful. It is important to ask this question to students who did apply for credit to determine if they received the assistance they needed in a timely fashion, as students who did not need to apply for exemptions may not be knowledgeable on the process. Information may also be presented at College or program orientation events.
2.4 Students know what is needed to ensure they will be able to demonstrate program outcomes and complete the program.Main sources of input: Student Focus Group; Monograph Review
The element explores information beyond Academic Admission requirements. Students need to know what the expectations and obligations are for them to advance in the program. This might include the necessity to submit health and immunization forms, proof of a current valid driver’s license, or a police records check. Identify any special requirements the program might have, ensuring that these are in line with College and MTCU guidelines, and note how the students are informed.
The program may also have special progression requirements, for example, students need to obtain a higher grade than just a pass to proceed on to a course for which it is a prerequisite. Identify and review these unique situations to ensure they are legitimate requirements. Note this information, and how it is communicated to students.
As in 1.8, the program may also have external relationships governing additional credentials and/or certifications for which students may qualify on completion of the program. Note this information, and how it is communicated to students.
2.5 Students know how they will be evaluated.Main sources of input: Student Focus Group; Faculty Focus Group; Curriculum Review
Students need to understand what to expect regarding evaluation methods, how to receive special accommodation, how to appeal a grade, and how progression is determined. Describe how the department informs students regarding assessment and progression. Examples of unique situations where students have received special accommodation can be briefly discussed. Cite information included in letters to students/applicants, within course descriptions, and course outlines. This question should be posed at the Student Focus Group.
2.6 Students indicate the learning requirements are relevant and meaningful.Main sources of input: Student Focus Group; Student Surveys
47
Student surveys and the Faculty program self-audit responses may provide some insight into student perceptions. Current students can be asked this question at the Student Focus Group.
2.7 Students indicate that assessment methods relate to the learning requirements.Main sources of input: Student Focus Group; Student Surveys
Student surveys and the Faculty program self-audit responses may provide some insight into student perceptions. Current students can be asked this question at the Student Focus Group
Criterion 3. Program-level learning outcomes are established for all programs of instruction, consistent with the programs’ intended purposes, and appropriate for the credential offered upon completion of the program.
3.1 Program learning outcomes are appropriate to the level at which the qualification is offered, consistent with the requirements of the Credentials Framework, and appropriate to the occupational requirements of the program graduates.
Main sources of input: Curriculum Review; Credentials Framework; External Stakeholder Focus Group
Program learning outcomes form the foundation of any program offered at Algonquin College. They provide the structure from which programs are developed and renewed, and as such, a review of the program’s learning outcomes is a fundamental component of PQR.
Review the Provincial Program Standard for the program, or, if not published, the PQAA can assist in locating the program description.
Vocational Learning Outcomes are routinely reviewed by Faculty and with the Program Advisory Committee. During PQR they are examined to ensure that the outcomes remain valid and are at the level appropriate for the credential, and appropriate for the occupational requirements of program graduates (Curriculum Mapping7 and with the External Stakeholders).
The Credentials Framework is included in Appendix B of this guide for reference. Care should be taken to review all the expectations for the credential offered – duration, structure, General Education requirements etc. This information will be useful for responding to other quality criteria elements.The PQAA will provide additional guidance when reviewing Apprenticeship, College Certificate and Degree programs.
In a situation where curriculum does not meet the College or Ministry Standard in any area, a plan to ensure the curriculum will conform in the
7 Note that Curriculum Mapping ensures that the review of the outcomes extends to the General Education component of the program, and the Essential Employability Skills outcomes. Information on EES leveling can be obtained from the EES committee, or through the EES Policy AA28 Essential Employability Skills. (Reports on General Education and EES are provided by the appropriate committees to the PQR Team Leader.)
48
subsequent academic year must be clearly identified in the recommendations and implementation plan. Changes will be finalized during the annual curriculum review period.
3.2 Program learning outcomes are consistent with MTCU Provincial Program Standards where they exist.Main sources of input: Program Standard, if exists
Program learning outcomes must meet or exceed the MTCU Provincial Program Standards if they exist for the program. The programs for which provincial program standards exist are found at http://www.tcu.gov.on.ca/pepg/audiences/colleges/progstan/ .
3.3 Program learning outcomes are reflected in course outlines.Main source of input: Curriculum Review – Course Outline Review Summary
This will be examined during Curriculum Mapping. All programs with course outlines loaded into COMMS should have program outcomes reflected in course outlines. It is an important part of PQR to review to ensure that program outcomes are easily identifiable, valid and mapped appropriately at the course level.
3.4 Program learning outcomes are used in the design and development of teaching and learning activities and student learning assessments.
Main sources of input: Faculty Focus Group; Curriculum Review – Course Outline Review Summary
This will be examined during Curriculum Mapping at the Faculty Focus groups. Explore how program learning outcomes inform the design, development and update of teaching and learning activities and student learning assessments, and note areas that could be improved. Relate the operational activities to annual curriculum review activities.
3.5 Program learning outcomes are used in prior learning assessment and recognition (PLAR).
Main sources of input: Faculty Focus Group; Curriculum Review – Course Outline Review SummaryAs part of annual curriculum review, course outlines are individually reviewed using the course outline review worksheet or equivalent. PLAR options are recorded as part of each course outline review and the findings are documented on the Course Outline Review Summary sheet. NB: how a student can challenge the course using PLAR needs to be identified on each course outline.
As each course is reviewed, it needs to be determined how faculty can evaluate the outcomes for the purposes of PLAR. Usually a department has two weeks to set up an appropriate challenge. Therefore, it is prudent that the challenge be determined in advance. The two weeks lead time allows for updating the challenge.
49
3.6 Changes to courses and program learning outcomes are introduced on a timely basis and are designed to maintain the relevance of the program.
Main sources of input: Faculty Focus Group; Curriculum Review.
Against this element, describe how the department determines what updates are necessary and how they are incorporated into the curriculum. For example, state as a result of feedback from student program councils, agency staff, and advisory committee members, changes are made annually as part of the annual curriculum review process. Refer to relevant meeting minutes to indicate how this information was gathered and used. Relate the operational activities to annual curriculum review activities.
3.7 The capabilities of program graduates, including knowledge, understanding, skills, and attitudes are consistent with the intended program learning outcomes.
Main sources of input: External Stakeholder Focus Group Faculty Focus Group; Curriculum Review.
Are the graduates meeting/demonstrating all outcomes? Describe how the program provides experiential learning opportunities that allow the learners to perform, with support, as a practitioner in their field. Discuss planned learning activities that allow the student to perform as a practitioner. In addition, review and discuss the student feedback as identified in student surveys or focus groups.
Review the evaluation strategies for courses in the final level of the program. Are the students ready to function in the workplace at the entry level expected? This is an important question to pose at the external stakeholder focus group meeting.
Criterion 4. Methods of program delivery, including the design, development and execution of teaching and learning activities and student evaluation strategies are consistent with the program learning outcomes.
4.1 Program delivery (including that which takes place off-site) is consistent with the nature of the program, the learning outcomes, and the needs of the students.
Main sources of input: Faculty Focus Group; Student Focus Group; Program Chair(s)/Academic Manager(s)
This element should be discussed with students and faculty. With the broad mix of demographics in the classroom, curriculum design needs to be flexible. As well, students are not equally prepared to learn online, in groups, or using independent studies. It is important to note successes as well as challenges faced in the classroom.
If the program is delivered on more than one campus, discuss how regional differences are accommodated. Outline how communication between faculty on different campuses is facilitated. Similarly, if the program is delivered both on-campus and online, discuss how
50
communication between those managing the different delivery modes is facilitated. (Aside: as much as possible, programs delivered at multiple campuses and/or online are reviewed in the same PQR cycle, ideally as a joint review.)
If a program is delivered off-site, describe accommodations made to ensure there is consistency in the ability of the students to demonstrate the program outcomes. This could include specifications as outlined in the Memorandum of Understanding and the process for ensuring that off campus graduates meet the learning outcomes.
4.2 There is a range of instruction methods consistent with a variety of learning styles and learner needs and abilities.
Main source of input: Faculty Focus Group
The Team Leader and faculty review the course outlines and the information in the course outline review form to evaluate this element for each course. Faculty need to discuss the current offering of learning methods, and determine whether the variety is sufficient for the learners in each course. The Program Overview report in COMMS may help with this review.
State any anecdotal information about special accommodations that the program implements to aid student success. For example, discuss strategies used to engage learners, support special needs, or provide valuable learning opportunities.
4.3 Learning activities are published and are matched to the learning outcomes.Main sources of input: Faculty Focus Group; Curriculum Review – Course Outline Review Summary
Learning activities and learning outcomes are published on each course outline. It is important to review the stated learning activities to ensure that the learning activities chosen do indeed map to the stated outcomes.
4.4 There is congruency between the course learning requirements and the program learning outcomes.
Main sources of input: Faculty Focus Group; Curriculum Review – Course Outline Review Summary
Faculty who teach in the program will be able to provide valuable insight into the sequencing of the courses and the culmination of the various course learning requirements to the program learning outcomes. Curriculum maps will demonstrate the distribution of the program outcomes across all levels and indicate if any areas need consideration, for example learning outcomes that have more than adequate coverage.
The analysis of the curriculum maps is discussed in the Curriculum Mapping section of this manual.
4.5 There is a match between course learning requirements, course learning activities, and learning resources.
51
Main sources of input: Faculty Focus Group; Curriculum Review – Course Outline Review Summary
This is an important element for faculty to review as part of the Course Outline Review process. It is important to ensure that the match is obvious. This information is captured in the Course Outline Review summary table.
4.6 Learners are provided the skills necessary to be successful with the learning strategies selected.
Main sources of input: Faculty Focus Group; Curriculum Review
As part of annual curriculum review activities, faculty review course learning activities and ensure that relevant skills are introduced at appropriate points in the Program of Study to ensure students have the necessary skills to be successful. Document how, for example, faculty provide orientation to online learning to ensure success in later courses, or, if group projects are required, how orientation to group work is provided. Strategies used for any other skills needed by students to be successful learners may also be captured.
It is important to review the Failure Rate report for courses where special skills are required and identify whether students are successful.
Note that the second English course is expected to provide students with presentation skills. This course should be reviewed to ensure the content includes the skills needed to meet the expectations of the workplace.
4.7 Evaluation criteria are published and students are aware of how and when they are going to be evaluated.
Main sources of input: Faculty Focus Group; Curriculum Review
The course outline review process will indicate how the evaluation criteria, including type and timing of evaluation, and marking schemes, such as rubrics, are shared with the student. Gaps should be identified and a plan to address put in place.
Penalties for late assignments need to be reviewed to ensure that penalties are consistent across the program or that any differences in specific courses or course sections are justifiable, published, and implemented equitably).
Briefly describe policies and how they impact student success. Indicate minimum standards for feedback to students.
4.8 There is a match between course learning requirements and evaluation methods, i.e., evaluation methods allow students to demonstrate the course learning requirements.
Main sources of input: Faculty Focus Group; Curriculum Review – Course Outline Review Summary
52
This match is assessed during course outline review. It is important to include examples about the evaluation methods used and how they allow students to demonstrate the outcomes. Highlight special accommodations made or innovative methods used by the program faculty. There are many resources available regarding learning assessments. One useful website is http://gototheexchange.ca/index.php/curriculum-at-course-level/assessing-the-achievements-of-learning maintained by the ABC program.
Include examples of evaluation design, and how evaluations map to the program learning outcomes. 4.9 There is a range of evaluation methods used consistent with a variety of learning
styles.Main sources of input: Faculty Focus Group; Curriculum Review; Student Focus Group
This element will need to be discussed with both faculty and students. The COMMS Program Overview report is a useful tool to visualize the student experience in each level. In the PQR report, highlight methods used that demonstrate meeting the needs of various learning styles, and highlight any innovative evaluation strategies used.
4.10 Evaluation methods are valid and reliable.Main source of input: Faculty Focus Group; Curriculum Review
Describe processes that faculty use to ensure that evaluation methods are reliable and valid, for example, using rubrics, examining negatively discriminating questions on multiple choice examinations prior to submitting final grades, or reviewing exam outcomes to determine questions that students answer poorly.
Pay particular attention to courses which have a failure rate of 25% or higher. It may be that the evaluation strategies are not aligned with the outcomes, or the course needs to be better placed. Actions for curriculum update should be noted and carried forward into the implementation plan.
4.11 Students indicate that feedback is timely and allows them to build on their learning.Main sources of input: Student Focus Group; Student Surveys
Student surveys may provide insight into past students’ experience. This element can be further explored at the student focus group. If the students indicate that feedback is not timely or helpful, note this in the PQR report being careful to maintain confidentiality. Recall that PQR is examining the program as a whole. Should matters arise that pertain to individuals, please work with the Chair/PQAA to ensure confidentiality is preserved.
4.12 Students perceive evaluation to be fair.Main sources of input: Student Focus Group; Student Surveys
53
Student surveys may provide insight into past students’ experience. This element can be further explored at the student focus group.
4.13 Practices for resubmissions, supplementals, and appeals are published, and are appropriate, fair, valid, and implemented consistently.
Main sources of input: Faculty Focus Group; Document Review
Describe the program policy and how it is published and implemented. In particular, department practices can be compared to Policy AA13 - Evaluation of Student Learning and Policy AA39 - Program Progression and Graduation Requirements.
4.14 Student workload and assessment is balanced across the term at both the course and program level.
Main sources of input: Faculty Focus Group; Curriculum Review
This should be examined by the Team Leader and at the faculty focus group meeting. Discuss here the efforts faculty members make to ensure a balance of workload and assessment of all courses. The COMMS Program Overview report is a useful tool to visualize the student experience in each level, and to help plan future workload balancing.
Further feedback may be requested at the student focus group meeting.
4.15 Learners can earn credit for up to 75% of the program hours using the PLAR process.
Main source of input: Curriculum Review – Course Outline Review Summary
Whether a course is eligible for PLAR is noted on each course outline, along with the method(s) students may use to provide evidence of learning achievement. The team leader notes whether the course is eligible on the Course Outline Review Summary Sheet. To determine whether the 75% threshold is being met, tally the number of hours in each PLAR-eligible course and compare this to the total number of program hours.
Where a course is not eligible for PLAR, capture the reason(s) why in the Course Outline Review Summary Sheet. This should be reflected also in the course outline.
If the program has less than the required number of hours with PLAR challenges available, note this against the element, and include a recommendation to address the shortfall in the PQR implementation plan.
Note that PLAR challenges need to be established using existing College and Ministry policies and fees structures. Assistance to define the challenges is available from the Centre for Organizational Learning. Recall that, on the course outline itself, it is only necessary to identify whether the course is challengeable, and to provide contact information. The actual challenge format should be identified and planned, although time will be needed and allowed to ensure the challenge tool is updated.
54
Aside: Once a challenge process is in place for a course, the PLAR flag needs to be set within GeneSIS for the course.
4.16 Academic policies and practices that provide for the development and continuous improvement of teaching and learning methods are valued, documented, and supported.
Main source of input: Summary of Faculty Credentials and experience, Program Self-Audit
Describe how program faculty have experimented with new teaching methods and how they evaluate the effect on student success.
Identify any relevant Professional Development that faculty have attended and found useful. This information will be found in the summary of faculty credentials and experience summarized by the Chair.
It is helpful to identify areas the faculty feel could be improved to help them develop their teaching skills. For example, if students identify online courses as challenging, a follow up action may be to assign PD for faculty to improve online course delivery.
Criterion 5. Programs conform to current government policy related to the design and delivery of programs of instruction, are consistent with accepted college system nomenclature / program titling principles, and maintain relevance.
5.1 The duration and structure of the program are consistent with the program learning outcomes and the credential offered.
Main source of input: Curriculum Review
The Credentials Framework document is included in Appendix B of this guide. Compare the program duration and outcomes to the framework to ensure the program is in the appropriate credential level.
5.2 Appropriate credit is allocated for each component of the program, and transfer and laddering options are stated.
Main sources of input: Faculty Focus Group; Curriculum Review
It is important for faculty to ensure that courses are organized and sequenced to support student learning, are allocated appropriate hours of study, and are consistent with published requisites. (Note that this review is formally part of annual curriculum review.)
Laddering opportunities for graduates to move into the next credential, either at the College or at university, as well as opportunities for receiving external certifications etc., should be noted here. Opportunities to specialize or select majors should be discussed if applicable.
5.3 Prerequisites do not unnecessarily hinder progress in the program.Main sources of input: Faculty Focus Group; Curriculum Review
55
Prerequisites may not be an issue for some programs. If there are situations where students, as a result of failing a prerequisite course, cannot continue in the program without waiting a whole year, it is valuable to review the prerequisite to ensure that the knowledge is indeed foundational and the student is not likely to be successful if allowed to progress without the prerequisite.
Note strategies that are in place to help students stay in the program, even though they may have to repeat a course.
5.4 The program title is consistent with college system nomenclature / titling principles.Main sources of input: Faculty Focus Group; Curriculum Review
Please check that the program title is consistent with that expected for the credential offered, for example, ‘technician’ is normally applied to a 2-year diploma program, ‘technologist’ to a 3-year advanced diploma. Note if an alternative name is attributed to the program, for example, for marketing purposes.
The program must reflect title in the MTCU Provincial Program Standards if they exist for the program. The programs for which provincial program standards exist are found at http://www.tcu.gov.on.ca/pepg/audiences/colleges/progstan/ .
5.5 The program has established articulation agreements.Main source of input: Document Review
Articulation agreements currently in place for the program should be reviewed to ensure that any major curriculum changes made since the last review have been reflected within the agreement.
The final PQR report should include a list and brief description of any articulation agreements that exist with other postsecondary institutions. If no articulation agreements currently exist and graduates from the program regularly seek university education, the implementation plan may include a recommendation to consider establishing an articulation agreement.
Information on degree pathways/articulation agreements is found at: http://www3.algonquincollege.com/degree-pathways/.
Note: Include in this section a list of any pre-College laddering opportunities, for example School to College to Work Initiative (SCWI) projects occurring in partnership with the program, (e.g. program specific Specialist High Skills Majors (SHSM) or Dual Credit initiatives).
5.6 The program conforms to the College requirement for the number of English courses.Main source of input: Curriculum Review
56
The minimum English hours of instruction for two-year and three-year programs is 90 hours, normally delivered in two 45-hour courses, and in one-year programs it is 45 hours. Criterion 5.6 is not applicable for locally approved College Certificate, Ontario College Graduate Certificates, Bachelor’s Degrees or Apprenticeship programs.
5.7 College designated targets regarding hybrid courses are met.Main sources of input: Faculty Focus Group; Curriculum Review
Online learning is an integral component of programs offered at Algonquin College. With some exceptions, all programs are to have a minimum of 20% of program hours conducted through online learning. Examine how online learning is integrated in the program, whether the 20% expectation is being met.
Related to this element is the examination of the positioning of online learning – is it appropriately sequenced through the Program of Study, with sufficient introduction and orientation? This is a topic to discuss at the faculty focus group.
5.8 All curriculum documentation is up-to-date including course outlines and the program monograph information.
Main sources of input: Faculty Focus Group; Curriculum Review
As part of annual curriculum review, course outlines are individually reviewed using the course outline review worksheet or equivalent. This review identifies course outlines that need to be updated. Any work required to ensure course outlines are complete will be discussed with the Chair and captured in the Implementation Plan. This includes refinements based on EES and General Education course reviews.
Program Monograph information is updated each year. If any areas have been designated as requiring modification as a result of this PQR, state what changes are being implemented. Otherwise, indicate the changes that are being made as a part of the annual update process.
5.9 Concepts of social, economic, and environmental sustainability are embedded in the program curriculum.
Main sources of input: Faculty Focus Group; Curriculum Review
Describe how concepts of sustainability are, or are planned to be, introduced into the program curriculum. If embedding of sustainability into the curriculum is a new area for consideration, faculty need to discuss plans with the Chair. The resulting implementation plan may recommend discussions at advisory committee meetings for the upcoming year to gauge required changes, with updates to the curriculum to be implemented in the following year.
Effective Fall 2013, all Ontario College Credentials offered at Algonquin College are expected to formally embed the Sustainability and Internationalization VLO “The graduate has reliably
57
demonstrated the ability to identify and apply discipline-specific practices that contribute to the local and global community through social responsibility, economic commitment and environmental stewardship” within the program. Course outline review will facilitate the areas where courses can include this learning. Note that inclusion of this outcome is not mandatory for preparatory General Arts and Science programs.
5.10 Work Integrated Learning, such as co-operative work placement, clinical/field placement, service learning and/or participating in applied research projects, is embedded in the program curriculum.
Main sources of input: Faculty Focus Group; Curriculum Review
Programs should include an experiential component, characterized by practical, hands-on learning such as co-operative education, clinical/field placement, service learning and/or participating in applied research projects.
Describe how Work Integrated Learning is, or is planned to be, embedded into the program, and carry forward into the PQR implementation plan as appropriate.
5.11 Students have opportunities to develop the skills, knowledge, and attitudes necessary to succeed in a global economy.
Main sources of input: Faculty Focus Group; Curriculum Review
Describe how concepts of cultural diversity, team work, flexibility, transfer of skills, interpersonal skills, problem-solving and decision-making ability, adaptability, creative thinking, self-motivation, the capacity for reflection, etc. are incorporated within the program curriculum. Some of these skills are currently embedded through the incorporation of the Essential Employability Skills and the College-Wide Sustainability and Internationalization.
Course outline review may highlight areas in the program where there are opportunities to include specific learning outcomes or course learning requirements that directly reflect this learning. These should be noted and the plans to update the specific course outlines should be included in the implementation plan.
Criterion 6. Human resources, physical resources, financial resources, student support resources, and technological structure to support student achievement of program learning outcomes wherever and however they are delivered exists and are available.
6.1 Program faculty members, as a whole, have adequate academic preparation and workplace experience to deliver a quality program.Main source of input: Summary report from Chair to Team Leader
Information regarding the preparation and experience of the faculty team is gathered by the Chair, who submits a summary report to the Team Leader. Student surveys will provide past student perceptions of faculty knowledge and skills.
58
Any concerns that faculty have should be identified and considered in the recommendations. Similarly, celebrate the experience and expertise that is available to support student learning.
6.2 Program faculty members are formally evaluated every three years.Main source of input: Summary report from Chair to Team Leader
Data to support this element are within the summary report that the Chair will submit to the Team Leader. Summarize the information in the final report.
6.3 Program faculty members engage in professional development activities that ensure they are current in their field and developing teaching expertise.
Main source of input: Summary report from Chair to Team Leader
Data to support this element are within the summary report that the Chair will submit to the Team Leader. Summarize the information in the final report.
Feedback should also be solicited during the Faculty focus group meeting regarding the availability and usefulness of professional development.
6.4 Program faculty members work within clear and well-structured instructional plans.Main source of input: Curriculum Review
Data to support this element are within the summary report that the Chair will submit to the Team Leader. Summarize the information in the final report.
All courses are expected to have documented course outlines and course section information within which faculty work. This will be verified during Curriculum Mapping. Should there be any issues at a program level, a plan to address should be made with the Chair, and captured within the PQR implementation plan.
6.5 Program faculty members participate in reflective practice.Main source of input: Summary report from Chair to Team Leader
Reflective practice means looking at what you do as a teacher in the classroom – thinking about why you do it, and thinking about if it works. It is a process of self-observation and self-evaluation and may include keeping a teaching diary or notes, peer observation, recording lessons, and student feedback.
Data to support this element are within the summary report that the Chair will submit to the Team Leader. Summarize the information in the final report.
Feedback could also be solicited during the Faculty focus group meeting regarding the reflective practice methods used, successfully or otherwise, by faculty.
6.6 Students consider faculty to be available.Main sources of input: Student Focus Group; Student Surveys
Students’ views are captured at the student focus group, and garnered from student surveys, including KPIs. Additionally, the Chair can learn if there is a generic or individual issue as part of the Student Course Feedback process. The Chair should not release confidential information; however, if survey data reflects that student perception is that faculty are not available,
59
assurance needs to be provided that any issues regarding faculty availability will be addressed in the Implementation Plan. Ask the PQAA for assistance with the writing of this response if necessary.
6.7 Students consider faculty to be adequately prepared for class.Main sources of input: Student Focus Group; Student Surveys
Students’ views are captured at the focus group, and garnered from student surveys, including KPIs. It is necessary here, as above, that any concerns raised by students are discussed in a confidential manner and are addressed in the Implementation Plan. The PQAA may be of assistance when determining how to document areas which need to be addressed.
6.8 Students consider faculty to promote a positive attitude to learning.Main sources of input: Student Focus Group; Student Surveys
A positive attitude to learning is considered to be an essential aspect of a quality learning environment. Please capture, if overall, the students consider program faculty to promote a positive learning environment. This question can be posed at the student focus group. Again, if concerns arise, these should be discussed in a confidential and respectful manner. The PQAA can assist in documenting appropriately.
6.9 All students are assigned an academic advisor.Main source of input: Program Chair/Academic Manager and/or Program Coordinator
Describe how academic advisors are assigned.
6.10 Academic advisors contact their students early in the term with an invitation to meet and to ensure that students know who their advisor is.
Main source of input: Student Focus Group; Academic advisor
Describe how students are informed about their academic advisor and in general, how the program implements Policy AA40 – Academic Advising. This is a good topic for the Student Focus Group. Not all departments use the same terminology, therefore a more generic question can be posed, like “Do you know where to get help if you have a problem with your program?”.
6.11 Labs, clinical facilities and placement facilities are complementary to, and integrated into, the program and allow the learner to demonstrate the learning outcomes.
Main sources of input: Program Chair/Academic Manager; Faculty Focus Group
Discuss how classrooms are used, the types of labs available, and placement activities used to provide learning opportunities for students, and allow them to demonstrate program outcomes. Celebrate quality facilities and identify changing or future needs that may have been identified.
6.12 Students indicate that there are adequate and accessible learning resource materials to allow them to be successful, including: textbooks in the bookstore;
60
online materials; print resources; equipment; technological infrastructure; and student support services.
Main sources of input: Student Focus Group; Student Surveys
Student surveys will provide insight into past student views. This quality criteria element can also be discussed at the Student Focus Group meeting. The Team Leader should also provide an assessment of the availability and adequacy of learning resource materials and student support.
6.13 The program is financially viable. Main source of input: Data review – ACA1000 Costing Year Detail by Program
A program is considered financially viable if it is contributing more than 25% after depreciation. This information is displayed in the first graphic on the ACA1000 – Costing Year Detail by Program data set. Given program budgets are developed using student enrolment projections, programs may face financial challenges if registration numbers are below projections.
6.14 The demand for the program has been sustained for the last five years. Main source of input: Data review - PQR1000 Registration Milestones by Program
The Chair and the Team Leader review the registration by milestone data and application/registration ratio to see the trend over the last five years. These data should be reviewed in the context of specific constraints of the individual program, for example, the maximum registration allowed, or any physical (space) constraints.
6.15 There is a future demand for graduates of this program.Main source of input: Program Data; External Stakeholder focus group
Issues about demand for the program, and/or demand for graduates can be identified from the External Stakeholder focus group. Registration and withdrawal statistics from the last 5 years will help identify any significant areas that may require follow up in the focus groups.
6.16 The learning environment is supportive and conducive to learning.Main source of input: Student Focus Group
This question must be asked at the Student Focus Group meeting. It is important that students feel safe asking questions, i.e., that the classroom environment is conducive to students being comfortable sharing their learning needs. Student surveys will provide insight into past students’ views.
6.17 The students are provided with the information they need to know to function safely in both the College and workplace learning environments.
Main sources of input: Student Focus Group; Faculty Focus Group
61
Describe any special orientation processes provided for learners to help them function in the classroom, lab, or workplace environments. Any special precautions taken in lab settings, for example WHMIS protocols, and use of safety equipment can be discussed here.
6.18 Program materials, including the program monograph, course outlines and course materials, are provided in an accessible ready format as per AODA regulation.
Main source of input: Document Review
Program materials are required to be available in an accessible ready format as per the AODA’s Integrated Accessibility Standard Regulations. With the Chair and faculty, review the current status of program materials, and gauge faculty awareness of the AODA. PD requirements and required updates to program materials (in order to comply with requirements of AODA) must be captured in the PQR Implementation plan.
62
CONDUCTING FOCUS GROUPSThe memos and templates mentioned in this section are found in Appendix C.
Focus groups are held to gather feedback from the participants about their perception of the program’s effectiveness. Once the Team Leader and the Chair have reviewed the available survey data and program documentation, they determine which elements of the evaluation criteria require additional input from members of the focus groups and prepare the agendas for the specific focus group meetings.
Each program Team Leader plans the focus group meetings and prepares the agendas for their own program meeting, and, excepting the faculty focus group, facilitates the meetings for a Team Leader from another School. The report of the focus group meetings will be written by the facilitator, with an opportunity provided to the program Team Leader and Chair to review the report for accuracy before it is finalized.
Note: Facilitator partners need to work together to set meeting dates and room logistics for shared focus group meetings,
The following pages contain detailed directions to assist in planning and facilitating a focus group meeting and writing a report on the feedback obtained. It is understood that facilitators will dress and act professionally for these meetings. In the event of illness, the Chair acts as a facilitator, so please ensure that each Chair is informed of upcoming focus groups.
For some programs it is not feasible to convene a focus group. In these instances, an alternate means of collecting the information may be considered, for example, through an online survey. The survey would be composed by the program Team Leader, but conducted, collated and written into the report by the partner facilitator.
Memos from the Chair and Team Leader At the outset of the PQR, the Chair sends a notice to the faculty to introduce the Team Leader, and to ask each staff member to fill out a faculty credentials and experience sheet.
Once the Chair has sent the PQR introductory notice to the faculty, the Team Leader contacts faculty members to invite them to the faculty focus group meeting, and to prepare by filling out the program self audit form, and to bring the most recent review of each program course that they teach.
The Chair will also send out an invitation to the External Stakeholders (normally the advisory committee plus some recent graduates and relevant employers) to attend a focus group meeting for PQR. The invitation to the student focus group meeting can be sent by the Team Leader.
Faculty Education, Experience, and Professional Development SurveyAhead of the faculty focus group, each member of faculty fills and sends to the Chair a credentials and experience sheet. The Chair then summarizes this and sends it to the
63
Team Leader, who uses it to prepare for the upcoming focus groups, and to respond to the elements of the evaluation criteria.
Program Self-AuditAhead of the faculty focus group, each member of faculty assesses the program against the quality criteria, and provides their assessment to the Team Leader to summarize for inclusion into Appendix B of the final PQR report, and use to prepare for the upcoming focus groups.
Course OutlinesEach course that contributes to the program needs to be self-assessed ahead of the faculty focus group meeting8. A tool for course writers (the “Course Outlines Check Up” tool, available in Appendix C) can be used to review each course, within which there is an option for peer review. The review of course outlines forms part of curriculum mapping.
Setting the Focus Group Agendas An example agenda for each focus group is provided in Appendix C. Agendas are adjusted based on the findings from the document review and data analysis performed ahead of the focus group meetings. Discussion points are defined or expanded to address gaps or ambiguities. Where there is adequate evidence that an element of a criterion is met, it does not need to be brought forward to the focus group.
Tips for Planning and Facilitating Focus Groups Preparing and Planning for the Meeting
Objectives - identify the major objectives of the meeting Questions - develop lead questions
o Use five or six questions from the list of possible items for discussion, provided for each focus group
o Focus on areas that assist in projecting the future of the programo Identify areas of improvement within the program
Date and Time - set date and time (two hour meeting plus social/networking time) Location - reserve a conference room and order light refreshments9
Invitees - generate guest list and send invitations (four weeks prior to the meeting) Agenda - set the agenda based on the information gathered to that point (begin with an
simple topic) Materials - determine the meeting recorder and prepare/order necessary materials (flip
chart paper or laptop/projector, markers, etc.)
Facilitating the Meeting Thank you – thank guests for their participation Introductions – establish positive rapport and comfortable environment by taking the time
for personal introductions
8 Course outline review happens annually as part of Annual Curriculum Review9 Expenditure must follow guidelines as per College Policy AD12 “Travel, Meals and Hospitality”
64
Overview – review the meeting agenda and purpose and set the ground rules Questions – ask each question and record the responses. Listen attentively and request
clarification as necessary then reflect back to ensure the notation is accurate. Participation – encourage participation by all attendees Timing – be mindful of the meeting duration, allowing time for review and
recommendations. Closing – thanks guests once again for their valuable feedback and close the meeting SMILE – be positive and face the group as much as possible
Post Meeting Tasks Organize – gather notes, number pages, and comment on any important observations Report - write the report within two weeks Ensure that focus group participants are provided feedback on the meeting.
65
Faculty Focus Group The Team Leader will convene a meeting of as many faculty members who teach in the program as possible, including part-time faculty and professors of service courses. Part-time faculty members will be reimbursed for attending this meeting as per guidelines earlier in this document.
Faculty Focus Group sessions should be standardized as much as possible for all programs. The agenda for each session should closely resemble that in Appendix C.
Ahead of the faculty focus group meeting, all faculty must provide a completed Program Self-Audit, which the Team Leader will
have summarized in order to set discussion items for the agenda. course teachers are to provide a review of their course, using the course outline
summary sheet or equivalents
The main agenda items for the faculty focus group meeting are reviewing the mapping of Vocational Learning Outcomes to courses review of Course Outlines against the 5 course specific elements of the quality criteria
(as per section Curriculum Mapping) review the interaction of course material in support of achievement of learning outcomes discussing issues (arising from Program Self-Audit)
A second faculty meeting may be planned after all the focus groups have been completed, depending on the number of faculty on the team. At this time, the Team Leader and the Chair can share with faculty recommendations made at the external stakeholder focus group or the student focus group meeting. The input from this meeting, should it be held, would be documented in a separate report and included in Appendix B, and conclusions and final recommendations formulated for the final report
Facilitating the focus groupThe Team Leader opens the meeting by welcoming everyone and providing an opportunity for introductions as necessary. After a brief overview of the Program Quality Review process, the Team Leader then reviews the Agenda, and explains the process and ground rules.
The ground rules for the discussion are:
Ask that only one person speak at a time. Members are asked to be as concise and focused as possible. All comments/perceptions should be freely stated and will be recorded by the Team
Leader/recorder All contributions are regarded as valid and no consensus is required with respect to the
perceptions and feedback stated and recorded. The Team Leader will attempt to achieve consensus only with respect to final
recommendations to be made to the program.
All elements of the discussion should be recorded during the meeting. The Team Leader may enlist the help of an external note taker, if one is available in the department. The meeting notes
66
will serve as a permanent record of the discussion and will provide the basis for the focus group report.
Preparing the focus group reportThe Team Leader will prepare the report for the meeting which s/he has facilitated, using the notes from the meeting as the basis for the report. The Chair and the Team Leader must meet to review the report before it is finalized.
The faculty focus group report includes
the agenda, date, and time of the meeting the names of the faculty members who attended the meeting a summary of the meeting – the highlights, key findings, and recommendations.
The final report does not contain the detailed discussion notes taken by the Team Leader/note taker.
67
External Stakeholder Focus Group Planning the External Stakeholder Focus Group The program Team Leader and the Chair meet to determine who to invite to the external stakeholder focus group. Participants include relevant Advisory Committee Members, additional employers, recent graduates and other guests as determined by the chair and Team Leader. Faculty may be invited to the meeting. The Chair sends the invitation. A draft invitation is included in Appendix C.
The program Team Leader prepares the agenda, including discussion questions, for the external stakeholder focus group meeting, however Team Leaders facilitate the external focus group for a program other than their own. Conduct of the external stakeholder focus group meetings should be standardized as much as possible for all programs. The agenda for each meeting will closely resemble the sample included in Appendix C.
Note that the focus group leader will prepare the report for the meeting which s/he has facilitated, using the meeting notes as the basis for the report.
After the focus group meeting has been held, a letter is sent to each participant by the Chair to thank participants. This letter can include
a brief summary of the recommendations a note that these recommendations may be included in the final implementation plan how the department will inform the participants of the availability of the focus group
report.
Time Frame for the MeetingThe working part of the meeting should be about two hours in length. Refreshments should be available at the beginning of the meeting to minimize interruptions once the meeting commences.
Leadership of the MeetingThe Chair of the department states the purpose of the focus group meeting, introduces the focus group leader, and invites the participants to introduce themselves. The focus group facilitator then reviews the agenda, explains the PQR process and the ground rules for the focus group meeting. The Chair of the department may then leave the meeting.
Ground Rules for the Meeting Ask that only one person speak at a time. Participants are asked to be as concise and focused as possible. All comments/perceptions should be freely stated and are recorded by the facilitator /
recorder. All contributions are regarded as valid and no consensus is required with respect to the
perceptions and feedback stated and recorded. The facilitator will attempt to achieve consensus only with respect to final
recommendations to be made to the program.
68
If consensus cannot be reached with respect to a recommendation, this should be noted. The final decision rests with the Chair.
All elements of the discussion should be recorded during the meeting. The focus group facilitator may enlist the help of an external recorder, if one is available in the department. The meeting notes will serve as a permanent record of the discussion and will provide the basis for the focus group report. Note that the actual meeting notes are not included in the report.
Writing the external stakeholder focus group reportThe focus group leader will prepare the report for the meeting which s/he has facilitated, using the discussion notes from the meeting as the basis for the report. The final report will include
the agenda, date, and time of the meeting. the names and capacity (e.g., the company they represent) of the attendees at the
external stakeholder focus group a summary of the meeting - highlights, key findings, and recommendations.
The intent is to give the reader a clear understanding of the input and viewpoints of the attendees of the meeting, without the heavy overhead of verbatim minutes.
A draft of the external stakeholder focus group report should be verified by the program Team Leader and the Chair before it is finalized. A copy of the meeting notes should be available for consultation during this review.
The draft report is to be provided by the focus group facilitator to the program Team Leader within two weeks of the meeting and copied to the PQAA. If the program Team Leader wishes the final report earlier, a date can be negotiated.
Possible Items for DiscussionBelow are elements of the quality criteria that may be relevant for discussion at the external stakeholder focus group meeting. Some of these elements may not need to be addressed, as earlier review of the survey data and program documents may have concluded that no further information is required.
Criterion 1Students are obtaining employment in their field.
What is the perspective of the group with respect to future employment opportunities for the next 5-10 years?
Graduates are successful in obtaining external licenses or credentials where relevant. Do the graduates understand the legal responsibilities of their job? Are they aware of
the accreditations required?’
Employers are satisfied with graduate performance.
69
Does the group feel that the graduates have adequate practical experience? Do the graduates understand the reality of the workforce? Are they able to relate the skills they have learned to the workforce
Criterion 2 Processes for granting of internal and external transfer of academic credits are clearly defined and available to students.
Does the group feel that current employees have opportunities to upgrade/receive credit for experience/have access to the courses needed to obtain a college credential?
Has anyone at the meeting used the advanced standing, external credit or Prior Learning Assessment and Recognition process?
Do people know where to look for transfer credit/PLAR infomation? Do they have suggestions about where they would expect to find it?
Students have adequate information to allow them to make informed choices about: selecting the correct program to meet their career aspirations; the financial commitment needed; the workload commitment needed; and the study options available to them.
Do employers have suggestions regarding the employment opportunities (or other program narrative information) as outlined in the program monograph?
Students know what is needed to ensure they will be able to demonstrate program outcomes and complete the program.
Do employers have any suggestions or changes to current job requirements like police records checks, special driver’s licences and/or immunizations?
Criterion 3Program Learning Outcomes are appropriate for the program’s intended purpose, and the credential offered upon completion.
Does the group feel that the current vocational learning outcomes are appropriate for the program and credential?
Are changes in the field anticipated such that the vocational learning outcomes should be changed in the next 5-10 years?
Does the group have any feedback on the relevancy of any of the provincial standards? Are graduates you receive ready to function at the entry level expected?
Criterion 4Learners are provided the skills necessary to be successful with the learning strategies selected.
Are there specific skills you require graduates to demonstrate? (Examples may be related to computer competencies, specific communications skills)
Criterion 5
70
The duration and structure of the program are consistent with the program learning outcomes and the credential offered.
Does the group feel that the program curriculum needs to change in the next 5-10 years in anticipation of a change in the knowledge base needed to work in the field?
Does the group have any feedback on the relevancy of any of these standards?
The curriculum conforms to the College Policy for the number of English Courses. What English skills would employers look for in our graduates? Any suggestions for
applications in the workplace that could be adapted to learning activities in the classroom?
Criterion 6Labs, clinical facilities, and placement facilities are complementary to and integrated into the program to allow the learner to demonstrate the learning outcomes.
Does the group have any recommendations about the labs, equipment, and placement facilities?
The students are provided with the information they need to know, to function safely in both the College and workplace learning environments.
Is there any new information the College should know about, for example, WHMIS changes, new safety regulations?
71
Student Focus Group Planning the Student Focus Group The program Team Leader invites students to participate in the focus group meeting. It is recommended that 20-30 students be invited, depending on the size of the program. There should be fairly equal representation from all years and all sections/groups. Students should respond to the invitation, allowing time to find an alternate if the student cannot attend. Students can be selected randomly from a class list – e.g., every fourth or fifth student, depending on the size of the class. Faculty is not invited to the meeting. A sample invitation is included in Appendix C.
The program Team Leader prepares the agenda, including discussion questions, for the student focus group meeting; however Team Leaders facilitate the student focus group for a program other than their own. Student focus group meetings should be standardized as much as possible for all programs. The agenda for each meeting will closely resemble the sample included in Appendix C.
Note that the focus group leader will prepare the report for the meeting which s/he has facilitated, using the meeting notes as the basis for the report. Note that the actual meeting notes are not included in the report.
After the focus group meeting has been held, a letter is sent to each participant by the Chair to thank participants. This letter can include
a brief summary of the recommendations a note that these recommendations may be included in the final implementation plan how the department will inform the participant of the availability of the focus group report.
Time Frame for the MeetingThe working part of the meeting should be about two hours in length. Refreshments should be available at the beginning of the meeting to minimize interruptions once the meeting commences.
Leadership of the MeetingThe Chair of the department states the purpose of the focus group meeting, introduces the focus group leader, and invites the participants to introduce themselves. The Chair then leaves, allowing uninhibited discussion to take place. Thereafter, the focus group leader reviews the agenda, and explains the PQR process and the ground rules for the focus group meeting.
Ground Rules for the Meeting Ask that only one person speak at a time. Participants are asked to be as concise and focused as possible. All comments/perceptions should be freely stated and are recorded by the
facilitator/recorder. All contributions are regarded as valid and no consensus is required with respect to the
perceptions and feedback stated and recorded.
72
The focus group leader will attempt to achieve consensus only with respect to final recommendations to be made to the program.
This is an opportunity to discuss global issues regarding the program. Any students who have individual issues regarding faculty are invited to make an appointment to discuss them with the Chair.
All elements of the discussion should be recorded during the meeting. The focus group facilitator may enlist the help of an external recorder, if one is available in the department. The meeting notes will serve as a permanent record of the discussion and will provide the basis for the focus group report. Note that the actual meeting notes are not included in the report.
Writing the Student Focus Group ReportThe focus group leader will prepare the report for the meeting which s/he has facilitated, using the transcriptions of the discussion notes as the basis for the report. The final report will include
the agenda, date and time of the meeting. the number of students, and their current academic level (it is not necessary to list the
actual names or the participants, level is important as it provides context for the responses)10.
a summary of the meeting - highlights, key findings, and recommendations.
The intent is to give the reader a clear understanding of the input and viewpoints of the attendees of the meeting, without the heavy overhead of verbatim minutes.
A draft of the student focus group report should be verified by the program Team Leader and the Chair before it is finalized. A copy of the meeting notes should be available for consultation during this review.
The draft report is to be provided by the focus group facilitator to the program Team Leader within two weeks of the meeting and copied to the PQAA. If the program Team Leader wishes the final report earlier, a date can be negotiated.
Possible Items for DiscussionBelow are elements of the quality criteria that may be relevant for discussion at the student focus group meeting. Some of these elements may not need to be addressed, as earlier review of the survey data and program documents may have indicated that no further information is required.
Criterion 1Students indicate that they are satisfied with the program.
What parts of the program provide you with the most satisfaction? What parts of the program provide the least satisfaction?
Issues raised at Program Councils are addressed in a timely fashion and feedback is provided to the council.
10 Note that the facilitator will provide the Chair the list of participants, in order to send the thank you letter indicated earlier
73
Are issues from Program Council meetings addressed and progress communicated back to students? ( Aside: Note that if an issue cannot be resolved, this must be communicated back to the Program Council.)
Learners progress through the program, achieve program outcomes, and graduate in a timely fashion.
Do you feel that the learning outcomes are achievable in the designated time frame for the program? Have you been successful in all your courses so far? If not, why not?
Criterion 2The qualifications and prerequisites required of the applicant are published and are appropriate to allow the student to be successful without limiting access to the program.
Was the program description provided useful? And the Success Factors? Did you know what you needed to know to get into the program? Did you have enough of a Mathematics foundation to succeed in level 01? Did you know anyone who wanted to get into the same program as you but didn’t – and
why?
Students have adequate information to allow them to make informed choices about: selecting the correct program to meet their career aspirations; the financial commitment needed; the workload commitment needed; and the study options available to them.
Does the course content match what you expected to learn? Do you expect these studies will lead to the type of career you are seeking? Were you prepared for the financial and workload commitments needed?
Students know how to get internal and external transfer of academic credits and recognition for prior learning.
Did you know how to get any exemptions for courses taken at another postsecondary institution or here at Algonquin College in another program?
Were you eligible for exemptions or PLAR?
Students know what is needed to ensure they will be able to demonstrate program outcomes and complete the program.
Does your program have special requirements you must have before you can be able to demonstrate learning outcomes/graduate?
Was the information you needed readily available?
Students know how they will be evaluated. Is the published information adequately available and meaningful to ensure you know
how you will be evaluated?
Students indicate that assessment methods relate to the learning requirements. Are the assessment methods related to the learning requirements? How do you know?
74
Criterion 3Program Learning Outcomes are appropriate for the program’s intended purpose, and the credential offered upon completion.
Did you/ do you know the expected outcomes for your program? Do you know where to locate them? If so, do you think they are appropriate for the program?
Criterion 4The program provides experiential learning opportunities that allow the learners to perform, with support, as a practitioner in their field.
Do you feel you are provided with adequate opportunities to perform as a practitioner in your field?
There is a range of instruction methods consistent with a variety of learning styles and learner needs and abilities.
Do the instruction methods support your learning style and your needs and abilities? Do you have any suggestions for improving this aspect of the program delivery?
Learning methods are published and are matched to the learning outcomes. Did you know what learning methods to expect and do you feel they match the learning
outcomes?
Learners are provided the skills necessary to be successful with the learning strategies selected.
Do you receive adequate preparation to be successful with the learning strategies being used, for example, online learning or group presentations?
Evaluation criteria are published and students are aware of how and when they are going to be evaluated.
Were you aware of when and how you were going to be assessed?
There is a match between course learning requirements and evaluation methods. Do you feel that the evaluation methods allow you to demonstrate the outcomes?
Students indicate that feedback is timely and allows them to build on their learning. Is feedback timely? Is it given is a way to aid future learning?
Students perceive evaluation to be fair. Are your evaluations fair? Are there any improvements/adjustments that could be
considered?
Evaluation methods allow students to demonstrate the course learning outcomes. Do the ways the professors evaluate you allow you to demonstrate the course learning
requirements? Are there other ways that may better allow you to demonstrate your learning?
75
Student workload and assessment is balanced across the term at both the course and program level.
Do you find the workload to be balanced across the term for all your courses?
There is a range of evaluation methods used consistent with a variety of learning styles. Do the evaluation methods used match your learning style? Can you offer suggestions about other evaluation methods that would suit you better?
Criterion 5Prerequisites do not unnecessarily hinder progress in the program.
Are there any courses that seem to obstruct progression in the program? If so, do you have any suggestions about how this could be improved?
Courses are organized and sequenced to support student learning, are allocated appropriate hours of study, and are consistent with published prerequisites.
Does the sequencing of the courses make sense? Do you have the foundational knowledge needed as you progress in the program?
There is congruency between the course learning requirements and the program learning outcomes.
Do you think that each of the courses contribute to the overall learning outcomes?
There is a match between the course learning requirements, course learning activities and learning resources.
Do the learning activities and learning resources support your success in demonstrating the course learning requirements?
Curriculum design maximizes flexibility of student learning. Do you feel that you have adequate flexibility to accommodate your learning? Do you have any suggestions regarding how the program delivery could be more flexible
to support your learning? (What do you consider to be needed for flexibility – courses available at different times to allow them flexible schedules; different learning offerings, e.g., online as well as in class; multiple resources available, a variety of learning activities; opportunities for PLAR challenges; laddering of curriculum?)
Criterion 6Students consider faculty to be available.
In your experience, has faculty been available?
Students consider faculty to be adequately prepared. In your experience, is faculty prepared for class?
Students consider faculty to promote a positive attitude to learning. Overall, has this been your experience with your program, and at the College?
Students are assigned an academic advisor.
76
Do you know who your academic advisor is?
Academic Advisors contact their students early in the term with an invitation to meet and to ensure that students know who their advisor is.
Have you met with your academic advisor this term or connected in another way? Do you have any suggestions about how your academic advisor can better support your learning?
Labs, clinical facilities and placement facilities are complementary to and integrated into the program and allow the learner to demonstrate the learning outcomes.
Do the labs, clinical, and placement facilities allow you to demonstrate the learning outcomes?
Students indicate that there are adequate and accessible learning resource materials including: textbooks in the bookstore; online materials; print resources; equipment; student support services; and technological infrastructure to allow them to be successful.
Are the learning resource materials accessible? Should anything be improved? Removed?
Do you have adequate access to student support services?
The learning environment is safe. Do you feel safe in the College? Is the classroom environment conducive to asking questions? Do you feel safe in all learning environments? Do you have adequate supervision when demonstrating new skills in a placement
environment? If there is an area where students do not feel safe, do they have suggestions about how
this could be resolved?
The students are provided with the information they need to know how to function safely both in the College and workplace learning environments.
Do you feel that you are well orientated to practice lab and workplace learning environments?
Are you offered specialized safety training? Can you identify any situations where you felt you were not able to practice your skills in
a safe manner?
77
COMPLETING THE FINAL REPORT – AN OVERVIEWAs the Program Quality review is being conducted, the Team Leader should
write the introduction section of the final report review the available documentation and data respond iteratively to the elements of the evaluation criteria (i.e., after document and
data review and after each focus group meeting was held) complete curriculum mapping identify program Strengths, Weaknesses, Opportunities, and Threats start to identify recommendations.
To close the report, the Team Leader needs to complete responses to the elements of the evaluation criteria write the conclusions and prepare a draft of the recommendations and implementation
plan review the report with the Chair with the Chair, develop the Implementation Plan11
with the Chair, write the Executive Summary send the draft report to the PQAA for review make updates as required, and resubmit to the PQAA for approval and archiving.
The draft report should be submitted to the PQAA and the Program Chair as a .docx file.
All items, such as the Program Monograph and COMMS curriculum maps, should be included electronically into this final report. If administrative assistance is required in compiling the final report, please contact the PQAA.
On receipt of the draft report, the PQAA and the Program Chair will review and advise the Team Leader of any updates required. Once the report has been finalized, the formal review and sign-off procedure is followed, where the report is reviewed by the PQAA, the Chair and finally the Dean. Once approved by the Dean, the timing for the annual follow up report is provided. A formal sign-off sheet is used, and is provided in Appendix C. The master copy of the PQR is stored electronically by Academic Development. A hard-copy can be provided on request.
11 The Chair is responsible for actioning the recommendations
78
ANNUAL FOLLOW UPEach spring, the PQAA sends out a notice requesting follow up from each Chair regarding current PQR actions. Chairs summarize the progress made, and update the program implementation plan, adjusting actions based on changes such as College expectations, program budget and availability of human resources.
In addition the Chair includes PQR recommendations in his/her annual performance contract. the PQAA presents a summary report which includes the highlights and
recommendations for each program involved in PQR for that cycle. This report is submitted to the Vice President Academic.
The PQAA presents a report which summarizes the progress on open action items from all previous PQRs. This report is also submitted to the Vice President Academic.
79
GLOSSARY AND LIST OF ACRONYMS
AODA Accessibility for Ontarians with Disabilities ActCAAT Colleges of Applied Arts and TechnologyCOMMS Course Outline Mapping and Management SystemCOL Centre for Organizational LearningGeneSIS The College Student Information SystemKPI Key Performance IndicatorsMTCU Ministry of Training, Colleges and UniversityOCAS Ontario College Application ServiceOCQAS Ontario College Quality Assurance ServicePLAR Prior Learning Assessment and RecognitionPQAA Program Quality Assurance AdministratorPQR Program Quality ReviewTL Team Leader
80
APPENDIX A – THE FINAL REPORT
The template for the final report is found online on the Academic Development – Program Quality Assurance web pages. Help text for the main sections of the report – the executive summary, the introduction, the conclusions, and the implementation plan – are provided on the following pages of this Appendix and in the final report template for ease of reference.
81
The Executive Summary is a one-page, single-spaced review of the highlights of the Program Quality Review. The focus of this summary is to highlight the overall status of the program, its special strengths and achievements, and any challenges the program is facing and the strategies recommended to address these challenges. Four to five paragraphs can cover the information needed:
the purpose of the report, background - a brief history of the program and delivery information, key findings - strengths and highlights, challenges (SWOT analysis), a brief summary of the conclusions and recommendations.
Executive Summary
Example Executive Summary
(This is fictitious but uses some real situations from recent reports.)
This report is a summary of the findings of the Program Quality Review Process for the 2008-200 academic year for the Housemaking program offered at Algonquin College.
The Housemaking Program was first offered at Algonquin College in 1981 as a one-year certificate. In 1992, the program was converted to a two year diploma program to meet the demands for more highly skilled and knowledgeable workers, and in 1997, the delivery mode was compressed into a non-semester diploma program format. The program prepares graduates to enter the housebuilding industry in the Ottawa and surrounding area. The program is also offered at the Perth Campus.
There has been a consistently high demand for the program and its graduates except for a minor slump in the housing depression in the early 90s. There are a number of local builders who have supported the program over the last 30 years and our graduate employment rate is consistently above 90%. The KPI and Course Assessment Surveys note high satisfaction rates with respect to the dedication and expertise of the program faculty. This commitment from both full-time and part-time faculty members has ensured the program has maintained a high standard of delivery.
Two major concerns will be addressed in the implementation plan. The first is the need for the students to be ready to enter the job market in the spring rather than the fall. The faculty will review the delivery mode over the next year and determine ways to deliver the curriculum to meet the demands of the students to move quickly through the program yet be available at the start of the construction season. The other most significant concern is the expectation to lose the coordinator and the other most senior full-time professor in the next year as both of these faculty members are eligible to retire. Succession planning will be a high priority to ensure we have excellent teaching and curriculum development skills in the department along with our current practical expertise. A review of course outlines indicates that some outlines are not as complete as would be desired. A plan is in place to ensure this will be resolved in the May-June 2014 planning session.
This program has enjoyed a high demand, and high employment rates. Industry projections in the Ottawa area suggest that this demand will continue. Advice from our industry partners on an annual basis will assist us in keeping this program strong over the next few years.
82
83
The introduction ‘sets the tone’ for the entire PQR report. It introduces the program as it stands at the start of the PQR review period, and includes
the purpose of the program – why does it exist, who does it serve in terms of the student body and the future employers,
the programs’ evolution from its first inception to today, relevant external accreditation organizations linked to the program, if applicable, number of faculty, both full-time and part-time, the nature of the student body, special contributions that the program offers in the community, variations in delivery like co-op options, campus options, Winter intakes, multiple intakes
etc., a discussion of the use of classroom, labs, computer labs, placement opportunities, field
trips, etc.
If the program has previously gone through PQR, re-use material as appropriate within the introduction, and note the key recommendations and progress against them as applicable. If PQR actions remain open, they should be considered for current applicability, and carried forward as appropriate.
Introduction
This section does not include any information accumulated during the current PQR.
84
Findings
EVALUATION OF THE PROGRAM AGAINST THE CRITERIA
Program Quality Review Criteria
85
The findings section is where each of the elements of the evaluation criteria are responded to by Yes/No or occasionally N/A.
Each element of the criteria requires a justification, i.e. explain why the response is Yes, or No or N/A.
Previous PQR reports within the department can be used for reference when writing. Also, the PQAA can provide guidance on how to respond to each of the elements of the criteria.
To avoid repetition, the evaluation criteria and their elements are not listed here.
Discuss the review of the program data and the identification of agenda items for the focus groups.
A brief paragraph can be dedicated to highlights of each of the three focus groups.
The last half or third of this section speaks to the evaluation of the various recommendations from each of the focus groups and the decisions taken for the formulation of the implementation plan. For example, students may suggest that the computer applications course is irrelevant and they do not like the hybrid courses. The external stakeholders provide feedback that the graduates are not computer literate. The final recommendation would probably indicate the need to make the computer applications course relevant to the industry by adding inventory control and invoicing software exercises and communication to the students that computer skills are desirable in the workplace. To address the students’ dislike of hybrid courses, a recommendation could identify the development of better orientation to Blackboard for both faculty and students. (Reminder: it is important to provide feedback to focus group participants regarding resolutions of issues raised at the focus group.) Program strengths, weaknesses, opportunities and threat can be noted.
Some recommendations may arise that will not be feasible to implement and therefore will not be actioned in the final recommendations. These need to be discussed here. For example, one group suggested that students have a university level course as a prerequisite for admission to the program, as the students need a strong science background. Given that this is not allowable by Ministry standards, and is not feasible to implement, the recommendation might be modified to ensure that highest Grade 12 College course is listed as a required course, or if this is already the case, the recommendation may not warrant any action. If the students have the high school subject at the grade twelve level, there may need to be a college level course in level 01 to help the students with the material. This should be explained here, i.e., recommendations from the Focus Groups should not just disappear without explanation and feedback.
Some recommendations will be desirable to implement but may have budget constraints. These recommendations can be maintained but identified as having fiscal constraints. Sometimes recommendations may be carried forward until such time that the funds are available or it is determined that a less costly solution is needed and implemented.
Conclusions
This part of the report addresses the analysis of the data, documents, and input from the focus groups.
Note: Recommendations are identified under the relevant pillar of the College’s Strategic Plan.
86
Recommendations
Recommendations resulting from the PQR are listed, even if they are not able to be implemented, and the pillar(s) of the Strategic Plan under which they fall is noted.
The four pillars of the Strategic Plan and their associated goals are: Applied Education and Training
o Goal 1: Deliver an exemplary applied education and training experience.o Goal 2: Create a unique suite of programs, products and services geared to meet
the needs and expectations of our clients and students.o Goal 3: Leverage technology to enhance the educational experience.o Goal 4: Provide opportunities for every full-time student to have a work
experience outside of the classroom. Student and Client Success
o Goal 5: Deliver exceptional service to our diverse student and client populations.o Goal 6: Leverage technology to automate and modernize our business
processes, fostering an environment of continuous improvement. Empowered People
o Goal 7: Attract, develop and retain employees who have the knowledge and skills to be fully contributing members of the College.
o Goal 8: Create and foster an environment in which the College’s model of leadership competencies and behaviours is supported.
Financial Sustainabilityo Goal 9: Align our funded operational expenditures with provincial funding.o Goal 10: Expand non-funded opportunities to increase revenue.o Goal 11: Leverage strategic business partnerships to meet the capital needs of
the College.o Goal 12: Create the technological foundation to align with the digital direction.
87
Implementation Plan
Actions arising from the stated recommendations are captured, and must be resourced and measureable. Any recommendations that are not able to be implemented at this time should be noted.
The PQR implementation plan is carried forward as a ‘living document’ to be used as part of the ongoing operations of the program, and is updated at least annually.
Recommendation 1:
Strategic Plan: <enter one or more references here…>
Action 1.1Assigned to:Timing
Action 1.2Assigned to:Timing
Recommendation 2:
Strategic Plan: <enter one or more references here…>
Action 2.1Assigned to:Timing
Action 2.2Assigned to:Timing
Recommendation 3:
Strategic Plan: <enter one or more references here…>
Action 3.1Assigned to:Timing
Action 3.2Assigned to:Timing
<…create more tables as applicable…>
88
APPENDIX B – THE CREDENTIALS FRAMEWORKExtracted from the “Minister’s Binding Policy Directive “Framework for Programs of Instruction”.
89
91
92
93
94
95
APPENDIX C – TEMPLATESThe following templates and review aids are available to download from the PQR Blackboard course. Contact the PQAA for access.
Introductory Memo from Chair Introductory Memo from Team Leader Faculty Education, Experience and PD Survey Faculty Education, Experience and PD Summary Program Self-Audit Course Outline Check Up Review Tool Course Outline Review Summary Sheet Curriculum Mapping – Terms and Concepts Analysis of VLO maps (feedback form) Analysis of EES maps (feedback form) General Education Feedback Form Sample Faculty Focus Group Agenda Sample Invitation to External Stakeholders Sample External Stakeholder Focus Group Agenda Sample Student Focus Group Invite Sample Student Focus Group Agenda Student Focus Group Purpose and Objectives Sample Thank You Letter to Focus Group participants PQR Approval (Submission Sign-off) form
The blank PQR final report template is also available to download at the above URL.
Introductory Email from the Chair
To: All faculty members of the program under reviewFrom: ChairCC: Team LeaderSubject: Program Quality Review - <program title here>
I am writing to inform you that <program title here> program is undertaking the Program Quality Review (PQR) process for <academic year here>. Programs go through PQR every five years as part of the quality assurance process at Algonquin College, to fulfill the mandate of the Ministry of Training, Colleges and Universities (MTCU).
To lead the review, <Team Leader name here> has been assigned the task of assessing the program against MTCU and college quality criteria, and will be in contact with you about PQR activities. Your input on course outlines, program outcomes, and curriculum mapping is an important part of the review process. Your input will contribute to strategies and recommendations that will enhance program quality, which will ultimately enhance the student experience.
Part of the PQR process is to review program staffing and participation in professional development. To that end, please follow the link provided to complete a brief survey on faculty education, experience, and professional development. The survey will be open for one week - please complete it by <enter survey closing date>. The information you provide is confidential and remains with the Chair. A summary of all responses is provided to the Team Leader to be included in the final PQR report. If anyone has any concerns about how this information will be used, please feel free to discuss them with me.
<insert link to survey here>
I encourage your full participation in PQR activities. Thank you for your support and cooperation.
97
Introductory Email from the Team Leader
To: ALL Faculty members of programFrom: Team Leader for Program Quality ReviewCC: Chair, PQRASubject: Invitation to faculty focus group
Dear <program name> faculty,
As part of Quality Assurance at Algonquin College, all programs participate in a cyclical Program Quality Review process. This year the <program name> program is undergoing review. The College is seeking your participation in the faculty focus group session. Below is a link to a Doodle poll where you can enter your availability.
http://doodle.com/..........................
The Program Quality Review process seeks input from a wide range of perspectives. The data collected will contribute valuable information to ensure that the program remains current and meets the changing needs of our students and the workplace.
I have enclosed a draft agenda for the focus group session. I anticipate that the focus group session will require up to two hours of your time, not including travel, for which part-time faculty are compensated at $30/hr.
To ensure full participation for this special meeting, I ask that you respond to the Doodle poll on or before <add closing date here>.
The format for the focus group will be discussion-based and the session will be facilitated by me. Your perceptions are important to the program and I hope that you will be able to attend.
In preparation for the meeting, your direct feedback on the quality criteria will be informative. Please follow the link below to the program self-audit survey. You may find that some items are outside your knowledge or experience. It’s okay to answer ‘No opinion’ and move onto the next item. Please complete the survey by <add closing date here>.
<include Fluid Survey link here>
Your feedback on the course outline for any courses you teach is also valuable. Though course writers review their outlines annually, Program Quality Review seeks to examine them in more detail. Please fill out the attached form and bring it to the meeting.
During the meeting we will discuss the program website and Vocational Learning Outcomes (VLOs). Take a moment to review the webpage and the program VLOs (attached). A great starting point or review of curriculum basics for colleges in Ontario is this 8 minute video on YouTube https://www.youtube.com/watch?v=awWQhUeAgdc.
Thank you for your ongoing support of our program.
98
Faculty Education, Experience, and Professional Development SurveyThis survey can be deployed online via a survey tool such as Fluid Surveys. The survey is initiated by the Chair of the program who will summarize the data for the final PQR report. This survey is considered confidential information once completed.
Faculty credentials and experience surveyThe purpose of this survey is to collect information about your educational credentials and professional experience. Information entered into the survey is confidential. It takes between 5 and 10 minutes to complete.
Your full name:
Program name:
Employment status:Part-time / Sessional
Full-time / Partial-load
Summary of education
Please select you highest level of educational attainment.Diploma
Bachelor's Degree
Master's Degree
PhD
Other
Please write the field or area of your highest credential.
e.g. BSc - Physics
Do you have other, relevant credentials? Please list them here.
99
Teaching experience
How many years have you taught in this program...Please enter a number. If you've taught less than a year, enter 0.3 for one semester or 0.6 for two semesters.
...as part-time or sessional faculty?
...as partial-load or full-time faculty?
Please enter your date of hire here:____/__/__ (YYYY/MM/DD)
Work experience
How many years of work experience do you have?Please enter a number.
Do you have other, relevant experience? Please describe here.
Professional development
Describe any professional development activities you have participated in over the last 5 years. Include formal courses, workshops, conferences, industry renewal opportunities, sabbatical leaves, etc.
Describe any reflective practices you have used in the last 5 years.Examples of reflective practice can include peer feedback, changing your lessons based on formal or informal student feedback, keeping a journal or record of teaching methods.
When was your last performance review?
If you have not had a review, skip this question.
____/__/__ (YYYY/MM/DD)
100
Summary of Faculty Members’ Credentials and Experience 2015-2016
Program summary informationProgram name
Total number of students in the program
Total number of faculty teaching in the program
Program faculty education summary
Number of Faculty with PhD or EdD
Number of Faculty with Master’s Degree
Number of Faculty with Bachelor’s Degree
Number of Faculty with Diploma
Number of Faculty other credentials
Teaching and workplace experience
Number of combined years of full-time / partial load teaching experience
Number of combined years of part-time / sessional teaching experience
Number of combined years of relevant workplace experience
Areas of concern regarding Faculty education and/or experience
Areas of strength regarding Faculty education and/or experience
101
Professional development
Number of Faculty members
who have attended professional development within the last two years
participating in reflective practice
Examples of reflective practices being used
Performance reviews
Number of Faculty with a written performance review done
in the last year
more than a year ago but within the last two years
more than two years ago but within the last three years
Number of Faculty with no written performance reviews within the last three years
102
Program Quality Review - Program Self-AuditThe self-audit is intended as a quick reference check for faculty to respond to and evaluate the program against the quality criteria.
Prior to filling this document, it is advised that the Team Leader provide faculty with the Vocational Learning Outcomes (VLOs) and the Essential Employability Skills (EES) outcomes for the program.
This survey can be deployed online via a survey tool such as Fluid Surveys. Each faculty member completes this program self-audit form and submits it to the Team Leader, who summarizes the input. The full survey is not included for brevity; find below a sample.
PQR - Program Self-AuditThe purpose of this self-audit survey is to capture faculty input. The criteria combine provincially mandated elements as well as college requirements. Your comments will help guide discussions that aim to improve program quality. As you go through the quality criteria, you may find that some elements are not relevant to you and your teaching experience. Provide comments for any quality criteria element that you can. Reviewing the Vocational Learning Outcomes (VLOs) and the Essential Employability Skills (EES) outcomes for your program will help you to complete the survey. These will be supplied by your Team Leader, and are also available via COMMS. It will take approximately 20 to 40 minutes to complete the survey. Your participation and contributions are appreciated. – The Algonquin Program Quality Review Team
Your full name
Criterion 1
Regular program quality assessment that involves faculty, students, industry representatives, and others as appropriate for the purpose of continual improvement is in place and happens.
1.1 Students indicate that they are satisfied with the program.
Yes
No
No Opinion
Other ______________________
Comments
103
1.2 Program Council meetings are held regularly in accordance with college policy. Issues raised are captured and feedback provided to students in a timely fashion.
Yes
No
No Opinion
Other ______________________
Comments
1.3 Learners progress through the program, achieve program outcomes and graduate in a timely fashion.
Yes
No
No Opinion
Other ______________________
Comments
1.4 Learners with a wide range of abilities demonstrate the expected learning outcomes.
Yes
No
No Opinion
Other ______________________
Comments
104
Course Outline Check-Up Review Tool for PQR
A course outline documents the essential elements of a course for an academic year. It also acts as a curriculum contract for students. The Course Outline Check-up Tool for PQR helps reviewers to identify strengths and areas for improvement in all parts of a course outline, and is an important part of quality assurance.
Reviewer Enter your full name
Course title
Course number Three letters followed by four numbers and possibly another letter. E.g. ENL1813S, DRA9251
General check-upAbsolutely
Somewhat
Needs Work
The rationale of the course (i.e. the why? what? how?) is clearly expressed in the course description.
☐ ☐ ☐
There is an appropriate number of CLRs (generally 4-6 for a 45-hour course, 6-8 for a 60-hour course).
☐ ☐ ☐
All CLRs complete the phrase "You will have demonstrated the ability to...".
☐ ☐ ☐
The entire course outline is free of spelling, grammar and syntax errors.
☐ ☐ ☐
Reading the course outline, a clear picture of the course comes to mind.
☐ ☐ ☐
Program Quality Review criteriaAbsolutely
Somewhat
Needs Work
There is congruency between the course learning requirements, and the program learning outcomes. (QC 4.4)
☐ ☐ ☐
There is a match between course learning requirements, course learning activities and learning resources. (QC 4.5)
☐ ☐ ☐
Evaluation methods allow students to demonstrate the course learning requirements. (QC 4.8)
☐ ☐ ☐
PLAR opportunities exist and are based on course learning ☐ ☐ ☐
105
requirements. (QC 2.3, 4.15)
Any other comments or observations?Please add any additional comments or observations on the course outline.
Course Outline Review Summary Sheet
PurposeThough course outlines are updated annually, PQR provides an opportunity to look at the outlines in detail, with a particular view to how they support the program outcomes, and whether all essential components of the outline are documented.This is an especially valuable exercise for non-course writers, who may not be aware of how the outline connects to the program and its learning outcomes.
InstructionsDistribute the form, Course Outline Check-up Tool for PQR, to faculty. Have faculty each review a course outline. Even better if it’s a course they don’t usually teach.
Use the completed forms to fill in the table below. List all courses, except for General Education and Service courses. Extend the form as necessary.
The numbers in the table refer to the items in the numbered list below. Answer Y (yes) or N (no) or N/A (not Applicable) for each numbered item.
Include comments from the EES review.
Retain this summary – it can be used as a work plan when outlines are next updated.
Include the section below in the PQR report.
Course outline review summary
1. There is congruency between the course learning requirements, and the program learning outcomes. <Relates to Quality Criteria element #4.4>
2. There is a match between course learning requirements, course learning activities and learning resources. <Relates to Quality Criteria element #4.5>
3. Evaluation methods allow students to demonstrate the course learning requirements. <Relates to Quality Criteria element #4.8>
4. PLAR opportunities exist and are based on course learning requirements. <Relates to Quality Criteria elements #2.3 and #4.15>
106
Course name and number 1 2 3 4
List strengths and recommendations for change/update.If PLAR is N/A, please indicate why.
Example:ZOO1004Intro to Zoology
N Y Y Y The CLRs do not fully match the VLOs.The course description needs a rationale statement.
Curriculum Mapping - Terms and Concepts
Teach Do you provide instruction/learning opportunities in this skill in your course?
Is this skill identified in one or more course learning requirements?
Do you devote a significant amount of time to facilitating student development of the skills and knowledge embedded in the outcome?
Assess Are there assignments, tests or projects which are designed to allow you to evaluate or assess student performance of this outcome or some of its elements?
Do you, in your evaluation of student performance, verify that this particular outcome (or a significant component of it) has been achieved?
Is this outcome reflected in your course outline in the course learning requirements and/or embedded knowledge and skills, and in the evaluation of the course outline?
107
Culminating Performance Culminating performances are tasks or activities designed to assess a learner’s ability to demonstrate one or more learning outcomes in their totality. While they do not necessarily occur at the end of a program of study, they do evaluate whether a learner is able to integrate and apply their learning to demonstrate the performance described in the learning outcome(s) at the exit level.
Is there an opportunity for you to evaluate the outcome in its totality?
Does the evaluation result in a final product or performance which allows you to determine whether the learner has integrated the knowledge and skills identified in the elements of performance?
Can you determine from this performance if the learner has demonstrated the outcome?
Program culminating performances should require learners to demonstrate learning similar to what would be expected of new graduates in as close to a “real world” context as possible.
If a culminating performance is assigned to a course, the professor teaching the course will be asked to evaluate whether or not the students successfully demonstrated the outcome(s) the culminating performance addresses
108
Analysis and Review of VLO Maps for PQR
Curriculum Mapping
Purpose
Provides an overview of the curriculum for the total program Identifies areas of redundancy where content is more than adequately covered Serves as a review tool to ensure that all program standards are aligned in the Program
of Study Helps faculty see the distribution of outcomes throughout the courses in the program
Location
COMMS Generate Reports Course Vocational Learning Outcome by Level and Program Department + Program Run
Mapping terms
Taught: There are learning opportunities for the VLO in the course.
Assessed: There are assignments, tests, or projects aligned with the VLO. To verify, cross-reference with course outlines.
Culminating Performance (CP): The student’s ability to demonstrate the VLO in its entirety, at the level of a graduate. They usually come near the end of program.
109
VLO mapping and program quality review
The form below facilitates the analysis and review of the VLO maps for the purpose of curriculum review. Looking at the VLO curriculum map, answer the questions. Anything mentioned in the COMMENTS or key changes section can be considered for inclusion in the recommendations and action plan that are part of the PQR report.
A. For each VLO, are there instances of Teaching and Assessment? YES NOOTHER
Comments:
B. Do all Assessments have prior or concurrent Teaching? YES NOOTHER
Comments:
C. Does each VLO have two Culminating Performances? YES NOOTHER
Comments:
D. Are Culminating Performances in identified in the appropriate courses? YES NOOTHER
Comments:
E. Is there an appropriate balance of Teaching and Assessment of VLOs in the program?YES NOOTHER
Comments:
Summary
Are the VLOs as mapped generally supported by the information in the course outlines?YES NOOTHER
110
Are changes required as a result of this review? YES NOOTHER
What key changes are required as a result of this review?
Analysis and Review of EES Maps for PQR
Program Name – Program Number
This sheet has been prepared to facilitate the analysis and review of the EES maps used in the curriculum review process. The items mentioned in the COMMENTS section should be reviewed and considered for inclusion in the recommendations and action plan that are part of the PQR report.
SUMMARY
Are the EESs as mapped generally supported by the information in the course outlines? YES NO
Unable to
Answer
What key changes are required as a result of this review?
A. Are there instances of teaching (T), assessment (A), and culminating performances (CP) for each of the EES outcomes? YES NO
Unable to
Answer
Comments:
B. Does the program meet the minimum requirement of two culminating performances (CP) for each of the EES outcomes? YES NO
Unable to
Answer
Comments:
C. Does the program map show any instances of assessment (A) with no previous or associated teaching (T) for any of the EES outcomes? YES NO
Unable to
Answer
Comments:
D. Are culminating performances (CP) found in courses that are likely to support the demonstration of the complete EES outcome at the level expected of a graduate?
YES NOUnable
to Answer
Comments:
E. Does the program map show a reasonable balance of teaching (T) and assessment (A) of EES outcomes in the program? YES NO
Unable to
Answer
111
Comments:
General Recommendations for program consideration:
F. Do the course learning requirements, learning activities and/or assessment strategies in the course outlines reflect the EES outcomes that are attached to the course? (i.e., Is it possible to visualize the performances based on the content of the course outlines?)
YES NOUnable
to Answer
Comments:
G. What trends or general concerns are evident in the course outlines for the program as a whole?
Recommendations for consideration related to specific course outlines:
LEVEL 1
LEVEL 2
<add levels as required>
Program Maps
The program maps used for this review were generated on DD-MM-YYYY, and are appended below.
Essential Employability Skills
1 Communicate clearly, concisely and correctly in the written, spoken and visual form that fulfills the purpose and meets the needs of the audience.
2 Respond to written, spoken or visual messages in a manner that ensures effective communication.
3 Execute mathematical operations accurately.
4 Apply a systematic approach to solve problems.
5 Use a variety of thinking skills to anticipate and solve problems.
6 Locate, select, organize and document information using appropriate technology and information systems.
7 Analyze, evaluate and apply relevant information from a variety of sources.
8 Show respect for diverse opinions, values, belief systems and contributions of others.
112
9 Interact with others in groups or teams in ways that contribute to effective working relationships and the achievement of goals.
10 Manage the use of time and other resources to complete projects.
11 Take responsibility for one's own actions, decisions and consequences.
113
Analysis and Review of General Education Maps and Courses for PQR12
Program Name:Program of Study Version(Academic Year):Program Code:School:Date of Report:Level General Education Courses
CODE + TITLENormative Hours
Choice Theme
Yes No
1
2
3
4
Collectively, do the general education courses in this Program of Study (POS)…
A provide learners with choice appropriate to the credential level?Ontario College Certificate - No choiceOntario College Diploma - One choiceOntario College Advanced Diploma - Two choices
Credential Level: CompliantNot compliant
B ensure learners have successfully completed study in the required number of theme areas?Ontario College Certificate - One themeOntario College Diploma - Two themesOntario College Advanced Diploma - Two themes
Number of themes by
end of POS:
CompliantNot compliant
C incorporate the required number of general education courses?Ontario College Certificate - One 45-hour courseOntario College Diploma - Three 45-hour coursesOntario College Advanced Diploma - Five 45-hour courses
Number of Gen Ed
courses in POS:
CompliantNot compliant
D include only course outlines that have been approved by the General Education Committee?
Please see attached course outline reviews
Yes/No CompliantNot compliant
COMMENTS:
General Education – Course Review Checklist (To be completed by General Education Committee reviewers)
Course Date:
12 This is the standard form for General Education reporting. The Committee may choose to provide an alternate format.
114
Number:
Title: Reviewers: General Education Committee
Theme #: Name of Theme:
Date of Last Review:
Other Programs Course Appears In:
Does this general education course include the following? Yes
No X
Comments
1 45 hours of instruction
2 Does the Course Description provide a student-focused description of the purpose, key topics, and major learning activities of the course?
3 Does it reflect the General Education theme?
4 Do EESs reflect General Education outcomes rather than Vocational Skills?
5 Do CLRs and EKSs begin with active verbs and clearly describe measurable and realistic course learning and objectives?
6 Do CLRs and EKSs reflect societal challenges in community, family, or contemporary life? Is there clear value beyond vocational interests?
7 Does the course explore questions related to issues and values that cover at least two of the following: Historical context Theoretical bases Ethics
8 Are Learning Resources reasonable and appropriate? Have open online resources been considered?
9 Are Learning Activities clearly described, varied, and appropriate? Are there opportunities for collaboration? For self-reflection?
10 Are Evaluation/Earning Credit strategies clear and detailed for students? Do they provide variety and opportunities for collaboration and self-reflection?
11 Do Evaluation/Earning Credit strategies support CLRs, EKSs and EESs?
12 Are opportunities for PLAR clearly outlined?
Comments:
APPROVED
APPROVED WITH RECOMMENDED CHANGES
RESUBMIT WITH CHANGES FOR FURTHER REVIEW
NOT APPROVED
115
Faculty Focus Group – Sample Agenda
Program name:Date:Location:
Introductions Objectives of the session Process/ground rules/groups Tasks/activities
Review program monograph/website and program promotional materials
Review mapping of vocational learning outcomes Review of course outlines and program of study Discussion of issues arising from the program self-audit survey Identification of agenda items for student and external stakeholder
focus groups Identification of recommendations for consideration Thank you and wrap-up
116
Faculty Focus Group – Sample Invitation Email
To: Faculty members of programFrom: Team Leader for Program Quality ReviewCC: Chair, PQRASubject: Invitation to faculty focus group
Dear <program name> faculty, As part of Quality Assurance at Algonquin College, all programs participate in a cyclical Program Quality Review process. This year the <program name> program is undergoing review. The College is seeking your participation in the faculty focus group session. Below is a link to a Doodle poll where you can enter your availability.
http://doodle.com/..........................
The Program Quality Review process seeks input from a wide range of perspectives. The data collected will contribute valuable information to ensure that the program remains current and meets the changing needs of our students and the workplace.
I have enclosed a draft agenda for the focus group session. I anticipate that the focus group session will require up to two hours of your time, not including travel, for which part-time faculty are compensated at $30/hr. To ensure full participation for this special meeting, I ask that you respond to the Doodle poll on or before <add closing date here>. The format for the focus group will be discussion-based and the session will be facilitated by me. Your perceptions are important to the program and I hope that you will be able to attend. In preparation for the meeting, your direct feedback on the quality criteria will be informative. Please follow the link below to the program self-audit survey. You may find that some items are outside your knowledge or experience. It’s okay to answer ‘No opinion’ and move onto the next item. Please complete the survey by <add closing date here>.
<include Fluid Survey link here>
Your feedback on the course outline for any courses you teach is also valuable. Though course writers review their outlines annually, Program Quality Review seeks to examine them in more detail. Please fill out the attached form and bring it to the meeting.
During the meeting we will discuss the program website and Vocational Learning Outcomes (VLOs). Take a moment to review the webpage and the program VLOs (attached). A great starting point or review of curriculum basics for colleges in Ontario is this 8 minute video on YouTube https://www.youtube.com/watch?v=awWQhUeAgdc.
117
Thank you for your ongoing support of our program. Sincerely,
Sample Invitation to External Stakeholders
Dear <invitee name here>,
You are receiving this email as you have been identified as a potential contributor to the Program Quality Review of <program title here>.
As part of the Quality Assurance process at Algonquin College, all programs participate in a five-year, cyclical Program Quality Review. During the review, the College seeks input from as broad a base as possible, and from a wide range of perspectives held by various sectors linked to the program. Your input will contribute valuable information and guidance to ensure that the program remains current and meets the changing needs of our students and the workplace.
I invite you to participate in the External Stakeholder Focus Group session which will be held on <date, time, and location here>. I have enclosed a draft agenda for the focus group session. The format for the Focus Group will be discussion-based, and the session will be facilitated by one of the College’s Program Quality Review Team Leaders. I anticipate that the focus group session will require two hours of your time, not including travel. <You may wish to include directions about parking.> Other participants will include members of the Program Advisory Committee, recent graduates, and employers.
To ensure full participation for this special meeting, I ask that you respond to this invitation on or before <RSVP date here>.
Your perceptions are important to the program and we hope that you will be able to attend. Thank you for your support.
Sincerely,
Chair
118
External Stakeholder Focus Group - Sample Agenda
Program Name:Date:Location:
Introductions Objectives of the session Process/ground rules Program Overview (Chair)
o Events/Highlights/Issueso Environmental Influences
Discussion Itemso Review of Program Monographo Perceived strengths and weaknesses of the program (based on the responses to
quality criteria)o Present and anticipated trends in the vocation/workplaceo Review of vocational learning outcomeso Actions needed to address future trends.
Recommendations arising
119
Student Focus Group – Sample Invitation
Dear <Program Name> Students;
As part of the Quality Assurance process at Algonquin College, all programs participate in a five-year, cyclical Program Quality Review. You are invited to participate in the Student Focus Group session, held on <**date**> from <**start time**> – <**end time**>, in room <**Room number**>.
The Student Focus Group Session is intended to generate feedback from students. Algonquin College would like to know about your experience and perception of the education you are receiving. This will help to identify factors that you feel impact your studies and learning environment. The data collected will contribute valuable information and guidance to ensure that the program remains current and meets the changing needs of our students and the workplace.
The Focus Group session will require two hours of your time. Light refreshments will be provided at the start of the session. The format for the Focus Group will be discussion-based and the session will be facilitated by one of the College’s Program Quality Review Team Leaders.
To ensure full participation for this special meeting, we ask that you respond to this invitation on or before <reply date>, to allow us to properly prepare for the session.
Please notify me at <**e-mail**>, of your desire to attend.
Thank you for your ongoing support of the <**Program Name**> Program.
Sincerely,
Program Quality Review Team Leader
120
Student Focus Group - Sample Agenda
Program Name:Date:Location:Facilitator:
Introductions
Program Quality Review, and Objectives of the Student Focus Group
Ground rules of meeting
Round table – student feedback on program quality (questions to be distributed)
Recommendations arising
Closing
121
Thank You Note to Focus Group Participants - Sample
Dear <participant name here>
Thank you for your recent participation in our focus group meeting, as part of the Program Quality review for the <program title here> program. Your contribution to our process for improving this program is greatly appreciated.
I have included a copy of the focus group report for your information. This will be included in the final PQR report, and used to help form overall program improvement recommendations.
Thank you for your ongoing support of the program.
Sincerely,
Chair
122
PQR Submission Sign Off Form
Program Name(s)/Code(s):
1. The Program Quality Assurance Administrator (PQAA) has reviewed the PQR report, and advises the Team Leader and Chair that the reviewed PQR Report is considered final.
______________________ _______________________ ______________________PQAA (Print Name) PQAA Signature Date
2. The Program Chair has reviewed and approved the contents of the PQR report.
______________________ _______________________ ______________________Chair (Print Name) Chair Signature Date
3. The Program Dean has reviewed and approved the contents of the PQR report.
______________________ _______________________ ______________________Dean (Print Name) Dean Signature Date
PQR Highlight Narrative for ASAC annual report
PQR Follow Up Report Due Date: ______________________________________________
123
124
APPENDIX D – Program Data
Key Performance Indicators (KPIs)
Student Satisfaction Survey This survey is administered by an appointed group of administrators to all students in class during a “Reference Week” identified in October or February. All information gathered is handled confidentially. Students are asked to comment on experiences with their Program, Courses, Teachers and the Facilities/Resources and Services available.
The four KPI Student Satisfaction Survey capstone questions have remained constant for the past 6+ years; the numbering changed in 2014:
KPI Question Numbering 2009-2013 Numbering 2014Overall, your program is giving you the knowledge and skills that will be useful in your future career
Q14S Q13S
The overall quality of the learning experiences in this program
Q26S Q24S
The overall quality of the services in the college
Q45S Q39S
The overall quality of the facilities/resources in the college
Q44S Q49S
Graduate Outcomes/Graduate Satisfaction Survey Approximately six months following graduation an independent consulting firm - Forum Research – contacts graduates by telephone to conduct this system-wide survey. Graduates are asked to comment on their experiences at the college as well as information on employment and salary levels to determine the success of a program, program development, and provide information to potential students. All information gathered is handled confidentially.
At that time, the graduate is asked for permission to speak to their employer (their front line supervisor to be exact). If the graduate grants the Forum Research interviewer permission to contact their employer then the employer will receive a call within a month or less of contacting the graduate. If the graduate declines permission to the Forum Research interviewer to contact their employer then the employer will not receive a call.
Employer Satisfaction Survey Employers are asked to comment on the graduate's workplace abilities (based on knowledge and skills acquired from their college program) as well as their perspective on how the workplace is changing. The employer is also given the opportunity to suggest what new skills are expected from today's college graduates. All information gathered is handled confidentially.
125
126
Algonquin College Performance Measures Application/Registration Ratio: Number of applications vs. number of registered students (both full-time and part-time) on Day 1 in the intake level.
The formula used to calculate this ratio is:
number of applications/number of registered students
The data is taken from the Ontario College Application Services (OCAS) report from GeneSIS (SRR058) for each semester: Spring, Fall, Winter.
Registered students are students with registration status of 0 or 700 (Registered or Withdrawn After Registration).
Intake/Projected Enrolment: Percentage of the actual enrolment (full-time only) over the projected enrolment (full-time only) in the intake level.
The formula used to calculate this percentage is:
actual enrolment/projected enrolment
The data is taken from the Download Registration Summary Data report from GeneSIS (SRR025) on audit days (Spring: June 30, Fall: November 01, Winter: March 01)
KPI Graduation Satisfaction: Percentage of students who rated Very Satisfied or Satisfied for the usefulness of the college education in achieving goals after graduation. Responses are aggregated from Q34 of the Graduate Satisfaction Survey: How would you rate your satisfaction with the usefulness of your college education in achieving your goals after graduation?
KPI Total Employment: Percentage of students who are employed after graduation from the college education. Responses are aggregated from question Q6 from the KPI Graduation Satisfaction survey: During the week of **the survey** were you employed or self-employed; employed or self-employed, but looking for another job; not employed but have accepted a job to start shortly; not employed but looking for a job; not employed, but not looking for a job.
KPI Related Employment Rank: Percentage of students who responded Yes or Yes, partially to related employment after graduation. Responses are aggregated from question Q20 from the KPI Graduation Satisfaction survey, only for students who are currently employed: Was this job related to the program that you graduated from?
KPI Employer Satisfaction: Percentage of employers who answered Very Satisfied or Satisfied for the graduate’s overall college preparation. Responses are aggregated from question Q74 from the KPI Employer Satisfaction survey: In general, how would you rate your satisfaction with this employee’s overall college preparation for the type of work he/she was doing.
127
Algonquin College Quality Measures KPI Student Satisfaction: Percentage of students who rated Very Satisfied or Satisfied for the overall program/college experience. This measure is the average of responses from the four capstone questions of the Student Satisfaction Survey:
o Overall, your program is giving you the knowledge and skills that will be useful in your future career
o The overall quality of the learning experiences in this programo The overall quality of the facilities/resources in the collegeo The overall quality of the services in the college.
Instructional Quality (This measure was used in Program Mix Review, and is not currently provided for PQR purposes. The definition is retained for reference):
Students who rated the overall quality of the course Excellent or Very Good. Aggregated from the capstone question in the Course Assessment surveys conducted during the fiscal year (i.e. Spring, Fall, and Winter terms in that order):
Overall, please rate the quality of this course.
Note that not every course within the program may be assessed each term, therefore this measure shows the summative rate for the courses in the program, extrapolated to the program as a whole.
Quality Average: Average of Student Satisfaction and Instructional Quality.
Additional data:
School Average: Average of all the values explained previously, rolled-up by school.
Benchmark: based on the values for the 8th position college in Ontario from the overall KPI survey.
128
