Effectively Sharing Your Program’s Story · QAR Appendices. Appendix A: Candidate Recruitment,...

Effectively Sharing Your Program’s Story

Writing the AAQEP Quality Assurance Report

February 27, 2020AAQEP Staff

Always Improving Together

Agenda: Sharing Your Program’s Story

I. Standards & Key QuestionsII. Quality Assurance Report ComponentsIII. Presenting Your Case for Quality: The Quality

Assurance Report

2

Inquiring Minds…

What are your QARquestions?

1. Download Socrative Student2. Enter Classroom Code WTIDVIM

3

Standards & Key Questions

AAQEP Standards: Dimensions of Quality

5

Four Key QuestionsStandard 1

At the end of the program, are completers ready to fill their target professional role effectively?

Standard 2 Were completers prepared to work in diverse contexts, have they done so successfully, and are they growing as professionals?

What evidence supports these claims?6

Four Key Questions (cont.)Standard 3

Does the program have the capacity (internally & with partners) to ensure that completers are prepared and succeed professionally?

Standard 4Is the program engaged in strengthening the education system in conjunction with its stakeholders and in keeping with its institutional mission?

What evidence supports these claims?

7

Evidence Requirements and Priorities

8

Multiple measures with reasonable continuity

Quality of evidence must be investigated and shared

Priority is given to direct performance measures

Indirect and downstream measures must be considered

Differentiation of evidence by license, level, location, and mode

Assessments appropriate to the program and context

Quality Assurance Report Components

Quality Assurance Report at a Glance

10

• Presents your case for quality• Explains abbreviated data table(s) w/ evidence to

support each individual aspect• Includes links to complete data tables/appendices• Provides analysis of data (often based on multiple

measures informing the same aspect)

• Present key program practices that correspond to aspects of Standards 3 and 4

• Describe and document key program practices• Provide full data tables (where appropriate)

QAR Narrative

Appendices

QAR Outline Introduction and Overview of the Program and the Study The Case for Standard 1: Candidate/Completer Performance The Case for Standard 2: Completer Competence and Growth The Case for Standard 3: Quality Program Practices The Case for Standard 4: Program Engagement in

Improvement Conclusion: Findings and Commitments Appendices

11

QAR Outline: Narrative IntroductionIntroduction and Overview of the Program and the Study Institution/program origin, context, commitments Program logic (rationale, standards alignment, curricular

coherence) Program overview (Program Specification Table, program

strands/options, enrollment, staffing) Self-study overview (summary of method, who participated, etc.)

Set the stage for your story—your case for quality.Does the introduction help a reviewer understand your program’s context?

12

QAR Outline: The Standards The Case for Standard 1: Candidate/Completer Performance

The Case for Standard 2: Completer Competence and Growth

The Case for Standard 3: Quality Program Practices

The Case for Standard 4: Program Engagement in Improvement

√ Clear presentation of evidence* supporting each aspect.√ Account for the evidence requirements for each standard.√ Articulate your program’s criteria for success.

*Standards 2 & 4: identify planned-for measures13

QAR Outline: ConclusionIn addition to summarizing what your evidence tells you about your program, the conclusion provides:

an explanation of any planned changes to the program based on the evidence presented

identification of new areas of investigation or inquiry, data sources to collect, strategies to explore, benchmarks for tracking progress

documentation of new initiatives or innovations, program commitments

This section will frame your Annual Report.14

QAR AppendicesAppendix A: Candidate Recruitment, Selection, and MonitoringAppendix B: Completer Support and Follow-Up PracticesAppendix C: Program Capacity and Institutional CommitmentAppendix D: Internal Audit of the Quality Control SystemAppendix E: Evidence of Data Quality

Appendices support the narrative by documenting key program practices.

15

A Review of the AppendicesFor each appendix we will cover: Focus Purpose Relationship to the narrative and

standard/aspect

16

Appendix A: Candidate Recruitment, Selection, and Monitoring

17

A program’s admissions, monitoring, and support process based on empirical evidence of the provider’s own choosingFocus

• To describe the cycle of admissions and monitoring throughout the program and

• To evaluate the effectiveness of the systemPurpose

Candidate Success

Are you admitting and supporting candidates in such a way that they can succeed?

In contrast to Appendix D (internal audit), is the system operating successfully (as intended)?

18

QAR (S3) – Appendix A Relationship

19

QARBrief narrative to describe

program’s process regarding:

Recruitment/Admissions

Selection (Program/state criteria)

Monitoring/Support

Appendix AProvides the necessary

documentation/evidence of:

Recruiting efforts, admissions data, demographics

Data on program acceptances

Data on benchmarks, gate keeping, tracking, advising, completion, etc.

Appendix B: Completer Support and Follow-Up Practices

20

Services offered to program completers to support their career entry and ongoing professional growth

Focus

To show how the provider supportscompleters as they enter and grow in the professional workplace

Purpose

QAR (S4) - Appendix B RelationshipQAR

Brief narrative to describe how the program:

Supports completers

Follows up with completers

Appendix BProvides the necessary

documentation/evidence of:

Innovations (what’s new to your program?)

Data on completer support and follow-up

Appendix C: Documentation of Program Capacity and Institutional Commitment

22

The program’s capacity and institutional commitment to deliver quality preparation through curricula, faculty, facilities, fiscal support student support services, student feedback, etc.

Focus

• To inform an accreditation decision based on both quality and confidence

• To make the case that program quality can continue between accreditation reviews

Purpose

Appendix C- QAR Relationship to Standard 3QAR

Brief narrative to describe how:

PART I: Evidence of capacity for maintaining program quality

PART II: Evidence of institutional commitment

Appendix cProvides the necessary evidence of:

DIMENSIONS of CAPACITY Curricula alignment with state/national standards: tableQualified faculty: table, CVs Adequate facilities: table/description Fiscal support: refer to parity table Support services: description/policies Means for student feedback: description/ policies

Parity table: is your program comparable to other programs? (next slide)

Faculty Qualifications Table (sample)Name Current

Rank/Title

Terminal degree

(Y/N)

Institution Granting Degree

Year of degree granted

Field of degree/ area of special-ization

Related qualifi-cations(link to

CV)

Coursenumbers

of regularly assigned courses

Years at provider

FACULTY

INSTRUCTIONAL STAFF

ADJUNCTS/LECTURERS

24

Parity Table (sample)Demonstrates Institutional Commitment

25

Capacity Aspect

Preparation Program

Institutional Norm or Comparable

Program

Analysis of Difference

CurriculumFaculty (ranks, load)Facilities Fiscal supportSupport ServicesStudent feedback

A Pathway to the Internal Audit: Appendix D Appendix D invites you

to: describe the quality

control processes used to evaluate and improve program quality

audit the system to ensure that it is working as intended

26

Appendix A:specified

admissions and

monitoring processes

Appendix B: described completer

support and follow up

efforts

Appendix C:

documented dimensions of program capacity

Appendix D: Internal Audit of the Quality Control System

27

• The quality control processes used to evaluate and improve program quality

• An audit to ensure the system is working as intended

Focus

To demonstrate the program’s capacity to monitor and maintain quality and engage in continuous improvementPurpose

Internal Audit: Appendix DTwo optional approaches

Broad-based approach10-20 program completers

Focused approach when a particular problem has already been

documented by evidence

28

QAR(S3) – Appendix D Relationship

29

QARBrief narrative to describe

program’s process:

Provide an overview of the internal audit,

process, and outcomes.

Appendix DProvides the necessary

documentation/evidence of:

DESCRIBE policies and practices that ensure quality

SAMPLE completers and audit the system starting with them.

Sample purposefully.

REPORT degree to which processes/polices were followed,

explain discrepancies, improvements

Internal Audit Map (sample)

30

Admissions

What process/ criteria

are used?

Advising

Expect-ations for

advancing through the program?

Monitoring

What are the criteria along the way? (with diversity of

placements)

Final Clinical

Placement

Were criteria met?

Completion Check

Process followed?

Criteria met?

Completer Follow-up

Who will be contacted

post-completion

and did that

happen?

For a sample of completers

Sample coursesWhat process/criteria are used?

Sample of placements, partnerships, supervisorsCriteria for placement/personnel/training

Appendix E: Evidence of Data Quality

31

Data quality of the programs’ measures/instrumentsFocus

To ensure the quality of evidence on which decisions are based is credible

Purpose

QAR (S3) – Appendix E Relationship

32

QARNarrative to describe:

Overview of how the program ensured data quality through

validity, reliability, trustworthiness, and fairness

Links to copies of locally developed measures

Appendix ESupporting evidence/

documentation addresses:

Program’s formal process to establish validity, reliability, trustworthiness, and fairness

How were faculty, internal and external stakeholders engaged in evaluating instruments and ensuring reliable administration?

What were the results of these investigations?

Aspects of Data QualityEvidence regarding assessments should be appropriate to type

ReliabilityReliable and reliably collected. How

are raters calibrated?

ValidityHow have/ might content validity be

established?

TrustworthinessFor qualitative data,

how is analysis done?

FairnessAddressing bias – do measures work

well for all candidates?

Data Quality

Presenting Your Program’s Story

Making Your Case for Quality:The Quality Assurance Report

How Many QARs Should I Write?Things to consider…

All in One

Coherent evidence set*

*Evidence must be disaggregated by level,

location, mode

More Than OneDifferent evidence sets

Varied governance

Autonomy of programs, use of feedback

35

Report FormatRequired components include:

Introduction, Standards, Conclusion, AppendicesNo page or character requirements/limitations, but

do remember that:Reviewers are humanQuality is more important than quantity

No one product design/presentation**but do consider the report will need to be archived

36

Perspective and How to Gain It…Who is your audience?Remember that your reviewers and the Accreditation

Commission will likely be unfamiliar with your context.

Review the QRT Report template. Does your self-study presentation help or hinder a reviewer?

Use your cohort members as critical friends (introduction!).

37

Anticipate Reviewer Needs

title sections embed links in narrative

file names, page

numbers

provide portions of data tables

terms, acronyms defined

graphics, visual relief

38

Is navigating your QAR intuitive?

A Note About DataWhen feasible, provide sample size and basic descriptive

statistics (range, mean, standard deviation) and the program’s definition of success

De-identify your dataData must be disaggregated by: Level, license, certificate program LocationMode of delivery

39

QAR Ideas to Remember Use the narrative to tell your story and make your case.

Although standards will be evaluated holistically, the aspects need to be clearly addressed by the evidence put forth.

If data is not yet available for aspirational Standards 2 and 4, provide newly or soon-to-be implemented plans to address these particular aspects.

Consider the evidence set provided for each aspect. Be judicious in selecting the best evidence to make your case.

40

QAR Ideas to Remember (cont.)Attention to limitations in the evidence (small sample

size, not fully validated measures) is expected and providers are encouraged to address these issues in continuous improvement plans.

Explanation and analysis of the evidence and how the program plans to use this evidence for program improvement. (What, why, how!)

41

QAR Ideas to Remember (cont.)Power in the “conclusion”: changes based on

evidence, new areas of investigation because of evidence, documentation of new initiatives and innovations.

Conclusion helps to shape the Annual Report. The analysis and implications guides your action plan.

42

And Finally…

The QAR is a celebration of your program’s work and a sign of your program’s commitment to

continuous improvement!

43

Questions?

44

Follow AAQEP on Social Media

45

twitter.com/aaqep1

linkedin.com/company/aaqep

THANK YOU!

Visit www.aaqep.org for updates and opportunities

Send questions and ideas to: aaqep@aaqep.org

46