WASC Evaluator Training Fall 2010

1

Evaluator Workshop Fall Visits 2010

Tuesday, May 25, 2010

Please join the audio portion of this training: 866-740-1260, Access Code: 7489001

ReadyTalk Help Desk: 800.843.9166 International Help: 303.209.1600

2

AnnouncementsThis presentation and the accompanying materials are available for download from:

http://www.wascsenior.org/evaltrainingFall2010

For assistance with Voice and Web connections please contact:

ReadyTalk Help Desk, 800.843.9166

International Help: 303.209.1600

Please mute your microphone if you are not speaking.If you have questions, please enter them into the Chat window.

3

WASC Evaluator WorkshopFall Visits 2010

4

Workshop Outcomes

• Know how to prepare for and conduct an effective visit

• Be prepared to produce a useful, high-quality team report

• Be ready to make sound judgments about institutions under the Standards

• Be familiar with resources that support your work on a team

5

Agenda

• Context for the Visit/Accreditation

• Preparing for the Visit

• Conducting the Visit

• Developing Team Recommendations

• Writing the Team Report

6

Context for Accreditation and Visits

• The continuing evolution of the WASC process and Standards

• The accountability movement– Moving from assessment for improvement to

assessment for accountability– New areas of emphasis

• The Handbook revision process

7

Recent Changes in the Institutional Review Process

and Standards • Changes to Institutional Review Process re: Student

Success, Program Review and EE Sustainability• Changes to CFRs• Clarifying the scope of the CPR visit to review the

“infrastructure” for assessment of student learning• Program Review and Program-Level Student Learning in

a systematic way

Tool: Table A and B (RB pg. 41and 46)

8

Covering the Impact of the Financial Recession on Institutions

Questions to ask the institution:

• How has the financial recession affected your institution?

• How has your institution responded?• What plans are in place in case the current state

of affairs becomes permanent?

9

Q&A

• Please feel free to type in your questions using the chat window or just chime in.

10

The Three-Stage Review Process

Institutional Proposal

Capacity & Preparatory

Review

Educational Effectiveness

Review

1. 2. 3.

11

The Three-Stage Review Process

1. Institutional Proposal

2. Capacity and Preparatory Review

3. Educational Effectiveness Review

12

Timeline for Three-Stage Review Process

Submitted 2Years beforeCPR review

Capacity And Preparatory

Review

18-24months toprepare for

EER(or less for

Candidacy orInitial Accreditation)

EducationalEffectiveness

Review

Extendedperiod oftime tosustain

Initiatives(7-10 years)

Proposal

13

Institutional Self-Review

• The heart of accreditation

• Built upon an effective internal process of– Evaluation– Self-reflection– Recommendations from previous reviews– Plans for action

14

Stage 1:

The Institutional Proposal

15

The Institutional Proposal

• Guides the entire accreditation review process• Connects institution’s context and priorities with

the Standards of Accreditation• Provides primary basis for both institution self-

review and team evaluation• Allows alignment of accreditation activities to

institutional strategic plan and key areas chosen for improvement

16

The Letter of Intent

• Submitted by institutions seeking Candidacy or Initial Accreditation, the LOI serves the same purpose as the proposal

• Includes suggestions from Eligibility approval letter

• Submitted to assigned WASC liaison, one year in advance of CPR Review

• Instructions are in “How to Become Accredited” on WASC website

17

Stage 2: The Capacity and Preparatory Review

18

Purpose of the CPR

• Review and verify the information in the institutional presentation (report and data)

• Evaluate key institutional resources, structures, processes in light of Standards

• Evaluate institution’s infrastructure to support and assess student learning

• Assess institution’s preparedness to undertake the Educational Effectiveness Review

19

Stage 3: The Educational Effectiveness

Review

20

Purpose of the EER

• Invite sustained engagement by the institution on the extent to which it fulfills its educational objectives

• Enable the Commission to make a judgment about extent to which institution fills its Core Commitment to Educational Effectiveness

21

The Two Core Commitments

22

Core Commitment 1

“The institution functions with clear purposes, high levels of institutional integrity, fiscal stability, and organizational structures to fulfill its purposes.”

23

Core Commitment 2

“The institution evidences clear and appropriate educational objectives and design at the institutional and program level. The institution employs processes of review, including the collection and use of data, that ensure delivery of program and learner accomplishments at a level of performance appropriate for the degree or certificate awarded.”

24

The Four Standards

Tool: Standards of Accreditation, RB p. 23 Standards at a Glance, RB p. 37

25

Standard 1:Defining Institutional Purposes and Ensuring Educational Objectives

• Institutional Purposes

• Integrity

26

Standard 2:Achieving Educational Objectives

Through Core Functions

• Teaching and Learning

• Scholarship and Creative Activity

• Support for Student Learning

27

Standard 3:Developing and Applying Resources

and Organizational Structures to Ensure Sustainability

• Faculty and Staff

• Fiscal, Physical, Information Resources

• Organizational Structures & Decision Making Processes

28

Standard 4:Creating an Organization Committed to

Learning and Improvement

• Strategic Thinking and Planning

• Commitment to Learning and Improvement

29

Expectations for Two Reviews

Capacity and Preparatory

• Preparatory = readiness for the Educational Effectiveness Review

• Capacity = purposes, integrity, stability, resources, structures, policies, processes

Educational Effectiveness

• Demonstrating student learning

• Demonstrating institutional learning

• Demonstrating evidence-based decision-making

Tool: Expectations for Two Reviews (RB pg. 20)

30

Q&A


31

Preparing for the Visit(Visit Guide, Part II, pp. 29-52)

32

Roles and Responsibilities of Team Members and Staff

• Role of Team Chair (RB pg. 193)

• Role of Team Assistant Chair (RB pg. 195)

• Role of assigned WASC staff liaison (VG pg. 7)

• Team assignments

Tool: Section 9 (Tips, Roles and Advice, RB p 189)

33

Timeline For CPR/EER Reviews

12 weeks 2 months

Institution mails report to team and

WASC

Team holds conference

call

Site visit held and team report

written

Institution responds to

errors of fact in team report

Institution responds to final team

report

Commission acts at

February or June meeting

Tool: CPR or EER Timeline (VG, pg. 29)

34

Pre-visit Preparation

• Read all the documents from WASC– Standards, CFRs, policies, visit guide, rubrics– Background documents re: institution and purpose of

the visit, including Proposal and/or last action letter/team report

• Read the institutional report• Review the data portfolio and exhibits

– What to look for and highlight?

35

Reviewing the Exhibits

• Enrollment data– Headcounts and FTE

• Graduation data• Faculty data• Key financial indicators• Inventory of Educational Effectiveness Indicators• Inventory of Concurrent Accreditation and Key

Performance Indicators

Tool: How to Review WASC Data Exhibits (RB pg. 61)

36

Reading the Report

• Has the institution done what it said it would do in its Proposal?

• Has it collected and analyzed data effectively?• Are its conclusions supported by evidence?• Are there serious problems or potential areas

of noncompliance?• Does the report contain recommendations for

further institutional action?

37

Worksheet for Team Pre-Visit Conference Call

• Organizes team’s responses to institutional materials

• Helps team make preliminary evaluation under the Standards

• Provides basis for team to work toward consensus

• Should be submitted in advance of call

Tool: CPR Conference Call Worksheet (VG pg. 42)

EER Conference Call Worksheet (VG pg. 47)

SV Conference Call Worksheet (SVG pg. 70)

38

• Evaluates quality of institutional report and alignment with Proposal and previous action letter(s)

• Identifies areas of good practice, improvement, and further inquiry

• Identifies issues, strategies, evidence needed• Identifies persons and entities to be interviewed • Makes or refines team assignments• Plans visit logistics

Team Pre-Visit Conference Call

39

Off-Campus Sites and Distance Education Programs(special requirement for some visits)

Prior to Visit: Sites/online programs will be identified and assignments made• Review substantive change action letters to determine if issues have been

identified• Develop plan for the review of the programs and/or sites

During Visit• Interview faculty, administrators and students• Evaluate facilities OR online infrastructure• Observe classes (can be done ahead for online)• Document visit and findings in the appendix • Discuss important findings with team for inclusion in report, as appropriate

Tools: Protocols (RB pg. 157, RB pg. 162) Forms (RB pg. 55, RB pg. 58)

40

Compliance Audit (special requirement for some visits)

• Required for:– Institutions seeking Candidacy and Initial

Accreditation– Some institutions under sanction

• Additional report submitted by institution in advance of the visit—with links to documents

• Prepare as appendix to report

Tool: Compliance Audit Checklist (RB, pg. 51)

41

Determining Strategy for CPR Visit

• What evidence is provided to show capacity and readiness for EE?

• What are the strengths and weaknesses of the evidence?

• What other evidence do you want to review to evaluate capacity and preparation for EE?

• Do any issues arise with regard to the Standards?

• Meetings: format/methodologies

42

Determining Strategy for EER Visit

• What evidence is provided to show EE?• Why was it chosen?• What are the strengths and weaknesses of the

evidence?• What other evidence do you want to see to

evaluate effectiveness?• Do any issues arise with regard to the

Standards?• Meetings: format/methodologies

43

Drafting in Advance of the Visit

• Assistant Chairs draft outline of team report and Section I

• Team members draft outline or text for which they are responsible, using institution’s report and data portfolio, with space for additional evidence, analysis and conclusions

Tool: Team Reports (VG pg. 54)

44

Q&A


45

Conducting the Visit

46

Launching the Visit: Team Executive Session

• Discuss preliminary findings• Identify major issues for exploration• Refine lines of inquiry for each meeting• Confirm team assignments• Discuss use of tools and rubrics• Review preliminary outline/draft team report• Discuss options for confidential team recommendation• Review schedule

47

Visit Schedule

• Executive sessions and debriefings with team only

• Meetings and interviews with key individuals and groups

• Open meetings with students, faculty and staff• Document review • Time for drafting report sections• Final exit meeting

Tool: Visit Schedule (VG pg. 40, SVG pg. 68)

48

Confidential Email Account

• Set up by WASC as extension of open meetings• Checked by Assistant Chair during visit• Important emails shared with team and

investigated• Comments included in team report only if the

institution has a chance to address them

Tool: Sample Notification re: Confidential Email Account (RB, pg. 151)

49

Approaches Used on Visits

• Document review• Interviews and meetings

– Mini-questionnaires– Techniques for small and large meetings – Fishbowl exercises

• Audits

Plan visit methodologies in advance as part of schedule.

50

Tips for Good Interviews

• Decide on a protocol for interview• Prepare questions and lines of inquiry in

advance • Ask questions that elicit information, stimulate

discussion, or require judgment• Avoid interrogation, leading questions, or

loaded language• Avoid consultation, giving solutions, or talking

about your institution• Let them do the talking

51

Addressing Student Success

• All CPR and EER reports should address this topic

• Teams should designate someone to study and write about student success

• Discussion should include summary of data, identification of issues, and plans for improvement

52

Evaluating Program Review and Student Learning

on EER Visits

Tool: Suggested Approaches for Evaluating Program Review (RB, pg. 156)

EER Toolkit

53

Rubrics: Assessment of Student Learning

1. Academic Program Learning Outcomes2. Use of Portfolios in Assessing Program

Outcomes3. Use of Capstones in Assessing Program

Outcomes4. Integration of Student Learning

Assessment into Program Review5. General Education Assessment Process

Tool: Assessment Related Rubrics (VG, pp. 77-88)

Program Learning Outcome Rubric

54

Educational Effectiveness Framework

• Use with team to evaluate institution’s “place” • Use language of rubric to describe the institution in

the report• Ask the institution to evaluate itself and discuss• Confer with team toward end of visit to mark a copy

of the EEF • Submit the marked EEF confidentially to WASC

Tool: Educational Effectiveness Framework (RB, pp. 160-161)

55

The Exit Meeting

• Team chair communicates commendations and key recommendations that will be included in report

• Chair may ask team members to participate• The meeting is not a dialog, discussion or

debate

56

Q&A


57

Developing Team Recommendations

58

Two Kinds of Recommendations

• Team recommendations at the end of team report, delivered at the exit meeting

• Confidential Team Recommendation to the Commission for action

Tools: Commission Decisions on Institutions

(Visit Guide pg. 89; SV Guide pg. Append. F)

Commission and Team Decision Matrix (RB pg. 177)

59

Team Report Recommendations

Should be:• Overarching and important• Supported by evidence • Linked clearly to Standards and CFRs• Supported by text in the report

- Distinguish recommendations from suggestions and observations embedded in the report

Tool: Educational Effectiveness Framework (RB pg. 160)

60

Producing Effective Team Reports

61

Report Preparation Logistics

• Follow report template

• Start writing before the visit

• Complete your sections on site and give to Assistant Chair for editing together

Tool: Team Report Templates (VG, pg. 55; SV Guide, Appen. K)

CPR Team Report

62

Using Evidence in Team Reports

• Use qualitative and quantitative evidence • Select evidence carefully and purposefully• Connect evidence to an assertion or question • Analyze information; do not just set forth data• Let evidence suggest improvements• Use evidence that speaks to the institution’s

themes and the team's questions

63

Team Use of the Standards and CFRs

• Team judgments must be linked to specific Standards and CFRs

• CFRs must be cited in reports • Standards and CFRs form the basis for

Commission decisions• Standards and CFRs provide a context for

continuous quality improvement

64

What is an effective team report?

• Reflects a thorough assessment of the institution’s capacity, preparation, and/or effectiveness

• Is evidence based• Cites the Standards and CFRs• Provides the basis for a sound and supportable

Commission decision• Identifies important areas for institution to

address

65

Tips for Writing Team Reports

• Consider multiple audiences: institution, Commission, and next team

• Know your areas of responsibility, including length and depth of your section

• Start writing before you arrive on campus• Address priorities and goals set by the institution• Address Commission’s concerns (last action letter)• Make commendations, but don’t overdo it• Use praise that doesn’t send wrong or mixed signal

66

More Tips on Team Reports….

• Be sure to check facts • Support findings and recommendations with

evidence --and tie them to CFRs• Ensure evidence is sound and valid• Distinguish recommendations from suggestions

or observations• Use formal language and tone (e.g., not

“we/they”)• Don’t mention personnel by name• Don’t prescribe solutions

67

Q&A


68

After the Visit

69

What happens next?

• Assistant Chair prepares draft for Chair, team and staff review; changes as needed

• Chair sends to institution for corrections of fact

• Chair finalizes draft and submits to WASC• Chair sends Confidential Team

Recommendation and completed EEF to WASC

• WASC sends report to institution

70

Then…

• Staff prepares draft action letter, which is reviewed by team Chair

• Commission Panel reads report and documentation including institution’s written response, meets with institutional representatives at Commission meeting

• Panel makes recommendation to Commission, and Commission acts

• Staff finalizes draft action letter on behalf of Commission

71

Also after the visit….

• Team members send reimbursement forms to WASC within 30 days– Hotel arranged and paid directly by institution– Travel / food reimbursed – Rental car must be approved in advance by

WASC staff– Spouse or assistant costs not covered– See policy for more details

• Team members should not have any contact with the institution – About the visit OR– Consult with the institution for one year

72

The Team’s Impact

• Peer review is the foundation of accreditation. • The team report forms the basis for the

Commission action and its letter.• The team report and action letter inform the

work of the institution for years to come.• Why were you chosen for a team?

73

Resources for Teams

• Appendices of Visit Guide

• Team Materials and Institutional Report mailed 10-12 weeks in advance of visit

• WASC Website: www.wascsenior.org

• WASC Email Advisory (sent prior to visit)

• WASC Staff

74

Q&A


75

Thank youfor your service to the

region

76

Announcements

The materials presented during this webinar and a recording of this session will be posted at:

http://www.wascsenior.org/evaltrainingFall2010