1

WASC Evaluator WorkshopFall Visits 2009

2

Workshop Outcomes

• Understand WASC’s three-stage process and how your visit fits into the process

• Be familiar with the WASC Standards and CFRs and how to use them in the review process

• Know how to prepare for and conduct an effective visit and produce a useful, high-quality team report

• Be prepared to make sound judgments about institutions under the Standards

• Be familiar with resources that support your work on a team

3

Agenda

• Context for the Visit/Accreditation• WASC Three-Stage Review Process• Standards and CFRs• Preparing for the Visit• Conducting the Visit• Developing Team Recommendations• Writing the Team Report• After the Visit

4

Context for Accreditation and Visits

• The Continuing Evolution of the WASC Process and Standards

• The Accountability Movement– Retaining Peer Review

• The Impact of the Economy

• Value Added, Collaboration, and Ongoing Efforts to Refine and Improve

5

Changes this Year: Revisions and Improvements

• Implement changes to Institutional Review Process re: Student Success, Program Review and EE Sustainability

• Implement changes to CFRs• Clarify the scope of the CPR visit to review the

“infrastructure” for assessment of student learning

• Examine Program Review and Program-Level Student Learning in a systematic way

• Allow teams more time together on visits

6

The WASC Review Process

7

WASC’s Three-Stage

Review Process

1. Proposal: Identifies priorities, themes/areas of emphasis, and outcomes. Aligns work with institutional plans and needs.

2. Capacity/Preparatory Review: Focuses on capacity (systems, policies, resources) and readiness for educational effectiveness.

3. Educational Effectiveness: Focuses on results, findings.

8

The Two Reviews

Capacity and Preparatory

• Preparatory = readiness for the Educational Effectiveness Review

• Capacity = purposes, integrity, stability, resources, structures, policies, processes

Educational Effectiveness

• Demonstrating student learning

• Demonstrating institutional learning

• Demonstrating evidence-based decision-making

9

The CPR and EER as a Whole

• The CPR evaluates what an institution has for infrastructure (staff/faculty, resources, processes, facilities, systems, structures).

• The EER evaluates how well that infrastructure works and the results that the institution achieves.

10

Navigating Multiple Purposes

Focusing on the institution Applying the Standards and CFRs

Focusing on Proposal themes/topics

Evaluating capacity and effectiveness under Standards; addressing team-identified issues

Reviewing the whole institution Focusing on specifics, e.g., distance education, samples of program reviews

Advancing institutional development

Addressing “compliance” matters

11

Navigating Multiple Purposes

Allowing flexibility and experimentation on visit

Ensuring consistency and fairness among visits and quality control of visits and reports

Using the CPR to evaluate EE readiness

Leaving evaluation of educational effectiveness until EER

Supporting institutional creativity and excitement

Reporting to the Commission and serving the public

12

The Special Visit

• Intended to monitor institutional progress on issues identified by the Commission

• May or may not be connected to a sanction• Limited to a few specific areas of concern• Intended to assess how institution will move

into compliance (if on sanction)

13

Changes in the Institutional Review Process

• Institutions will cover the following in their reports:– Student Success (at CPR and EER)– Program Review (at EER)– Sustainability of EE (at EER)

• Teams should address in the team report

Tool: Table B (RB pg. 47)

14

The Team’s Impact

• Peer review is the foundation of accreditation.

• The team report forms the basis for the Commission action and its letter.

• The team report and action letter inform the work of the institution for years to come.

• Why were you chosen for a team?

15

Working with the Standards and CFRs

16

Standards and CFRs

• Two Core Commitments: Capacity and Educational Effectiveness

• Standards: Broad, holistic, encompassing• Criteria for Review: More specific and

detailed• Guidelines: Ways to demonstrate compliance

with the relevant CFR

17

Revisions to CFRs

• Institutions required to submit summary showing how they address revised CFRs

• Teams should review to see if institution is addressing important new requirements

Tool: Table A (RB pg. 41)

18

Team Use of the Standards and CFRs

• Team judgments must be linked to specific Standards and CFRs

• CFRs must be cited in reports • Standards and CFRs form the basis for

Commission decisions• Standards and CFRs provide a context for

continuous quality improvement

19

Standards at a Glance

20

STANDARD 1:Defining Institutional Purposes and Ensuring Educational Objectives

Institutional PurposesIntegrity

21

STANDARD 2:Achieving Educational Objectives

Through Core Functions

Teaching and LearningScholarship and Creativity

Support for Student Learning

22

STANDARD 3: Developing and Applying Resources

and Organizational Structures to Ensure Sustainability

Faculty and StaffFiscal, Physical & Information Resources

Organizational Structures & Decision-Making Processes

23

STANDARD 4: Creating an Organization Committed

to Learning and Improvement

Strategic Thinking and Planning Commitment to Learning and

Improvement

24

Two Visits: Different Views of a CFR (2.6)

The institution demonstrates that its graduates consistently achieve its stated levels of

attainment and ensures that its expectations for student learning are embedded in the standards

faculty use to evaluate student work.

25

CFR 2.6: Two Views

Capacity and Preparatory

Has the institution defined expected levels of attainment for SL?

Are they embedded in the standards and measures for student work?

How does the institution know if students are meeting expectations?

What data are collected and how analyzed?

How is student learning measured?

Educational Effectiveness

What do data show about student learning?

Are data disaggregated and analyzed?

Did the students learn what the faculty intended them to learn? At what levels of performance?

Has the institution used data to make changes and/or improvements?

26

Preparing for the Visit(Visit Guide, Part II, pp. 29-52)

27

Timeline For CPR/EER Reviews

12 weeks 2 months

Institution mails report to team and

WASC

Team holds conference

call

Site visit held and team report

written

Institution responds to

errors of fact in team report

Institution responds to final team

report

Commission acts at

February or June meeting

28

Roles and Responsibilities of Team Members and Staff

• Role of team chair (RB pg. 189)

• Role of team assistant chair (RB pg. 191)

• Role of assigned WASC staff liaison (VG pg. 7)

• Team assignments

29

Pre-visit Preparation

• Read all the documents from WASC– Standards, CFRs, policies, visit guide, rubrics– Background documents re: institution and purpose of

the visit, including Proposal and/or last action letter/team report

• Read the institutional report• Review the data portfolio and exhibits

– What to look for and highlight?

Tools: Timeline (VG pg. 8, VG pg. 29)

30

Reviewing the Exhibits

• Enrollment data– Headcounts and FTE

• Graduation data• Faculty data• Key financial indicators• Inventory of Educational Effectiveness Indicators• Inventory of Concurrent Accreditation and Key

Performance Indicators

Tool: How to Review WASC Data Exhibits (RB pg. 61)

31

Reading the Report

• Has the institution done what it said it would do in its Proposal?

• Has it collected and analyzed data effectively?• Are its conclusions supported by evidence?• Are there serious problems or potential areas

of noncompliance?• Does the report contain recommendations for

further institutional action?

32

Developing Visit Strategies and Lines of Inquiry

• What are areas needing clarification and/or more information?

• What are the major issues challenging the institution?

• What is raised by the themes that needs to be verified or explored?

• What are the strategies that will be most effective?

33

Worksheet for Team Conference Call

• Organizes team’s responses to institutional materials

• Helps team make preliminary evaluation under the Standards

• Provides basis for team to work toward consensus

• Should be submitted in advance of call

Tool: Team Worksheet (VG pg. 43)

34

Team Conference Call

• Evaluates quality of institutional report and alignment with Proposal and previous action letter(s)

• Identifies areas of good practice, improvement, and further inquiry

• Identifies issues, strategies, evidence needed• Identifies persons and entities to be interviewed • Makes or refines team assignments• Plans visit logistics

35

Off-Campus Sites and Distance Education Programs

Prior to Visit: Sites will be identified and assignments made• Review substantive change action letters to determine if issues

have been identified• Develop plan for the review of the programs and/or sites

During Visit• Interview faculty, administrators and students• Evaluate facilities OR online infrastructure• Observe classes• Document visit and findings in appendix • Discuss important findings with team for inclusion in report, as

appropriateTools: Protocols (RB pg. 158, RB pg. 160)

Forms (RB pg. 55, RB pg. 58)

36

Compliance Audit

• Required for:– Institutions seeking Candidacy and Initial

Accreditation– Some institutions under sanction

• Additional report submitted by institution in advance of the visit—with links to documents

Tool: Compliance Audit Checklist (RB, pg. 51)

37

Determining Strategy for CPR Visit

• What evidence is provided to show capacity and readiness for EE?

• Why was it chosen?• What are the strengths and weaknesses of the

evidence?• What other evidence do you want to review to evaluate

capacity and preparation for EE?• Do any issues arise with regard to the Standards?• Meetings: format/methodologies

38

Determining Strategy for EER Visit

• What evidence is provided to show EE?• Why was it chosen?• What are the strengths and weaknesses of the

evidence?• What other evidence do you want to see to evaluate

effectiveness?• Do any issues arise with regard to the Standards?• Meetings: format/methodologies

39

Drafting in Advance of the Visit

• Assistant Chair draft outline of team report and context sections

• Team members draft outline or text for which they are responsible, using data from institution, with space for additional data, analysis and conclusions

Tool: Team Reports (VG pg. 53)

40

Conducting the Visit

41

Process of Visit

• Team meets at start of visit to confirm roles, assignments, logistics, and agenda

• Team meets frequently re: observations, emerging recommendations, and issues

• Team members draft sections of report and turn in to assistant chair on the last day

• Team agrees on report recommendations and confidential recommendation to Commission

42

Visit Schedule

• Executive sessions and debriefings with team only

• Meetings and interviews with key individuals and groups

• Open meetings with students, faculty and staff

• Document review • Time for drafting report sections• Final exit meeting

43

Confidential Email Account

• Set up by WASC as extension of open meetings

• Checked by assistant chair during visit• Important emails shared with team and

investigated• Comments included in team report only if the

institution has a chance to address them

44

Approaches Used on Visits

• Document review• Interviews and meetings

– Mini-questionnaires– Techniques for small and large meetings– Fishbowl exercises

• Audits

Plan visit methodologies in advance as part of schedule.

45

Document ReviewDO as much as possible in advance:

Use to:• Check compliance• Evaluate the level of institutional engagement• Examine the evolution of a policy or process• Identify direct and indirect evidence of

student and organizational learning• Confirm report claims

46

Interviews

Use to:• Gather information• Explore issues• Build relationships with members of the

institution• Validate impressions and observations

47

Tips for Good Interviews• Decide on a protocol for interview• Prepare questions and lines of inquiry in

advance • Ask questions that elicit information, stimulate

discussion, or require judgment• Avoid interrogation, leading questions, or

loaded language• Avoid consultation, giving solutions, or talking

about your institution• Let them do the talking

48

Alternative Forms of Interview

• Fishbowl• Brainstorm/free discussion on a salient topic• Go-round• Bundling• Audit

49

Evaluating Program Review and Student Learning

on EER Visits

50

51

52

53

54

Rubrics re: Assessment of Student Learning

1. Academic Program Learning Outcomes

2. Use of Portfolios in Assessing Program Outcomes

3. Use of Capstones in Assessing Program Outcomes

4. Integration of Student Learning Assessment into Program Review

5. Assessing General Education

55

Expectations for Two Reviews

• Use questions re: Student Learning on page 2

• Use as a monitor to be sure you are within the proper scope of the visit

56

Educational Effectiveness Framework

• Use with team to evaluate institution’s “place” • Use language of rubric to describe the institution

in the report• Ask the institution to evaluate itself and discuss• Confer with team toward end of visit to mark a

copy of the EEF • Submit the marked EEF confidentially to WASC

57

The Exit Meeting

• Team chair communicates commendations and key recommendations that will be included in report

• Chair may ask team members to participate• The meeting is not a dialog, discussion or

debate

58

Lunch

Meet your team and assigned

WASC staff liaison

59

Case Studies

• CPR group• EER group• Special Visit group

60

Developing Team Recommendations

61

Two Kinds of Recommendations

• Confidential Team Recommendation to the Commission for action

• Team recommendations at the end of team report, delivered at the exit meeting

Tools: Commission Decisions on Institutions

(Visit Guide pg. 84; SV Guide pg. Append. F)

Commission and Team Decision Matrix (RB pg. 173)

62

Team Report Recommendations

Should be:• Overarching and important• Supported by evidence • Linked clearly to Standards and CFRs• Supported by text in the report

- Distinguish recommendations from suggestions and observations embedded in the report

Tool: Educational Effectiveness Framework (RB pg. 176)

63

Confidential Recommendation to Commission

• CPR– Proceed to EER or reschedule EER visit– Conduct a Special Visit (not preferred)– Add time to EER visit– Issue a notice of concern or impose

a sanctionTool: Confidential Team Recommendation Form

(VG pg. 70)

64

Confidential Recommendation to Commission

• EER– Grant Candidacy, Initial Accreditation or

Reaffirmation of Accreditation for specified term

– Sanction or Notice of Concern– Interim Report or Special Visit

Tool: Confidential Team Recommendation Form

(VG pg.72)

65

Confidential Recommendation to Commission

• Special Visit– Varies with status of institution – Next steps– Removal or continuation of sanction

(note two-year limit on sanctions)

66

Producing Effective Team Reports

67

Report Preparation Logistics

• Follow report template

• Start writing before the visit

• Complete your sections on site and give to assistant chair for editing together

68

What is an effective team report?

Reflects a thorough assessment of the institution’s capacity, preparation, and/or effectiveness

Is evidence basedCites the Standards and CFRsProvides the basis for a sound and supportable

Commission decision Identifies important areas for institution to address

69

Using Evidence in Team Reports

• Use qualitative and quantitative evidence • Select evidence carefully and purposefully• Connect evidence to an assertion or question • Analyze information; do not just set forth data• Let evidence suggest improvements• Use evidence that speaks to the institution’s

themes and the team's questions

70

Tips for Writing Team Reports

• Consider multiple audiences: institution, Commission, and next team

• Know your areas of responsibility, including length and depth of your section

• Start writing before you arrive on campus• Address priorities and goals set by the institution• Address Commission’s concerns (last action letter)• Make commendations, but don’t overdo it• Use praise that doesn’t send wrong or mixed signal

71

More Tips on Team Reports….

• Be sure to check facts

• Support findings and recommendations with evidence --and tie them to CFRs

• Ensure evidence is sound and valid

• Distinguish recommendations from suggestions or observations

• Use formal language and tone (e.g., not “we/they”)

• Don’t mention personnel by name

• Don’t prescribe solutions

72

After the Visit

73

What happens next?

• AC prepares draft for chair, team and staff review; changes as needed

• Chair sends to institution for corrections of fact

• Chair finalizes draft and submits to WASC• Chair sends Confidential Team

Recommendation and completed EEF to WASC

• WASC sends report to institution

74

Then…• Staff prepares draft action letter, which is reviewed

by team chair

• Commission Panel reads report and documentation including institution’s written response, meets with institutional representatives at Commission meeting

• Panel makes recommendation to Commission, and Commission acts

• Staff finalizes draft action letter on behalf of Commission

75

Also after the visit….• Team members send reimbursement forms to WASC

within 30 days– Hotel arranged and paid directly by institution– Travel / food reimbursed – Rental car must be approved in advance by

WASC staff– Spouse or assistant costs not covered– See policy for more details

• Team members should not have any contact with the institution – About the visit OR– Consult with the institution for one year

76

Some common complaints about visits

• Some team members not well prepared• The team “did not understand us”• The CPR team moved into EER “territory”• Team did not review all the evidence• Team changed the schedule at the last minute,

or did not stay on schedule

77

Some common complaints about team reports

• The recommendations were too specific or were unfair

• The report did not show that the team reviewed the evidence

• The recommendations were not based on good evidence or supported in report text

• The report did not address all the important issues or themes

78

Resources for Teams

• Appendices of Visit Guide

• Team Materials and Institutional Report mailed 10-12 weeks in advance of visit

• WASC Website: www.wascsenior.org

• WASC Email Advisory (sent prior to visit)

• WASC Staff

79

Breakout Groups

• Chairs

• Assistant Chairs

• Evaluators/New Evaluators

80

Remaining Concerns, Questions, Comments?

And…evaluate!

81

Thank you for your service to the region