Wasc evaluator training webinar spring 2011
Preview:
DESCRIPTION
Citation preview
- 1. Evaluator Workshop Spring Visits 2011 Wednesday, December 8,
2010 Please join the audio portion of this training: 866-740-1260,
Access Code: 7489001 ReadyTalk Help Desk: 800.843.9166
International Help: 303.209.1600
- 2. Announcements This presentation and the accompanying
materials are available for download from:
http://www.wascsenior.org/spring2011 For assistance with Voice and
Web connections please contact: ReadyTalk Help Desk, 800.843.9166
International Help: 303.209.1600 Please mute your microphone if you
are not speaking. If you have questions, please enter them into the
Chat window.
- 3. WASC Evaluator Workshop Spring Visits 2011
- 4. Workshop Outcomes
- Know how to prepare for and conduct an effective visit
- Be prepared to produce a useful, high-quality team report
- Be ready to make sound judgments about institutions under the
Standards
- Be familiar with resources that support your work on a
team
- 5. Agenda
- Context for the Visit/Accreditation
- Developing Team Recommendations
- 6. Context for Accreditation and Visits
- The continuing evolution of the WASC process and Standards
- The accountability movement
-
- Moving from assessment for improvement to assessment for
accountability
- The Handbook revision process
- 7. Recent Changes in the Institutional Review Process and
Standards
- Changes to Institutional Review Process re: Student Success,
Program Review and EE Sustainability
- Clarifying the scope of the CPR visit to review the
infrastructure for assessment of student learning
- Program Review and Program-Level Student Learning in a
systematic way
- Tool: Table A & B (EVG pg 65 & 71, SVG pg. 63 &
69)
- 8. Covering the Impact of the Financial Recession on
Institutions
- Questions to ask the institution:
- How has the financial recession affected your institution?
- How has your institution responded?
- What plans are in place in case the current state of affairs
becomes permanent?
- 9. Q&A
- Please feel free to type in your questions using the chat
window or just chime in.
- 10. The Three-Stage Review Process Institutional Proposal
Capacity & Preparatory Review Educational Effectiveness Review
1. 2. 3.
- 11. The Three-Stage Review Process
- Capacity and Preparatory Review
- Educational Effectiveness Review
- 12. Timeline for Three-Stage Review Process Submitted 2 Years
before CPR review Capacity And Preparatory Review 18-24 months to
prepare for EER (or less for Candidacy or Initial Accreditation)
Educational Effectiveness Review Extended period of time to sustain
Initiatives (7-10 years) Proposal
- 13. Institutional Self-Review
- The heart of accreditation
- Built upon an effective internal process of
-
- Recommendations from previous reviews
- 14. Stage 1: The Institutional Proposal
- 15. The Institutional Proposal
- Guides the entire accreditation review process
- Connects institutions context and priorities with the Standards
of Accreditation
- Provides primary basis for both institution self-review and
team evaluation
- Allows alignment of accreditation activities to institutional
strategic plan and key areas chosen for improvement
- 16. The Letter of Intent
- Submitted by institutions seeking Candidacy or Initial
Accreditation, the LOI serves the same purpose as the proposal
- Includes suggestions from Eligibility approval letter
- Submitted to assigned WASC liaison, one year in advance of CPR
Review
- Instructions are in How to Become Accredited on WASC
website
- 17. Stage 2: The Capacity and Preparatory Review
- 18. Purpose of the CPR
- Review and verify the information in the institutional
presentation (report and data)
- Evaluate key institutional resources, structures, processes in
light of Standards
- Evaluate institutions infrastructure to support and assess
student learning
- Assess institutions preparedness to undertake the Educational
Effectiveness Review
- 19. Stage 3: The Educational Effectiveness Review
- 20. Purpose of the EER
- Invite sustained engagement by the institution on the extent to
which it fulfills its educational objectives
- Enable the Commission to make a judgment about extent to which
institution fills its Core Commitment to Educational
Effectiveness
- 21. Q&A
- Please feel free to type in your questions using the chat
window or just chime in.
- 22. The Two Core Commitments
- 23. Core Commitment 1
- The institution functions with clear purposes , high levels of
institutional integrity , fiscal stability , and organizational
structures to fulfill its purposes.
- 24. Core Commitment 2
- The institution evidences clear and appropriate educational
objectives and design at the institutional and program level. The
institution employs processes of review , including the collection
and use of data , that ensure delivery of program and learner
accomplishments at a level of performance appropriate for the
degree or certificate awarded.
- 25. The Four Standards Tool: Standards of Accreditation (EVG
pg. 49, SVG pg. 47) Standards at a Glance (EVG pg. 61, SVG pg.
59)
- 26. Standard 1: Defining Institutional Purposes and Ensuring
Educational Objectives
- 27. Standard 2: Achieving Educational Objectives Through Core
Functions
- Scholarship and Creative Activity
- Support for Student Learning
- 28. Standard 3: Developing and Applying Resources and
Organizational Structures to Ensure Sustainability
- Fiscal, Physical, Information Resources
- Organizational Structures & Decision Making Processes
- 29. S tandard 4: Creating an Organization Committed to Learning
and Improvement
- Strategic Thinking and Planning
- Commitment to Learning and Improvement
- 30. Expectations for Two Reviews Tool: Expectations for Two
Reviews (EVG pg. 44, SVG pg. 42)
- Educational Effectiveness
- Demonstrating student learning
- Demonstrating institutional learning
- Demonstrating evidence-based decision-making
- Preparatory = readiness for the Educational Effectiveness
Review
- Capacity = purposes, integrity, stability, resources,
structures, policies, processes
- 31. Q&A
- Please feel free to type in your questions using the chat
window or just chime in.
- 32. Preparing for the Visit
- 33. Roles and Responsibilities of Team Members and Staff
- Role of Team Chair (EVG pg. 259, SVG pg. 193)
- Role of Team Assistant Chair (EVG pg. 261, SVG pg. 195)
- Role of assigned WASC staff liaison (EVG pg. 9)
- The Role of the Team Chair & Assistant Chair/Team Editor
(EVG pg. 8)
- Section 10 (Tips, Roles and Advice)
- 34. Timeline For CPR/EER Reviews 12 weeks 2 months Institution
mails report to team and WASC Team holds conference call Site visit
held and team report written Institution responds to errors of fact
in team report Institution responds to final team report Commission
acts at February or June meeting Tool: CPR or EER Timeline (VG, pg.
75, SVG pg. 73)
- 35. Pre-visit Preparation
- Read all the documents from WASC
-
- Standards, CFRs, policies, visit guide, rubrics
-
- Background documents re: institution and purpose of the visit,
including Proposal and/or last action letter/team report
- Read the institutional report
- Review the data portfolio and exhibits
-
- What to look for and highlight?
- 36. Reviewing the Exhibits
- Inventory of Educational Effectiveness Indicators
- Inventory of Concurrent Accreditation and Key Performance
Indicators
- Tool: How to Review WASC Data Exhibits (EVG pg. 97, SVG pg.
90)
- 37. Reading the Report
- Has the institution done what it said it would do in its
Proposal?
- Has it collected and analyzed data effectively?
- Are its conclusions supported by evidence?
- Are there serious problems or potential areas of
noncompliance?
- Does the report contain recommendations for further
institutional action?
- 38. Worksheet for Team Pre-Visit Conference Call
- Organizes teams responses to institutional materials
- Helps team make preliminary evaluation under the Standards
- Provides basis for team to work toward consensus
- Should be submitted in advance of call
- Tool: CPR Conference Call Worksheet (EVG pg. 84)
- EER Conference Call Worksheet (EVG pg. 88)
- SV Conference Call Worksheet (SVG pg. 82)
- 39.
- Evaluates quality of institutional report and alignment with
Proposal and previous action letter(s)
- Identifies areas of good practice, improvement, and further
inquiry
- Identifies issues, strategies, evidence needed
- Identifies persons and entities to be interviewed
- Makes or refines team assignments
Team Pre-Visit Conference Call
- 40. Off-Campus Sites and Distance Education Programs (special
requirement for some visits)
- Prior to Visit: Sites/online programs will be identified and
assignments made
-
- Review substantive change action letters to determine if issues
have been identified
-
- Develop plan for the review of the programs and/or sites
-
- Interview faculty, administrators and students
-
- Evaluate facilities OR online infrastructure
-
- Observe classes (can be done ahead for online)
-
- Document visit and findings in the appendix
-
- Discuss important findings with team for inclusion in report,
as appropriate
- Tools: Distance Ed. Summary (EVG pg. 197, SVG pg. 147)
- Off-Campus Site Summary (EVG pg. 194, SVG pg. 138)
- 41. Compliance Audit (special requirement for some visits)
-
- Institutions seeking Candidacy and Initial Accreditation
-
- Some institutions under sanction
- Additional report submitted by institution in advance of the
visitwith links to documents
- Prepare as appendix to report
- Tool: Compliance Audit Checklist (EVG pg. 93, SVG pg. 87)
- 42. Determining Strategy for CPR Visit
- What evidence is provided to show capacity and readiness for
EE?
- What are the strengths and weaknesses of the evidence?
- What other evidence do you want to review to evaluate capacity
and preparation for EE?
- Do any issues arise with regard to the Standards?
- Meetings: format/methodologies
- 43. Determining Strategy for EER Visit
- What evidence is provided to show EE?
- What are the strengths and weaknesses of the evidence?
- What other evidence do you want to see to evaluate
effectiveness?
- Do any issues arise with regard to the Standards?
- Meetings: format/methodologies
- 44. Drafting in Advance of the Visit
- Assistant Chairs draft outline of team report and Section
I
- Team members draft outline or text for which they are
responsible, using institutions report and data portfolio, with
space for additional evidence, analysis and conclusions
- Tool: Guidelines for Drafting Preliminary Report Narratives
Prior to the Visit (EVG pg. 92, SVG pg. 86)
- 45. Q&A
- Please feel free to type in your questions using the chat
window or just chime in.
- 46. Conducting the Visit
- 47. Launching the Visit: Team Executive Session
- Discuss preliminary findings
- Identify major issues for exploration
- Refine lines of inquiry for each meeting
- Discuss use of tools and rubrics
- Review preliminary outline/draft team report
- Discuss options for confidential team recommendation
- 48. Visit Schedule
- Executive sessions and debriefings with team only
- Meetings and interviews with key individuals and groups
- Open meetings with students, faculty and staff
- Time for drafting report sections
- 49. Confidential Email Account
- Set up by WASC as extension of open meetings
- Checked by Assistant Chair during visit
- Important emails shared with team and investigated
- Comments included in team report only if the institution has a
chance to address them
Tool: Sample Notification re: Confidential Email Account (EVG pg.
169, SVG pg. 117)
- 50. Approaches Used on Visits
-
- Techniques for small and large meetings
- Plan visit methodologies in advance
- 51. Tips for Good Interviews
- Decide on a protocol for interview
- Prepare questions and lines of inquiry in advance
- Ask questions that elicit information, stimulate discussion, or
require judgment
- Avoid interrogation, leading questions, or loaded language
- Avoid consultation, giving solutions, or talking about your
institution
- 52. Addressing Student Success
- All CPR and EER reports should address this topic
- Teams should designate someone to study and write about student
success
- Discussion should include summary of data, identification of
issues, and plans for improvement
- 53. Evaluating Program Review and Student Learning on EER
Visits Tool: EE Toolkit (available online) Suggested Approaches for
Evaluating Program Review (EVG pg. 174)
- 54. EER Toolkit
- 55.
- 56.
- 57.
- 58. Rubrics: Assessment of Student Learning
- Academic Program Learning Outcomes
- Use of Portfolios in Assessing Program Outcomes
- Use of Capstones in Assessing Program Outcomes
- Integration of Student Learning Assessment into Program
Review
- General Education Assessment Process
Tool: Rubrics for Evaluating the Effectiveness of Assessment
Practices (EVG pg. 180)
- 59. Educational Effectiveness Framework
- Use with team to evaluate institutions place
- Use language of rubric to describe the institution in the
report
- Ask the institution to evaluate itself and discuss
- Confer with team toward end of visit to mark a copy of the
EEF
- Submit the marked EEF confidentially to WASC
Tool: Educational Effectiveness Framework (EVG pg. 178)
- 60. The Exit Meeting
- Team chair communicates commendations and key recommendations
that will be included in report
- Chair may ask team members to participate
- The meeting is not a dialog, discussion or debate
- 61. Q&A
- Please feel free to type in your questions using the chat
window or just chime in.
- 62. Developing Team Recommendations
- 63. Two Kinds of Recommendations
- Team recommendations at the end of team report, delivered at
the exit meeting
- Confidential Team Recommendation to the Commission for
action
- Commission Decisions on Institutions
- (EVG pg. 233, SVG pg. 167)
- Commission and Team Decision Indicators
- (EVG pg. 243, SVG pg. 177)
- 64. Team Report Recommendations
- Overarching and important
- Linked clearly to Standards and CFRs
- Supported by text in the report
-
- Distinguish recommendations from suggestions and observations
embedded in the report
- Tool: Educational Effectiveness Framework
- 65. Producing Effective Team Reports
- 66. Report Preparation Logistics
- Start writing before the visit
- Complete your sections on site and give to Assistant Chair for
editing together
Tool: Section 7 (Producing Effective Team Reports)
- 67. Using Evidence in Team Reports
- Use qualitative and quantitative evidence
- Select evidence carefully and purposefully
- Connect evidence to an assertion or question
- Analyze information; do not just set forth data
- Let evidence suggest improvements
- Use evidence that speaks to the institutions themes and the
team's questions
- 68. Team Use of the Standards and CFRs
- Team judgments must be linked to specific Standards and
CFRs
- CFRs must be cited in reports
- Standards and CFRs form the basis for Commission decisions
- Standards and CFRs provide a context for continuous quality
improvement
- 69. What is an effective team report?
- Reflects a thorough assessment of the institutions capacity,
preparation, and/or effectiveness
- Cites the Standards and CFRs
- Provides the basis for a sound and supportable Commission
decision
- Identifies important areas for institution to address
- 70. Tips for Writing Team Reports
- Consider multiple audiences: institution, Commission, and next
team
- Know your areas of responsibility, including length and depth
of your section
- Start writing before you arrive on campus
- Address priorities and goals set by the institution
- Address Commissions concerns (last action letter)
- Make commendations, but dont overdo it
- Use praise that doesnt send wrong or mixed signal
- 71. More Tips on Team Reports.
- Support findings and recommendations with evidence --and tie
them to CFRs
- Ensure evidence is sound and valid
- Distinguish recommendations from suggestions or
observations
- Use formal language and tone (e.g., not we/they)
- Dont mention personnel by name
- 72. Q&A
- Please feel free to type in your questions using the chat
window or just chime in.
- 73. After the Visit
- 74. What happens next?
- Assistant Chair prepares draft for Chair, team and staff
review; changes as needed
- Chair sends to institution for corrections of fact
- Chair finalizes draft and submits to WASC
- Chair sends Confidential Team Recommendation and completed EEF
to WASC
- WASC sends report to institution
- 75. Then
- Staff prepares draft action letter, which is reviewed by team
Chair
- Commission Panel reads report and documentation including
institutions written response, meets with institutional
representatives at Commission meeting
- Panel makes recommendation to Commission, and Commission
acts
- Staff finalizes draft action letter on behalf of
Commission
- 76. Also after the visit.
- Team members send reimbursement forms to WASC within 30
days
-
- Hotel arranged and paid directly by institution
-
- Rental car must be approved in advance by WASC staff
-
- Spouse or assistant costs not covered
-
- See policy for more details
- Team members should not have any contact with the
institution
-
- Consult with the institution for one year
- 77. The Teams Impact
- Peer review is the foundation of accreditation.
- The team report forms the basis for the Commission action and
its letter.
- The team report and action letter inform the work of the
institution for years to come.
- Why were you chosen for a team?
- 78. Resources for Teams
- Appendices of Visit Guide
- Team Materials and Institutional Report mailed 10-12 weeks in
advance of visit
- WASC Website: www.wascsenior.org
- WASC Email Advisory (sent prior to visit)
- 79. Q&A
- Please feel free to type in your questions using the chat
window or just chime in.
- 80. Thank you for your service to the region
- 81. Announcements
- The materials presented during this webinar and a recording of
this session will be posted at:
- http://www.wascsenior.org/spring2011