Pieces of the Puzzle Perspectives on the Accreditation Visit……

Pieces of the Puzzle

Perspectives on the Accreditation Visit……

Central ThemeAccreditation and Assessment are

critically linked in contemporary accreditation models.

They were parallel themes in recent past but have “collided” to form a new accountability in higher education.

Ratcliff JL et al, How Accreditation Influences Assessment, Josey Bass 2001

How does accreditation and assessment come together?

Two key questions: How should learner outcomes be

demonstrated in the accreditation process? How should assessment information be

used to show improvement in programs, services, and student/resident learning?

Accreditation Purposes Fostering excellence through the development of

criteria and guidelines for assessing effectiveness

Encouraging improvement through ongoing self-study and planning

Ensuring external constituents that a program has clearly defined goals and appropriate objectives, maintains faculty and facilities to attain them, demonstrates it is accomplishing them, and has the prospect for continuing to do so


Accreditation Purposes Provides advice and counsel to new and

established programs in the accrediting process

Ensures that programs receive sufficient support and are free from external influence that may impede their effectiveness and the freedom of inquiry


Program Accreditation Needs to fulfill two dimensions of program

quality: There should be some evidence that the programs have

clear goals and courses of study to attain them The process should demonstrate that responsibilities

associated with the goals are being carried out. Peers interpret quality in the context of the

program’s own aims and activities Central issue: Program review is conducted as a self

study process. Central assumption: Evaluators from outside cannot

effectively examine and judge until those inside have reached a judgment about its quality.


Program Review – Audit Style Issues identified by “outside party” Buy-in from program director?

Reactive – Fear about the inspection

Approach – “cover the basics, get by this one and on to the next”

Not stimulated to think about “points of excellence” – only points of deficiency

Outcome from Inspection: Fear relieved – “get on to the next one.”

Yardstick: STANDARDS – Basic/Minimal

Program Review Assessment Style

Self Study and Peer assessment against target goals: Minimal standards = musts Growth goals Improvement goals Outcome goals

Buy-in for Program Director Proactive – ownership of goals

Outcome of Inspection External validation of internal assessment (agree or not) How much progress has been made against goals?

Yardstick: Goals achieved or improvement beyond minimal standards – Points of Excellence

Federal and Local Rules

OPTI StandardsBasic Standards

For

I R

Specialty College

Standards

Practicality?

Last Inspection?

Time?

Salary?

Patient Care?

Productivity?

Family?

What about the Residents?

How am I Doing as a PD?

Malpractice?



For

I R

Specialty College

Standards

YES!!

??? ???

What role?

What am I supposedTo do with this?



For

I R

Specialty College

Standards

Do they agree???

?

?

?

?

Which one takesprecedence?

Experience with the Audit Program Review….

Attempts to help program directors and DMEs to prepare for upcoming inspections.

Attempts to find “common areas” of support need across OPTI programs of same discipline

Based on BASIC STANDARDS – developed “Crosswalks” to assist in review of individual standards and collective standards.

Standard Number Standard Text Suggested Evidence Supporting Documentation

Standard Met (Yes or No?)

Standards Crosswalk and Worksheet

Specialty:______________Completed by:_____________Date:_____

# Standard Previously cited? Findings and Evidence Recommendations and Suggested Actions

Program Pre-Inspection FindingsHospital_______________Completed by:________________

Standards often do not: Consider practical issues

A complete research paper per year of residency (human subject approvals take almost half that time).

Ability of community hospital to afford “out rotations” since the BBA 1997

Ability of trainer to (again) teach subjects which were dropped from practice in time past.

Consider federal rules or procedures Institutional Review Board and HIPPA approval for

human subject research Documentation standards

Standards often do not: Reflect the need to include the following in the

curriculum and assess effectiveness: Ethics Communication skills

Require the assessment of outcomes of study Clinical skill development or OSCEs In Service Examination and Analysis (individual and

program wide). Progress exams or oral exams

Require that a program develop an ongoing quality assessment program and benchmarks for process and learning outcomes goals.

Experiences with Accreditation Visits – an OPTI view ..example: Program Review

40 “areas of concern” – standards not met in crosswalk

Affiliation agreements for major components not completed (for over 10 years)

Manual in disarray…never distributed to residents for past three years

PD mostly practices at alternate hospitals (where residents not welcome) for >70% time

Etc.

Inspector “Wonderful program

with dedicated PD and hospital”

No major deficiencies Recommend increase

in number of trainees Commendable PD

attention to program Recommend full five

year accreditation.

Net result:: “OPTI is too tough….see, We are OK!!”

Experiences with Accreditation Visits – an OPTI view ..example: Program Review

36 “areas of concern” on Standards Crosswalk

No resident manual in print/no written curriculum

Resident work hours = 2 weeks on call continuously

2 yr program required by PD/1 yr by specialty college….few recent trainees complete the two years. Certificates difficult to obtain.

Certification rates = <%50 pass rate on first attempt.

Inspector exit notes: “Excellent program

which should be a model for others”

This program should be expanded to train more residents.

No major deficiencies

Net result:: PD who had been working with OPTI to correct

findings has stopped. “We are doing fine….see how well we did on the inspection? Why won’t you

sponsor more positions?”

Experiences with Accreditation Visits – the “Inspector Factor”

Inspectors Often not familiar with Basic Standards for Interns and

Residents or OPTI standards Often spend as little as 4-6 hours in the entire process Do not “dig deep enough” to see issues Accept inspectors workbook without verifying content.

Inspectors workbooks and standards often do not reflect the same “intent” or priority for the issue in question – produce confusion Ranking of standards is evident by threat of automatic

probation if question answered “NO”. “Inter-rater reliability” seems low

Conclusion – Audit Review “Dissatisfaction” with end result

Gets through the current “audit” inspector approach

Fails to stimulate long term goals or real improvement

Something is missingMay promote a punitive rather than a

supportive atmosphere –

“Begin with the end in mind”- What is the goal? Get from one accreditation visit to the next?

Minimal level thinking Threshold is the lowest possible point to keep the

program afloat (?) Continuous improvement over time against

declared goals. Determine a base line Set realistic but challenging goals Measure progress….set new goals (meeting minimal thresholds is never a problem)

Goal –get from one accreditation visit to the next?

Data driven Minimalist thinking “audit mentality” Promotes bashing and

blaming Sets interested parties at

odds Promotes a pessimistic

environment and inhibits creative thinking

Goals are at the minimal threshold – “Basic”

Goal – Assessment for CQI and Outcomes Markers of

Excellence “Internalized goals”

born in self study and assessment

“Buy-in” for all stakeholders

Goals for the Workshop Recreate the “Basic Standards” to allow and

encourage growth, goal setting and a reach for higher goals.

Revise and recreate basic standards to ensure basic elements are present and to foster agreement between: External standards/Internal standards External validation/Internal validation

To foster better understanding of the “fit” between the AOA standards and laws which apply to residency training.


OPTI Standards

Basic StandardsFor

I R

Specialty College

Standards

Goal

Documents

Pieces of the Puzzle Perspectives on the Accreditation Visit……