Welcome

The National Cooperative Agreement on

Clinical Workforce Development

Presented by the

the Community Health Center, Inc.

& the National Nurse-Led Care Consortium

WEBINAR 4: How did WE do? Evaluating the Student Experience

June 6, 2017

Let’s Review..

Why Start or Expand Student Experience Programs at Your Health Center?

Creating a Process that Works for YOU: Infrastructure for a Successful Student Training Program

Spring 2016 #2Spring 2016 #1

How to Make it Work for the Students

Spring 2016 #3

How to Create Life Changing Experiences for Students

Spring 2016 #4

Developing a Student Training Process: On-Boarding and Orientating to Your Health Center Community

Spring 2017 #1

Team Based Care 101 for Health Professions Students

Spring 2017 #2Spring 2017 #3

Enhancing the Student Experience Through Effective Precepting

How did WE do? Evaluating the Student Experience

Spring 2017 #4

SpeakersFrom Community Health Center, Inc.:

Anna Rogers, Director of the National Cooperative Agreement

Reema Mistry, MPH, Project Coordinator, National Cooperative Agreement

Amanda Schiessl, Interprofessional Student Coordinator

From National Nurse-Led Care Consortium:

Kristine Gonnella, MPH, Director for Training and Technical Assistance

Cheryl Fattibene, Chief Nurse Practitioner Officer

From UPENN School of Nursing

Carrie Doherty, Director of Implementation-CMS/GNE Demonstration

From Community College in Philadelphia:

Laureen Tavolaro-Ryler, MSN CNS

From the National Nurse Practitioner Residency and Fellowship Training Consortium

Candice Rettie, PhD, Executive Director

Learning Objectives:

1. Participants will describe two key components for a

student to successfully interface with HR in a FQHC

environment.

2. Participants will identify two reasons why providing

evaluative feedback will enhance the student training

program

Get the Most Out of Your Zoom Experience

• Send your questions using Q&A function in Zoom

• Look for our polling questions

• Recording and slides are available after the presentation on our website within

one week www.chc1.com/nca

Program Evaluation: Improving Performance

Presented by Candice S. Rettie, PhD NCA Webinar June 2017

Overview of the SessionDefinitions and Process of Good Program Evaluation

Designing Meaningful Evaluation for Your FQHC Educational Program– Crosswalk evaluation with curriculum– Integrated throughout the learning experience– Creates explicit expectations for learner– Documents programmatic success– Fosters improvement positive growth, creativity and innovation

Characteristics of Useful Evidence

Learning ObjectivesKnowledge:

– Understand the purpose of evaluation– Know the characteristics of good evaluation– Understand the process of evaluation– Understand the connection with curriculum

Attitude:– Embrace the challenge– Value the outcomes

Skills– To be gained by partnering with local experts and

collaborate focusing on local program

Definitions:Evaluation: systematic investigation of merit, worth, or significance of effort; Program evaluation: evaluate specific projects and activities that target audiences may take part in;Stakeholders: those who care about the program or effort.

Approach: practical, ongoing evaluation involving program participants, community members, and other stakeholders.

Importance: 1. Helping to clarify program plans;2. Improving communication among participants and partners;3. Gathering the feedback needed to improve and be accountable for program

outcomes/effectiveness;4. Gain Insight about best practices and innovation;5. Determine the impact of the program;6. Empower program participants and contribute to organizational growth.

1. Develop a Written Plan Linked to Curriculum2. Collect Data3. Analyze Data4. Communicate and Improve

4 Basic Steps to Program Evaluation

Fitting the Pieces Together: Program Evaluation

Program Curriculum

PreceptorFacultyStaff

Learner

Institution

Overall Program

Crosswalk Curriculum and Evaluation

Program Evaluation Feedback Loops

Learner performance

Instructor and staff performance

Program curriculum performance

Programmatic and Institutional performance

Evaluation Process How Do You Do It?

Steps in Evaluation:

Engage stakeholders

Describe the program

Focus the evaluation design

Gather credible evidence

Justify conclusions Analyze, synthesize and interpret findings, provide alternate explanations

Feedback, follow up and disseminate Ensure use and share lessons learned

Level 1: Reaction (Satisfaction Surveys) Was it worth time; was it successful? What were biggest strengths/weaknesses? Did they like physical plant?

Level 2: Learning (Formative: Performance Observation/checklist; Monitored skill demonstration) Observable/measurable behavior change? Can they teach others/patients?

Level 3: Behavior (Observations/interviews; 3-12 weeks later) New or changed behavior after the program? Can they teach others? Are trainees aware of change?

Level 4: Results (Program Goals/Institutional goals; 3-12 mo later) Improved employee retention? Increased productivity for new employees? Higher morale?

Kirkpatrick Model of Evaluation

Relevance of Evaluation

• What will be evaluated?

• What criteria will be used to judge program performance?

• What standards of performance on the criteria must be reached for the program to be considered successful?

• What evidence will indicate performance on the criteria relative to the standards?

• What conclusions about program performance are justified based on the available evidence?

Questions Guiding the Evaluation Process

Credible evidence -- Raw material of a good evaluationBelievable, trustworthy, and relevant answers to evaluation questions

Indicators (evidence)Translate general concepts about program and expected effects into specific, measurable parts. (eg: increase in patient panel / billable hours over 1 year)

SourcesPeople, documents, or observations (eg: trainees, faculty, patients, billable hours, reflective journals). Use multiple sources -- enhances the evaluation's credibility. Integrate qualitative and quantitative information -- more complete and more useful for needs and expectations of a wider range of stakeholders.

QuantityDetermine how much evidence will be gathered in an evaluation. All evidence collected should have a clear, anticipated use.

LogisticsWritten Plan: Methods, timing (formative and summative), physical infrastructure to gather/handle evidence.Must be consistent with cultural norms of the community, must ensure confidentiality is protected.

Goals of Good Evaluation

• Purposeful, systematic

• Formative (on-going) and summative (final) data collection

• Conduct quality assurance

• Determine training effectiveness

• Communicate results

• Build capacity

Take Away Points:Partner with your local university education experts

Crosswalk the evaluation with the curriculum• Anchored in the goals and objectives of the curriculum

Accuracy, Utility, Feasibility, Propriety• Measurable and observable criteria of acceptable performance

• Multiple, expert ratings/raters: Multiple observations give confidence in findings and provide an estimate of reliability (reproducibility or consistency in ratings).

• Validity achieved with synthesis of measurements that are commonly accepted, meaningful, and accurate

Formative and summative

Conclusions need to be relevant and meaningful

The Community Tool Box, (Work Group for Community Health at the U of Kansas): incredibly complete and understandable resource, provides theoretical overviews, practical suggestions, a tool box, checklists, and an extensive bibliography.

Pell Institute: user-friendly toolbox that steps through every point in the evaluation process: designing a plan, data collection and analysis, dissemination and communication, program improvement.

CDC has an evaluation workbook for obesity programs; concepts and detailed work products can be readily adapted to NP postgraduate programs.

Another wonderful resource, Designing Your Program Evaluation Plans, provides a self-study approach to evaluation for nonprofit organizations and is easily adapted to training programs. There are checklists and suggested activities, as well as recommended readings.

NNPRFTC website – blogs: http://www.nppostgradtraining.com/Education-Knowledge/Blog/ArtMID/593/ArticleID/2026/Accreditation-Standard-3-Evaluation

Resources:

POLLING QUESTION

Do you use an evaluation tool to assess the student training experience at your health center?

How did WE do? Evaluating the Student

Experience

LAUREEN TAVOLARO - RYLEYCOMMUNITY COLLEG E OF PHILA DELPHIA

CH ERYL FAT TIBENE AND KRIST INE G ONNELLANATI ONAL NURSE - LED CARE CONSORTIUM

CAROLINE D OUG H ERTYUPENN S C HOOL OF N URS I N G

National Nurse-Led Care Consortium (NNCC)

About us:

Membership organization that supports nurse-led care and nurses at the front lines of care.

Supported by the Health Resources and Services Administration (HRSA) under grant number U30CS09736, a National Training and Technical Assistance Cooperative Agreement (NCA).

Focus of discussion:

1. Discuss how we identify and assess successes and challenges in our student training program?

2. How do we adjust processes to improve upon the student training program and experience?

CMS GNE Demonstration Project

Goals:

• Test feasibility, effectiveness, and cost of increasing APRN production with Medicare payment model

• Increase APRNs in primary care settings, underserved areas and providing care to Medicare beneficiaries

• Increase clinical training opportunities

Evaluating the Student Experience1. Identify current process

• Initial interface• Administrative onboarding• Clinical onboarding• Training• Off-boarding

2. Evaluate steps of process: What is and isn’t working?• Informal discussions• Focus groups• Surveys• Observations

3. Communicate outcomes of evaluation• Provider• Administrative• Clinic staff• Student• Academic partnership

4. Adjust process based on outcomes

Survey:

Student:

1. This experience was different than my other clinical rotations. yes/no, If yes: how was it different?

2. I would recommend this experience to my classmates.

3. Please rate this training experience (1-5)

4. I felt more likely to ask questions during this rotation than others.

5. I received more feedback on my performance during this rotation than others.

6. Suggestions for future semesters

Survey

Preceptor:

1. This experience was different than my precepting experiences in the past. If yes, how was it different?

2. Please rate this training experience (1-5)

3. This experience allowed me to spend more time with my patients compared to when I have had a student with me in the past.

4. This experience makes me more likely to commit to precepting a student in future semesters.

5. Suggestions for future semesters

STUDENT

EXPERIENCE

Identify

Process

Evaluate

Process

Communicate

Outcomes

Adjust

Process

Evaluating the Student Experience

Community College of Philadelphia: 19130 Zip Code Project

Quality Safety and Education for Nursing (QSEN)

Community College of Philadelphia: 19130 Zip Code Project

• Clear Objectives with Community Partners

• Clear Objectives with Students

• Measurement of the Objectives

• Share Documents

• End of Semester Wrap up

Community College of Philadelphia: 19130 Zip Code Project

• Continuous Evaluation of Service Learning

• Assessment of the Community

• Curriculum Review• Shift in Community Based

Care

Reminders

Complete our survey!

Sign up at www.chc1.com/NCA