Assessing Your Learner

Lawrence R. Schiller, MD, FACGDigestive Health Associates of Texas

Baylor University Medical Center, Dallas

Why do we assess trainees?

• It is an essential part of the learning process– Teacher needs to know what needs to be taught– Learner needs to know strengths to build on,

weaknesses to be corrected• It provides “quality control” for education– Effectiveness of teaching– Appropriateness of content

Education Science• Cognitive psychology and neuroscience

provide insights into learning process– Principles for learning– “Experience is the best teacher” (if guided)– Learning with understanding • More than memorizing facts• Based on pre-existing knowledge

– Active learning• Learner is invested in process• Learner takes control of own education

Education Science• Cognitive psychology and neuroscience

provide insights into learning process– Principles for learning– “Experience is the best teacher” (if guided)– Learning with understanding • More than memorizing facts• Based on pre-existing knowledge

– Active learning• Learner is invested in process• Learner takes control of own education

EVIDENCE-BASED TEACHING

Skills-attainment Continuum

NoviceAdvanced beginner

CompetentProficient

Expert

After Dreyfus

How experts differ from novices

• Experts notice features and meaningful patterns of information that are not noticed by novices

• Experts have acquired content knowledge that is organized with deep understanding of field

• Experts’ knowledge reflects contexts of applicability (conditioned by circumstances)

• Experts can retrieve knowledge with little attentional effort

Adaptive Expertise

• “Metacognition”: thinking about how you are thinking about a given problem

• “Internal conversation” focusing on what additional information is needed for effective decision-making

• Self-assessment of how one is doing in dealing with a clinical issue

Skills-attainment Continuum

NoviceAdvanced beginner

CompetentProficient

Expert

After Dreyfus

Elements of Competence

• Deep foundation of factual knowledge• Conceptual framework for applying

knowledge• Organization of knowledge to facilitate

retrieval and application• (Adequate technical/motor skills)

Types of Assessment

• Formative (feedback)– Measure of knowledge and/or skills

• Summative (appraisal/global assessment)– Integrated review of all aspects of work– May incorporate targets for future performance

Purposes of Assessments

• Evaluating pre-existing knowledge

• Monitoring learning• Determining level of

achievement• Profiling strengths and

weaknesses

• Assuring that predetermined minimal qualifications are met

• Deciding about promotion or repetition of course

• Selection of best candidates for program

FORMATIVE SUMMATIVE

Educational Objectives

• Expected results of learning, not methods of instruction– Should be observable, measurable, feasible

• Standards for assessment should be set ahead of time and discussed with learner– Norm-referenced (comparison to peers)– Criterion-referenced (absolute standard)– Limen-referenced (minimal threshold set)

Miller, 1990

Assessment Tools

“No single assessment method can provide all the data required for judgment of anything so complex as the delivery of professional services by a successful physician.”

Assessment Tools

• Validity– The extent to which an assessment measures

what it is supposed to measure• Domains of validity– Content validity– Concurrent validity– Predictive validity– Construct validity– Face validity

Assessment Tools• Written tests– Progress testing – Mastery testing

• Oral examinations• In-training assessment• Objective structured clinical exam (OSCE)• Global rating scales– Behaviorally anchored scales– Likert scales

Workplace-based Assessments

• Checklists• Milestones/Logs• Mini-clinical evaluation

exercise (mini-CEX)• Multisource feedback

(360o evaluations)• Directly observed

procedural skills (DOPS)• Procedure-based

assessments (PBA)

• Case-based discussions• Significant event

analysis• Chart audits• Patient satisfaction

surveys• Review of recorded

patient visits• Portfolio reviews

ACGME General Competencies

• Patient Care• Medical Knowledge• Practice-based Learning and Improvement• Interpersonal and Communication Skills• Professionalism• Systems-based Practice

Review with Learners

• Plan for the session– Collect data– Organize thoughts– Review standards against which trainee is to be judged

• Decide whether assessment is formative or summative

• Consider emotional intelligence aspects– The capacity for recognizing our own emotions and those

of others for motivating ourselves and others, and for managing emotions well in ourselves and others to enhance work performance

Emotional Intelligence

• EI Competencies are learned capabilities• Self-Awareness– Emotional self-awareness, accurate self-

assessment, self-confidence• Self Management– Self-control, transparency, optimism, adaptability,

orientation to achievement, initiative• Social Awareness– Empathy, organizational awareness, altruism

Emotional Intelligence

• Relationship Management– Developing others– Inspirational leadership– Influence decisions of others– Catalyze change– Manage conflict– Promote teamwork and collaboration

Emotional Intelligence

• Relationship Management– Developing others– Inspirational leadership– Influence decisions of others– Catalyze change– Manage conflict– Promote teamwork and collaboration

YOU NEED TO KNOW YOUR LEARNER

Review with Learners• Maintain a relaxed, positive atmosphere• Private and confidential; timely• Descriptive rather than judgmental• Focus on observed behaviors, not intentions or

interpretations; be specific rather than general• Involve learner in dialogue, listen to what

learner says (it provides the basis for action).• Help the learner understand any problems and

make plans to address issues in the future

Questions?

Summary

• Assessment is the process of measuring an individual’s performance during education

• Assessments can be – Formative: designed to improve performance– Summative: designed to assess competence

• Timely and appropriate assessment shared honestly with the learner is the best way to improve future performance