Upload
spiro
View
54
Download
1
Tags:
Embed Size (px)
DESCRIPTION
Self- Assessment. Karen V. Mann PhD ‘ Fall Together’ Department of Family Medicine University of Calgary October 22 nd , 2013. Objectives:. Participants will: Explore the concept of self- assessment: how do people know “how they are doing”? - PowerPoint PPT Presentation
Citation preview
Self- Assessment Karen V. Mann PhD
‘Fall Together’Department of Family Medicine
University of Calgary October 22nd, 2013
1
Objectives:Participants will:Explore the concept of self- assessment: how do people know “how they are doing”?Discuss an evidence-based model for understanding informed self-assessment Discuss conditions that can facilitate informed self-assessment and use of external feedback.Consider the implications and apply these conditions and approaches to practical examples. Identify and develop practical educational strategies to enhance informed self-assessment and feedback adoption, both for learners and themselves.
2
How do we know “how we are doing?” Consider how you make judgments about
your performance as a physician, or teacher: What sources of information do you use?
What challenges do you experience in assessing your own performance ?
3
The self-assessment literature “in a majority of the relevant studies, physicians do not
appear to accurately self-assess …” (Davis, et al., 2006)
“…the most important educational activity related to self-assessment … should entail helping people overcome their personal belief that they can rely on it.” (Eva & Regehr, 2008)
What’s worse, those with the least proficiency in a domain are the least accurate self-assessors (Kruger, Dunning 1999; Violato, Lockyer 2006)
Self-assessing on one’s own is a little like…
5
Background
“The person and the situation; perspectives of social psychology” (Ross, Nisbett 1991):
1. Situational factors are powerful influences of behaviour.
2. It is actually the person’s perceptions of situational factors which are influential.”
3. The social system and individual cognition/ perceptions are in a dynamic state of flux. Therefore making inferences about individual behaviour is complex/ difficult.
6
Understanding informed self-assessment: Social psychological theory
Understanding informed self-assessment: Constructivist learning theory Learning takes place in contexts Learners form or construct much of what they
understand as a function of their Experiences in situations Social interactions
7
Understanding informed self-assessment: Humanist learning theory
Learning is personal growth, self-actualization Includes affective and cognitive domains Focuses on personal goals and accomplishments
Learning is self-directed (which assumes some degree of self-assessment)
Honest and accurate external feedback is crucial to facilitate improvement and to accurate self-
assessment(Larsen, et al., 2008; Boehler, et al., 2007; Rees,Shepherd, 2005)
Self-assessment: “a global judgement of one’s ability in a particular domain” (Eva & Regehr, 2010)
Informed Self-Assessment: “a set of processes through which individuals use external and internal data to generate an appraisal of their own performance”. (Sargeant et al, 2011)
Background: The terms we use
Funding: Research grants: Medical Council of Canada Additional funding: ABIM, Dalhousie University
Research team: Joan Sargeant PhD, Dalhousie University (CA) Kevin Eva PhD, McMaster University (CA) Karen Mann PhD, Dalhousie University (CA) Heather Armson MD, University of Calgary (CA) Jocelyn Lockyer PhD, University of Calgary (CA) Elaine Loney, MSc, Qual Research Consultant (CA) Cees van der Vleuten, PhD, Universiteit Maastricht (NL) Timothy Dornan MD, PhD, Manchester University (UK) Eric Holmboe MD, Am Board Internal Medicine (USA) Benjamin Chesluk PhD, Am Board Internal Medicine (USA)
11
Informed self-assessment: Informed self-assessment: Summary of researchSummary of research
12
Informed self-assessment: Informed self-assessment: Research objectives:
To understand - 1. how students, residents and physicians perceive
and use self-assessment in clinical learning and practice
2. their views of activities aimed at guiding self-assessment and factors that influence their use
3. how external feedback might be better tailored to the needs of health professionals
13
Research design and methods International qualitative study using focus groups
drawing upon principles of grounded theory Purposive recruitment from identified programs
using specific approaches to enhance self-assessment
14
Participating programs
Undergrad (students)
UK, 3rd yr. medical students NL, final yr. medical studentsBelgium, final yr midwifery students
Post grad (residents)
UK, Foundation Program trainees US, Internal medicine residents
Physicians CA, family doctors, MSF program (PAR) CA, family doctors, PBSG learning US, internal medicine, chart audit
15
Results: Participants 17 focus groups
2 for each of 8 programs except 3 for Canadian MSF program
134 participants 53 undergraduate/ student learners (3 programs) 32 postgraduate/ resident learners (2 programs) 49 practicing doctors/ consultants (3 programs)
16
Results – what we learned:
Self-assessment and “informing” self-assessment are complex activities
They are: multi-faceted contextualized dynamic non-linear
External & Internal Conditions –e.g.External - learning climate,
relationshipsInternal - emotions, confidence, perceived credibility of feedback
Tensions - Within people
Between peopleWithin the learning environment
Sources of Information -
External – feedback, people,
processesInternal –
“self”
Interpretation of Information -
Reflect, compare,
calibrate, filter, assimilate
Responses to Information-
IgnoreRejectSeek
Accept
Dimensions of Informed Self Assessment
External & Internal Conditions –e.g.External - learning climate,
relationshipsInternal - emotions, confidence, perceived credibility of feedback
Tensions - Within people
Between peopleWithin the learning environment
Sources of Information -
External – feedback, people,
processesInternal –
“self”
Interpretation of Information -
Reflect, compare,
calibrate, filter, assimilate
Responses to Information-
IgnoreRejectSeek
Accept
Dimensions of Informed Self Assessment
19
1. Sources of information: external (FEEDBACK)
I feel like I’m … kind of working under my own steam, and thinking, “I hope I’m doing enough, doing the right thing”. I got a ‘satisfactory’ but it might be nice for someone to give a bit more feedback about how we’re actually doing, because sometimes you feel very alone. (Medical Student B3)
20
1. Sources of information:external
Anytime that you have to interact with a colleague, be it at a consultant level, personal, group practice level or as a teacher, that always provides feedback. It is a general amalgamation of all those things that gives you an assessment of how you are doing. (Physician L8)
21
1. Sources of information:internal
… you are sometimes aware when you are talking to patients that you are vague or are filling in gaps that shouldn’t be gaps. I think we all do that to come across as being coherent, but one should be able to recognize when one is doing that, and fill in the gaps... (Physician K3)
External & Internal Conditions –e.g.External - learning climate,
relationshipsInternal - emotions, confidence, perceived credibility of feedback
Tensions - Within people
Between peopleWithin the learning environment
Sources of Information -
External – feedback, people,
processesInternal –
“self”
Interpretation of Information -
Reflect, compare,
calibrate, filter, assimilate
Responses to Information-
IgnoreRejectSeek
Accept
Dimensions of Informed Self Assessment
23
2. Interpretation of information (REFLECTION)
If at the end of the day, if you haven’t asked yourself some questions, then that’s a problem. If you go through the whole day and think you know all about all the problems that faced you, then you don’t know. You have to ask yourself why, because you can’t know everything. (Physician M5)
External & Internal Conditions –e.g.External - learning climate,
relationshipsInternal - emotions, confidence, perceived credibility of feedback
Tensions - Within people
Between peopleWithin the learning environment
Sources of Information -
External – feedback, people,
processesInternal –
“self”
Interpretation of Information -
Reflect, compare,
calibrate, filter, assimilate
Responses to Information-
IgnoreRejectSeek
Accept
Dimensions of Informed Self Assessment
25
3. Responses to information
. . . somebody gave me a ‘below-standard’ score for something, and at first you’re like, ‘Oh, my life, I can’t believe it! I’m not fit to be a doctor!!” …then it actually makes you think, ‘yeah, okay, I am weaker in that area, I need to work on it’. And it does kind of give you a scare - ‘Okay, in future that’s something that I’m going to have to do.’ (Postgraduate E6)
26
3. Responses to information
“But for some patients… I thought I’d been keeping up with things, and I found (from the audit) that actually, no… I am missing things.” (Physician P1)
External & Internal Conditions –e.g.External - learning climate,
relationshipsInternal - emotions, confidence, perceived credibility of feedback
Tensions - Within people
Between peopleWithin the learning environment
Sources of Information -
External – feedback, people,
processesInternal –
“self”
Interpretation of Information -
Reflect, compare,
calibrate, filter, assimilate
Responses to Information-
IgnoreRejectSeek
Accept
Dimensions of Informed Self Assessment
28
4. External and internal conditions/ influences: Climate
Because there is that level of trust within the group, I don't mind my peers knowing that I might not know the answer to something. I don't feel they would judge me … (Physician M2)
29
External and internal conditions/ influences: Internal
“. . . When you are more confident, you can ask for more detailed feedback.” (Medical student H9)
External & Internal Conditions –e.g.External - learning climate,
relationshipsInternal - emotions, confidence, perceived credibility of feedback
Tensions - Within people
Between peopleWithin the learning environment
Sources of Information -
External – feedback, people,
processesInternal –
“self”
Interpretation of Information -
Reflect, compare,
calibrate, filter, assimilate
Responses to Information-
IgnoreRejectSeek
Accept
Dimensions of Informed Self Assessment
5. Tensions Overarching tension –”Between people”:
Wishing to learn and improve vs. wishing to appear knowledgeable and confident among peers and superiors
“Tensions in learning environment”: Providing genuine assessment feedback vs
“playing the evaluation game” “Tensions within self”
Wanting feedback yet fearing disconfirming information
31
32
5. Tensions: Within people
“… there is this sense of intimidation that you don’t want to speak up in case you might just be deficient in your medical knowledge.” (Physician M3)
33
5. Tensions: Within learning environment
“I know you’ve got to play the game, but that’s how I always feel when I’m doing it. I’m playing the game. I don’t feel like I’m learning and evolving and becoming a better person because I’ve done it.” (Postgraduate E2)
External & Internal Conditions –e.g.External - learning climate,
relationshipsInternal - emotions, confidence, perceived credibility of feedback
Tensions - Within people
Between peopleWithin the learning environment
Sources of Information -
External – feedback, people,
processesInternal –
“self”
Interpretation of Information -
Reflect, compare,
calibrate, filter, assimilate
Responses to Information-
IgnoreRejectSeek
Accept
Dimensions of Informed Self Assessment
Overarching finding: No one activity was generally effective in informing
self-assessment Effectiveness was moderated by external factors,
internal perceptions, tensions
The value of a specific activity was in how it was used, not in the activity itself
Summary
Question for discussion What is your experience in helping learners to use self-assessment effectively?
What are the challenges? What have you found helpful?
36
Discussion and activity
Considering our discussion of self- assessment, today, and the factors that influence it…
What strategies might you use with your learners?How would you incorporate it?
37
Helping learners (1) Providing structured opportunities e.g., MSF, reflection on portfolios Modeling self-assessment Asking reflective questions – helping learners to
reflect on performance Providing effective feedback
38
Information, provided by an agent (e.g., teacher, peer, book, parent, self, experience) regarding aspects of one’s performance or understanding.
Its goal: to identify the gap between current and desired performance and to identify a route to get there.
(Hattie & Timperley, 2007)
Effective Feedback
Focus feedback on the task not the learner
Provide elaborated feedback to enhance learning
Give feedback in manageable units
Be specific and clear
Make feedback as simple as possible and no simpler(Shute, 2008)
40
Effective Feedback: What We Have Learned
Zone of proximal development (Vygotsky) The distance from where a learner is currently
in their development to the level they can perform at with assistance.
The importance of scaffolding Scaffolding provides a framework for the
learner to achieve the task.
41
Effective feedback: two helpful concepts
Helping learners (2) Assisting with identifying sources of information Promoting the seeking of feedback Facilitating the incorporation and use of feedback Recognizing the emotional content of feedback Addressing the tensions ( within people, between
people and in the environment)
42
Tips: Educational strategies to enhance informed self-assessment
Selecting assessment & feedback strategies: Feedback (formal, informal) is required to inform self-
assessment To work, feedback needs to be timely, specific, credible,
authentic, provided by informed & engaged supervisors, guided by standards
Use of objective feedback data as possible; e.g., observed, chart audit, practice review
Assessment processes need to be credible, authentic, meaningful (not simply “jumping through another hoop”)
Mann et al , 2011 43
Tips: Educational strategies to enhance informed self-assessment
Faculty development: Increase awareness that feedback (formal,
informal) is required to inform self-assessment Provide timely, relevant, specific, feedback Engage learners in their own self-assessments Practice providing constructive feedback that will
help learners see how they can improve Create a safe learning environment where it’s OK
to say “I don’t know” or ask for helpMann et al , 2011 44
Tips: Educational strategies to enhance informed self-assessment
Learner/ physician development: Help them to identify valid sources of external performance
feedback Help them to develop skills in critically interpreting/ analyzing
the feedback data, and judging their performance Help them be aware of and manage their emotional response
to corrective feedback Teach skills for seeking specific feedback from supervisors that
will help them improve Empower learners to seek feedback Encourage physicians to seek feedback from colleagues
Mann et al , 2011 45
Summary and next steps What will you take away?
Thank you!
46
Informed self-assessment: Publications
Sargeant J, Armson H, Chesluk B, Dornan T, Holmboe E, Eva K, Mann K, Lockyer J, van der Vleuten C, Loney, E. Processes and dimensions of informed self-assessment: A conceptual model. Acad Med. 2010; 85(7):1212-20.
Lockyer J, Armson H, Chesluk B, Dornan T, Holmboe E; Loney E, Mann K, Sargeant J. Feedback Data Sources that Inform Physician Self Assessment, Med Teach, 2011; 33(2): Pages e113-e120 (DOI: 10.3109/0142159X.2011.542519).
Sargeant J, Eva KW, Armson H, Chesluk B, Dornan T, Holmboe E, Lockyer J, Loney E, Mann K,. Features of assessment learners use for informed self-assessments of clinical performance. Med Ed 2011; 45(6): 636-647. (DOI: 10.1111/j.1365-2923.2010.03888.x)
Mann K, van der Vleuten C, Armson H, Chesluk B, Dornan T, Eva K, Holmboe E, Lockyer J, Loney E, Sargeant J. Tensions in informed self-assessment: How the desire for feedback and reticence to collect/use it create conflict . Acad Med 2011, 86 (9):1120-1127
Eva KW, Holmboe E, Lockyer J, Loney E, Mann K, Sargeant J. Factors influencing responsiveness to feedback: On the interplay between fear, confidence, and reasoning processes. Adv Health Sci Ed 2012 Mar;17(1):15-26