47
Self- Assessment Karen V. Mann PhD Fall Together’ Department of Family Medicine University of Calgary October 22 nd , 2013 1

Self- Assessment

  • Upload
    spiro

  • View
    54

  • Download
    1

Embed Size (px)

DESCRIPTION

Self- Assessment. Karen V. Mann PhD ‘ Fall Together’ Department of Family Medicine University of Calgary October 22 nd , 2013. Objectives:. Participants will: Explore the concept of self- assessment: how do people know “how they are doing”? - PowerPoint PPT Presentation

Citation preview

Page 1: Self- Assessment

Self- Assessment Karen V. Mann PhD

‘Fall Together’Department of Family Medicine

University of Calgary October 22nd, 2013

1

Page 2: Self- Assessment

Objectives:Participants will:Explore the concept of self- assessment: how do people know “how they are doing”?Discuss an evidence-based model for understanding informed self-assessment Discuss conditions that can facilitate informed self-assessment and use of external feedback.Consider the implications and apply these conditions and approaches to practical examples. Identify and develop practical educational strategies to enhance informed self-assessment and feedback adoption, both for learners and themselves.

2

Page 3: Self- Assessment

How do we know “how we are doing?” Consider how you make judgments about

your performance as a physician, or teacher: What sources of information do you use?

What challenges do you experience in assessing your own performance ?

3

Page 4: Self- Assessment

The self-assessment literature “in a majority of the relevant studies, physicians do not

appear to accurately self-assess …” (Davis, et al., 2006)

“…the most important educational activity related to self-assessment … should entail helping people overcome their personal belief that they can rely on it.” (Eva & Regehr, 2008)

What’s worse, those with the least proficiency in a domain are the least accurate self-assessors (Kruger, Dunning 1999; Violato, Lockyer 2006)

Page 5: Self- Assessment

Self-assessing on one’s own is a little like…

5

Background

Page 6: Self- Assessment

“The person and the situation; perspectives of social psychology” (Ross, Nisbett 1991):

1. Situational factors are powerful influences of behaviour.

2. It is actually the person’s perceptions of situational factors which are influential.”

3. The social system and individual cognition/ perceptions are in a dynamic state of flux. Therefore making inferences about individual behaviour is complex/ difficult.

6

Understanding informed self-assessment: Social psychological theory

Page 7: Self- Assessment

Understanding informed self-assessment: Constructivist learning theory Learning takes place in contexts Learners form or construct much of what they

understand as a function of their Experiences in situations Social interactions

7

Page 8: Self- Assessment

Understanding informed self-assessment: Humanist learning theory

Learning is personal growth, self-actualization Includes affective and cognitive domains Focuses on personal goals and accomplishments

Learning is self-directed (which assumes some degree of self-assessment)

Page 9: Self- Assessment

Honest and accurate external feedback is crucial to facilitate improvement and to accurate self-

assessment(Larsen, et al., 2008; Boehler, et al., 2007; Rees,Shepherd, 2005)

Page 10: Self- Assessment

Self-assessment: “a global judgement of one’s ability in a particular domain” (Eva & Regehr, 2010)

Informed Self-Assessment: “a set of processes through which individuals use external and internal data to generate an appraisal of their own performance”. (Sargeant et al, 2011)

Background: The terms we use

Page 11: Self- Assessment

Funding: Research grants: Medical Council of Canada Additional funding: ABIM, Dalhousie University

Research team: Joan Sargeant PhD, Dalhousie University (CA) Kevin Eva PhD, McMaster University (CA) Karen Mann PhD, Dalhousie University (CA) Heather Armson MD, University of Calgary (CA) Jocelyn Lockyer PhD, University of Calgary (CA) Elaine Loney, MSc, Qual Research Consultant (CA) Cees van der Vleuten, PhD, Universiteit Maastricht (NL) Timothy Dornan MD, PhD, Manchester University (UK) Eric Holmboe MD, Am Board Internal Medicine (USA) Benjamin Chesluk PhD, Am Board Internal Medicine (USA)

11

Informed self-assessment: Informed self-assessment: Summary of researchSummary of research

Page 12: Self- Assessment

12

Informed self-assessment: Informed self-assessment: Research objectives:

To understand - 1. how students, residents and physicians perceive

and use self-assessment in clinical learning and practice

2. their views of activities aimed at guiding self-assessment and factors that influence their use

3. how external feedback might be better tailored to the needs of health professionals

Page 13: Self- Assessment

13

Research design and methods International qualitative study using focus groups

drawing upon principles of grounded theory Purposive recruitment from identified programs

using specific approaches to enhance self-assessment

Page 14: Self- Assessment

14

Participating programs

Undergrad (students)

UK, 3rd yr. medical students NL, final yr. medical studentsBelgium, final yr midwifery students

Post grad (residents)

UK, Foundation Program trainees US, Internal medicine residents

Physicians CA, family doctors, MSF program (PAR) CA, family doctors, PBSG learning US, internal medicine, chart audit

Page 15: Self- Assessment

15

Results: Participants 17 focus groups

2 for each of 8 programs except 3 for Canadian MSF program

134 participants 53 undergraduate/ student learners (3 programs) 32 postgraduate/ resident learners (2 programs) 49 practicing doctors/ consultants (3 programs)

Page 16: Self- Assessment

16

Results – what we learned:

Self-assessment and “informing” self-assessment are complex activities

They are: multi-faceted contextualized dynamic non-linear

Page 17: Self- Assessment

External & Internal Conditions –e.g.External - learning climate,

relationshipsInternal - emotions, confidence, perceived credibility of feedback

Tensions - Within people

Between peopleWithin the learning environment

Sources of Information -

External – feedback, people,

processesInternal –

“self”

Interpretation of Information -

Reflect, compare,

calibrate, filter, assimilate

Responses to Information-

IgnoreRejectSeek

Accept

Dimensions of Informed Self Assessment

Page 18: Self- Assessment

External & Internal Conditions –e.g.External - learning climate,

relationshipsInternal - emotions, confidence, perceived credibility of feedback

Tensions - Within people

Between peopleWithin the learning environment

Sources of Information -

External – feedback, people,

processesInternal –

“self”

Interpretation of Information -

Reflect, compare,

calibrate, filter, assimilate

Responses to Information-

IgnoreRejectSeek

Accept

Dimensions of Informed Self Assessment

Page 19: Self- Assessment

19

1. Sources of information: external (FEEDBACK)

I feel like I’m … kind of working under my own steam, and thinking, “I hope I’m doing enough, doing the right thing”. I got a ‘satisfactory’ but it might be nice for someone to give a bit more feedback about how we’re actually doing, because sometimes you feel very alone. (Medical Student B3)

Page 20: Self- Assessment

20

1. Sources of information:external

Anytime that you have to interact with a colleague, be it at a consultant level, personal, group practice level or as a teacher, that always provides feedback. It is a general amalgamation of all those things that gives you an assessment of how you are doing. (Physician L8)

Page 21: Self- Assessment

21

1. Sources of information:internal

… you are sometimes aware when you are talking to patients that you are vague or are filling in gaps that shouldn’t be gaps. I think we all do that to come across as being coherent, but one should be able to recognize when one is doing that, and fill in the gaps... (Physician K3)

Page 22: Self- Assessment

External & Internal Conditions –e.g.External - learning climate,

relationshipsInternal - emotions, confidence, perceived credibility of feedback

Tensions - Within people

Between peopleWithin the learning environment

Sources of Information -

External – feedback, people,

processesInternal –

“self”

Interpretation of Information -

Reflect, compare,

calibrate, filter, assimilate

Responses to Information-

IgnoreRejectSeek

Accept

Dimensions of Informed Self Assessment

Page 23: Self- Assessment

23

2. Interpretation of information (REFLECTION)

If at the end of the day, if you haven’t asked yourself some questions, then that’s a problem. If you go through the whole day and think you know all about all the problems that faced you, then you don’t know. You have to ask yourself why, because you can’t know everything. (Physician M5)

Page 24: Self- Assessment

External & Internal Conditions –e.g.External - learning climate,

relationshipsInternal - emotions, confidence, perceived credibility of feedback

Tensions - Within people

Between peopleWithin the learning environment

Sources of Information -

External – feedback, people,

processesInternal –

“self”

Interpretation of Information -

Reflect, compare,

calibrate, filter, assimilate

Responses to Information-

IgnoreRejectSeek

Accept

Dimensions of Informed Self Assessment

Page 25: Self- Assessment

25

3. Responses to information

. . . somebody gave me a ‘below-standard’ score for something, and at first you’re like, ‘Oh, my life, I can’t believe it! I’m not fit to be a doctor!!” …then it actually makes you think, ‘yeah, okay, I am weaker in that area, I need to work on it’. And it does kind of give you a scare - ‘Okay, in future that’s something that I’m going to have to do.’ (Postgraduate E6)

Page 26: Self- Assessment

26

3. Responses to information

“But for some patients… I thought I’d been keeping up with things, and I found (from the audit) that actually, no… I am missing things.” (Physician P1)

Page 27: Self- Assessment

External & Internal Conditions –e.g.External - learning climate,

relationshipsInternal - emotions, confidence, perceived credibility of feedback

Tensions - Within people

Between peopleWithin the learning environment

Sources of Information -

External – feedback, people,

processesInternal –

“self”

Interpretation of Information -

Reflect, compare,

calibrate, filter, assimilate

Responses to Information-

IgnoreRejectSeek

Accept

Dimensions of Informed Self Assessment

Page 28: Self- Assessment

28

4. External and internal conditions/ influences: Climate

Because there is that level of trust within the group, I don't mind my peers knowing that I might not know the answer to something. I don't feel they would judge me … (Physician M2)

Page 29: Self- Assessment

29

External and internal conditions/ influences: Internal

“. . . When you are more confident, you can ask for more detailed feedback.” (Medical student H9)

Page 30: Self- Assessment

External & Internal Conditions –e.g.External - learning climate,

relationshipsInternal - emotions, confidence, perceived credibility of feedback

Tensions - Within people

Between peopleWithin the learning environment

Sources of Information -

External – feedback, people,

processesInternal –

“self”

Interpretation of Information -

Reflect, compare,

calibrate, filter, assimilate

Responses to Information-

IgnoreRejectSeek

Accept

Dimensions of Informed Self Assessment

Page 31: Self- Assessment

5. Tensions Overarching tension –”Between people”:

Wishing to learn and improve vs. wishing to appear knowledgeable and confident among peers and superiors

“Tensions in learning environment”: Providing genuine assessment feedback vs

“playing the evaluation game” “Tensions within self”

Wanting feedback yet fearing disconfirming information

31

Page 32: Self- Assessment

32

5. Tensions: Within people

“… there is this sense of intimidation that you don’t want to speak up in case you might just be deficient in your medical knowledge.” (Physician M3)

Page 33: Self- Assessment

33

5. Tensions: Within learning environment

“I know you’ve got to play the game, but that’s how I always feel when I’m doing it. I’m playing the game. I don’t feel like I’m learning and evolving and becoming a better person because I’ve done it.” (Postgraduate E2)

Page 34: Self- Assessment

External & Internal Conditions –e.g.External - learning climate,

relationshipsInternal - emotions, confidence, perceived credibility of feedback

Tensions - Within people

Between peopleWithin the learning environment

Sources of Information -

External – feedback, people,

processesInternal –

“self”

Interpretation of Information -

Reflect, compare,

calibrate, filter, assimilate

Responses to Information-

IgnoreRejectSeek

Accept

Dimensions of Informed Self Assessment

Page 35: Self- Assessment

Overarching finding: No one activity was generally effective in informing

self-assessment Effectiveness was moderated by external factors,

internal perceptions, tensions

The value of a specific activity was in how it was used, not in the activity itself

Summary

Page 36: Self- Assessment

Question for discussion What is your experience in helping learners to use self-assessment effectively?

What are the challenges? What have you found helpful?

36

Page 37: Self- Assessment

Discussion and activity

Considering our discussion of self- assessment, today, and the factors that influence it…

What strategies might you use with your learners?How would you incorporate it?

37

Page 38: Self- Assessment

Helping learners (1) Providing structured opportunities e.g., MSF, reflection on portfolios Modeling self-assessment Asking reflective questions – helping learners to

reflect on performance Providing effective feedback

38

Page 39: Self- Assessment

Information, provided by an agent (e.g., teacher, peer, book, parent, self, experience) regarding aspects of one’s performance or understanding.

Its goal: to identify the gap between current and desired performance and to identify a route to get there.

(Hattie & Timperley, 2007)

Effective Feedback

Page 40: Self- Assessment

Focus feedback on the task not the learner

Provide elaborated feedback to enhance learning

Give feedback in manageable units

Be specific and clear

Make feedback as simple as possible and no simpler(Shute, 2008)

40

Effective Feedback: What We Have Learned

Page 41: Self- Assessment

Zone of proximal development (Vygotsky) The distance from where a learner is currently

in their development to the level they can perform at with assistance.

The importance of scaffolding Scaffolding provides a framework for the

learner to achieve the task.

41

Effective feedback: two helpful concepts

Page 42: Self- Assessment

Helping learners (2) Assisting with identifying sources of information Promoting the seeking of feedback Facilitating the incorporation and use of feedback Recognizing the emotional content of feedback Addressing the tensions ( within people, between

people and in the environment)

42

Page 43: Self- Assessment

Tips: Educational strategies to enhance informed self-assessment

Selecting assessment & feedback strategies: Feedback (formal, informal) is required to inform self-

assessment To work, feedback needs to be timely, specific, credible,

authentic, provided by informed & engaged supervisors, guided by standards

Use of objective feedback data as possible; e.g., observed, chart audit, practice review

Assessment processes need to be credible, authentic, meaningful (not simply “jumping through another hoop”)

Mann et al , 2011 43

Page 44: Self- Assessment

Tips: Educational strategies to enhance informed self-assessment

Faculty development: Increase awareness that feedback (formal,

informal) is required to inform self-assessment Provide timely, relevant, specific, feedback Engage learners in their own self-assessments Practice providing constructive feedback that will

help learners see how they can improve Create a safe learning environment where it’s OK

to say “I don’t know” or ask for helpMann et al , 2011 44

Page 45: Self- Assessment

Tips: Educational strategies to enhance informed self-assessment

Learner/ physician development: Help them to identify valid sources of external performance

feedback Help them to develop skills in critically interpreting/ analyzing

the feedback data, and judging their performance Help them be aware of and manage their emotional response

to corrective feedback Teach skills for seeking specific feedback from supervisors that

will help them improve Empower learners to seek feedback Encourage physicians to seek feedback from colleagues

Mann et al , 2011 45

Page 46: Self- Assessment

Summary and next steps What will you take away?

Thank you!

46

Page 47: Self- Assessment

Informed self-assessment: Publications

 Sargeant J, Armson H, Chesluk B, Dornan T, Holmboe E, Eva K, Mann K, Lockyer J, van der Vleuten C, Loney, E. Processes and dimensions of informed self-assessment: A conceptual model. Acad Med. 2010; 85(7):1212-20.

Lockyer J, Armson H, Chesluk B, Dornan T, Holmboe E; Loney E, Mann K, Sargeant J. Feedback Data Sources that Inform Physician Self Assessment, Med Teach, 2011; 33(2): Pages e113-e120 (DOI: 10.3109/0142159X.2011.542519).

Sargeant J, Eva KW, Armson H, Chesluk B, Dornan T, Holmboe E, Lockyer J, Loney E, Mann K,. Features of assessment learners use for informed self-assessments of clinical performance. Med Ed 2011; 45(6): 636-647. (DOI: 10.1111/j.1365-2923.2010.03888.x)

Mann K, van der Vleuten C, Armson H, Chesluk B, Dornan T, Eva K, Holmboe E, Lockyer J, Loney E, Sargeant J. Tensions in informed self-assessment: How the desire for feedback and reticence to collect/use it create conflict . Acad Med 2011, 86 (9):1120-1127

Eva KW, Holmboe E, Lockyer J, Loney E, Mann K, Sargeant J. Factors influencing responsiveness to feedback: On the interplay between fear, confidence, and reasoning processes. Adv Health Sci Ed 2012 Mar;17(1):15-26