Embed Size (px)
Citation preview
Reliability and Validity of the Disruptive Behavior
Instrument for Physicians with Disruptive Behavior
Philip Hemphill, Ph.D.Professional Enhancement Program (PEP)
Pine Grove Behavioral Health and Addictive Services
FSPHP Presenter Disclosure Slide
“Reliability and Validity of the Disruptive Behavior Instrument for
Physicians with Disruptive Behavior”
Philip Hemphill, PhD
is a paid consultant of
Pine Grove Behavioral Health and Addiction Services
FSPHP Annual Conference and Meeting
Fort Worth, TexasApril 23-26, 2012
Why use a MSF/360?1. Facilitate insight of strengths and
weaknesses for an individual.2. Enhance organizational cultural
change.3. Provide summative assessment of
performance.4. Evaluate potential.5. Improve team effectiveness.6. Identify training & coaching needs.7. Make organizational values explicit.8. Measure progress after training &
coaching.
MSF/360 Essentials Proven validity & reliability. Useful in assessing “humanistic” skills. Weak or no associations between self-rated
assessment and external assessment.
“…methods such as multisource feedback (360) evaluations may bea necessary next step, particularly when interpersonal, communication
skills, or professionalism needs to be evaluated (p. 1101).”
Source: Davis et al. (2006). Accuracy of physician self-assessment compared with observedMeasures of competence.JAMA, 296 (9), 1094-1102.
Key Players in MSF/360
TargetIndividual
Raters Boss
Facilitator Vendor
MSF/360 Internal Process
Feedback
Is it consistent with self-perception?
Emotional Response
Reflection and internal processing of emotion and content.
Decision to accept and act, or not
Action for learning, changeSource: Sargeant et al. (2009)Advances in Health Sc Ed.
Relevant Literature Review of Multi-Source Feedback Measures
The Fundamental Interpersonal Relations Orientation–Behavior™ (FIRO-B®) In the late 1950s William Schutz, PhD. developed the
FIRO-B theory to aid in the understanding and predicting of how high-performance military teams would work together.
In developing the FIRO-B theory, Schutz began with the premise that "people need people." He used the term interpersonal to indicate any interaction, real or imagined, occurring between people. He used the term need to describe a psychological condition that, if not satisfied, leads to a state of discomfort or anxiety.
He posited that interpersonal needs could be grouped into three categories: Inclusion, Control, and Affection. The FIRO-B model describes the interaction of these three categories of interpersonal need along two dimensions: expressed and wanted.
Awareness Scales and were designed primarily to help individuals with their self-awareness and of understanding their relation to other people (Schutz, 1978).
Relevant Literature Review of Multi-Source Feedback Measures
1. Developing insight into strengths and weaknesses, for the individual (Farh & Dobbins, 1989; Hazucha et al., 1993; Martocchio & Judge, 1997; Fletcher, 1999; Keeping et al., 1999; London et al., 1999; McCarthy & Garavan, 1999) and the organization (London & Beatty, 1993; Lepsinger & Lucia, 1998; Steensma et al., 1998; Tornow & London,1998) to act upon.
2. Enhancing culture change (Cardy & Dobbins, 1994; O’Reilly, 1994; Crossley & Taylor, 1995; Church & Bracken, 1997; Bracken et al., 1998; DeSimone, 1998).
3. Summative assessment of performance (Newble et al., 1999).
4. Evaluating the potential of individuals, for instance to use in career advice or selection (Tornow, 1993; Lindsey et al.,1997; Facteau et al., 1998).
5. Enhancing team effectiveness (by allowing opportunity to comment) (Garavan et al., 1997; Towers-Perrin, 1998).
6. Identifying training needs for the system (as opposed to the individual) (Towers-Perrin, 1998).
Development of Disruptive Behavior Instrument
2004 – Developed list of disruptive behaviors and boundary violation behaviors. Approximately 70 items…
5 people monitored over 12 months – Instrument too long; complaints from participants; descriptive data….
2005- Cut instrument down to 50 items(1 Fctr)
2008- Cut down to current version Been using current version for past 4
years…
Definition of Disruptive Behavior
Disruptive behavior is any inappropriate behavior, confrontation, or conflict, ranging
from verbal abuse to physical or sexual harassment. Disruptive behavior causes strong
psychological and emotional feelings, which can adversely affect patient care.
Rosenstein A, O’Daniel M. (2008). Managing disruptive physician behavior: Impact on staff relationships. Neurology, 70, 1564-1570.
Disruptive Behavior Items Avoids taking responsibility for actions Inappropriately blames others Places other interests above the safety and
welfare of community members Suspects, without sufficient basis, that others are
exploiting, harming, deceiving, or out to “get” him
Seems to have a preoccupation with unjustified doubts about the loyalty or trustworthiness of others
Reads hidden, demeaning, or threatening meanings into benign remarks or events
Perceives attacks on his character/reputation not apparent to others and is quick to react angrily/counterattack
Shows rigidity and stubbornness Displays an unpredictable mood Responds to constructive criticism in an
immature/irresponsible fashion Takes too long to requests from others Arrives late Fails to complete paperwork in a timely manner Responds to requests in an avoidant, unreliable,
and/or uncooperative manner
Displays a sense of entitlement Shows perfectionism that interferes with task
completion Behaves in a sexually inappropriate manner (verbally or physically)
Lacks empathy Intimidates others or attempts to intimidate others
Exhibits physically aggressive or assaultive behaviors
Exhibits arrogant behaviors and/or has an arrogant attitude
Demonstrates an inflated sense of self-importance
Takes advantage of others in order to get what he wants
Is reluctant to delegate tasks or to work with others unless they submit to exactly his way of doing things
Expresses anger/hostility inappropriately Exhibits passive-aggressive behavior Uses abusive, belittling, condescending,
demeaning, and/or threatening language
Client signs release allowing for electronic
correspondence, and client given monitoring
instructions and materials with due date.
Client identifies MC,
gives MC instructions and rating
materials, and with MC identifies
approx. 15 raters..
Ratee email addresses
Collection of surveys online.
Rating process is
repeated at predetermine
d intervals until four
uneventful reports
Client, MC, and other relevant
stakeholders review the feedback report
Significant impairment is present and
PHP/LAP intervenes
CLIENT SIGNS/ UPDATES
MONITORING CONTRACT WITH PHP/LAP AND/OR
No significant impairment is
present
Client attends
treatment and rating process is
temporarily suspended
PHP/LAP recommends interventions
Client participates
in interventions
while continuing
rating process
PHP/LAPrecommends
intensive/residential
treatment
Client completes treatment
Report produced and
emailed to persons
indicated on release.
MONITORINGRATING FLOWCHART
33 questions 28 Likert-type questions regarding ratee’s
behavior 1 question concerning how many hours rater
works with ratee 1 question about the rater’s title 3 open-ended questions that solicit
comments
Client 43 Overall Ratings Over Time
0.0
5.0
10.0
15.0
20.0
25.0
30.0
35.0
1 2 3 4 5
Rating Period
Ove
rall
Sco
re
Aggression: Verbal Blaming Others Disobeying Rules Lack of Opennes to Others Response to Criticism
Client 109 Overall Ratings Over Time
0.0
5.0
10.0
15.0
20.0
25.0
30.0
35.0
1 2 3 4 5Rating Period
Ove
rall
Sco
re
Aggression: Verbal Blaming Others Disobeying Rules
Lack of Openness to Others Response to Criticism Sexual and Inappropraite Remarks
Survey Dimensions Appropriate
Documentation Arrogance Citizenship Compliance Empathy Exploitation of Others Expression of Anger Intimidation Job Attitudes Multicultural
Sensitivity Openness to Others Perfectionism Respect for Others Response to Criticism
Responsibility for Actions
Rigidity Satisfaction with Ratee Sexual
Appropriateness Stability of Mood Stress Management Supervision Suspiciousness Tardiness Team Impact Threatening Behavior Verbal Aggression Working with Others
Disruptive Behavior Instrument
Descriptive Characteristics of Sample: N = 22 ratees (498 surveys)
Average Age = 38 years Male = 19 ; Female = 3
Survey Breakdown by rater: Self-Report = 53 Staff = 328 Supervisor = 37 Peer = 79
Number of Times each patient was assessed: Range – 1-8 times
Disruptive Behavior Instrument
Aggression• Aggression: Non-
verbal• Aggression: Physical• Aggression: Verbal• Aggression: WrittenInterpersonal Difficulties• Intimidation• Lack of Empathy• Lack of Multicultural
Sensitivity• Lack of Openness to
Others• Problematic
Perfectionism• Response to Criticism• Rigidity• Suspiciousness
Sexually inappropriate behavior• Sexual Inappropriateness• Sexual/Inappropriate
RemarksSelf Regulation• Mood Instability• Poor Stress
Management
Disruptive Behavior• Arrogance• Blaming Others• Disobeying Rules• Disrespect for
Staff/Colleagues• Ease of Working with
Ratee• Employee Turnover
Intentions• Exploitation of Others• Poor Citizenship• Tardiness
• Originally hypothesized to be based
on 5 factors
Disruptive Behavior Instrument
Results of hypothesized 5-factor loadings; 59% cumulative total variance explained
1 2 3 4 5Q11 .746 .014 .000 .118 .293
Q12 .746 -.134 -.006 -.095 .036
Q4 .733 .080 -.023 .249 -.110
Q6 .732 -.055 .023 -.120 .267
Q5 .726 -.194 -.040 -.060 .064
Q1 .661 -.261 -.115 -.049 -.159
Q3 .620 -.306 -.183 -.033 -.194
Q23 .616 -.056 .057 -.095 .257
Q7 .560 -.210 .171 -.039 .017
Q17 .550 -.349 -.090 -.054 .014
Q14 .515 .058 .267 .204 -.345
Q2 .513 -.257 -.255 .102 -.090
Q9 .510 -.335 .238 -.066 .024
Factor 1
1 2 3 4 5
Q24.207 .702 .195 -.417 -.288
Q15-.179 .693 -.093 .108 .005
Q18-.114 .683 -.211 -.032 -.054
Q13-.136 .672 .115 -.050 .000
Q21-.163 .633 -.222 .077 -.019
Q20-.209 .622 -.179 .021 .134
Q8-.187 .574 .025 -.079 .140
Q16-.157 .546 -.236 .026 -.002
Factor 2
1 2 3 4 5
Q26-.482 -.315 .702 -.063 -.029
Q10.235 .100 .632 .201 .034
Q19.083 -.321 .431 .141 -.011
Factor 3
1 2 3 4 5Q25 .160 .091 .212 .718 .095
Q28 -.107 -.102 .049 .638 -.076
Factor 4
1 2 3 4 5Q22 .109 .078 .011 .025 .860
Factor 5
Raw Data Eigenvalues, Mean & Percentile Random Data Eigenvalues
Ncases 498; Nvars 27; ; Ndatsets 1000; Percent 95 Root Raw Data Means Percentile1.000000 10.462501 1.456515 1.5280562.000000 1.836325 1.386788 1.4370433.000000 1.408055 1.335702 1.3800894.000000 1.231678 1.291131 1.3312215.000000 1.178393 1.253103 1.2879056.000000 .966591 1.216903 1.2475327.000000 .890236 1.183359 1.2121078.000000 .805858 1.152611 1.1805259.000000 .786433 1.121760 1.14696810.000000 .707731 1.092700 1.11756811.000000 .671235 1.064358 1.08790412.000000 .647008 1.037406 1.06168113.000000 .567647 1.010437 1.03366114.000000 .518775 .984026 1.00712015.000000 .514043 .958262 .98167816.000000 .454498 .932558 .95516717.000000 .414122 .907291 .92983718.000000 .395714 .882943 .90681119.000000 .363423 .857912 .88259420.000000 .337990 .832012 .85569221.000000 .312618 .806413 .83104722.000000 .294869 .780470 .80649623.000000 .281812 .753717 .78018024.000000 .268155 .725304 .75336825.000000 .258453 .695053 .72401726.000000 .230247 .661538 .69425027.000000 .195588 .619728 .658277
Disruptive Behavior Instrument
Recent analyses reveal that the items are loaded on three distinct factors
1 2 3Q12 .780 -.068 -.031
Q5 .770 -.099 .002
Q4 .763 .194 .128
Q11 .742 .054 .305
Q1 .740 -.085 -.162
Q6 .732 -.051 .117
Q3 .717 -.077 -.194
Q17 .618 -.210 -.054
Q23 .609 -.079 .132
Q2 .603 .008 -.044
Q7 .575 -.254 .026
Q9 .528 -.416 .016
Q14 .522 -.007 -.006
Factor 1
1 2 3
Q18-.163 .727 -.066
Q21-.208 .697 .029
Q15-.251 .666 .098
Q20-.275 .628 .098
Q16-.192 .621 -.005
Q19.066 -.531 .171
Q13-.234 .496 .028
Q24.125 .487 -.398
Q8 -.276 .443 .073
Factor 2
1 2 3
Q25.127 .043 .636
Q22.025 -.034 .604
Q28-.089 -.034 .408
Q10.144 -.276 .319
Factor 3
* Excluded Item 26
1. Aggression: Non-Verbal Interpersonal Aggression2. Aggression: Physical Interpersonal Aggression3. Aggression: Verbal Interpersonal Aggression4. Aggression: Written Interpersonal Aggression5. Arrogance Interpersonal Aggression6. Blaming Others Interpersonal Aggression7. Disobeying Rules Interpersonal Aggression8. Disrespect forStaff/Colleagues
Professionalism
9. Ease of Working With Ratee Interpersonal Aggression10. Employee TurnoverIntentions
Critical Item
11. Exploitation of Others Interpersonal Aggression12. Intimidation Interpersonal Aggression13. Lack of Empathy Professionalism14. Lack of MulticulturalSensitivity
Interpersonal Aggression
15. Lack of Openness to Others Professionalism16. Mood Instability Professionalism17. Poor Citizenship Interpersonal Aggression
18. Poor Stress Management Professionalism19. Problematic Perfectionism Professionalism20. Response to Criticism Professionalism21. Rigidity Professionalism22. Sexual Inappropriateness Critical Item23. Suspiciousness Interpersonal Aggression24. Tardiness Professionalism25. Sexual and inappropriateRemarks
Critical Item
26. Substance Abuse Professionalism27. Sexual Inappropriateness Critical Item28. Prescribing medications Critical Item
Disruptive Behavior InstrumentFactor 1: Interpersonal Aggression
Items: 1, 2, 3, 4, 5, 6, 7, 9, 11, 12, 14, 17, 23
Factor 2: Professionalism
Items: 8, 13, 15, 16, 18, 19, 20, 21, 24, 26
Critical Items:
Items: 10, 22, 25, 28
Disruptive Behavior Instrument
Total Variance Explained
Factor Extraction Sums of Squared Loadings
Rotation Sums of Squared Loadings
Total % of Variance
Cumulative %
Total
1 10.463 38.750 38.750 9.296
2 1.836 6.801 45.551 7.498
3 1.408 5.215 50.766 1.655
Extraction Method: Principal Component Analysis.a. When components are correlated, sums of squared loadings cannot be added to obtain a total variance.
Disruptive Behavior Instrument
Component Correlation Matrix
Factor 1 2 3
1 1.000 -.543 1.35
2 -.543 1.000 -.031
3 .135 -.031 1.000
Extraction Method: Principal Component Analysis. Rotation Method: Promax with Kaiser Normalization.
Disruptive Behavior Instrument
Cronbach’s Alpha (α) N of Items
Factor 1 .922 13
Factor 2 .879 9
Factor 3 .305 4
Reliability for Three Factors
Poor reliability for items on Factor 3 suggests that a 2 Factor structure may be a better fit.
Disruptive Behavior Instrument2 Factors
Interpersonal Aggression• Aggression: Non-verbal• Aggression: Physical• Aggression: Verbal• Aggression: Written • Arrogance• Blaming Others• Disobeying Rules • Exploitation of Others • Intimidation• Lack of Multicultural
Sensitivity• Poor Citizenship • Suspiciousness
Professionalism• Disrespect for
Staff/Colleagues• Lack of Empathy• Lack of Openness to
Others• Mood Instability• Poor Stress
Management• Problematic
Perfectionism• Response to Criticism• Rigidity• Tardiness
Disruptive Behavior Instrument
Instead of a third factor, items were named “critical items”.
5 Critical Items: “I think about quitting or transferring to a different job because of the ratee.”
“Touches or looks at one or more patients/coworkers in an inappropriate/sexually suggestive manner.”
“Makes sexual remarks in the workplace that are inappropriate or tells jokes that makes others uncomfortable.”
“Performs unnecessary exams on female and/or male patients.”
“Demonstrates lax boundaries when prescribing controlled substances, giving out samples, or storing samples of controlled substances.”
Disruptive Behavior Instrument
Conclusions/Limitations:We have 2 main factors and 5 critical items.
Two factors:Interpersonal AggressionProfessionalism
Future Directions: Measuring and refining Inter-rater
Reliability Measuring and refining Test-Retest
Reliability How is this data utilized? What are cutoff scores? Can we track disruptive behavior
without treatment? Can instrument be utilized outside of
medicine?
We do not deal much in fact when we are contemplating ourselves.
Mark Twain
