Design and Validation of a Lean Leader Behavior Self-Assessment Tool

Laura Winner, DrPH, RN Senior Director, Lean Sigma Deployment

Erica Reinhardt, MS, RD Lean Six Sigma Black Belt, Asst. Director, Lean Sigma Deployment

© The Johns Hopkins University, The Johns Hopkins Hospital, and Johns Hopkins Health System

June 12, 2019

Armstrong Institute for Patient Safety and QualityJohns Hopkins Medicine

2

MissionWe partner with patients, their loved ones and all interested parties to end preventable harm, to continuously improve patient outcomes and experience, and to eliminate waste in health care.

Disclosures

3

None

Background

4

• Johns Hopkins Bayview Medical Center sought to prepare healthcare leaders for their role managing in a lean management system environment

Identified the principle-based behaviors we wanted from managers & staffReviewed the literature for existing lean assessment tools

Review of Literature

5

• Majority of lean assessments were at the organization level• Focused on implementation of Lean tools and methods

Lean Assessment Tool Industry

Lean Production Checklist Manufacturing

Lean Construction Maturity Model Construction

Lean Manufacturing: Performance Evaluation Audit Manufacturing

Lean Index Wood products industry

Lean Enterprise Self-Assessment Tool (LESAT) Aerospace

Shingo Prize for Manufacturing Excellence Manufacturing

Shingo Prize for Operational Excellence Expanded to other industries in 2008

Study Aims

6

• Aim 1– Develop and validate a tool to assess adoption of ideal

Lean behaviors at an individual level

• Aim 2– Pilot the validated tool at Johns Hopkins Bayview Medical

Center.

Setting

7

Acute Care Medical Centers within the Johns Hopkins Health System in Maryland.

Methods: Tool DevelopmentDrawing from Lean Management Thought Leaders

8

• Shigeo Shingo– Industrial engineer, expert in Toyota Production System

• John Toussaint– Physician expert in Lean Management in healthcare

• David Mann– Organizational psychologist, expert in Lean Management Systems

Methods: Tool Development Shingo Guiding Principles for Enterprise Excellence

9

Shingo Institute, https://shingo.org

Methods: Tool Development Toussaint & Ehrlich Lean Behavioral Dimensions

10

Toussaint JS, Ehrlich, SP. Five Changes Great Leaders Make to Develop an Improvement Culture. NEJM Catalyst, April 27, 2017.

Methods: Tool Development Toussaint & Berry Lean Principles

11

6 principles:• Lean is an Attitude of Continuous Improvement• Lean is Value-Creating• Lean is Unity of Purpose• Lean is Respect for the People Who Do the Work• Lean is Visual• Lean is Flexible Regimentation

Toussaint JS, Berry LL. The Promise of Lean in Health Care. Mayo Clinic Proceedings 2013; 88(1):74-82.

Methods: Tool Development Mann’s Managerial Elements

12

• Leader Standard Work• Visual Controls• Daily Accountability• Discipline

Mann D. Creating a Lean Culture. 3rd edn. New York: CRC Press, 2015..

Methods: Tool Development Cross Walk of Themes

13

Shingo

Toussaint & Ehrlich

Toussaint & Berry

Mann

Continuous Improvement

Seek PerfectionEmbrace Scientific

Thinking Focus on Process

PerseveranceCuriosity

Attitude of Continuous

ImprovementVisual

Visual ControlsDaily Accountability

Create Value

Flow & Pull Value Assure Quality at the

SourceCreate Value for the

Customer

Value-Creating

Unity of Purpose

Think SystemicallyCreate Constancy of

Purpose

Unity of Purpose

Respect for People

Respect Every Individual

Lead with Humility

Humility

Respect for the People Who Do the

Work

Flexible Regimentation

Self-DisciplineWillingness

Flexible Regimentation

Leader Work StandardDiscipline

Methods: Tool Development Lean Leader Principle-Based Behavioral Domains

14

• Promote Continuous Improvement• Create Value• Inspire Unity of Purpose• Demonstrate Respect for People• Embrace Flexible Regimentation

Methods: Tool Development Constructing questions

15

• Brainstormed specific behaviors– JHBMC leaders and staff – Health system lean experts

• Combined lists• Categorized behaviors by domain • Paired down to the essence of each domain• Resulted in 27 items

Methods: Tool Development

16

• Section 1: You, Your Peers, and Your Leadership– 27 items (81 responses)

• Section 2: Your Organization– 5 items

• Section 3: You (Demographics)

Methods: Tool DevelopmentContent Validity

17

• Requested participation from 9 Lean experts– Upon agreement to participate, emailed excel version of the tool

• Received 8 responses– 2 Health Services Researchers– 1 Expert in Lean Manufacturing– 5 Experts in Lean Healthcare

• Responses collated for analysis

Methods: Tool DevelopmentContent Validity

18*Polit et al.: For 6-10 reviewers, I-CVI is valid if at least 0.78. I-CVI used by Marsteller et al. at 0.75 threshold.

• Relevancy assessment of each item using the following four-point Likert scale:

4 -Highly Relevant3 -Quite Relevant 2 -Somewhat Relevant1 -Not Relevant

• Calculated Item-level Content Validity Indices (I-CVI) – I-CVI = # in agreement / # responses– Included items with I-CVI of at least 0.75*

In Agreement

Results & Data AnalysisSection 1: You, Your Peers, Your Leadership

19

• In this section, think about the frequency with which YOU, YOUR PEERS, and YOUR LEADERSHIP exhibit the following behaviors.

• 5 Domains– Continuous Improvement– Create Value– Unity of Purpose– Respect for People– Flexible Regimentation

Results & Data Analysis: Content Validity

20

Initial VersionPost-Content

Validity ChangeSection 1: You, Your Peers, and Your Leadership 27 26 -1

Continuous Improvement 7 4 -3Create Value 5 5 0Unity of Purpose 4 4 0Respect for People 8 9 +1Flexible Regimentation 3 4 +1

Section 2: Your Organization 5 5 0Section 3: You (Demographics) n/a n/a n/a

# of items

Results & Data Analysis: Scale Content Validity

21

• Calculated Scale-level Content Validity Indices (S-CVI)– S-CVI = average of I-CVIs = 0.956*

*Polit et al.: S-CVI is valid if at least 0.90.

Strengths & Limitations

22

Strengths• Private collection of the content validity feedback from experts

Limitations• Selection Bias: Experts chosen may have opinions that differ

from others

Next Steps

23

• Face Validity– 2 focus groups of acute care leaders identified by Senior Director of Innovation &

Continuous Improvement• “More” Lean exposed• “Less” Lean exposed

– Feedback on usability and sensibility• Do the questions make sense to you?• What do you think about how we are trying to measure a culture of continuous

improvement?• Is there a concept you think does not apply to you?

• Application of tool at Acute Care Medical Center

Future Research

24

• Based on results from the pilot, consider opportunities to validate the tool in other settings

References

25

• Mann D. Creating a Lean Culture. 3rd edn. New York: CRC Press, 2015.

• Marsteller JA, Hsu YJ, Chan KS, Lubomski LH. Assessing content validity and user perspectives on the Tea Check-up Tool: expert survey and user focus groups. BMJ Quality & Safety Published Online First: 12 April 2016 doi:10.1136/bmjqs-2015-004635.

• Nightingale D, Abdimomunova L, Shields T, Perkins N, Srinivasan J, Valerdi R. LAI Enterprise Self-Assessment Tool LESAT Version 2.0 February 2012. Massachusetts Institute of Technology. 2012.

• Polit DF, Beek CT, Owen SV. Is the CVI an acceptable indicator of content validity? Appraisal and recommendations. Res Nurs Health 2007;30:459–67.

• Shingo Application Guidelines. Behavior Assessment Scale. https://shingo.org

• Toussaint JS, Berry LL. The Promise of Lean in Health Care. Mayo Clinic Proceedings 2013;88(1):74-82.

• Toussaint JS, Ehrlich, SP. Five Changes Great Leaders Make to Develop an Improvement Culture. NEJM Catalyst, April 27, 2017.

• Urban W. The Lean Management Maturity Self-Assessment Tool Based on Organisational Culture Diagnosis. Procedia –Social and Behavioral Sciences. 2015;213:728-733.

26

Your Questions