Reliability and checklists - Stanford Universityweb.stanford.edu/~kunz/Chalmers/Reliability.pdf · Reliability and checklists John Kunz CIFE, ... Joe MEP Engineer Define and share

Center for Integrated Facility Engineering

Reliability and checklists

John Kunz

CIFE, Stanford University


Reflections - Tuesday


+/∆ Tuesday


Pre-plan examples

• Sketch

• Build checklist

• Do

• Monitor and challenge

Problems Outcomes Resources

Problems

for session

to focus on

Desired outcomesIntended

Participants

Participant

discipline

Pre-session

assignments

Member of

pre-plan

team

(yes/no)

Role in ICE

session

Member of

post-

session

wrapup

team

(yes/no)

Agenda items

Outcome intent

met?

(Yes/Partial/No)

Meeting

space,

technologies,

models, tools

Specify

architectural

spaces first

floor

Spaces to model are

specified

Mary Architect Owner share

project goals and

objectives

Yes Discipline expert Yes Review project

goals and

objectivesPartial

Smart Boards

Size building

systems

Systems in first floor are

sized and specified

Joe MEP Engineer Define and share

team charter

No Discipline expert No Specify spaces,

systems,

components to

model

Yes

Excel

Identify building

construction

components

for first floor

Components (first floor)

that take > 1 hour to

install are listed and

added to spec of

components to model

Hamid PM PM assure

availability of BIM

authoring and

review tools for

team members

Yes Facilitator Yes assign modeling

tasks to individuals

No

Meeting space

with tables,

chairs for team

of 10

ICE pre-plan

Participants Agenda


Product sketches

• How to clarify their meaning for project


Smartboard example

6


Submission-2

Comments and questions

http://www.stanford.edu/~kunz/Chalmers/Agenda13.htm




Reflections - Thursday

8


Agenda – Thursday

• Reflections, review of Tuesday

• Checklists for reliability

• Submission-1 Examples

• ICE session: project definition w/organization

analysis and optimization

• Initial student presentations

• SimVision charrette to analyze and optimize a

predefined baseline organization


Reliability Overview

Session Objectives

[Lecture/

discussion;

Understand and experience:

• Reliability issues in AEC globally

• Invisibility of reliability and risk in projects

• Checklists to improve reliability

• Risk assessment to identify reliability

10


Reliability

Data from Chile post-earthquake

– ~500K/~5M homes

damaged or destroyed: <2σ

– ~4 joint failures /~100 in

(collapsed) buildings: 2σ

– 4/~10,000 post-1985

buildings collapsed in major

damage area: >3σ

Conclusions: Chile and

developed economies

• We can design and

construct structures well

• For structures,

– Statistical performance

OK to good

– Society needs higher

reliability from AEC

– Reliability of (design-

construction-operation)

management is the issue

11


Invisibility of reliability and risk in projects

Project success factors:

Cost

Schedule

Quality

Safety

• Reliability -

Indicates that projects predict, measure and manage

against these factors

12


Risk assessment

• Do formal risk assessment periodically during project

– Small number of “experts”

• Consider factors that affect life-cycle risk/success

– Market and demographic changes

– Practice (methods) changes

– Natural and societal challenges

– Design/construction risks

• Create mitigation activities to address biggest risks

– Flexibility (buffer) in design

– Checklists in design/construction processes

13


• Spectrum of task types:

– Simple: many people can quickly learn to do the

work, e.g., shovel gravel

– Specialized: large amounts of training need to learn

to do the (virtuoso) work, e.g., Master-builder

(pilot/surgeon); high strength welding

– Complex: High uncertainty and knowledge required

exceeds that of any individual, e.g., construction

(flight/surgical) team; design structural system

• Reliability management a requirement in situations with complexity:

– w/complexity, checklists required for success

Reliability issues

14


Causes of low-reliability

• Knowledge missing: we do not know how to do something we

are trying to do

– The fact that virtually all buildings have actual

energy >> predicted suggests that we lack some

important knowledge

• Incompetence: we do not do something we know how to do

– The fact that most buildings in Chile withstood big

earthquakes suggests that we know how to design

and build for earthquake performance

15


Checklists address the competence problem

Good Checklists foster teamwork and discipline. They

• Define

– Tasks (“stupid” things) to do

– Collaboration (Information to exchange) among multiple

responsible stakeholders

• Allow specialists freedom to manage unpredicted conditions

• Need to balance

– Freedom (judgment) vs. discipline (procedure)

– Craft flexibility vs. protocol structure

– Specialty action vs. group collaboration

• Enable practice to “monitor and challenge”

16


Reliability in other industries

• Airline operations, surgery:

– Lots of performance data, e.g., on-time records; “evidence

based medicine”

– Checklists, e.g.,

• Flight checklist for entire aircraft crew; Pause time for

checks in surgery

• Carefully crafted by industry, company, local groups

• Apply to very small steps in processes that have

statistical reliability problems; not used for many steps

• Designed to provide structure and professional flexibility

• Enable extremely high process reliability when used

broadly

17


Culture of effective Checklist use

• Power: for complex, non-routine problems, push to source

of greatest actionability: the periphery

– Away from center

• Senior management:

– Sets goals and objectives

– Measures progress

– Facilitates (and assures) communication

• Team members:

– Have shared responsibility to act, to check: monitor

and challenge

– Act professionally: selfless, skilled, trustworthy,

disciplined

18


Culture of effective Checklist use

• Initiate activation phenomenon

– Share names (if not already known)

– Share concerns and observations at start of each

major activity, e.g., the day

more engagement in checklist use

• Of marginal help, at best

– Published books of standards

– Financial incentive for performance

19


Effective Checklists

• Precise: Clear, relatively short (~5-9 major steps), uncluttered

– Include “Do Confirm” steps

• Often part of a family of checklists, e.g., checklist for each day, for

different circumstances

• Have strengths:

– Help experts remember important steps in complex procedures

– Clarify priorities and sequences

– Prompt teamwork

– Enable rapid dissemination of new knowledge

– Do work – often dramatically

• Have limits:

– Cannot do skilled work

– Cannot force conformance

– Require training

– Require careful development

20


AEC Checklists that can help reliability

• Examples:

– Gantt charts, deliverable schedules, punch lists, w/

• Tasks (process steps)

• Responsibilities: who does task; who helps

• Options, e.g., resources, start/end times, next steps

(successors)

– Commitments:

• Formal: RFIs, formal meetings, punch lists

• Informal: emails, informal meetings

– VDC checklists: process to specify functional intent,

design content and behaviors of project models …

and projects

21


AEC Reliability data

• Large amounts of data are needed to identify process steps that affect life

or company health and safety

– Some industries have accident cause data, e.g., OSHA, NTSB

– Some performance data exist in AEC, e.g.,

• RS Means cost

• OSHA Total Construction safety Incidence Rates per 100

Full-time Workers:

– 1998 1999 2000 2001 2002 2003 2004 2005 2006 2007

– 8.8 8.6 8.3 7.9 7.1 6.8 6.4 6.3 5.9 5.4

– Process reliability data largely invisible, even within companies

– Process reliability must be exceptionally high (> 6σ) to assure

acceptable lifecycle product reliability

22


Management

practices

Hospital

Operations

Management

Performance

and target

management

Talent and People

management

Example dimensions

evaluated

Performance dialogue

and review

Interconnection

of targets

Consequence

measurement

Target balance

Performance tracking

3.5) Performance is

continuously tracked and

communicated, formally

and informally, to all staff

using a range of visual

management tools

1.1) Measures tracked do not

indicate directly if overall

hospital objectives are

being met. Tracking is

ad hoc

2.3) Most key performance

indicators are tracked

formally. Tracking is

overseen by senior staff

Dimension

Scoring criteria

Management Quality Survey: 20 questions in

three broad areas [Bloom]

24

Center for Integrated Facility Engineering 25

Intervention aimed to improve 38 core textile

management practices in 6 areas – for example:

India project

25

Center for Integrated Facility Engineering 26

Intervention aimed to improve 38 core textile

management practices in 6 areas – for example:

Frequent monitoring

Management Context

India project

26


Treatment plants

Control plants

Share

of

key t

extile

manag

em

ent

pra

ctices a

dopte

d

Excluded plants

(not treatment or control)

Experiment gave treatment plants free consulting for 5 months

- adoption of these 38 management practices rose .2

.3

.4

.5

.6

-10 -8 -6 -4 -2 0 2 4 6 8 10 12 Months after treatment

Shar

e o

f 3

8 m

anag

emen

t p

ract

ices

ad

op

ted

Intervention

Treated

Control

27

India project

27


02

04

06

08

01

00

12

01

40

-20 -10 0 10 20 30 40weeks since diagnostic phase

2.5th percentile

Performance improved – e.g., quality defects down 50%

Control plants

Treatment plants

Weeks after the start of the diagnostic

Qu

alit

y d

efe

cts

in

de

x (

hig

he

r sco

re=

low

er

qu

alit

y)

Start of

Diagnostic

Start of

Implementation

Average (+ symbol)

97.5th percentile

Average (♦ symbol)

97.5th percentile

End of

Implementation

2.5th percentile

India project

28 28


Example Checklist Step in detailed

schedule Checks to make

before start of step

29

Monitor and Challenge


Example Checklist Step in detailed

schedule Checks to make

before start of step

30

Monitor and Challenge


Example Checklist

Step in detailed

schedule

31


Impacts of Safe Childbirth Checklist

Single centre testing, Gokak, India

Preliminary findings

• 3 months of pre-

intervention data

• Data on ~400

childbirths collected

(admission to

discharge)

• 3 weeks of checklist

implementation data

Indicator (selection) Baseline Post-

Intervention

Washing hands and

wearing gloves when

doing vaginal examination

or delivery

<10% >90%

Checking baby’s

temperature after birth <5% >90%

Routinely administering

Oxytocin within 1 minute

after birth

<5% >90%

32


Checklist templates and VDC

We can use checklists to do Project definition (v) and define the:

1. Functional objectives – what you want for your project;

2. Scope or form – what you will or did do;

3. Behaviors – predictions and measurements of what you did

We can use these (actionable) checklist templates to:

• Specify work we do, e.g., BIM (create checklist)

• Verify work (e.g., BIM content) wrt specification (w/checklist) –

monitor and challenge

• Store results of analysis (in checklists)

• Evaluate how well the designed/built scope responds to objectives

for the Product, Organization and Process, i.e., manageable

elements of the project (using a checklist)

33

Center for Integrated Facility Engineering Inspired by Science and Decisions: Advancing Risk

Assessment, National academies press, 2009.

Questions to consider in Project Definition …

actionability for team, for us

• What is the problem? … what actions can a manager take?

– Manager can adjust functions, scope and details of product,

organization and process

– Project definition defines them

• For whom is it a problem?

– Project definition describes the organization and identifies the

stakeholders in the decision(s) that may/will face the problem

• Who is the decision maker(s)

– Project definition identifies stakeholders who can act

• Why do the modeling and analysis?

– Project definition identifies impact of performance on assessed

goodness|objectives, which allows team to identify impacts of choices

• So what?

– Adjust modeling and analysis level of detail to fit the impact of the

decision on form or scope, schedule, quality and risk

34


Product breakdown structure: Design scope or form of Product

Content of Commitments – BIM content template

35

Building

Components Spaces Systems

HVAC Patient

rooms x 400

Services

e.g., CT

iRoom work

space

Public areas

Foundation

Above-

ground steel

Wall

systems

Skin

Windows

Roof

Legend

Type of product

Product sub-type

Note: functional intent of

product element is implicit

A LOD

B LOD

Specification of BIM

contentBIM elements

Level of detail (LOD) -

AIALevel of detail LOD) - comments

Due date

of next

BIM

version

BIM

content

conforms

to spec

[Yes/No]

Components Foundation 100 (Conceptual) Minimum dimension > 4"

Components Above-ground steel 100 (Conceptual) Minimum dimension > 4"

Components walls - Exterior, basic200 (Approximate

geometry)Minimum dimension > 4"

Components walls - Interior - Partition 100 (Conceptual) Minimum dimension > 4"

Components Roof 300 (Precise geometry) Architectural, not construction detail

Components Windows 300 (Precise geometry) Architectural, not construction detail

Components Doors200 (Approximate

geometry)

Spaces Rooms 100 (Conceptual) at least 2 rooms; size of each 150 - 250

ft2

BIM Content Specification


Work breakdown structure: Design scope or form of Work

Content of Commitments – production plan template

36

Tasks

Design Build Commission Operate

Design: building

elements

[Architect]

Assess:

Behaviors

[Owner]

Design: Building

systems

[HVAC/MEP designers]

Design: Building

systems [Architect]

Legend

Type of process task

Task sub-type

Note: functional intent of

task element is implicit

A LOD

B LOD

Task Priority Short descriptionResponsible

team

Budget (FTE-

hours)

Coordination

dependent

team(s)

Approval

teamDue date

Done on-

timeComments/ model image(s)

1

A (Contract

requirement) Add door in BIM PM 20 Architect PM 1/1/2010 No

2

Commitments - example


Metrics Implementation template

(C) 2013

37

P: Process; O: Outcome

Prediction

Name CommentTarget

valueTolerance: +-∆

How to use in

managementSource of data

Type [P,

O]

Stakeholder

s who saw

data last

week

Collection

frequency Objective Weight

Predicted/

measured

value (how

you are

doing)

Assesed

value

MQuality: POE satisfaction wrt

program (%)100 5

Guide

commissioning,

next job

Client

assessmentO Owner only

Turnover time

+ 6-24 months40 86 1

ECost conformance to plan (item

actual - predicted/predicted)100 5 Plan next job

Client

assessmentO PM only Turnover time 25 98 3

T Schedule conformance to plan (%) 100 10 Plan next jobClient

assessmentO Alll on team Turnover time 35 ? NA

RPredicted Cost conformance to plan

(item actual - predicted/predicted)100 5

Attention

management

Periodic project

progress reportP Subteam only Weekly 15 99 3

IProduction schedule conformance

to plan (%)90 10

Attention

management

Periodic project

progress reportP Alll on team Weekly 10 75 2

CAssessed Quality conformance to

plan (% of items with rating >=4 on

scale 1:5)

100 10Attention

management

Periodic project

progress reportP Subteam only Weekly 15 95 3

SStakeholder participation that is

timely and meaningful (%) 90 10

Adjust plan by

stakeholder review

and assessments

Periodic project

stakeholder

survey

P Alll on team Weekly 10 90 3

Metrics

Intent Evaluation


How to use VDC templates for project definition

and as checklists

Generally, in <= 20 minutes each,

• Sketch

• Review

• Implement in template(s)

• Review: add checks

For identified areas of greatest risk

Documents

Reliability and checklists - Stanford Universityweb.stanford.edu/~kunz/Chalmers/Reliability.pdf · Reliability and checklists John Kunz CIFE, ... Joe MEP Engineer Define and share