Shake, Rattle and Roles: Earthquake Engineering as HRO

Dan HornJeremy BirnholtzNovember 5, 2003

Presentation Outline

Introduction High Reliability Organizations Earthquake Engineering

MethodsFindingsImplicationsNext steps

High Reliability Organizations

Weick (1999) lists 3 key characteristics Environment rife with errors to be

detected Constant monitoring and reporting No anomaly too small

Reluctant to simplify interpretations Integrate multiple, redundant sources

Ongoing sensitivity to operations Collective mindfulness, heedful interrelating,

the “bubble” etc.

Why study HROs here?

Error detection is a key user goal, and therefore a key aspect of designLessons are broadly applicable Weick suggests that ordinary orgs

become HRO-like in crises Newell (1997) suggests that under

extraordinary circumstances, the ordinary user becomes extraordinary

Experimental Structural Earthquake Engineering (EE)

Large scale physical test equipmentMany forces that are complex and interactingPotential danger!

Structural Labs as HROs

We argue that structural EE labs are a form of HRO3 types of risk faced in these labs Catastrophic specimen failure Losing laboratory and field autonomy;

as (Galison, 1997) discusses in physics Risk of significant social cost;

concentrated social risk (Sims, 1999)

NEES: Telepresence

Teleobservation Watching tests from a distance Potentially many observers Passive vs. active

Teleoperation Controlling apparatus during test Remote operations

Methods

Interviews with 75 earthquake engineers Faculty, students and technicians Questions about

Sequence of a typical test What they are looking at during a test

Ongoing observation of EE testsCoding for themes (Huberman and Miles, 1994)

Findings

Local Failure detection Variable Likelihood of failure Integrating sensory cues Multiple collocated persons

Beliefs about telepresence Remote Failure Detection: One story

Variable likelihood of failure

More likely early and late: “I’m always there for the first test on a

particular specimen, because I need to train the students on the things they need to do … like making sure the test frame is not creating a physical anomaly. Students have a tendency to just roll forward without checking these things.”

Early failure Dangerous and Costly

Late failure Predictable and can be prepared for this

Integration of sensory cues

Students rely on visual cues Visual displays of data (e.g. graphs) Walking around and looking at the specimen “if we can’t explain the graphs, we stop

immediately. If we get data that are surprising, but not crazy we’ll keep going”

More experienced integrate more cues Sound: “Even after [we had fixed a problem with

the test setup], there was still a lot of noise. I might have pushed the [emergency stop] button, as it was very noisy.”

“Feel”

Multiple Collocated Persons

Reliance on multiple viewpoints “different accounts of what happened, like

people’s reports at the scene of a car accident“

Technicians say they will “send somebody out to stand in a particular place and keep an eye on things.”

Frequent interaction “When things go awry, we tend to powwow

in the lab…to sort out what’s going on.” “You have to argue”

Beliefs About Telepresence

Value in remote observation Dog-and-pony show value “Real” researchers plan to be at their

tests

Fears of remote operation Don’t mess with my actuators Low-fidelity means low value

Remote Participation: One Story

One faculty member had remote participants in his shake table test when he was a graduate student. He had primitive video via html and people watching could email him.Valuable in that “you’re concentrating on one thing like maybe

running the test, and someone emails you and says, ‘hey what’s that that’s going on?’ and you look right there and you get a whole other opinion about what’s going on”

Email is low-cost, persistent and not real-time: “Don’t have 20 people yelling at you at once.”

Implications

Additional local observers are valuable But: tend to have more correlated views

Remote observers are constrained Cues are less rich Cues are mediated

Can we design representations that exploit these constraints? Increase statistical probability of error

detection

Formal Model

P(D|F)=1-(1-P(d1|F))…(1-P(dn|F))

Person 1: .5Person 2: .4

P(D|F)=1-(.5)*(.6)=.7 = 70%

ASSUMES STATISTICAL INDEPENDENCE

Next step: Lab Studies

BackgroundTask Expected results

Experiment: Background

Tower of Hanoi (TOH) Problem

Rule 1: Only move 1 piece at a time Rule 2: Only move smallest FROM a peg Rule 3: Only place smallest ONTO a peg

Experiment: Background

Waitress and Orange TOH Isomorph (Zhang & Norman, 1994)

Converts part of external representation into internal representationLeads to less efficient performance

Experiment: Background

Distributed Representations in TOH (Zhang, 1998) Waitress and Orange Problem Different Levels of Knowledge

Expert: R123 Mid-level: R12 or R13 Novice: R1

Pairs – Taking Turns R123-R1 vs R12-R13 vs R123-R123 vs

R123

Zhang’s Results: Solution Time

Solution Times (Seconds)

0

100

200

300

400

500

R123-R1 R12-R13 R123-R123 R123

Zhang’s Results: Steps

Solution Steps

0

10

20

30

R123-R1 R12-R13 R123-R123 R123

Zhang’s Results: Errors

Errors

0

1

2

3

R123-R1 R12-R13 R123-R123 R123

Zhang’s Results: Summary

Two experts are always better than oneTwo mid-levels are never better than one expertA novice is as helpful as a second expert in reducing errors

Beyond Zhang: Proposed Study

Payoff (to be piloted) +$1.00 per solved problem -$0.10 per move -$0.40 per error

“Local” person Knows all rules Complex display, full information Makes all moves

Beyond Zhang: Proposed Study

“Remote” person(s) Knows all rules Complex vs. Simple display (full vs.

partial info) Can only reject moves

IllegalInefficientEither illegal or inefficientRejections cost $0.20 each

Proposed Study

Expected Outcomes (Remote Interface) Simple > Complex in Error Detection Simple < Complex in Inefficiency

Detection

Implications

How these lessons will inform design of these systems Could show value for remote folks Allow us to take this to EE context New expert-remote participants Must overcome “speedbump of

inefficiency”

Questions?