Not Dead Yet: Designing Great Experiences with Bad Data

1 0 0 1 0 1 1 0 1 0 1 1 0 1 1 0 0

1 1 0 1 1 1 0 0 1 1 0 1 1 0 1 0 1

0 1 1 0 1 0 0 1 0 1 1 0 0 1 1 1 0

1 1 0 1 0 0 1 1 0 1 0 1 1 1 0 0 1

0 1 0 0 1 0 1 0 1 1 1 0 0 1 0 1 1

0 0 1 0 0 1 0 1 0 0 1 0 1 0 1 1 0

0 1 0 1 1 0

0 1 1 1 0 0

1 0 1 0 0 1

0 1 0 0 1 1

0 0 1 0 1 0

1 0 0 1 0 1

1 1 0 1 0 0 1 1 0 1 0 1 1 1 0 0 1

0 1 0 0 1 0 1 0 1 1 1 0 0 1 0 1 1

0 0 1 0 0 1 0 1 0 0 1 0 1 0 1 1 0

0 1 0 0 1 1

0 0 1 0 1 0

1 0 0 1 0 1

1

0

1

1

1

0

1

1

0

1

0

1

1

1

0

1

1

0

NOT DEAD YET…Designing Great Experiences with Bad Data@ SONIA KOESTERER

PART ONE

Why are we here?

PERSONAL BACKGROUND

Why does a designer want to talk to you about data?

SHOW OF HANDS

How many of you design happy paths?

HAPPY PATH (NOUN)

A default positive scenario, which demonstrates the sequence of activities executed if everything goes as expected.

I MADE A SWITCH

Consulting Healthcare

QUICK OVERVIEW

What exactly does Collective Health do, anyway?

Traditional healthcare is disconnected.

We connect all the pieces.

Live Concierge ServiceWeb Applications Mobile Apps

And help you make sense of your health benefits.

MY AWAKENING

HEALTHCARE DATA IS F CKED *

THE PROBLEM

HEALTHCARE $ = 1/5 FINANCIAL $

1 0 0 1 0 1 1 0 1 0 1 1 0 1 1 0 0

1 1 0 1 1 1 0 0 1 1 0 1 1 0 1 0 1

0 1 1 0 1 0 0 1 0 1 1 0 0 1 1 1 0

1 1 0 1 0 0 1 1 0 1 0 1 1 1 0 0 1

0 1 0 0 1 0 1 0 1 1 1 0 0 1 0 1 1

0 0 1 0 0 1 0 1 0 0 1 0 1 0 1 1 0

1 0 0 1 0 1 1 0 1 0 1 1 0 1 1 0 0

1 1 0 1 1 1 0 0 1 1 0 1 0 1

0 1 1 0 1 0 0 0 1 1 1 0

1 1 0 1 0 1 1 0 0 1

0 1 0 0 1 0 1 0 1 1

0 0 1 0 0 1 0 1 0 0 1 0 1 0 1 1 0

PART TWO

What does data failure mean for design?

DATA FAILURE

Erodes trust & can be dangerous.

1 0 0 1 0 1 1 0 1 0 1 1 0 1 1 0 0

1 1 0 1 1 1 0 0 1 1 0 1 1 0 1 0 1

0 1 1 0 1 0 0 1 0 1 1 0 0 1 1 1 0

1 1 0 1 0 0 1 1 0 1 0 1 1 1 0 0 1

0 1 0 0 1 0 1 0 1 1 1 0 0 1 0 1 1

0 0 1 0 0 1 0 1 0 0 1 0 1 0 1 1 0

1 0 0 1 0 1 1 0 1 0 1 1 0 1 1 0 0

1 1 0 1 1 1 1 0 1 1 0 1 0 1

0 1 1 0 1 0 1 0 0 1 1 1 0

1 1 0 1 0 0 1 1 1 0 0 1

0 1 0 0 1 0 0 0 1 0 1 1

0 0 1 0 0 1 0 1 0 0 1 0 1 0 1 1 0

1 0 0 1 0 1 1 0 1 0 1 1 0 1 1 0 0

1 1 0 1 1 1 0 0 1 1 0 1 1 0 1 0 1

0 1 1 0 1 0 0 1 0 1 1 0 0 1 1 1 0

1 1 0 1 0 0 1 1 0 1 0 1 1 1 0 0 1

0 1 0 0 1 0 1 0 1 1 1 0 0 1 0 1 1

0 0 1 0 0 1 0 1 0 0 1 0 1 0 1 1 0

1 0 0 1 0 1 1 0 1 0 1 1 0 1 1 0 0

0 1 1 0 1 0 0 1 0 1 1 0 0 1 1 1 0

0 1 0 0 1 0 1 0 1 1 1 0 0 1 0 1 1

1 00

1 1 0

10 1 1 10

0

1 0 1 0 0 1 01

1 1 00 1 0 0 1 0 1 0 1 1 1

When you say edge case, you're just defining the boundaries of what you care about.

BETH DEAN

We need a process for understanding & fixing failures in design.

Solution

Problem Space

FAIL

FAILFAIL

FAIL

Why not borrow concepts from Manufacturing, Software Engineering, Systems Engineering & Safety Engineering?

Solution

Problem Space

FAIL

FAIL FAIL

FAIL

Overflow

PART THREE

How do we get there?

REMEMBER

Data doesn’t know it has a problem.

1

1

10

0

0

EXAMPLE ONE

Imagine FBI agents turn up at your home and forcefully declare it to be the site of illicit activities…

FRAUDFRAUD

Imagine FBI agents turn up at your home and forcefully declare it to be the site of illicit activities…

FRAUDFRAUDA quiet Kansas home wound up with 600-million IP addresses and a world of trouble. KASHMIR HILL, FUSION.NET

http://fusion.net/story/287592/internet-mapping-glitch-kansas-farm/

1 0 0

1

01

1 0 1 0 11

0 1 100

1 1

0

1 1 1

0

0 1 1 0 1 1 0 1 0 1

0 1 1 0

1

00

1 0

1

1 00

1 1 1

01

1

0 1 0

01

10 1 01 1 1 0 0 1

01 0

01 0 1 0 1 1 1 0

01 0 1 1

0

0 1 0 0 1 0

1

0 0 1 0 1

0

1 1 0

0 1

0

1 1 0

0 1 1 10

0

1 0 1

0 0 1

0

10 0

1 1

0 0

1

0 1 0

1 0

0

1 0 1

11

0 1 0 0 1 1 0 1 0 11

1 0 0 1

0 1

0

0 1 0 1 0

1

1 100

1 0 1 1

0 0 1 0 0 10

1 0 0 1 01 0 1 1 0

0 1 0 0 1 1

0 0 1 0 1 0

1 0 0 1 0 1

1

0

1

11

0

1

1 01

0 11

1

0

1

1

0

1 0 0

1

01

1 0 1 0 11

0 1 100

1 1

0

1 1 1

0

0 1 1 0 1 1 0 1 0 1

0 1 1 0

1

00

1 0

1

1 00

1 1 1

01

1

0 1 0

01

10 1 01 1 1 0 0 1

01 0

01 0 1 0 1 1 1 0

01 0 1 1

0

0 1 0 0 1 0

1

0 0 1 0 1

0

1 1 0

0 1

0

1 1 0

0 1 1 10

0

1 0 1

0 0 1

0

10 0

1 1

0 0

1

0 1 0

1 0

0

1 0 1

11

0 1 0 0 1 1 0 1 0 11

1 0 0 1

0 1

0

0 1 0 1 0

1

1 100

1 0 1 1

0 0 1 0 0 10

1 0 0 1 01 0 1 1 0

0 1 0 0 1 1

0 0 1 0 1 0

1 0 0 1 0 1

1

0

1

11

0

1

1 01

0 11

1

0

1

1

0

PROBLEM ONE

Data is messy. Sloppy data, uneven data, varying levels of precision

WHAT IF WE COULD

Learn from our mistakes?

Experiment at a small scale before launching at scale.

LEAK BEFORE FAILURE

*Not a healthy motto for life outside of work.

LEAK BEFORE FAILURE

BE LEAN, BUT FIRST GET FAT.

EXAMPLE TWO

Imagine going to the hospital to have your tonsils removed, but coming back with a leg operation.

Imagine going to the hospital to have your tonsils removed, but coming back with a leg operation.

Hospitals and medical practices mix up 8% of patient identities. JOHN MCQUAID, STATNEWS.COM

https://www.statnews.com/2016/01/28/patient-mixups-universal-identification/

1 1 0 1 1 0 1 0 0 1 1 0 0

1 1 0 1 1 0 0 1 1 0 1 1 0 1 0 1

0 1 1 0 1 0 0 1 1 1 0 0 1 1 1 0

1 1 0 0 1 1 0 1 0 1 1 1 1

0 1 0 0 1 1 0 1 1 1 0 0 1 0 1 1

0 0 1 0 0 1 0 1 0 0 1 0 1 1 1 0

0 1 0 1 1 0

1 0 0

1 0 1 0 0 1

0 1 0 0 1

0 0 1 0 1 0

1 0 0 1 1

1 0 1 0 0 1 1 0 1 0 1 1 1 0 0 1

0 1 0 0 1 1 1 0 0 1 0 1

0 0 1 0 1 0 1 0 0 1 0 1 0 1 1 0

0 1 0 0 1 1

0 0 1 0 1 0

1 0 0

0

1

1

1

0

1

1

1

0

1

1

0

1

1

1 1 0 1 1 0 1 0 0 1 1 0 0

1 1 0 1 1 0 0 1 1 0 1 1 0 1 0 1

0 1 1 0 1 0 0 1 1 1 0 0 1 1 1 0

1 1 0 0 1 1 0 1 0 1 1 1 1

0 1 0 0 1 1 0 1 1 1 0 0 1 0 1 1

0 0 1 0 0 1 0 1 0 0 1 0 1 1 1 0

0 1 0 1 1 0

1 0 0

1 0 1 0 0 1

0 1 0 0 1

0 0 1 0 1 0

1 0 0 1 1

1 0 1 0 0 1 1 0 1 0 1 1 1 0 0 1

0 1 0 0 1 1 1 0 0 1 0 1

0 0 1 0 1 0 1 0 0 1 0 1 0 1 1 0

0 1 0 0 1 1

0 0 1 0 1 0

1 0 0

0

1

1

1

0

1

1

1

0

1

1

0

1

1

PROBLEM TWO

Data is incomplete. Blank fields, disappearing feeds, and uncollected information

WHAT IF WE COULD

Just say no?

Design forcing functions to get the data you need.

POKA–YOKE

KNOW YOUR ANTI-USE CASE.

EXAMPLE THREE

Imagine you come home to find that your house is being treated as a playground…

Imagine you come home to find that your house is being treated as a playground…

Man’s home mistakenly set as a Pokémon GO gym. MARCUS LESHOCK, WGNTV

BOON SHERIDAN @BOONERANG

http://wgntv.com/2016/07/11/mans-home-mistakenly-set-as-pokemon-go-gym/

1 0 1 0 1 1 0 1 0 1 1 0 1 1 0 0

1 1 0 1 1 1 1 1 0 1 1 0 1 0 1

0 1 1 1 0 0 1 0 1 1 0 0 1 1 1 0

1 1 0 1 0 0 1 1 0 1 0 1 1 1 0 0 1

0 1 0 0 1 0 1 1 1 1 0 0 1 0 1 1

0 0 1 0 0 1 0 1 0 0 1 0 1 0 1 1 0

0 1 0 1 1 0

0 1 1 1 0

1 0 1 0 0 1

0 1 0 0 1 1

0 0 1 0 1 0

1 0 1 0 1

1 1 0 1 0 0 1 1 0 1 0 1 1 1 0 0 1

0 1 0 0 1 0 1 0 1 1 1 0 1 0 1 1

0 1 0 0 1 0 1 0 0 1 0 1 0 1 1 0

1 0 0 1 1

0 0 1 0 1 0

1 0 0 1 0 1

1

0

1

1

1

0

1

1

0

0 0

0

0

0

0

0

0

0

0

1

0

1

1

1

0

1

1

0

0

1

1 1

0

1

1

1

1

1

1

1

0 1 0 1 0 1 1 0 1 1 0 0

1 1 0 1 1 0 1 1 0 1 0 1

0 1 1 0 0 1 0 1 1 0 0 1 1 1 0

1 1 0 1 0 0 1 1 0 1 1 1 0 0 1

0 1 0 0 1 0 1 1 1 0 0 1 0 1 1

0 0 1 0 0 1 0 1 0 0 1 0 1 0 1 1 0

0 1 0 1 1

0 1 1 1

1 0 1 0 0 1

0 1 0 0 1 1

0 0 1 0

1 0 1

1 1 0 1 0 0 1 1 0 1 0 1 0 0

0 0 1 0 1 0 1 1 1 1 0 1 1

1 0 0 1 0 1 0 0 1 0 1 0 1 1 0

1 0 0 1 1

0 0 1 0 1 0

1 0 0 1 0

0

1

1

1

0

1

0

0 0

0

0

0

0

0

0

0

0

1

0

1

1

1

0

1

0

0

1

1 1

0

1

1

1

1

1

1

1

0

0

0

0

0

0

0

0

0

0

1

1

0

1

1

1

1

1

1

1

0

0 0

0

0

0

0

0

0 0

1

1 1

0

1

1

1

1

1 1

0

0

0

0

0

0

0

0

0

0

1

1

0

1

1

1

1

1

1

1

PROBLEM THREE

Data is fickle. Temporary data, old data, information no longer valid

WHAT IF WE COULD

Measure twice, and cut once?

Design the means to capture data from redundant sources.

REDUNDANCY

GET AHEAD OF CHANGES.

EXAMPLE FOUR

Imagine going to the doctor for a cough, and being told they can’t help you because you’re dead…

Imagine going to the doctor for a cough, and being told they can’t help you because you’re dead…

deceased

deceasedOver 12,000

Americans are erroneously declared dead each year. MARISSA FESSENDEN, SMITHSONIAN.COM

http://www.smithsonianmag.com/smart-news/about-12200-people-are-erroneously-declared-dead-every-year-us-government-180955549/?no-ist

1 0 0 1 0 1 1 0 1 0 1 1 0 1 1 0 0

1 1 0 5 1 1 0 0 1 1 0 1 1 0 C A T

0 1 1 0 1 0 0 1 0 1 1 0 0 1 1 1 0

1 1 0 1 0 0 1 1 0 1 0 2 1 1 0 0 1

0 1 0 0 1 0 1 0 1 1 1 0 0 1 0 1 1

0 0 1 0 0 1 0 1 0 0 1 0 1 0 1 1 0

0 1 0 1 1 2

0 1 1 1 0 0

1 0 1 0 0 1

0 1 0 0 9 1

0 0 1 0 1 0

1 0 0 1 0 1

1 1 0 1 0 0 1 1 0 1 0 1 1 1 0 0 1

0 1 8 0 1 0 1 0 1 1 1 0 2 1 0 1 1

0 0 1 0 0 W T F 0 0 1 0 1 0 1 1 0

C A T 0 1 1

0 0 1 0 1 2

1 0 0 1 0 1

3

0

1

1

1

0

1

1

0

4

0

1

1

1

0

1

1

8

1 0 0 1 0 1 1 0 1 0 1 1 0 1 1 0 0

1 1 0 5 1 1 0 0 1 1 0 1 1 0 C A T

0 1 1 0 1 0 0 1 0 1 1 0 0 1 1 1 0

1 1 0 1 0 0 1 1 0 1 0 2 1 1 0 0 1

0 1 0 0 1 0 1 0 1 1 1 0 0 1 0 1 1

0 0 1 0 0 1 0 1 0 0 1 0 1 0 1 1 0

0 1 0 1 1 2

0 1 1 1 0 0

1 0 1 0 0 1

0 1 0 0 9 1

0 0 1 0 1 0

1 0 0 1 0 1

1 1 0 1 0 0 1 1 0 1 0 1 1 1 0 0 1

0 1 8 0 1 0 1 0 1 1 1 0 2 1 0 1 1

0 0 1 0 0 W T F 0 0 1 0 1 0 1 1 0

C A T 0 1 1

0 0 1 0 1 2

1 0 0 1 0 1

3

0

1

1

1

0

1

1

0

4

0

1

1

1

0

1

1

8

PROBLEM FOUR

Data is imperfect. Typos, duplications, bad formatting, oh my!

WHAT IF WE COULD

Trust but verify?

Design feedback loops to prevent harmful mistakes.

SUCCESSIVE CHECKS

DO THE RIGHT THING.

IN SUMMARY

What are the big ideas you can take home?

1. First be fat.Pilot new offerings at a small scale to learn where the weak points are and gracefully recover.

2. Know your anti-use case. Define what you don’t want your service to enable, and create forcing functions to prevent disastrous mistakes.

3. Get ahead of changes. Future-proof your service by continuously gathering updated data from multiple sources.

4. Do the right thing.Verify the validity of key transitions where major status, account, or behavior changes occur for your users.

INTERESTING THOUGHTS FROM SMART PEOPLE

1. Fat Thinking and Economies of VarietyVenkatesh Rao

2. Go Home, Data, You’re DrunkDanielle Malik

3. Goodhart’s Law and Why Measurement is HardDavid Manheim

If you torture the data long enough, it will confess.

RONALD H. COASE

Thank you. @SoniaKoesterer