The Art of Better

CTO, New Iron, Austin, TXJanelle Klein

©2015 New Iron Group

The Art of Better

Creation Date: 10/05/2015

Yay!

What is this talk about?

How to quantify developer pain

How to identify what’s causing the pain

How to practice avoiding the pain

Idea Flow Learning Framework

Why should I care?

Great Team Disciplined with Best Practices Constantly Working on Improvements+

Project FAILURE

Technical Debt Mistakes

I thought the main obstacle was Technical Debt

?Most of our mistakes were in the

most well-written parts of the code.

Mistakes

We made significantly more mistakes in code that we didn’t write ourselves.

Lower Familiarity

More Mistakes=

There had to be more to the story...

Complex(So*ware(

...pain...

This is what I knew...

What made development feel painful?

Unexpected Behavior

Problem Resolved

Tracking Painful Interaction with the Code (Friction)

Troubleshooting

Progress

5 hours and 18 minutes of troubleshooting...

PAINFUL

What Causes Unexpected Behavior (likeliness)?

What Makes Troubleshooting Time-Consuming (impact)?

Semantic Mistakes

Stale Memory Mistakes

Association Mistakes

Bad Input Assumption

Tedious Change Mistakes

Copy-Edit Mistakes

Transposition Mistakes

Failed Refactor Mistakes

False Alarm

Non-Deterministic Behavior

Ambiguous Clues

Lots of Code Changes

Noisy Output

Cryptic Output

Long Execution Time

Environment Cleanup

Test Data Creation

Using Debugger

Most of the pain was caused by human factors.

What causes PAIN?

What Causes Unexpected Behavior (likeliness)?

What Makes Troubleshooting Time-Consuming (impact)?

Non-Deterministic Behavior

Ambiguous Clues

Lots of Code Changes

Noisy Output

Cryptic Output

Long Execution Time

Environment Cleanup

Test Data Creation

Using Debugger

What causes PAIN?

Most of the pain was caused by human factors.

Semantic Mistakes

Stale Memory Mistakes

Association Mistakes

Bad Input Assumption

Tedious Change Mistakes

Copy-Edit Mistakes

Transposition Mistakes

Failed Refactor Mistakes

False Alarm

PAIN occurs during the process of understanding and extending the software

Complex(So*ware(

PAIN

Not the Code.

Optimize Developer Experience

We spent tons of time working on improvements…

Our Improvements didn’t make much difference.

It took us 2 years to turn the project around.

The Key: Visibility

Consulting…

Builds were breaking

Releases were painful

Productivity slowing to a crawl

Begging for timePAIN

The Team Believed…

Problems Building Up in the Code

PAIN

The Improvements didn’t make much difference.

Brainstorming a List

“Technical Debt” Backlog

What does “better” really mean?

Mentorship = Teaching “Better”

The Interview

GeorgeMe

“Testability”

The Code Review

Eh… could you explain your reasoning?

!=

Different conceptual models for “Better”

“Better” “Better”

14:230:00

George’s Painful Experience

The Experience Review

What made troubleshooting take so long?

The Haystack Principle

“Optimize for small manageable haystacks.”

Lots of unvalidated changes

Easier to find the needle.

7:01

Iterative Validation with Unit Tests7:010:00

Small Haystacks

14:230:00

Skipping Tests and Validating at the End

Big Haystack

“If I decide to skip the unit tests, how will that affect my haystack size?”

14:230:00

George’s Painful Experience

The Experience Review

What made troubleshooting take so long?

The Art of Better

Avoiding Pain

With a data-driven feedback loop for learning what works, we can find our way to better.

Input: Constraints

Target: Minimize Pain

Output: Actual Pain

short-term loop long-term loop

1. Visibility

2. Clarity

3. Awareness

F ocus!


Systematically Optimize Decisions

Input: Constraints


Output: Actual Pain


1. Visibility

2. Clarity

3. Awareness

F ocus!

Target - The direction we’re trying to go…


Input: Constraints


Output: Actual Pain


1. Visibility

2. Clarity

3. Awareness

F ocus!

Output - The experience we’re trying to improve


Input: Constraints


Output: Actual Pain


1. Visibility

2. Clarity

3. Awareness

F ocus!

Input - The constraints that limit our short-term choices…


Input: Constraints


Output: Actual Pain


1. Visibility

2. Clarity

3. Awareness

F ocus!

1. Visibility - See the pain

1. Visibility

1. Visibility


Input: Constraints


Output: Actual Pain


1. Visibility

2. Clarity

3. Awareness

F ocus!

1. Clarity

2. Clarity

2. Clarity - Explain what’s causing the pain


Input: Constraints


Output: Actual Pain


1. Visibility

2. Clarity

3. Awareness

F ocus!

3. Awareness

3. Awareness - Learn to avoid the pain


Input: Constraints


Output: Actual Pain


1. Visibility

2. Clarity

3. Awareness

F ocus!

Focus on the biggest pain…

F ocus!F ocus!


Input: Constraints


Output: Actual Pain


1. Visibility

2. Clarity

3. Awareness

F ocus!



Process Control in Manufacturing

Quality Target

Lower Control Limit

Upper Control Limit

X"

This is “Out of Control”

Lower Variability = Better Control

We don’t improve productivity by trying to go faster, we improve productivity by improving control

“Pain Control” in Software Development

Optimal Friction

Upper Control Limit X"

“Out of Control”

20min

0m

Average Pain per Incident

This is Better

Target

Control Limit

How do I implement “better”?

Three Stages of Mastery

Visibility

Clarity

Awareness


Visibility

Clarity

Awareness

Visibility

Learn to see the pain by visualizing the pain with Idea Flow Maps

The Visibility Principle

We can’t solve a problem that we don’t see.

George Lakoff

Metaphors We Live ByJeff Hawkins

On Intelligence

+

How do we “see” (and not see)?

Observe Pattern

“Do I know this pattern?”

Metaphors are patterns of conceptual “Shapes”

Three Different Types of Metaphors

Object Patterns (Thing)

Context/Relationship Patterns (Space)

Process Patterns (Time)

We “see” the world through a fabric of metaphor.

Technical Debt

Debt is a “Thing”

“This looks like a tech debt thing.”

1. Categorize patterns by concept type

“This looks like a painful tech debt thing.”

2. Categorize variations by intensity.

Pain level = 8

= Little Debt

= Medium Debt

= Big Debt

Our Beliefs about Developer Pain

PAIN occurs during the process of understanding and extending the software

Complex(So*ware(

PAIN

Pain occurs over TIME.

What do you think happens when we make this transition?



nasa.com/solar-portraits

Metaphors are Filters

http://nasa.com/solar-portraits

Try on the Purple Glasses!

Complex(So*ware(

Idea Flow is the flow of ideas between the developer and the software

“Idea Flow” = Process Pattern

The Idea Flow Process

Conflict Confirm

Rework'

Learn'

Validate(

Modify'

Progress Loop Conflict Loop

Troubleshoot'

Confirmed Expectations

Our brain works like a giant prediction engine…

Violated Expectations

Our brain works like a giant prediction engine…

Process Pattern Match

Process Pattern Mismatch

Confirmation Conflict

The Idea Flow Process

Conflict Confirm

Rework'

Learn'

Validate(

Modify'


Troubleshoot'

Conflict Confirm

Rework'

Learn'

Validate(

Modify'


Troubleshoot'

“Friction” in the Idea Flow Process

“Idea Flow Map”

“Friction”

A “conflict” starts with an unexpected observation and ends when understanding is resolved.

Idea Flow Mapping Tools

(Open Source, Supported GA ~January 2016)github.com/ideaflow/tools

http://ideaflow.org/tools

Let’s Record an Idea Flow Map!

Typical Idea Flow Maps

Single Problem

Multi-Problem

Reading Visual Indicators in Idea Flow Maps

Le#$Atrium$

Le#$Ventricle$

Right$Ventricle$

Right$Atrium$

What’s$causing$this$pa7ern?$

Similar to how an EKG helps doctors diagnose heart problems...

...Idea Flow Maps help developers diagnose software problems.

Problem-Solving Machine

Reading Visual Indicators in Idea Flow Maps

= Solution Strategy

The “Heart” of Software Development(the problem-solving machine)

= Solution Strategy


Choose a general strategy

= Solution Strategy


Understand the system

= Solution Strategy


Code & work out the kinks

= Solution Strategy


Back to the drawing board

The type of friction depends on where the disruptions are in the problem-solving process

Trial and Error

The Friction is Here

Troubleshooting

Progress

Learning

Rework

Struggling with Understanding the Code


Troubleshooting

Progress

Learning

Rework

Struggling with Feedback


Troubleshooting

Progress

Learning

Rework

Idea Flow Maps visualize the experienced difficulty of a problem-solving task.

See the pain

Visibility


Visibility

Clarity

Awareness


Visibility

Clarity

Awareness

Clarity

Learn to explain the causes by expanding your vocabulary of patterns

The Clarity Principle

By naming the patterns,

we start to recognize the patterns.

What patterns do I see?


Context/Relationship Patterns (Space)


Define More Shapes

The Ten Pains of Software Development

Troubleshooting

Progress

Learning

Rework


Design-Fit Pain - When the new feature doesn’t fit well into the existing design.


Requirements Pain - Bad assumptions about what functionality to build


Collaboration Pain - Challenges collaborating with other developers on the team.


Modeling Pain - When it’s difficult to build a conceptual model of the existing software.


Cognitive Pain - Challenges with complexity and intense thinking


Alarm Pain - Challenges with false alarms and test maintenance


Experiment Pain - Challenges with getting feedback by running experiments


Execution Pain - When changing the code is highly mistake-prone


An amplifying effect on other problems

What type of pain did we have in the

Troubleshooting

Learning

Rework

Experiment*Pain*

!Numerous!Feedback!Loops!

Itera&ve(Narrowing(

Ambiguous(Clue(

Non6determinis&c(

Lots(of(Test(Cases(

Lots(of(Bugs(

Bad(Input(Assump&on(

Uncertainty(

Difficult(to(Interpret(

Experiment*Pain*

!Slow!Feedback!Loops!

Long%Execu+on%

Noisy%Output%

Limited%Output%

Complex%Behavior%

Non<determinis+c%Behavior%

Lots%of%Code%Changes%

Difficult%to%Interpret%

Test%Input%Genera+on%

Environment%Setup%

Environment%Cleanup%

Have%to%significantly%modify%a%test%

Debugger%

Breakdown the Causes of Pain into Patterns

Tags: #problemtag

Tag the Examples

100 hours

50 hours

Troubleshooting

Learning

Rework

Why categorize problems this way?

Focused Effort = Visible Improvement

Explain the Story of What Happened

JourneyBeginning Ending

Think about the Experience as a Problem-Solving Journey

What were the biggest challenges on the journey?

Constraints

Difficulties

Bad Strategies

Uncertainty

Journey

Constraints


Difficulties

Journey


JourneyBad Strategy


Journey Good Strategy


Journey Uncertainty


What were the challenges?

What other choices did we have?

Explain the Story of What Happened

Our level of understanding is constrained by the diversity of our conceptual metaphors.

Explain the causes of pain

Clarity


Visibility

Clarity

Awareness


Visibility

Clarity

Awareness

Learn to avoid the pain by running strategy experiments.

Awareness

Clarity without Awareness

This is Hindsight Bias. It’s not that easy!

14:230:00

Lots of changes at once will make troubleshooting difficult

“That was a dumb thing to do. I’ll avoid that next time.”

The ChallengeWe tend to make auto-pilot decisions and do what’s familiar

How do we break decision habits?

Imagine your brain is a decision-making engine

written in code.

BreakpointStop and Think!

Imagine your brain is a decision-making engine

written in code.

BreakpointStop and Think!

What’s a Decision Principle?

1. How do I evaluate my situation?

2. What should I optimize for?

Answers Two Questions

The Haystack Principle

“Optimize for small manageable haystacks.”

“If I decide to skip the unit tests, how will that affect my haystack size?”

Lots of unvalidated changes

Easier to find the needle.

The Awareness Principle

We can’t avoid the pain,

if we can’t predict the pain when making decisions.

Test the Model (Experiments)

Refine the Model (Patterns)

False&Predic,on&

Scien,fic&Model&

Creating an explicit model makes our beliefs testable.

The Scientific Method

Test the Model (Experiments)

Refine the Model (Patterns)

False&Predic,on&

Scien,fic&Model&

The Scientific Method

Creating an explicit model makes our beliefs testable.

Visibility Process

Clarity Process

Awareness Process

The Checklist ManifestoAtul Gawande

Running LeanAsh Maurya

+

Strategy Experiments with Checklists

14:230:00

I want to avoid this…

Thinking Checklist

Is my current approach likely to cause a big haystack?

Situation: start of subtask

Let’s Make a Checklist!

“What question could I ask my future self to recognize similar risks in the future?”

“In what situation would I ask the question?”

0:00

Stop and Think:


Predict: Small haystack

Strategy Experiments

18:120:00

Stop and Think:


Predict: Small haystack

False Prediction


18:120:00

Stop and Think:


False Prediction


High-Risk Situations

1. Unraveling sweater 2. Integration-heavy change 3. High state variation 4. Minimum scope is big

Q: Is my current approach likely to cause a big haystack?

Start of Subtask

Strategy Types (“do”)

1. DependencyAnalysis 2. IncrementalIntegrationTest 3. DataDrivenTest 4. IsolateHardToTestCode

Situation Types (“see”)

1. UnravelingSweater 2. HeavyIntegrationLogic 3. HighStateVariation 4. CoupledExternalDependencies

Haystack Decisions

Codify What Works

DoSee

Best Practices are Backwards

We don’t encounter solutions, we encounter problems.

Do See We need a new data structure!!

Learn to avoid the pain

To improve the quality of decisions, practice asking the right questions at the right time.

Awareness


Visibility

Clarity

Awareness

See

Explain

Predict

What decision principles apply to the example?

Experiment Time

Time Warp Bias

Setup Experiment Analyze Results & Decide Next Experiment

Execute

“Optimize experiments for human cycles.”

“How could I reduce the time spent on human cycles?”

Was it time-consuming to setup the inputs? What made it time-consuming?

Was it fast or slow to execute the experiment? What made it time-consuming?

Was it time-consuming to analyze the results? What made it time-consuming?

Execution

Decide Next Action

Setup Inputs

How could we have reduced the human cycles?

Time Warp Questions

Code Sandwich

Behavior Complexity

Observability

Ease of Manipulation

“How can I reduce the thickness of my code sandwich?”

“Optimize for low diagnostic difficulty.”

The thickness of the sandwich increases

troubleshooting difficulty

What behavior were we trying to understand? Was the core behavior complex or simple?

How difficult was it to see what was going on? What made it difficult to see?

How difficult was it to manipulate the behavior? What made it difficult to tinker?

Observability

Ease of Manipulation

Behavior Complexity

How could we reduce the thickness of the code sandwich?

Code Sandwich Questions

Cloud of Possibilities

“How can I generate clues to eliminate some possibilities?”

“Optimize the experiments to generate useful clues.”

Was the cloud of possibilities big or small?

What were some of the clues we saw? Were some of the clues more useful than others?

Scope

Narrowing Speed

How could we have have narrowed the possibilities faster?

Cloud of Possibilities Questions

Input: Constraints


Output: Actual Pain


1. Visibility

2. Clarity

3. Awareness

F ocus!



Why go to all this trouble?

Accelerate productivity improvements by focusing on what matters most.

Make the improvements permanent by improving your decision habits.

Make the case to management for spending time on improvements by quantifying the indirect costs

Open Mastery 2016 We Believe in Peer Mentorship!

Yay!

Free Online Training and Community Support:


Individuals Free Membership

Companies Paid Membership

Open Mastery Guild Membership

IFM Tools

Anonymized data

Developers Learning Together!

IFM Tools

IFM Tools IFM Tools

IFM Tools

IFM Tools

Community Focus

Integrated #HashTag Glossary

IFM Tools

IFM Tools

IFM Tools IFM Tools

IFM Tools

IFM Tools

Idea Flow Analytics PlatformPrivate data

Companies Learning Together!

Revenue from membership fees to fund core development

@janellekz

Open Mastery 2016Join us! @openmastery

[email protected]

Thank you!

Free e-book if you sign up before publish day! (Jan 2016)

Tweet about #ideaflow!

IFM Tools available at: github.com/ideaflow/tools

mailto:[email protected]

http://ideaflow.org/tools