How to Break the Software Rewrite Cycle

Software Rewrite Cycle How to Break

CTO, New Iron Janelle Klein

My Career

Maintenance Team My First Project

Rewrite Team The Next Generation

My Career

Maintenance Team My First Project

Rewrite Team The Next Generation

My Career

My 2nd Project Third Generation Rewrite

My Career

My 3rd Project I Kicked Off the Rewrite!!

My Career

My 4th Project 3rd Generation Rewrite

Consulting...

My Career

Does anyone not have this challenge?

We Start with the Best of Intentions

High Quality Code

Low Technical Debt

Easy to Maintain

Good Code Coverage

Stages of Escalating Project Risk

Product Owner: “We’ve got more important things to do.”

Manager: “Good job everyone! Keep up that great work ethic!”

Manager: “We need to go faster! Let’s hire more developers.”

Developer: “I give up. I don’t care anymore if the project fails.”

The further down the path, the more tempting it is...

With invisible problems and business pressure that doesn’t let up, we just keep repeating the same cycle.

Software Rewrite Cycle

“A description of the goal is not a strategy.” -- Richard P. Rumelt

What’s wrong with our current strategy?

Our “Strategy” for Success

High Quality Code

Low Technical Debt

Easy to Maintain

Good Code Coverage

RESET “A good strategy is a specific and coherent response to—and approach for overcoming—the obstacles to progress.”

-- Richard P. Rumelt

The problem is we don’t have a strategy...

So what are the biggest obstacles that cause the software rewrite cycle?

What specific approach are we going to take to overcome the obstacles?

How I turned my project around...

We did all the “right” things.

How I turned my project around...

We did all the “right” things.

...and brought down production 3x in a row.

Technical Debt Mistakes

I thought the main obstacle was Technical Debt

Mistakes ?

Most of our mistakes were in the most well-written parts of the code.

We made significantly more mistakes in code that we didn’t write ourselves.

Lower Familiarity

More Mistakes =

There had to be more to the story...

...pain...

This is what I knew...

What made development feel painful?

Unexpected Behavior

Problem Resolved

Tracking Painful Interaction with the Code (Friction)

Troubleshooting

Progress

5 hours and 18 minutes of troubleshooting...

PAINFUL

The Amount of PAIN was Driven By...

What Causes Unexpected Behavior (likeliness)?

What Makes Troubleshooting Time-Consuming (impact)?

What causes PAIN?

Familiarity Mistakes

Stale Memory Mistakes

Semantic Mistakes

Bad Input Assumption

Tedious Change Mistakes

Copy-Edit Mistakes

Transposition Mistakes

Failed Refactor Mistakes

False Alarm

Non-Deterministic Behavior

Ambiguous Clues

Lots of Code Changes Noisy Output

Cryptic Output

Long Execution Time

Environment Cleanup

Test Data Generation

Using Debugger

What causes PAIN?

Stale Memory Mistakes Semantic Mistakes

Copy-Edit Mistakes

False Alarm

Ambiguous Clues

Lots of Code Changes Noisy Output

Cryptic Output

Long Execution Time

Environment Cleanup

Using Debugger

What causes PAIN?

Stale Memory Mistakes

Semantic Mistakes

Copy-Edit Mistakes

False Alarm

Once we understood causes, most problems were avoidable.

Ambiguous Clues Lots of Code Changes Noisy Output

Cryptic Output

Long Execution Time

Environment Cleanup

Using Debugger

An Avoidable Problem

Iterative Validation with Unit Tests 7:01 0:00

14:23 0:00

Skipping Tests and Validating at the End

Iterative Validation with Unit Tests 7:01 0:00

14:23 0:00

Skipping Tests and Validating at the End

Urgency Leads to High-Risk Decisions

If I make no mistakes I save ~2 hours.

If I make several mistakes I lose ~8 hours.

An Avoidable Problem

Software is a Reflection of our Decisions

Technical Debt is just a Symptom of the Problem

Software is a Reflection of our Decisions

How much work does it take to mop the floor?

How much work does it take to complete a software task?

= Side-Effects from Ignoring the Risk

Trade-off Decisions

Writing Unit Tests Troubleshooting Mistakes Direct Cost Indirect Costs

We make high-risk decisions because the indirect costs are hard to quantify.

Likeliness of Event

Potential Impact x =

Troubleshooting Learning Rework

Side-Effects in Software

Troubleshooting Risk

Learning Risk

Rework Risk

Ignore Tango (risk not obvious)

Cycle of Escalating

Ignore Risk

Tangos Multiply

Cycle of Escalating

Ignore Risk

More Mopping Work

Cycle of Escalating

Ignore Risk

Further Behind on Deliverables

Cycle of Escalating

Productivity plummets because of the overwhelming side-effects

Ignore Risk

I can’t go any faster!

Case Study 1: Healthy project about 10 months old

Troubleshooting

Progress

Learning

Rework 10-20% friction

Effects of Escalating Risk

Case Study 2: Thrashing project about 18 months old

Troubleshooting

Progress

Learning

0:00 28:15

12:23 0:00

Case Study 3: Post-meltdown project about 12 years old

Troubleshooting

Progress

Learning

7:07 0:00

0:00 19:52

Case Study 1

Case Study 2

Case Study 3

2 days

3 days

We can’t see these effects by measuring velocity or task lead-time.

What specific approach are we going to take to overcome the obstacles?

Obstacle: Business Pressure “We aren’t given time to work on improvements!”

(nobody understands the problems)

Developers are stuck because they don’t have control.

What are we supposed to do?

Management is stuck because they don’t have visibility.

Organizational Deadlock

1. Don’t ask for Permission 2. State your Goal "I want to make the business case to management for fixing things around here. No more chaos and working on weekends, this needs to stop. But I need data to make the case so I need everyone's help." 3. State the Plan "Here's what I'm thinking. I want to run an experiment to record data for one month on all the time we spend troubleshooting. We can look at the data together and identify our biggest problems, then I’ll write it up and present the case to management to get things fixed.” 4. Enlist the Team “Will you guys help me make this happen?”

Make the Decision to Lead

1. Don’t ask for Permission 2. Make the Goal Clear to Your Team "I want to make the business case to management for fixing things around here. No more chaos and working on weekends, this needs to stop. But I need data to make the case so I need everyone's help." 3. State the Plan "Here's what I'm thinking. I want to run an experiment to record data for one month on all the time we spend troubleshooting. We can look at the data together and identify our biggest problems, then I’ll write it up and present the case to management to get things fixed.” 4. Enlist the Team “Will you guys help me make this happen?”

Share the PAIN!

Over the last month, we’ve spent ~50% of our time troubleshooting

Troubleshooting JavaScript Errors

Share the PAIN!

With 25 developers on the team, rushing to save a couple hours caused over 1000 hours of developer downtime

Blame the System...

The likelihood of mistakes and the potential impact

is difficult to quantify and explain.

The product owner can’t help making

ill-informed decisions.

Blame the System...

“...we keep deferring the problems and taking risks because we get stuck in a cycle of urgency.”

Constant Urgency

Blame the System...

“...now the problems are out of control.”

Take Responsibility

Dedicated resources (1 or 2 developers)

Control of all decisions for improvement work

Make a commitment to show results in 3 months

Ask for a 3-Month Trial

Now the challenge is getting results.

Obstacle: Knowing What to Fix

Idea Flow Learning Framework “Idea Flow” is a metaphor for the

human interaction in software development

Idea Flow Map

Idea Flow Learning Framework

Troubleshooting Learning Rework

Strategy for improving software predictability by optimizing developer experience.

Need a Feedback Loop...

Mentorship Process

1.  Make the Problems Visible •  Understand the Problems •  Improve Decision Habits

Idea Flow Learning Framework Strategy for improving software predictability

by optimizing developer experience.

Visibility Framework Visibility helps us Identify and the Problems

2. Record an Idea Flow Map

Idea Flow Mapping Commands

Troubleshooting

Progress

Learning

Rework

Troubleshooting

Progress

Learning

Rework

Troubleshooting

Progress

Learning

Rework

Troubleshooting

Progress

Learning

Rework

3. Reflect on Decisions Learn to Read the Visual Indicators in Idea Flow Maps

Similar to how an EKG helps doctors diagnose heart problems...

...Idea Flow Maps help developers diagnose software problems.

Problem-Solving Machine

Learn to Read the Visual Indicators in Idea Flow Maps

3. Reflect on Decisions

= Solution Strategy

The “Heart” of Software Development (the problem-solving machine)

= Solution Strategy

Depending on where the disruptions are in the process, we see a different effect in Idea Flow.

"How did you evaluate the possible options and choose a strategy?”

"What was wrong with the different strategies you tried?”

"What was the discovery that made you choose a different direction?"

Problems with Evaluating Alternatives

"Were you working with something that you were unfamiliar with?”

"Did you run into code that was noisy, ambiguous, or misleading?”

"What do you think made it difficult to learn?"

Problems with Modeling

"Did your task involve changes to complex code or business rules?”

"Were there a lot of details that you had to keep in your head?”

"What was causing the complexity in the validation cycles?"

Problems with Refining

"What experiments did you run to troubleshoot the problem?”

"How many times did you run the experiment?"

"How long did it take to get through each experiment cycle?"

Problems with the Validation Cycle

"Was there something in the code that made these changes especially mistake-prone?”

"How familiar were you with the language and tools you were working with?”

"Were you tired or distracted when you did the work?"

Problems with Execution

1. Make the Problems Visible

The goal of visibility is to ask the right questions so we can identify problems with our strategy.

Visibility Framework Visibility helps us Identify and the Problems

Brainstorming a List

“Technical Debt” Backlog

Our improvements don’t make much difference

Clarity Framework Understand what’s Causing your Biggest Problems

Understand what’s Causing your Biggest Problems Clarity Framework

1. Choose a Focus

Categorize the problem by type to see the impact

Focus on the problem with the highest friction

1. Choose a Focus

Tags: #problemtag

Categorize the problem by type

1. Choose a Focus

“But what problem categories do we use?”

The Ten Pains of Software Development

Design-Fit Pain - When the new feature doesn’t fit well into the existing design.

Requirements Pain - Bad assumptions about what functionality to build

Collaboration Pain - Challenges collaborating with other developers on the team.

Modeling Pain - When it’s difficult to build a conceptual model of the existing software.

Cognitive Pain - Challenges with complexity and intense thinking

Alarm Pain - Challenges with false alarms and test maintenance

Experiment Pain - Challenges with getting feedback by running experiments

Execution Pain - When changing the code is highly mistake-prone

Have an amplifying effect on other problems

Add up Friction by Pain Type

100 hours

50 hours

Troubleshooting

Learning

Rework

Focused Improvements Visible Feedback

Create a Vocabulary of Patterns

Pain Types

Problem Types

Strategy Types

Symptoms Causes Fixes

2. Breakdown the Patterns

What Causes Experiment Pain?

100 hours

50 hours

Create a Vocabulary of Patterns

Why did it take an hour to find a typo?

Slow Feedback Loop - Ran experiments by firing up application and sending email

Numerous Feedback Loops - Problem was tricky to trackdown, sent 23 emails

Tags: #ExperimentPain

What Causes Experiment Pain?

What Strategies Do We Use?

Iterative Unit Testing

Vertical Slices

Talk to the Familiar

Incremental Integration Test

Intelligent Logging

Isolate Hard-to-Test Code

Tight Control

State Controller

Front Load the Risk

How do we know if we’re making things better?

What does “better” really mean?

“Better” following best practices

“Better” solving the problems

Best Practices (solution-focused)

Decision Principles (problem-focused)

What’s a “Decision Principle”?

Answers two questions:

How do we evaluate our situation?

What are we trying to optimize for?

Decision Principles

“You know that thing that happens when you make too many changes at once and it's really hard to troubleshoot problems? We want to avoid that.”

“If I decide to skip the unit tests, how will that affect my haystack size?”

Haystack Effect

The number of unvalidated changes has a huge impact on the

difficulty of tracking down a problem.

“You know when you run an experiment and it’s really hard to figure out what’s going on? We want to avoid that.”

“How can I design my code sandwich to make this easier to troubleshoot?”

Setup the Inputs

Interpret the Outputs

Code Sandwich

By reducing the thickness of my code sandwich I can reduce diagnostic difficulty.

Decision Principles

We can learn what works by running “Strategy Experiments”

3. Run Strategy Experiments

1. Decide on a Strategy 2. Predict the effects in Idea Flow 3. Explain the results

“I’m going to use Iterative Unit Testing to mitigate the risk of a big haystack."

Prediction: My strategy of #IterativeUnitTesting seemed successful, troubleshooting should take <15m

Explain Why? Situation Pattern: #HeavyIntegrationLogic

“I’m going to use Iterative Unit Testing to mitigate the risk of a big haystack."

2. Understand the Problems

The goal of understanding the problems is to identify the “critical few”

that will make the biggest difference.

Pareto’s Law - (80/20 Rule)

Clarity without Awareness...

“Well that was a stupid thing to do. Clearly, I shouldn’t have created a big haystack because troubleshooting would obviously be difficult.”

We have to be aware in the moment

we make the decision.

Stop and Think Practice asking the right question at the right time

“What does your code sandwich look like for this experiment?”

“What are you planning to observe in order to understand the behavior?”

“What parts of the system are you trying to manipulate in order to see variations in behavior?”

Situation

Developer struggling with an experiment (poke the developer)

Ask Questions

Awareness Framework

Think: What questions could I ask my future self to recognize this problem in the future?

Ask the right question at the right time

3. Improve Decision Habits

The goal of improving decision habits is to make the improvements permanent.

Idea Flow Learning Framework (Software Control = Predictability)

So how do we break the software rewrite cycle?

We Learn.

Thank you!

If you liked the talk, please tweet about #ideaflow!

Free e-book if you sign up by July 15th!

@janellekz janelle@newiron.com

Twitter: Email:

How to Break the Software Rewrite Cycle

Software

Sikh Genocide - INDIA: Break the cycle of impunity …sikhgenocide.org/downloads/ASA2000203.pdfAmnesty International January 2003 AI Index: ASA 20/002/2003 INDIA Break the cycle of

Break The Cycle Campaign Study & Analysis

Break the Cycle Now!

Spanish Recipe Activities Listen to authentic recipes and rewrite Cloze Recipe card for rewrite Sample rewrite

Low-cycle fatigue life of a thermal break system under

Value of Client's life to break the vicious cycle

Multi-cycle datapath - University Of MarylandMulti-cycle datapath Multi-cycle implementaion: break up instructions into separate steps Each step takes a single clock cycle Each functional

What are YOU doing to break this vicious cycle?

Break the Always Cycle

SAScon Slides - How will you break the cycle of Bad Marketing?

BREAK THE CYCLE OF DOMESTIC VIOLENCE

Break the cycle

INDIA: Break the cycle of impunity and torture in Punjab

BREAK THE CYCLE (A California Not-For-Profit Organization ...Break the Cycle Los Angeles, California I have audited the accompanying statements of financial position of Break the Cycle

Pantene - Break The Cycle Case Study

How to Break the Software Rewrite Cycle

Break the cycle of Spiritual Osmosis

Break the cycle of domestic abuse instead....Break the cycle of domestic abuse instead. Men’s Referral Service 1300 766 491 Domestic Violence Crisis Line 1800 800 098

Break the Germ Cycle: Hand Hygiene Compliance among

Break the Cycle Press Kit