05/03/2023

Henry Cipolla

Data Driven Development

2

Title Clickbait

305/03/2023

Agenda: Define Data Drive Development Explore DDD in Product Management Explore DDD in Engineering Implementation

Last chance to leave!

405/03/2023

Good buzzword Means different things to different people

For the next 40 minutes it will be:

The grown up version of the LEAN

movement

Defining Data Driven Development

505/03/2023

Picking the right metrics and measuring [your output] against them to get to [your goal] in order to build better products.

Specifically:

605/03/2023

Product Manager

Output = Customer Experience

Goal = Delight

Engineer

Output = Product Code

Goal = Velocity and Quality

Again, different things to different people

DDD for PMs

05/03/2023

805/03/2023

KPIs are too high level– Users, revenue

Conflicting success metrics– Longer sessions, use of utility functions

Poor instrumentation– No chart instantly shows if goal was achieved

Limited AB Testing– AB is only used on the web for content testing

PM: Signs that you are not actually data driven

905/03/2023

1. Define success of this feature / story– Not just usage. Consider NPS, retention, and other behavior

2. Define the chart in your analytics tool that tracks success– Usually this means adding tagging to the story

3. Define the variations to test– Don’t test colors and positions. And do test expected failures

4. Revisit the results frequently

Recommended DDD approach

1005/03/2023

Onboarding Experience– First session length, sessions until successful task

Meal calorie estimator– Pounds of weight lost, meals entered per customer

Music shuffler– New exposures per listener, retention against control

Example feature success metrics

1105/03/2023

Relevant example of a descriptive chart

1205/03/2023

Ask a user how likely they are to refer the app on a scale of 1 to 10

Look at what features and attributes users share on either end

Thank the user by reacting to the feedback. If they score low, ask why. If high, have them rate you

Example NPS Survey

DDD for Engineers

05/03/2023

1405/03/2023

Dashboards only exist to impress investors & scare sales Features are not revisited after they are released Only bad retrospectives happen (postmortems) Eng process is not measured and tested like the product Velocity might be tracked but it isn’t improving and output

doesn’t match expectations

ENG: Signs that you are not data driven

1505/03/2023

Engineering has its own KPIs– Velocity and quality

“Product reflects the org”– Measuring the process can predict product troubles

Small time frames create easy measuring points– There is a reason LEAN and AGILE tend to come together

Your process is a product – be data driven

1605/03/2023

ENPS in disguise with some extras After every sprint (2 weeks) we ask engineers to

anonymously(ish) provide 3 1-5 ratings for:– The company

– Their crew / team

– Their work

Our approach: “Awesomeness”

1705/03/2023

The results are aggregated to describe how things are going

Provides insight into things you wouldn’t expect, like the efficacy of morale events

Makes a great dashboard

Overall Awesomeness

Implementation

05/03/2023

19

1. Business KPIs – track them visibly

2. Define success metrics per-feature, and track them - this will often require platform-specific tools

3. Expose these metrics – daily emails work great

4. Have a retrospective on feature launch. Did the feature hit its goals? Were multiple versions launched – who won?

5. Do this for every feature, frontend and backend as well as every team

How to get started

2005/03/2023

PMs feel more confident experimenting because there is a framework for measuring these experiments

Tradeoff conversations between feature and teams are easier because everybody speaks the same language

Problems are found faster because there are more opportunities for their discovery

Organizations scale easier because there is measurement on all efforts

Benefit of it all

Q&A ?

05/03/2023