Team Intelligence For Data-Driven Product Development ... · PDF fileTeam Intelligence For Data-Driven Product Development usenotion.com Great KPIs are: • Actionable: Create a goal

Team Intelligence For Data-Driven Product Development usenotion.com

http://usenotion.com



Table of ContentsAcquisition

Activation

Referral

Retention

Revenue

Team

p.7

p.12

p.17

p.21

p.26

p.31

Conversion From Lead Source 8

Conversion to Signup 9

Conversion from Trial 10

Customer Acquisition Cost 11

Average Sessions per User 13

Average Session Time 14

MAU/WAU/DAU 15

Feature Utilization 16

Net Promoter Score 18

Product Invite Conversion 19

Social Media Referrals 20

Customer Churn 22

Complaints per Customer 23

Feature Requests 24

Revenue Churn 25

Average Revenue per User 27

Average Revenue per Employee 28

Customer Lifetime Value 29

Monthly Recurring Revenue 30

Backlog Burn Rate 32

Cost of Development per Unit 33

Cycle Time 34

Data Removal Efficiency 35

Lead Time 36

Team Health 37

Throughput 38

Total Defects 39

Work in Progress 40

Velocity 41




The AARRRT! of Great Productimportant to think about the best fit for your users’ goals. There are a number of KPIs that are commonly tracked in many of the most successful technology companies. You may have come across business KPIs like ARPU or LTV, but sometimes it’s hard to see how those metrics are relevant to the process of creating and building the product.

This guide will take you through many of the most popular KPIs for product develop-ment, business, and customer success. Our goal is to teach you how measuring and track-ing these metrics will help you to make better, more profitable products.

For product people, finishing features or dealing with bugs can take up so much time that it’s hard to see the bigger picture. But to create really loveable, effective product for your customers, you need to understand how the work your team is doing fits in with the needs of your customers and the broader vision of your company.

Data is a great way to measure and test the effectiveness of your work, but many of us feel like we don’t have time or even know where to start when collecting and comparing data.

KPIs FTW!Key Performance Indicators vary acrossdifferent kinds of companies, and it’s




Great KPIs are:• Actionable: Create a goal for the KPI. If the

indicator’s value changes, you’ll know what actions to take next.

• Measurable: Choose KPIs that are specific and can easily be tracked and recorded.

• Comparable: Knowing how your KPIs inter-relate can help you discover how moving one lever can affect other outcomes.

• Valid: Indicators aren’t really useful unless they are based on good data and correct assumptions. Testing can help to ensure that the basis of your KPIs connects with the goals you’re trying to achieve.

Performance indicators need to give youactionable information about how to improve.

To create good indicators, first assess yourobjectives and your strategy to meet goals that further the objective.

Every KPI should have a direct correlationto performance towards the goal. The KPImay help the team detect problems orimpediments to the goal and also help driveawareness of actions to further the goal.So think about how a data point based on measuring something will tell you whetheryou are reaching a goal.

The Recipe for Great KPIs



Leading indicators tend to be predictive and can change quickly. They help you under-stand the direction things are going. Leading indicators are hard to measure and easy to influence.

Ideally, every decision-maker and stakeholder in your business will be able to see KPIs in a dashboard that shows where you are meeting goals, where you are off-target, and what factors correlate.

When a team is able to meet or achieve a KPI, then the business goals have been met for that particular objective. The next step is to set the goal higher for that same KPI—or focus on a new KPI with a new objective.

• User-focused: KPIs work best when you understand that improving them directly impacts the success of your customer.

• Owned: Make sure there is someone who is accountable for each KPI so it doesn’t get lost.

• Shared: KPIs are best used to help the entire organization understand how things are going and what issues need to be addressed.

Lagging indicators track output. They are easy to measure but hard to improve or in-fluence, since they deal with past work. Often these are your financial numbers, and they give you a concrete picture of where you are today.




typically you will be reviewing those numbers at least weekly.

We built Notion to give you an easy, beauti-ful way to look at KPIs with your team. In the following guide, you’ll see the recipes we’ve created to make tracking your indicators straightforward and simple.

How often should you measure?Most KPIs need to be reviewed at least monthly—though we’ve found that data-focused PMs are looking at these numbers every day—to get a picture of where they are going and what might be moving the needle.

With campaign-focused metrics, considerthe goals you’re trying to hit and track often enough to see if you need to iterate. Some metrics won’t change frequently—such as Revenue per Employee—and some are tied to data you may only be collecting periodically—such as Net Promoter Score. Your Team metrics will be tied to your development cycle, and





The Metrics

Discovering where customers hang out andwhat kinds of messages motivate them can be useful for product personas. And knowing how much it costs to acquire customers is important to compare with how much value the productdelivers to users.

AcquisitionAcquisition metrics help you understand which sources or channels are the most useful fordriving customer growth. Typically, thesenumbers are marketing and sales-focused, but they are valuable for product as well.

Marketing folks will tell you about TOFU (the Top Of the FUnnel, not a tasty soy product). Basically,you need to identify real indicators that suggest your efforts to bring new customers to theproduct are successful.

8

Acquisition Metrics

Conversion from Lead SourceLead Sources are purely marketing metrics, but you

may want to keep them in mind, simply because

understanding what’s driving customers to fill out

a landing page form or reply to an email can be

informative to what users expect from the product.

Unfortunately, product teams tend to ignore

marketing metrics and often vice-versa. More

communication between groups can lead to better

customer onboarding, better retention and ultimately,

better product.How to calculate

9

Acquisition Metrics

Conversion to Signup RateConversions can be helpful to keep an eye on

since they can give you a good picture of what users

are looking for. For example, if you have great

conversions at signup but a low conversion rate

from trial, it could be that you are promising some-

thing you can’t deliver on to customers, that you

are initially selling to the wrong person, or product

usability is a problem.

Based on the action you want your user to

accomplish, you can measure conversions for

any behaviour, but signing up and paying for your

service are key metrics.

How to calculate

10

If you have lots of customers signing up for a 14 or

30-day trial and your Conversion from Trial rate is

low, you may want to conduct surveys or user

interviews to find out where you’re going wrong.

Growth Hacker Lincoln Murphy suggests tracking

the “Common Conversion Activities” your success-

ful customers perform, such as Feature Utilization,

Lead Source and reading your blog or helpcenter

materials. Compare these metrics in a dashboard

like Notion’s to assess what actions to take.

Acquisition Metrics

Conversion From Trial Rate

How to calculate

11

Acquisition Metrics

Customer Acquisition Cost (CAC)

This metric measures how much it costs, on

average, for your company to acquire a user.

Sometimes companies, especially in ecommerce,

also measure Cost Per Action (CPA) which applies

to the cost of signup rather than the total costs

associated with acquisition, which include sales

salaries, promotions and marketing campaigns.

In general, early-stage companies will spend as little

as possible on customer acquisition until they are

sure they have a working product with product-

market fit. Once you are prepared to spend money

to acquire customers, you’ll need to be very clear on

who to spend it on, so some form of user personas

are recommended. You’ll also need to understand

the LTV (customer lifetime value) so you know if

spending more makes sense.

How to calculate

The Metrics

Every company will define “active” and “utilization” differently. A user could be “active” if they login on a regular basis, or perform an action likeinstalling code or making a profile.

Some products need hours of use every day to make it useful, whereas some will work just fine if users visit weekly or a couple times a month. We’re curious what works best for you, so drop us a line and we’ll incorporate your thoughts into a future lesson in the School of Little Data.

So, your marketing and sales teams are amazing and you have tons of users signing up. All done, right? If you’re working on product, this is where the journey actually gets interesting. By under-standing what your users are actually doing, you can see if they are understanding the vision you have to make them successful, or just getting stuck. Data can give you insight into where your product is most useful and where it’s superfluous or even annoying. With this knowledge, you can reduce Churn and increase LTV.

Activation

http://school.usenotion.com

13

Activation Metrics

Average Sessions Per UserA user’s session can be defined as the interval of

time in which you are receiving events from that

user if the time between events is less than 30

minutes.

Average Sessions per User is a simple metric that

will help you understand how your user in engaging

with your product. In many cases, it’s great when a

customer is using your product frequently. Based

on the job your product does, you can speculate

how many times your customer would optimally use

your product, and set goals.

How to calculate

14

Activation Metrics

Average Session Time can give you insight into the

engagement of your customers. After all, the more

time they are on the site, the better, right?

However, AST typically includes time that your

customer is “in the product” but may not be using

it all. You can analyze this metric better in conjunc-

tion with Average Sessions per User and specific

product utilization metrics.

Average Session Time

How to calculate

How to calculate

15

Activation Metrics

A number of measurements can help you understand

how users are exploiting your product, and which

features they don’t care—or possibly know—about.

These will tend to be somewhat specific to your

product, but generally, you will want to collect data

on how many customers used a feature, whether

they used it repetitively (x2-∞), or whether they “con-

verted” within the feature (performed the action you

designed for, such as creating a report). Typically this

metric will be most relevant for the onboarding peri-

od, the first 30–90 days.

Feature Utilization

16

Activation Metrics

This metric seems basic until you have to define

“Active.” You might imagine that the user is “Active”

if they visited a dashboard or installed a piece of

code. But different types of users may be measured

by different definitions of “Active”. Ideally, you will

define it by an action or set of actions that suggest

the customer is meaningfully using the product.

DAU, WAU, and MAU are often considered vanity

metrics. You may find it useful, though, to compare

the MAU/DAU ratio to see how “sticky” your product

is. It’s harder to retain customers who only use your

product once in a while.

Daily/Weekly/Monthly Active Users (DAU/WAU/MAU)

How to calculate

The Metrics

in your product or analytics from your marketing site or from social media.

If you have sharing or referral features in-app, you will want to understand who is activatingreferrals and to whom you are being referred.

ReferralMaking your product shareable makes it much cheaper and easier to acquire new customers.So you’ll likely want to understand the ways your customers are referring you.

Some of the most successful companies rely on referrals within the product. Dropbox famously grew by offering a credit for referrals, and Uber let users give free rides to friends. Make sure your referrals can be tracked and added to your tracking system. This may involve tracking events

18

0-6 Detractors: customers who are likely to conveynegative impressions to their network

7-8 Passives: customers who are mildly positive toindifferent about the product

9-10 Promoters: customers with high satisfactionwho will actively evangelize about the product

Referral Metrics

NPS was developed to give a metrically significant,

simple way of understanding customer satisfaction.

In traditional NPS surveys, you ask your customers

“How likely are you to recommend our product?”

on a scale from 0-10, with 10 as the most likely.

Typically, this is the only question in the survey, with

the option of providing qualitative feedback by

asking an open-ended question like “Why did you

pick that number?”

A principle of both Lean methodology and NPS is

“closing the loop,” meaning that customers who

offer feedback should be contacted and heard.

This process gives you an opportunity to make a

detractor into a promoter, and also gives you

insights into the customer’s experience.

Asking for an assessment of “satisfaction with”

versus “likely to recommend” has been shown to

be as or more effective to NPS.

Net Promoter Score (NPS)

How to calculate

How to calculate

19

Referral Metrics

Many software products include a way to invite new

users, since ideally, your existing customers are the

most effective source of generating qualified new

customers.

You want to make sure that the methods you’re

using to generate these referrals are effective.

Tracking how well your inproduct referral engines

are functioning can help you understand what

methods work best, what customers are promoting

you to others and how to better encourage new

customers to use the product.

Product Invite Conversion

20

Referral Metrics

Typically social media stays in the “marketing

domain” and is certainly a “vanity metric,” but

keeping track of engagement numbers like shares

via Twitter or Facebook relative to product release

cycles can help you understand your customers

as well as help in thinking about ways to generate

shares within the product.

You can also think about sentiment when tracking

social media. It’s great when users are engaged,

but a little less awesome when they are all unhappy.

On the other hand, negative sentiment can give you

visceral feedback on your product pain points.

Referrals via Social Media

How to calculate

The Metrics

Another key KPI related to Retention is Churn, which can tell you if your rate of losing customers is a cause for concern.

Retention matters for SaaS companies far more than for many other businesses. For example, if you sell diamond engagement rings, you likely don’t expect or need your customers to come back often. On the other hand, a company that sells subscription-based marketing software will likely lose a customer if the user doesn’t return to the product on a regular basis.

Retention can be the most important product metric for SaaS and is often overlooked. Build-ing retention into your product should start with your first touch with a user, ensuring that they have a complete understanding of the ways your product can help them and how to achieve those goals.

To measure retention, you will typically look at activation metrics (i.e. how you defined an “Active User”) and calculate the average instances ofactivation per user or per account.

Retention

22

Retention Metrics

Customer ChurnA typical solution is to calculate churn as (number

of customers who churned) divided by the sum

of (customers at the beginning of the period) +

(customers at the end of the period) divided by 2.

Unfortunately, this won’t give much of a picture of

the trend of churn: if there is a significant difference

between January and March, it will be hidden. One

way to attack this problem is to calculate churn with

a daily average, then multiply over the timeframe

you want to measure.

Churn is a key metric that touches everyone at the

company, so experiment with different formulas

until you find the one that works best for you and

make sure everyone understands your goals.

How to calculate

23

Retention Metrics

Complaints per CustomerAlso known as “Active Issues,” this metric lets you

know how many issues customers are having at

a given time. Knowing what issues customers are

having can help you make better product decisions,

while understanding Resolution Rate (how fast

issues are resolved) can be instructive about the

kinds of solutions needed.

For example, you may simply need to improve UI

so that customers better understand how to use

a feature. On the other hand, there may be more

complicated issues related to product-market fit

or persistent technical issues. Using user issues in

the development process is essential to creating

products your customers love.

How to calculate

24

Retention Metrics

Feature RequestsFeature Requests might be considered more of a

qualitative metric than an actionable measure. That

said, tracking requests by category can give you a

sense of how well your product is loved or even

understood.

For example, if you notice that there are persistent

feature requests for a feature you already have, you

need to do a better job of uncovering the feature for

new and existing customers. If you notice growing

feature requests in a certain category, you may want

to investigate further to find out if there is market

potential for growth in that area.

25

Retention Metrics

Revenue ChurnCompare Customer Churn and Revenue Churn to

help you understand your user trends. For example,

you could have a net increase in revenue with an

increase in customer churn if you gained a number

of people on a $99/month plan but lost many more

customers at your $5/month level. Low-value

customers are typically more likely to churn because

they’ve made less of an investment overall in your

product.

New sales don’t count. You can also calculate

Revenue Churn in regards to certain segments,

like RC based on downgrades or RC based on

cancellations.

How to calculate

The Metrics

Revenue metrics also help you plan product de-velopment, since you don’t want to build a lot of features if no one is there to pay for them. At an earlier-stage company, you may not be able totell much from a snapshot of revenue, butcomparing these metrics from month to month can be a good way to see if you are heading in the right direction.

You love your product and making and improving it can be fun and challenging. But at the end of the day, you need to make money so you can eat and pay for things like cool quadcopter drones (or, you know, whatever YOU need). Revenuemetrics help you see how well your product is resonating with customers based on whether they find it valuable enough to pay for it.

Revenue

27

Measure the average amount of revenue each customer pro-

duces in connection with data on upgrades and downgrades

as well as Churn to understand how you can increase ARPU.

ARPU is sometimes considered a vanity metric. If you’re just

dividing the sum of your MRR by your number of customers

(the simplest way to calculate ARPU), you might not learn

enough that’s actionable.

To improve the way you use ARPU, consider a cohort analy-

sis, or just compare signups from the last six months versus

older customers (or whatever sample makes sense for you).

If ARPU for new customers is higher than for ones who have

been around a while, then you might be on a better track

than if the reverse is true. ARPU can reflect whether you are

selling to the right people at a company or if there is prod-

uct-market fit. Your ARPU needs to be high enough that you

can afford to acquire the type of customers you serve.

You will use ARPU to calculate another KPI, LTV (customer

lifetime value).

Average Revenue Per User (ARPU)

How to calculate

28

Revenue Metrics

Customer Lifetime Value (LTV)LTV gives your marketing team a baseline on CAC

(Customer Acquisition Cost). You don’t want to

spend more on getting a customer than they are

worth in their lifetime. Typically, the taget ratio of

LTV:CAC is 3:1 or better.

For more accurate results, you can compare LTV of

more recent customers to your earlier customers to

see if you are trending higher.

Another interesting aspect of LTV is churn. Statisti-

cally, lower-value plans tend to churn more, since

the customer tends to be less invested. So let’s

all charge $10,000 a month! Unfortunately, the

corollary generally tends to be fewer customers

on pricier plans. You’ll need to look at the total

revenue per plan level to get a more balanced

picture.

How to calculate

29

Revenue Metrics

Monthly Recurring Revenue (MRR)Though the more punk rock among you may

associate this acronym with “Maximum Rock & Roll,”

Monthly Recurring Revenue is a bedrock of business

KPIs. You know that you need to know how much

money is coming in each month, but then what?

Comparing MRR with factors like team performance,

customer satisfaction, and feature utilization can

offer useful insights and areas to test.

How to calculate

30

Average Revenue Per EmployeeThis metric can help you in two ways. First of all, you

can compare it to industry standards to see how

you compare to your competitors. For example, the

average RPE for publicly traded SaaS companies is

about $200k.

Even more importantly, it can help you to under-

stand when you have the opportunity to hire, and

what effect adding or subtracting staff has on the

overall growth of your company. As a lagging

indicator, RPE is more helpful in understanding

how the company is performing than as a metric to

optimize for.

Revenue Metrics

How to calculate

The Metrics

KPIs such as DRE, Total Defects/Bugs, lines of code, and other metrics are typical ProductQuality measurements. These metrics can be helpful to understand the technical quality of your product, but it’s a good idea not to become too obsessive about metrics like these relative to others that focus more on the larger goals you have in delivering success to your customers. Development teams can also look at Churn and Customer Complaints to get a better overallpicture of product quality.

For Product Managers, team statistics can help to drive quality, efficiency and accountability during product development. In the past, it was hard to visualize the success of the development teamin relation to other metrics. After struggling with this issue in our previous companies, we createdNotion to help you track and compare your team’s success, their morale, and effectiveness against business metrics like revenue or custom-er behavior.

Team

32

Team Metrics

Your Backlog Burn Rate tells you how fast your team

is accomplishing the tasks and stories in the back-

log. This metric gives context to your Velocity and

Sprint Burndown charts so you can determine if

your team is prepared to tackle upcoming projects.

Use this metric to help plan new features and up-

coming sprints.

Read more about Backlog Burn Rate in the School of

Little Data.

Backlog Burn Rate

How to calculate



33

Team Metrics

This KPI measures how much it costs you to actually

make your product, primarily in terms of staff and

software.

Use this KPI to compare with MRR and ARPU to see

if you are pre- or post-breakeven. You don’t have

to know the exact salaries of your team—you can

simply take their estimated average annual salary x

1.15 (adding 15% overhead for benefits and payroll

taxes).

Cost of Development Per Unit

How to calculate

34

Team Metrics

Cycle Time measures the completion rate of your

product. It is always a subset of Lead Time, which

measures the time from a ticket being created to

the time the work is completed. Cycle TIme mea-

sures the actual time it took to build.

Cycle TIme is the inverse of Throughput. It’s related

to Lead Time by the following equation: Lead Time =

Cycle Time * WIP, according to Little’s Law.

Cycle Time

How to calculate

35

Team Metrics

Defect Removal Efficiency (DRE)DRE is a percentage of the bugs identified and

corrected internally compared to the total bugs

in the complete release life cycle. So, DRE is the

percentage of bugs caught by code reviews, unit

tests, and QA.

To use a DRE effectively, you may want to classify

bugs by common sources, like Design, Build,

Requirements, or Deployment to Production. You

can also create an overall DRE to assess general

quality in the product.

Read more about DREs in the School of Little Data.

How to calculate


How to calculate

36

Team Metrics

Lead Time is the average time it takes between the

creation of a ticket and its completion. You can cal-

culate Lead Time as Cycle Time multiplied by WIP or

as WIP divided by Throughput.

Some consider Lead Time to be the overall measure

of product development process. Being quick and

responsive to new customer stories can give a

product an edge in a competitive market.

On the other hand, good product management

requires thoughtful feature development rather

than answering every customer’s demand, so Lead

Time may be deceptive. A good practice is to

compare this metric in relation to actual customer

satisfaction or NPS.

Lead Time

How to calculate

37

Team Metrics

How your team actually feels about the work they

are doing can be instructive for the health and

success of the product itself.

By tracking how your team feels about product

quality, morale, and efficiency, you can correlate

team health with product development. Of course,

it’s a great idea to have a sense of how your team

feels for other reasons, like keeping them happy

and productive.

To facilitate this metric, Notion includes the Team

Polling feature, a lightweight survey tool that keeps

track of how team members assess accuracy,

defects, morale and other aspects of their work.

Read more about Team Health in the School of Little

Data.

Team Health



38

Team Metrics

Throughput is a simple metric that compliments

Velocity when used correctly. Velocity alone doesn’t

reflect the quality of work along with how much

effort it took to do, but by adding Velocity and

Throughput, you’ll be able to understand the real

results you are getting.

Software development doesn’t ever “finish” so you

need to have a metric that helps you understand

how much you are getting done over time.

Track throughput over long periods as well as

sprints to help you ensure you are consistent and

sustainable. Throughput should be tied to your

specific goals rather than inconsistent factors like

story points, which can vary over different teams.

Throughput

How to calculate

39

Team Metrics

Bugs are a problem for camping or cookouts and

they’re even worse when they start piling up in your

product.

Having a picture of how many bugs you have is criti-

cal to product planning and will often correlate with

Customer Complaints. This metric is a component of

DRE.

Total Defects/Bugs

How to calculate

40

Team Metrics

Your WIP is a simple key performance indicator

used to track how much work your team is doing at

any given moment.

Typically, too much in your WIP will lead to decreased

quality and performance for your team. Limiting

context shifting for your developers leads to better

and faster work, and limiting your WIP to essential

components can improve your overall efficiency.

Learn more about WIP in the School of Little Data.

Work In Progress (WIP)

How to calculate


41

Team Metrics

Velocity is a leading indicator that measures the

effort needed to deliver value. Units can consist

of features, user stories, requirements or backlog

items. Use this metric along with Throughput and

Cycle Time to evaluate your development process.

If Velocity is constant, it can be helpful in planning

sprints. For example, if you plan a sprint over a two-

week period and one of those days were a holiday,

you might estimate a 10% reduction in Velocity

(assuming you’re not working weekends!)

Don’t focus on maximizing Velocity, since your DRE

can go down significantly if you’re only worrying

about what you got done, versus the quality of your

work.

Strong product management and ownership, fre-

quent and regular release cycles, and effective UX

design prior to development will improve velocity.

Velocity

How to calculate


get in the way of a good experience. You may need to ramp up if your customers generally need more features. And you need to under-stand clearly how everything you are building will address real user stories.

No one metric can tell you everything, solook at your KPIs in relation to each other. We built Notion to give you a clear, sharable visual interface to compare your KPIs and increase product insight for your whole team.Let us know how we can help you think about setting up or analysing your KPIs. Now that you’ve done all this learning, we’d like to offer one more recipe: The Pirate’s Perfect Dark & Stormy. Don’t forget to stir!

KPIs help you understand your progress. A good Product Manager also needs to consider the customer’s experience through the pro-cess of using a product. When you decide what KPIs to track, consider how your customers are actually interacting with you and what would show you that they are successfully using your product. Don’t focus on “vanity metrics” that don’t really help you understand the needs of the customer and if you’re offering the value that they’re looking for.

When you look at your team’s success, keep in mind that your main goal is to create value for your users. You may need to slow down if you’re generating a lot of bugs or errors that

The Map to Real Treasure




AARRRT!

How to calculate



Documents

Team Intelligence For Data-Driven Product Development ... · PDF fileTeam Intelligence For Data-Driven Product Development usenotion.com Great KPIs are: • Actionable: Create a goal