View
4.660
Download
0
Category
Preview:
DESCRIPTION
How do you measure API success? What KPIs do APIs need? What mistakes should I avoid? Find out what you should, and shouldn't, be measuring as part of your API program in this Business of APIs Conference NYC talk. Dive into a breadth of API metrics, the 6 keys to better API metrics, and the traps to beware of (the important do's and don'ts). Also real-world API case studies show who measures what.
Citation preview
KPIs for APIs
or, why API Calls are the new Web Hits
and you may be measuring all wrong
John Musser / @johnmusser / API Science / apiscience.com Business of APIs Conference, NYC, 2014
1995
Why do we need a Web site?
2000 2005 2010
Of course we have a Web site
Of course we have an API
Why do we need an API?
1995
Why do we need a Web site?
2000 2005 2010
Of course we have a Web site
Of course we have an API
Why do we need an API?
You are here
Remember hit counters?
Hits Page Views Visitors Conversions
Evolution of web site metrics
Calls ? ? ?
Evolution of API metrics?
Great APIs get measured
But measure what?
Well, it depends…
Why do you have an API?
Extend product
BizDev / LeadGen Marketing channel
Upsell opportunity
Content acquisition
Device and mobile support
API as Product Distribution channel
Increase stickiness
New line of business
Accelerate internal projects
Drive innovation
User acquisition
Partner opportunities Drive traffic
Increase footprint
Who is your API for?
API consumer continuum
You Your partners & customers
Everyone else
Internal Open Private
The answers to who & why
should drive your KPIs
The answers to who & why
should drive your KPIs
Traffic Total calls
Top methods Call chains Quota faults
Developers Total developers
Active developers Top developers Trending apps
Retention
Service Performance Availability Error rates
Code defects
Marketing Dev registrations Dev portal funnel Traffic sources Event metrics
Support Support tickets Response times
Business Direct revenue
Indirect revenue Market share
Costs
Many ways to think about API KPIs…
Customer Satisfaction
NPS Churn
Community Forum activity Social media
(both for API and apps built on it)
Quality Defects
Error prone APIs Repair rate
API Maturity Stability
Maturity metric Change rate
Innovation Number of products
Number of apps Speed to market
More ways to think about API KPIs…
Channel Call volume: Mobile,
web, others Revenue by channel
There are a lot of API KPIs
you could measure
Don’t drown in API KPIs
Great APIs prioritize
what to measure
KPI SECRET #1
Choose your “KPI lens” CFO lens CMO lens CEO lens Ops lens
QA lens CRO lens PM lens PR lens
CFO lens: financial metrics Total direct and indirect revenue
Margin ROI
Marginal cost Capex / Opex investment
CLTV per developer
DevOps lens: operational metrics Performance Availability Error rates
Faults by API type Defect backlog
API traffic trends
CMO lens: marketing metrics Developer acquisition cost Developer demographics
Portal visitor count & traffic sources SEO / SEM metrics
Social media metrics Evangelism and event metrics
Know your Developer Funnel
KPI SECRET #2
Know your Developer Funnel
KPI SECRET #2
(for public APIs)
Web sites have funnels
Acquisi9on
Ac9va9on
Referral
Revenue
Reten9on
How do users find you?
Does the user have a good first experience?
Do users return?
How do you make money?
Do users tell others?
Dave McClure’s “Startup Metrics for Pirates”
APIs have funnels too
Acquisi9on
Ac9va9on
Referral
Revenue
Reten9on
How do developers find you?
Does the developer have a good first experience?
Do developers return?
How do you and developers make money?
Do developers tell others?
Potential KPIs
Acquisi9on
Ac9va9on
Referral
Revenue
Reten9on
Dev portal site: sources, traffic vol, uniques
# dev registraCons, API keys issued
# acCve devs, API call volume by dev, # apps
$ per dev, $ per app, # end users
# dev referrals, dev/app growth, NPS
Measure your API ecosystem
KPI SECRET #3
Apps Developers App Users
API 1
API 2
API 3
Your APIs You
Apps Developers App Users
API 1
API 2
API 3
Your APIs You
Total app users Revenue per user User growth rate
User churn
Apps Developers App Users
API 1
API 2
API 3
Your APIs You
Total apps Trending apps
Revenue per app Channel: mobile/web
Apps Developers App Users
API 1
API 2
API 3
Your APIs You
Total developers Active developers
Revenue per developer Top developers
Stuck developers
Apps Developers App Users
API 1
API 2
API 3
Your APIs You
Total API calls Top API calls
Top API packages Internal usage
Apps Developers App Users
API 1
API 2
API 3
Your APIs You
Total revenue Partnerships Market share
Innovation
Most APIs obey the
Power Law
KPI SECRET #4
The Power Law of APIs
Developers
API c
all vo
lume
The Power Law of APIs Most of your API traffic, revenue, success, etc….
Developers
API c
all vo
lume
Think quality, not just quantity Use your API KPIs, to find who, how, where, when.
Developers
API c
all vo
lume
API KPIs need buy-in
KPI SECRET #5
Don’t let API KPIs live in a vacuum
Who are your stakeholders?
Who’s your sponsor?
Get executive buy-in on your top 3-5 KPIs
Measuring internal API usage often just
as important as external
KPI SECRET #6
API usage from internal apps
External usage
Guardian: 70% internal calls
Evernote: 99% internal calls
API usage from internal apps
External usage
API usage from devices
External usage
Netflix: 99.9% internal calls
Netflix: 99.9% internal calls
Why Enable Netflix experience
Who Netflix device and UI teams
Beware of API vanity metrics
KPI WARNING #1
API Billionaire Club
5 billion API calls / day (April 2010)
1 billion API calls / day (Q1 2012)
5 billion API calls / day (October 2009)
13 billion API calls / day (May 2011)
1 billion API calls / day (January 2012)
1.1 billion API calls / day (April 2011)
1 billion API calls / day (May 2012)
5 Billion API calls/day
“What if, all things being equal, instead of 30 billion API calls a month,
we served 5 billion?” Daniel Jacobson, Netflix
Beware of API KPI traps
KPI WARNING #2
Not having a metrics process (have a metrics plan & a feedback loop)
Using the wrong metrics (don’t be afraid to change what you track)
Unrealistic expectations (“Absolutely, we’ll have X developers on-board by X date”)
3 common API KPI traps
One important metric you may have missed...
What’s your TTFHW? Time To First “Hello World” aka: how long from zero to 60?
What’s your TTFHW? Developer experience (DX) can make or break your API.
What’s your TTFHW? Too subjective to measure?
What’s your TTFHW? Don’t forget about qualitative metrics
What’s your TTFHW? User testing.
Surveys. Forums.
Social media…
Real world API KPIs
“We are seeing a trend where a person who uses another app with Evernote is
50% more likely to move on to our premium (paid) service”
KPI: Conversions
“Apps that have an API integration with Evernote are seeing much more usage:
Pocket readers normally read 15% of what they save, but Evernote users read 80% of
what they save in Pocket.”
KPI: Engagement
API now accounts for 20-25%
of the charity’s annual revenue
KPI: Revenue
KPI: Photo Edits
KPI: Apps
3,500 Apps Aviary, March, 2013
Goal: “more eyeballs on our content”
API calls (requests) Response volume (ex: # stories) Impressions (via image beacon) Loyalty
MoreKPIs
KPI: Impressions
“We sell Vodka. But people buy drinks.”
Goal: “ensure top quality drinks consistently delivered to people
regardless of where people encounter Absolut”
KPI: Recipe delivery
“We sell Vodka. But people buy drinks.”
Goal: “ensure top quality drinks consistently delivered to people
regardless of where people encounter where people encounter
Absolut”
KPI: Recipe delivery APIs let you be where your customers are.
Measure how.
Calls ? ? ?
Evolution of API metrics?
Who & why drives KPIs Have a KPI lens & process
Prioritize & repeat
Thank You
john@apiscience.com @johnmusser
Race car: hCp://www.flickr.com/photos/lim_lik_wei/3270522646/ Stopwatch: hCp://www.flickr.com/photos/purplemaMish/3020016417/ Stop sign: hCps://www.flickr.com/photos/9toperez/5791194358 Power law: hCp://en.wikipedia.org/wiki/Power_law
Photo Credits
Recommended