Designing & Optimizing the DNA of a Killer App (Dan Olsen, Startonomics SF 2008)

Copyright © 2008 Olsen Solutions LLC

Designing & OptimizingDesigning & Optimizingthe DNA of a Killer Appthe DNA of a Killer AppDan Olsen, CEO & Founder, YourVersionDan Olsen, CEO & Founder, YourVersionStartonomics Startonomics · San Francisco · Oct 2, 2008 · San Francisco · Oct 2, 2008

Copyright © 2008 Olsen Solutions LLCCopyright © 2008 Olsen Solutions LLC

Building the DNA of a Killer App with Building the DNA of a Killer App with Metrics and OptimizationMetrics and Optimization

How to integrate metrics & optimizationHow to integrate metrics & optimization Using metrics to optimize value creationUsing metrics to optimize value creation

Business ValueBusiness Value Customer ValueCustomer Value UI DesignUI Design

Illustrate with real world examplesIllustrate with real world examples


Adding Metrics and Optimization to your Product Process

PlanPlan

DesignDesign

DevelopDevelop

BusinessObjectives

ProductObjectives

Prioritized Feature List

Scoping

Requirements & Design

Code Test Launch

Site Level

Feature Level

OptimizeOptimize Metrics & User Feedback


Optimization through Iteration:Optimization through Iteration:Continuous ImprovementContinuous Improvement

Measurethe metric

Analyzethe metric

Identify top opportunities

to improve

Design & develop the enhancement

Launch theenhancement

Learning

Gaining knowledge:

• Market

• Customer

• Domain

• Usability


OptimizingOptimizingBusiness Value Creation Business Value Creation

with Metricswith Metrics


Approaching Business as an Approaching Business as an Optimization ExerciseOptimization Exercise

Given reality as it exists today,Given reality as it exists today,optimize our business resultsoptimize our business resultssubject to our resource constraints.subject to our resource constraints.

Profit = Profit = RevenueRevenue - Cost - Cost

Unique VisitorsUnique Visitors x x Ad Revenue per VisitorAd Revenue per Visitor

Impressions/VisitorImpressions/Visitor x Effective CPM / 1000 x Effective CPM / 1000

Visits/Visitor x Pageviews/Visit x Impressions/PVVisits/Visitor x Pageviews/Visit x Impressions/PV

New VisitorsNew Visitors + Returning Visitors + Returning Visitors

Invited VisitorsInvited Visitors + Uninvited Visitors + Uninvited Visitors

# of Users Sending Invites x Invites Sent/User x Invite Click-through Rate# of Users Sending Invites x Invites Sent/User x Invite Click-through Rate


Define the Equation of your BusinessDefine the Equation of your Business“Peeling the Onion”“Peeling the Onion”


How to Track Your MetricsHow to Track Your Metrics Track each metric as daily time seriesTrack each metric as daily time series

Create ratios from primary metrics: X / YCreate ratios from primary metrics: X / Y Example: How good is your registration page?Example: How good is your registration page? Okay:Okay: # of registered users per day# of registered users per day Better:Better: registration conversion rate =registration conversion rate =

# registered users / # uniques to reg # registered users / # uniques to reg pagepage

DateDateUnique Unique VisitorsVisitors

Page Page viewsviews

Ad Ad RevenueRevenue

New User New User Sign-upsSign-ups ……

4/24/084/24/08 10,10010,100 29,60029,600 2525 490490

4/27/084/27/08 10,50010,500 27,10027,100 2424 480480

……


Sample Signup Page Yield DataSample Signup Page Yield Data

Daily Signup Page Yield vs. TimeNew Registered Users divided by Unique Visitors to Signup Page

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

1/31 2/14 2/28 3/14 3/28 4/11 4/25 5/9 5/23 6/6 6/20 7/4 7/18 8/1 8/15 8/29 9/12 9/26 10/10

Da

ily S

ign

up

Pa

ge

Yie

ld

Changedmessaging

Added questionsto signup page

Started requiringregistration


Using Metrics to Optimize the Using Metrics to Optimize the Equation of your BusinessEquation of your Business

What are the metrics for your business?What are the metrics for your business? Where is current value for each metric? Where is current value for each metric? How many resources to “move” each metric?How many resources to “move” each metric?

Developer-hours, time, moneyDeveloper-hours, time, money Which metrics have highest ROI opportunities?Which metrics have highest ROI opportunities?

Retu

rnRe

turn

InvestmentInvestment

Retu

rnRe

turn

InvestmentInvestmentRe

turn

Retu

rnInvestmentInvestment

Metric AMetric AGood ROIGood ROI

Metric BMetric BBad ROIBad ROI

Metric CMetric CGreat ROIGreat ROI

• Which metric has highest ROI opportunity?


Case Study:Case Study:Optimizing Viral Loop MetricsOptimizing Viral Loop Metrics

ActiveUsers

Prospective Users

Invite Click

Succeed

Inviteclick-through rate

Conversion rate

Don’t Click

Fail

Invites per sender

% of users sending invites

• Multiplied together, these metrics determine your viral ratio

Users

% of users who are active

= 15%= 2.3

= 85%

Registration Process


Maximum Upside Potential of a MetricMaximum Upside Potential of a Metric

0

100%

0

100%

0

?

Registration conversion rate

% of users sending invitations

Avg # of invites sent per sender

2.3

85%

15%

Max possible improvement

0.15 / 0.85 = 18% 0.85 / 0.15 = 570% ? / 2.3 = ?%


Doubling Number of Invitations Sent per Sender by Doubling Number of Invitations Sent per Sender by Adding Address Book ImporterAdding Address Book Importer


OptimizingOptimizingCustomer Value Creation Customer Value Creation

with Metricswith Metrics


The Fuzzy Math of Customer ValueThe Fuzzy Math of Customer Value You create customer value with a product thatYou create customer value with a product that

Satisfies customers’ needsSatisfies customers’ needs Is easy to useIs easy to use Has a good priceHas a good price Is better than other alternativesIs better than other alternatives

Customer value = customer satisfactionCustomer value = customer satisfaction Not fuzzy: # of users and frequency of useNot fuzzy: # of users and frequency of use Understanding user needs and satisfactionUnderstanding user needs and satisfaction

Quantitative: site analytics, usage metrics, surveysQuantitative: site analytics, usage metrics, surveys Qualitative: support emails, usability sessions (best)Qualitative: support emails, usability sessions (best)


What is Your Value Proposition?What is Your Value Proposition? Which user benefits are most important?Which user benefits are most important? How well does your product deliver?How well does your product deliver?

Benefit 1

ImportancImportancee to User to User

Current Current UserUser

SatisfactionSatisfaction

Benefit 2

User BenefitUser BenefitUpsideUpside

PotentialPotential

Benefit 4

Benefit 5

Benefit 6

High

High

Med

Low

Low

Low

High – +

Doesn’t matter

Benefit 3 Lowopportunity

opportunity

?

?


Analyzing Product Ideas by ROIAnalyzing Product Ideas by ROI

Investment (developer-weeks)

Ret

urn

(V

alu

e C

reat

ed)

Idea C

Idea B

Idea D

Idea A

Idea F

1

1

2 3 4

2

3

4?


Prioritizing Product Ideas by ROIPrioritizing Product Ideas by ROI

Investment (developer-weeks)

Ret

urn

(V

alu

e C

reat

ed)

Idea B

Idea C

Idea A

1

1

2 3 4

2345678

5


The UI Design IcebergThe UI Design Iceberg

VisualDesign

InteractionDesign

InformationArchitecture

ConceptualDesign

Recommended reading: Jesse James Garrett’s“Elements of User Experience” chart, free at www.jjg.net

What most people seeand react to

What good PMs and Designers think about


UI questions are never yes/no! (not binary)UI questions are never yes/no! (not binary) Should ask: “What percentage of users …?”Should ask: “What percentage of users …?” UI changes impact your metricsUI changes impact your metrics

Impact can be positive, negative, small, largeImpact can be positive, negative, small, large Seek high-ROI UI changesSeek high-ROI UI changes

Approaching UI Design AnalyticallyApproaching UI Design Analytically Typical UI design question:Typical UI design question:

““When using web pages, do users scroll down?”When using web pages, do users scroll down?”- Yes- Yes- No- No


Put Key Conversion Actions Above The FoldPut Key Conversion Actions Above The Fold

Landing Page ALanding Page A Landing Page BLanding Page B

Key conversion action is above the fold

Key conversion action is below the fold

The Fold


The Fold Isn’t Binary EitherThe Fold Isn’t Binary Either

600 px

768 px

1024 px

Data courtesy of ClickTaleData courtesy of ClickTaleFree trial at www.clicktale.comFree trial at www.clicktale.com

% o

f Use

rs The chrome steals about 170 pixels


Case Study: Account Signup Process Case Study: Account Signup Process RedesignRedesign

100%

62.3%58.8%

50.9%

34.4% 32.7%

0%

20%

40%

60%

80%

100%

% o

f Use

rs%

of U

sers

Sign in /Sign in /RegistrationRegistration

Account TypeAccount Type Cash vs.Cash vs.MarginMargin

5 Partner5 PartnerPagesPages

3 Partner3 PartnerPagesPages

Biggest drop


OpenAccount

Sign in

Account Selection

Register

56%

44%

Forget Password

Registration Process

45% drop off(20% of total)

36% overall drop off for

this step

70%(32% of Total)

17% drop off (10% of total)

20% drop off(6% of total)

30%(14% of Total)

80%(26% of Total)

55%(24% of Total)

64%of Total

Analysis of Sign In/Registration FlowAnalysis of Sign In/Registration Flow

Change Password

83%(46% of Total)

Abandonment Rate (7 Day Moving Average)

0%

10%

20%

30%

40%

50%

60%

70%

80%

10/7

/02

10/1

4/02

10/2

1/02

10/2

8/02

11/4

/02

11/1

1/02

11/1

8/02

11/2

5/02

12/2

/02

12/9

/02

12/1

6/02

12/2

3/02

12/3

0/02

1/6/

03

1/13

/03

1/20

/03

Ab

and

on

men

t R

ate

(7 D

ay M

ovi

ng

Ave

rag

e)

Steps 1-2


Redesigned User Flow Improved Redesigned User Flow Improved Registration Conversion RateRegistration Conversion Rate

37% improvement in conversion rate

ReleasedNew Design


Summary: Building the DNA of a Killer Summary: Building the DNA of a Killer AppApp

Define what success meansDefine what success means Customer value propositionCustomer value proposition Equation of your businessEquation of your business

Instrument your site and track key metricsInstrument your site and track key metrics Identify opportunities and prioritize by ROIIdentify opportunities and prioritize by ROI Launch, learn, and iterateLaunch, learn, and iterate

Copyright © 2008 Olsen Solutions LLC

[email protected]@yourversion.com

Questions?Questions?

www.yourversion.comwww.yourversion.com


AppendixAppendix


Taking Optimization to The Next LevelTaking Optimization to The Next Level

Multivariate testing, aka A/B testingMultivariate testing, aka A/B testing New challenges emergeNew challenges emerge

Are we at local vs. global optimum?Are we at local vs. global optimum? How do we identify top new ideas to test? How do we identify top new ideas to test?

(creativity and innovation)(creativity and innovation) Are there assumptions we should Are there assumptions we should

challenge?challenge? Are we missing the forest for the trees?Are we missing the forest for the trees?


Is the site up when I want to use it?

Is the site fast enough?

Does the functionality work?

How compelling and easy to use is the functionality?

Olsen’s Hierarchy of Web User Needs(adapted from Maslow)

Customer’s Perspective What does it mean to us?

Uptime

Page Load Time

Absence of Bugs

Feature Set

Usability & Design

Decreasing

Dissatisfaction

Increasing S

atisfaction


Kano Model: User Needs & SatisfactionUser SatisfactionUser Satisfaction

User DissatisfactionUser Dissatisfaction

Performance Performance (more is better)(more is better)

Delighter (wow)Delighter (wow)

NeedNeednot metnot met

NeedNeedfully metfully met

Must HaveMust Have

Needs & features Needs & features migrate over timemigrate over time


Importance vs. Satisfaction Ask Users to Rate for Each Feature

98

8784

8679 847055 80

7280

75

4150

55

60

65

70

75

80

85

90

95

100

40 50 60 70 80 90 100

Satisfaction

Imp

ort

an

ce

Recommended reading: “What Customers Want” by Anthony Ulwick

BadBad

GreatGreat

Technology

Designing & Optimizing the DNA of a Killer App (Dan Olsen, Startonomics SF 2008)