K E E P I N G Y O U W AT C H I N G H O U S E O F C A R D S

B R I A N H O LT @ H O LT B T

bit.ly/techsummitpr16

W H AT D O Y O U R U S E R S W A N T ?

Y O U H AV E N O * * * * I N G C L U E .

N E I T H E R D O T H E Y.

S O W H Y C A N ’ T I T R A C K M Y D ATA B E F O R E A N D A F T E R I A D D F E AT U R E S ?

B O O K S A L E S O N M Y W E B S I T E

0

17.5

35

52.5

70

1 2 3 4 5 6 7 8

launched new feature

W H AT A C T U A L LY H A P P E N E D

launched new feature

Oprah put my book in her book club

0

15

30

45

60

1 2 3 4 5 6 7 8

A B A B A B A B A B A B A B A B

W H AT A C T U A L LY H A P P E N E D

0

17.5

35

52.5

70

1 2 3 4 5 6 7 8

A B A B A B A B A B A B A B A B

started my a/b test

Oprah put my book in her book club

vs

2 0 % L I F T I N V I E W I N G

W H Y ?

W E H AV E O U R T H E O R I E S B U T O F T E N I T ’ S T O U G H T O E S TA B L I S H C A U S A L I T Y.

A / B T E S T I N G I S N O T A S O L U T I O N .

Y O U R N E W A / B T E S T

R E D C E L L

💩

💩B L U E C E L L

💩💩💩💩💩💩💩💩

G R E E N C E L L

💰

I F Y O U R E F I N E A 💩 E X P E R I E N C E V I A A / B T E S T I N G , Y O U E N D U P W I T H A R E F I N E D

💩 E X P E R I E N C E

T R Y E N T I R E LY N E W E X P E R I E N C E S T O O 💰

A / B T E S T S• DO

• Verify hypothesis

• Indicate positive directions to go in

• Identify areas of consumer sensitivity

• Identify the key metrics you’re tracking (revenue versus sign-ups)

• DON’T

• Create your hypothesis after you have run the test

• Introduce your bias

• Assume that A/B testing will give you answers

D O N ’ T B E S H O R T- S I G H T E D

• Some tests may indicate positive short-term results but hide long-term wins.

• e.g. SEO win versus monetization win

A D V I C E ?

• Simpler is almost always better

• Faster

• Less options and less choices

• Less friction

• Make choices you know most users will make; allow other users the escape hatch*

A / B T E S T I N G H A R D PA R T S

• Be willing to throw work out

• Be willing to do ideas and features you don’t like

• Realize your consumers will tell you one thing but act another way

• Be willing to ignore the indication of the results

K E E P I N G Y O U W AT C H I N G H O U S E O F C A R D S ?

• We A/B test everything at Netflix. Every button, every layout, every algorithm, every color. It’s all A/B tested to be the least friction possible.

• That keeps the interface intuitive and out of the way between you Frank Underwood’s hostile Washington DC takeover.

H O W D O I A / B T E S T ?

• Open Source

• PlanOut (Facebook)

• SixPack (SeatGeek)

• Abba (Stripe)

• Proctor (Indeed)

H O W D O I A / B T E S T ?

• Commercial

• Optimizely

• Google Analytics Experiments

• KISSmetrics

B R I A N H O LT @ H O LT B T