Web Analytics Congress 2013 a/b-test workshop

Slides: http://ondi.me/wac2013

O N L I N E D I A L O G U E G u i d i n g y o u t o d i g i t a l m a t u r i t y

Wo r k s h o p We b A n a l y t i c s C o n g r e s s 2 0 1 3 / 0 3 / 1 3 , U t r e c h t , t h e N e t h e r l a n d s

A/B-test ing

Slides: http://ondi.me/wac2013

1.  Methodo logy 2.  Too l s

3.   Tes t i ng t i ps

To n W e s s e l i n g C h i e f C o n v e r s i o n O f fi c e r

O n l i n e D i a l o g u e

A/B-test ing

@tonwesseling @onlinedialogue

1995

T h e fi r s t g r a p h i c a l w e b b r o w s e r s


#measure

L o g fi l e a n a l y s e s


Why?

To o p t i m i z e


To improve

To g r o w


We asked

… a n d i t w a s f u n t o d o !

  Where can we improve?   We took small steps   So we could act now!


T h e y d i d s o m e t h i n g e l s e …


A m a z i n g

People

Knowlegde


O u t o f r e a c h


1999

O v e r s t o c k . c o m d e s i g n c h a n g e


2002



2005



2010



2013



What happend to:

… a n d i t w a s f u n t o d o !

  Where can we improve?   We took small steps   So we could act now!


What ’s the d i f fe rence?

W i t h t h e o w n e r o f t h e l o c a l b a k e r y s t o r e ?

@tonwesseling @onlinedialogue D i a l o g u e m o d e l – i m p r o v e d i a l o g u e i n s i g h t s

Company goals Customer goals €€€

Company

Behavior Behavior

Prospect / client

Mul< purpose

Mul<-‐ channel

Mul< phase

External influencers

Social environment Compe<tors Media

Knowledge

ACtude

Internal influencers

Time & Money

Personality

Internal influencers

Internal influencers Profile

Context

Knowledge

Tools

PlaHorm

Mo<va<on


How to communicate

W i t h t h e b r a i n s o f y o u r v i s i t o r s


Be l i ke B i l l

I n G r o u n d h o g d a y


Be l i ke B i l l

A n d c o n v i n c e t h e b r a i n


Do you know th is man?

W. E d w a r d s D e m i n g


Deming Cyc le

Q u a l i t y m a n a g e m e n t


Focus on qua l i t y

W h e n i t i s d e fi n e d b y t h i s r a t i o i t l e a d s t o r e s u l t s

results of work efforts quality = ___________________ total cost


Focus on qua l i t y

W. E d w a r d s D e m i n g

“When you focus just on costs they will rise”


Observat ion

J a p a n s t a r t e d t o l e a r n

  +O PDCA   Like the Toyota production system


The Toyota way

C o n t i n u o u s i m p r o v e m e n t

  Challenge  (We form a long-term vision, meeting challenges with

courage and creativity to realize our dreams.)

  Kaizen  (We improve our business operations continuously,

always driving for innovation and evolution.)

  Genchi Genbutsu  (Go to the source to find the facts to make correct

decisions.)


Kaizen

c o n t i n u o u s i m p r o v e m e n t p r o c e s s

改善


K a i z e n – G o o d c h a n g e

“ Change for the better ”


K a i z e n – G o o d c h a n g e

How to perform experiments using the scientific method


Hypotheses

S t a t i s t i c a l h y p o t h e s i s t e s t i n g

Hypothesis"

Alternative"


A/B-test ing

S m a l l s t e p s t o l e a r n h o w t o i m p r o v e t h e d i a l o g u e

Winner!"

Variation B"

Variation A"

Default"


A/B-test ing matur i t y

Yo u s h o u l d s t a r t a t l e v e l 1 b u t m o v e f a s t t o l e v e l 3

Yeah, we kunnen

testen!"

Fase 1" Hoe halen we drempels weg?"

Fase 2" Welke informatie zoekt de gebruiker?"

Fase 3" Gespreksontwerp toepassen"

Fase 4" Gesegmenteerde dialogen"

Fase 5"

Laaghangend fruit

Direct resultaat

Leren

Personaliseren

Mo<veren

Maak niet uit wat we testen als we maar

testen!"

Waar kunnen we nu direct het verschil maken?"

Willen begrijpen hoe je de

bezoeker beter kunt servicen"

De stap van pagina

optimalisatie naar user flow optimalisatie"

Niet dezelfde dialoog voor

elke bezoeker"


SAMEN mode l

L e t ’s d o i t t o g e t h e r


SAMEN mode l

Te l l t h e s t o r y “ n a v e r t e l l e n ”

S" Start reason"

A"Analyze"

M"Make"

E"Evaluate"

N"Notify"


S t a r t r e a s o n

“Without an analyzed reason to start, it does not make any sense to start at all.”!


A n a l y z e

“Do you understand what is happening?”!


M a k e

“You don’t make perfect designs in 1 time.!!Step by step we’ll grow to the final result.”!!Hypotheses based design!!


E v a l u a t e

“Not only the conversion, look for insights!”!


N o t i f y ( N a v e r t e l l e n )

“If someone can not reproduce your learnings… !Then, there were no learnings...”!


Too ls !

E v e r y t h i n g i s a l r e a d y h e r e a s S a a S


Too ls !

T h e t o o l s w e u s e t h e s e d a y s


Star t reason

To o l s

Gebruikers Onderzoek (SurveyMonkey), Gebruikers wensen (Uservoice), Continue gebruikers feedback (GetSatisfaction, Kampyle), Analytics

trendbreuk notificaties (analytics pakket), Benchmark vergelijken (Trendspot), Markpositie veranderd (Mozenda / Yahoo Pipes), Social

(Finchline, Momentum), Intern (Basecamp / Yammer). "


Star t reason

S u r v e y m o n k e y


Star t reason

G e t s a t i s f a c t i o n . c o m


Star t reason

K a m p y l e . c o m


Star t reason

Tr e s h o l d s ! ( G o o g l e A n a l y t i c s )


Star t reason

Tr e n d s p o t ( B e n c h m a r k i n g )


Star t reason

M o z e n d a . c o m : c o m p e t i t v e i n f o r m a t i o n


Star t reason

M o m e n t u m ( s o c i a l i n s i g h t s )


Star t reason

I n t e r n a l i n s i g h t s ( Ya m m e r. c o m )


Ana lyze

To o l s

Analytics pakket, Mousemaps en Screen recordings (Clicktale), Clickmaps (Crazyegg, Clicktale), Survey (SurveyMonkey), tijdelijke gebruikersfeedback (Qualaroo, 4Q en eigen tool (met Google Analytics)), Usability feedback (Feedbackarmy.com, Usertesting.com, Conceptfeedback.com), Interne feedback (notableapp.com)."


Ana lyze

G e t y o u r t a g m a n a g e m e n t r i g h t ! ( Q u b i t o p e n t a g )


Ana lyze

C l i c k t a l e . c o m


Ana lyze

Q u a l a r o o . c o m


Ana lyze

U s e r t e s t i n g . c o m


Ana lyze

N o t a b l e a p p . c o m


Make

To o l s

Wire-Frames (Mockflow.com), Concept testing (Verifyapp.com), Clickable demo’s / usability (Soldifyapp.com), Usability feedback (Feedbackarmy Usertesting, Conceptfeedback, usability test), Interne feedback (Notableapp.com), Test bouwen / uitvoeren (via CMS & script of VWO of Optimizely), Targeting (Personyze, Conductrics)."


Make

M o c k fl o w. c o m


Make

Ve r i f y a p p . c o m


Make

S o l d i f y a p p . c o m


Make

V i s u a l w e b s i t e o p t i m i z e r . c o m ( t e s t i n g )


Make

P e r s o n y z e . c o m ( t a r g e t i n g )


Make

C o n d u c t r i c s . c o m ( R T D )


Eva luate

To o l s

Analytics pakket (segmenten), Mousemaps en Screen recordings (Clicktale), Clickmaps (Crazyegg, Clicktale), Survey voor NPS / CES (SurveyMonkey)."


Eva luate

D i v e i n t h a t d a t a !


Eva luate

D o n ’ t f o r g e t t h e w h y !


Eva luate

M e a s u r e l i f e t i m e v a l u e i n d i c a t o r s


Not i f y

To o l s

Learnings bord, Learnings boekje, Powerpoint single slide learning (voor mail, maar ook voor op TV’s), Gedrags persona’s / Huisstijl update, Test / learnings archief (spreadsheet), intern (intranet, yammer, basecamp) en externe publicaties."


Not i f y

S p r e a d k n o w l e d g e

Publish & Promote

your insights!


Test t ips

B u t y o u w i l l l e a r n f r o m e v e r y t e s t !

“ Not every test

will improve conversion ”


Test t ips

M o v e o n a n d y o u ’ l l s e e

“ You will learn

you should target ”


Psycho logy

W h e e l o f p e r s u a s i o n


Test t ips

A b n a m r o . n l : L a n d i n g p a g e s


Test t ips

C a l i f o r n i a c l o s e t s . c o m : L a n d i n g p a g e s


Test t ips

P e r r y s p o r t . n l – l a n d i n g p a g e m e s s a g i n g


Test t ips

A l p h a e c o l o g i c a l . c o m : F o r m l o c a t i o n


Test t ips

J u l e s . f r : S h o p p h o t o s


Test t ips

K r a s . n l : F o o t e r


Test t ips

We h k a m p . n l : P r i n c i n g


Test t ips

K i c k z . c o m : p r i c e f o c u s ?


Test t ips

D e s k a w a y. c o m – “ u g l y b r o t h e r ” p r o d u c t


Test t ips

C D F n e t w o r k s . c o m – F o r m l a y - o u t


Test t ips

G s m s t u n t e r . n l – i n t e r n a l s e a r c h


Test t ips

B i j e n k o r f . n l : p o i n t - o f - a c t i o n s


Test t ips

A c t i e v a n d e d a g . n l : C TA m o b i l e p r o o f


Test t ips

B a s e c a m p h q . c o m – n e x t a c t i o n


Test t ips

K r a s . n l - c h e c k m a r k s


Test t ips

V G Z . n l - c h e c k m a r k s


Test t ips

H o t e l s c h i p h o l . n l - c h e c k m a r k s


Test t ips

T h e c o r k s c r e w. i e - d i s c o u n t


Test t ips

B o l . c o m – l i k e p o s i t i o n i n g


Test t ips

I n s h a r e d . n l – S t i c k y c o n t e n t


To do:

T h a n k y o u ! S l i d e s : h t t p : / / o n d i . m e / w a c 2 0 1 3

  Optimize your dialogue   Focus on quality

  Spread your knowledge

Business

Web Analytics Congress 2013 a/b-test workshop