KABOOSE.com – Valentine’s Day 2009 AB Testing Campaign - Omniture Cafe April 2009

corporate.kaboose.com

KABOOSE Valentine’s Day 2009 A/B Testing Campaign

Case StudyApril 01, 2009


Introduction:

Jose DavilaWeb Analytics - A/B Testing Strategist

Kaboose Inc.


About Kaboose Inc.


Business GoalsOnline Publisher:

• Increase the reach of our network of web properties

• Engage more our audience: • View more content, • Stay longer • Use our services• Interact.

• Business Model:• Advertisement...


Questions & ChallengesBefore A/B Testing:

• Testing process was lengthy and cumbersome.

• Most of the decisions were made with limited testing.

• Potentially great ideas were not being tested.

• Organization ready for further optimization.


Questions & ChallengesValentine’s Day Traffic:

• Valentine’s Day is an important seasonal source of traffic.

• Previous optimization for traffic and engagement (Marketing, Editorial, SEO, Web Analytics).

• Question: Can we further improve engagement on our Valentine’s page? (PVs/Visit, stickiness)

• List of “improvement” ideas to test: Valentine’s Day landing page.


Valentine’s Day Page• Control Recipe: The original page.


Valentine’s Day Page• Control Recipe: The original page.


Test StrategyValentine’s Day Landing Page:

• Test variations of the center content.

• Incorporate multiple “improvement” ideas.

• Content area broken down in three modules/blocks.

• Variations for each module/block (based on previous tests)

• Success metrics: PVs/Visit, stickiness


Test Strategy• Modules/blocks of content...


Recipes: Module 1 -Top ModuleA: Feature

Rotator• Flash based rotator.

B: Static Image and Links

• Minimalist version.

C: Tab Based Hub

• Flash based. • Static.




Recipes: Module 2 - Middle ModuleA: Small Index• List of linked categories

and short description.

B: Extended Index• List of categories with

additional links.




Recipes: Module 3 - Bottom ModuleA: Empty space B: Bottom content

element• Group of three additional links

with images and a short descriptions


The Results...!Can you guess?

• From the 12 possible combinations (+ control) the winner was….

… nobody on our team was able to guess it …

First some interesting findings….


The Results...!Some interesting findings…• All the new versions performed better.

• One recipe showed the highest performance across the board(+ 54% Lift – PVs/Visit)

• More influential modules/elements identified.

• Behavioral patterns from key visitor segments extracted


The Results...!Which recipe won?

• Module 1: Tabbed image features (C)• Module 2: Small Index (A)• Module 3: Blank (A)

• Recipe CAA had the highest performance: engagement + stickiness

• Recipe CAA pushed live

• Over 45% increase in engagement (PVs/Visit) during the whole campaign (including test).


Long Term Outcomes.From our A/B testing experience.

• Start small + Value + Allocate resources

• Build awareness based on a sequence of success stories

• Learning process (“failures” can be extremely valuable)

• Don’t assume “best practices” are the best solution. Test them.

• Create testing business culture

• A/B Testing is now part of our process.


Questions?


Thank you!

Jose DavilaWeb Analytics - A/B Testing [email protected]

Kaboose Inc.

mailto:[email protected]

Documents

KABOOSE.com – Valentine’s Day 2009 AB Testing Campaign - Omniture Cafe April 2009