Funnel Analysis in Hadoop at Etsy

Steve Mardenfeld, Wil Stuckey, Matt Walker

Tuesday, March 12, 13

Handmade MarketPlace

Data Driven Development

• Use data to make informed decisions

• Use data to evaluate the efficacy of our products

Funnel AnalysisFunnel Analysis

Registration Funnel

5 0 , 0 0 0 1 0 , 0 0 0

1 0 0 % 2 0 %

Registration Funnel

5 0 , 0 0 0 1 0 , 0 0 0

1 0 0 % 6 0 %4 5 , 0 0 0

1 5 , 0 0 0

3 3 %1 0 0 % 9 0 % 2 9 % 2 0 %

Funnels ++

• Funnels are more than just an optimization tool

• Use them to understand different pathways throughout our site

• Partition and compare these pathways by attributes

• A/B tests, categories, queries, cohorts

• Attribution model

Attribution Models

• Tie conversions and successes to specific products and actions

• Use to gain understanding of our users’ interaction with Etsy

• Help us to measure gains in A/B testing

• Easily compare different varieties of the same product

• Attribution techniques for internal and external attribution

Attributions Illustrated

• last product interaction - browse

• co-occurrence - search and browse

• direct funnel attribution - search

Indirect Attribution Illustrated

• direct funnel attribution - shop home

• indirect funnel attribution - search

Aggregate Results

Search Clicked Listing Purchased5 0 , 0 0 0 2 5 , 0 0 0 5 , 0 0 0E v e n t s

Direct

Aggregate Results

Search Clicked Listing Purchased

Search Clicked Listing Shop Clicked Listing

5 0 , 0 0 0

2 5 , 0 0 0

5 , 0 0 0

1 0 , 0 0 0 5 , 0 0 0

E v e n t s

Direct

IndirectPurchased

3 , 0 0 0

Aggregate Results

Search Clicked Listing Purchased

Search Clicked Listing Shop Clicked Listing

5 0 , 0 0 0

2 5 , 0 0 0

5 , 0 0 0

1 0 , 0 0 0 5 , 0 0 0

E v e n t s

Search Clicked Listing5 0 , 0 0 0 2 5 , 0 0 0E v e n t s

Direct

Indirect

Combined

Purchased3 , 0 0 0

Purchased8 , 0 0 0

Segmenting Within Funnels

Search Clicked Listing5 0 , 0 0 0 2 0 , 0 0 0C o u n t s

Old AlgorithmPurchased

1 5 , 0 0 0

Search Clicked Listing1 0 0 , 0 0 0 6 0 , 0 0 0C o u n t s

New HotnessPurchased

1 5 , 0 0 0

Segmenting Within Funnels

Search Clicked Listing1 0 0 % 5 0 %S t e p

1 0 0 % 5 0 %To t a l

Old AlgorithmPurchased

4 0 %2 0 %

Search Clicked Listing1 0 0 % 6 0 %S t e p

1 0 0 % 6 0 %To t a l

New HotnessPurchased

2 5 %1 5 %

Segmenting Across Funnels

* Clicked Listing1 0 0 % 5 0 %S e a r c h

1 0 0 % 4 0 %B r o w s e

Purchased4 0 %3 0 %

1 0 0 % 6 0 %H o m e 3 6 %

1 0 0 % 6 2 %A c t i v i t y F e e d

1 0 0 % 4 7 %Ta s t e Te s t2 8 %3 1 %

1 0 0 % 4 5 %S e a r c h A d s 3 8 %

Democratized FunnelsHow do we make this awesomeness available for everyone?

Awesome infrastructure but...

• Must be an engineer to write your own queries

• Engineering resources become the bottleneck

• Hard to scale as the company grows

So we want to:

• Allow the data engineers to focus on higher priority things.

• Allow people to answer their own questions.

How do we do this?

The Path of Funnel Tools

google spreadsheets internal tools

• Funnel Cake

• Feature Funnel

Funnel Cake

Version 1

Real Time

Lets do it in PHP!

It’s the “Etsy Way”

• Already have existing infrastructure

• Operationally stable

• The same tools everyone else is using means a better adoption rate across Engineering

• Event Stream

• Code runs on every page view

• Simple matching system

• (ab)Used memcached as a temporary storage

• Rolled up to DB every min (near real time)

What happened?

Broader adoption!Over 75 Funnels were created

Shiny Visualized Results

• Funnels had to be setup ahead of time (no backfills)

• Reconciliation is hard

• Limited to events in our web clickstream (ios/android would be excluded)

• Scaling and Operational issues

• Difficult to maintain multiple stacks

Turns out that we don’t make Product decisions in real time

http://mcfunley.com/whom-the-gods-would-destroy-they-first-give-real-time-analytics

Version 2

Getting back on the elephant

• Able to carry over the User Interface from v1

• Standardized event sessionization

• Operationally supported infrastructure

• Nightly batch process

Feature Funnel

Attribution for ALL

Built in metrics

• Click through rate

• Listing impressions

• Purchase rate

• Purchase from shop rate

• Favorite from shop rate

• More...

Built in metrics

• Click through rate

• Listing impressions

• Purchase rate

• Purchase from shop rate

• Favorite from shop rate

• More...

- Direct Attribution

- Indirect Attribution

Segmentation

• Primarily used for A/B test analysis

• Arbitrary segmentation

How do we get it?

select (clicks/visits * 100.0) as "CTR" from feature_funnel where event_type = 'search' and ab_test = 'sitewide' and epoch_s = 1361318400 and group_name = 'ALL_GROUPINGS';

The Future?

Features

• Eliminate the need for an engineer to write the queries

• Robust segmentation

• Not be limited to visit sessions

• Run Ad Hoc queries

Build your own Funnels

show the builder ui here

Mechanics of Funnel Analysis

• Event sequences

• Typically visits to website

• Sessionization scheme left to developer

Funnel

• Search

• Listing

• Cart

• Only discussed event types

• Funnel steps require additional constraints:

• Listing referred by search

• Added that listing to cart

Searchquery “dinosaur”

listing_ids 119469855, 90583707, ...

loc “http://www.etsy.com/search/handmade/patterns?q=dinosaur&order=most_relevant&view_type=gallery&ship_to=ZZ”

... ...

Listinglisting_id 119469855

ref “http://www.etsy.com/search/handmade/patterns?q=dinosaur&order=most_relevant&view_type=gallery&ship_to=ZZ”

shop_id 7415158

... ...

Cartadded_listing_id 119469855

cart_listing_ids 119469855, ...

cart_type guest

... ...

• Listing referred by search

• Added that listing to cart

Search

query “dinosaur”

listing_ids 119469855, 90583707, ...

loc “http://www.etsy.com/search/handmade/patterns?q=dinosaur&order=most_relevant&view_type=gallery&ship_to=ZZ”

... ...

shop_id 7415158

... ...

shop_id 7415158

... ...

listing.ref == search.loc

cart_type guest

... ...

cart_type guest

... ...

cart.added_listing_id == listing.listing_id

• Search

• Listing & Referred

• Cart & AddedListing

Pattern Matching

• Apply query to event sequence

• Select out tuples of matching events

What About Incomplete Funnels?

nullnull

Funnel Analysis

• Replace events with 1

• Replace nulls with 0

• Sum

nullnull

min(1, 1)min(2, 1)min(3, 1)

What Do We Keep?

Search

listing_ids 119469855, 90583707, ...

loc“http://www.etsy.com/search/handmade/patterns?q=dinosaur&order=most_relevant&view_type=gal

lery&ship_to=ZZ”

... ...

Listing

listing_id 119469855

ref“http://www.etsy.com/search/handmade/patterns?q=dinosaur&order=most_relevant&view_type=gal

lery&ship_to=ZZ”

shop_id 7415158

... ...

added_listing_id 119469855

cart_type guest

... ...

Segmenting Funnels

Search

listing_ids 119469855, 90583707, ...

loc“http://www.etsy.com/search/handmade/patterns?q=dinosaur&order=most_relevant&view_type=gal

lery&ship_to=ZZ”

... ...

Listing

lery&ship_to=ZZ”

shop_id 7415158

... ...

cart_type guest

... ...

Listing

lery&ship_to=ZZ”

shop_id 7415158

... ...

cart_type guest

... ...

cart_type guest

... ...

listing_idquery

query listing_id cart_type

Segmented Funnel Analysis

• Extract segmenting properties

• Compute indicators as before

• Group on segmenting properties and sum

MapReduce

• Work is done map-side

• Common first step in our jobs

• Expensive computation limited to first round mappers

Event Sequence Pattern Matching

Components

• Predicate: matches/rejects events

• Query: tuple of predicates

• Match: tuple of events

Match Predicates

• Select an event based on:

• Full event sequence

• Prior matched events

• Current candidate

Match Predicate DSL

• Combine predicates with logical operators

• val Query = Seq(Search, Listing & Referred, Cart & AddedListing)

Semantics

• Fixed number of events in match

• Arbitrary number of matches per sequence

• Collect and extend all partial matches

Search Listing Shop

Listing Cart Home

Search Listing Search

Search, Listing, Shop, Listing, Cart, Home, Search, Listing, Search

Search

Cart & AddedListing

Listing & Referred

Search

Cart & AddedListing

Listing & Referred

SearchSearch

Listing, Shop, Listing, Cart, Home, Search, Listing, Search

Cart & AddedListing

Listing & Referred

SearchSearch

Listing, Shop, Listing, Cart, Home, Search, Listing, Search

Cart & AddedListing

Listing & Referred

Search

Listing

Search

Shop, Listing, Cart, Home, Search, Listing, Search

Cart & AddedListing

Listing & Referred

Search

Listing

Search

Shop, Listing, Cart, Home, Search, Listing, Search

Cart & AddedListing

Listing & Referred

Search

Listing

Search

Listing, Cart, Home, Search, Listing, Search

Cart & AddedListing

Listing & Referred

Search

Listing

Search

Listing, Cart, Home, Search, Listing, Search

Cart & AddedListing

Listing & Referred

Search

Listing

Search

Cart, Home, Search, Listing, Search

Listing & Referred

Cart & AddedListing

Search

Listing

Search

Cart, Home, Search, Listing, Search

Listing & Referred

Cart & AddedListing

Search

Listing

Search

Home, Search, Listing, Search

Listing & Referred

Cart & AddedListing

Search

Listing

Search

Home, Search, Listing, Search

Listing & Referred

Cart & AddedListing

Search

Listing

Search

Search, Listing, Search

Listing & Referred

Cart & AddedListing

Search

Listing

Search

Search, Listing, Search

Listing & Referred

Cart & AddedListing

Search Search

Listing

Search

Listing, Search

Listing & Referred

Cart & AddedListing

Search Search

Listing

Search

Listing, Search

Listing & Referred

Cart & AddedListing

Search Search

Listing Listing

Search

Listing & Referred

Cart & AddedListing

Search Search

Listing Listing

Search

Listing & Referred

Cart & AddedListing

Search Search Search

Listing Listing

Search

Listing & Referred

Cart & AddedListing

nullnull

Match Tree

• Purely functional data structure

• Holds matched events and indices

• Match prefixes shared

Match Tree Algorithm

• Fold over sequence accumulating tree

• May extend any non-terminal node

• Each level in tree corresponds to predicate

Practicality

• Explodes, but

• Queries are constant length

• Sequences are bounded (visits)

• Predicates constrain growth

Summary

• Why funnels are interesting

• What we’ve built with them at Etsy

• Our approach to funnel analysis

Questions?

• Steve Mardenfeld

• Wil Stuckey @quiiver

• Matt Walker @data_daddy

Funnel Analysis in Hadoop at Etsy

Technology

Etsy How-To

Etsy craft party oc

Getting Started with Etsy How to Sell Your Products on Etsy · Getting Started with Etsy Part 1: How to Sell Your Products on Etsy Part 2: Etsy Selling Checklist Part 3: 50 Handmade

Etsy Shop Graphics Tutorial

Etsy + Kiva Zip Workshop: How to Crowdfund your 1st Etsy Startup

Continuous Deployment at Etsy

Etsy Seller Workshop

Etsy desconstruction

Cascading talk in Etsy (

Etsy On-Side Trading

Marketing Project Showcasing Etsy

Etsy Activity Feeds Architecture

Solr & Lucene at Etsy

Etsy Business Template

Doodle Button Etsy Wholesale

Etsy Business case presentation

Business case etsy; bell

Trend Book For Etsy

Etsy Vintage Kimono Lookbook

ETSY 1Q2018 Presentation final version · ETSY 1Q2018 Presentation final version Created Date: 20180508203529Z