85
Site Search Analytics in a Nutshell Louis Rosenfeld [email protected] @louisrosenfeld Webdagane 10 September 2013

Site Search Analytics in a Nutshell

Embed Size (px)

Citation preview

Page 1: Site Search Analytics in a Nutshell

Site Search Analytics in a Nutshell

Louis Rosenfeld

[email protected] • @louisrosenfeld

Webdagane • 10 September 2013

Page 2: Site Search Analytics in a Nutshell

Hello, my name is Lou

www.louisrosenfeld.com | www.rosenfeldmedia.com

Page 3: Site Search Analytics in a Nutshell

Let’s look at the data

Page 4: Site Search Analytics in a Nutshell

No, let’s look at the real dataCritical elements in bold: IP address, time/date stamp, query, and # of

results:

XXX.XXX.X.104 - - [10/Jul/2006:10:25:46 -0800] "GET /search?access=p&entqr=0&output=xml_no_dtd&sort=date%3AD%3AL%3Ad1&ud=1&site=AllSites&ie=UTF-8&client=www&oe=UTF-8&proxystylesheet=www&q=lincense+plate&ip=XXX.XXX.X.104 HTTP/1.1" 200 971 0 0.02

XXX.XXX.X.104 - - [10/Jul/2006:10:25:48 -0800] "GET /searchaccess=p&entqr=0&output=xml_no_dtd&sort=date%3AD%3AL%3Ad1&ie=UTF-8&client=www&q=license+plate&ud=1&site=AllSites&spell=1&oe=UTF-8&proxystylesheet=www&ip=XXX.XXX.X.104 HTTP/1.1" 200 8283 146 0.16

Page 5: Site Search Analytics in a Nutshell

No, let’s look at the real dataCritical elements in bold: IP address, time/date stamp, query, and # of

results:

XXX.XXX.X.104 - - [10/Jul/2006:10:25:46 -0800] "GET /search?access=p&entqr=0&output=xml_no_dtd&sort=date%3AD%3AL%3Ad1&ud=1&site=AllSites&ie=UTF-8&client=www&oe=UTF-8&proxystylesheet=www&q=lincense+plate&ip=XXX.XXX.X.104 HTTP/1.1" 200 971 0 0.02

XXX.XXX.X.104 - - [10/Jul/2006:10:25:48 -0800] "GET /searchaccess=p&entqr=0&output=xml_no_dtd&sort=date%3AD%3AL%3Ad1&ie=UTF-8&client=www&q=license+plate&ud=1&site=AllSites&spell=1&oe=UTF-8&proxystylesheet=www&ip=XXX.XXX.X.104 HTTP/1.1" 200 8283 146 0.16

What are users searching?

Page 6: Site Search Analytics in a Nutshell

No, let’s look at the real dataCritical elements in bold: IP address, time/date stamp, query, and # of

results:

XXX.XXX.X.104 - - [10/Jul/2006:10:25:46 -0800] "GET /search?access=p&entqr=0&output=xml_no_dtd&sort=date%3AD%3AL%3Ad1&ud=1&site=AllSites&ie=UTF-8&client=www&oe=UTF-8&proxystylesheet=www&q=lincense+plate&ip=XXX.XXX.X.104 HTTP/1.1" 200 971 0 0.02

XXX.XXX.X.104 - - [10/Jul/2006:10:25:48 -0800] "GET /searchaccess=p&entqr=0&output=xml_no_dtd&sort=date%3AD%3AL%3Ad1&ie=UTF-8&client=www&q=license+plate&ud=1&site=AllSites&spell=1&oe=UTF-8&proxystylesheet=www&ip=XXX.XXX.X.104 HTTP/1.1" 200 8283 146 0.16

What are users searching?

How often are users failing?

Page 7: Site Search Analytics in a Nutshell

SSA is semantically rich data, and...

Page 8: Site Search Analytics in a Nutshell

SSA is semantically rich data, and...

Queries sorted by frequency

Page 9: Site Search Analytics in a Nutshell

...what users want--in their own words

Page 10: Site Search Analytics in a Nutshell

A little goes a long wayA handful of queries/tasks/ways to navigate/features/ documents meet the needs of your most important audiences

Page 11: Site Search Analytics in a Nutshell

A little goes a long wayA handful of queries/tasks/ways to navigate/features/ documents meet the needs of your most important audiences

Not all queries are distributed equally

Page 12: Site Search Analytics in a Nutshell

A little goes a long wayA handful of queries/tasks/ways to navigate/features/ documents meet the needs of your most important audiences

Page 13: Site Search Analytics in a Nutshell

A little goes a long wayA handful of queries/tasks/ways to navigate/features/ documents meet the needs of your most important audiences

Nor do they diminish gradually

Page 14: Site Search Analytics in a Nutshell

A little goes a long wayA handful of queries/tasks/ways to navigate/features/ documents meet the needs of your most important audiences

Page 15: Site Search Analytics in a Nutshell

A little goes a long wayA handful of queries/tasks/ways to navigate/features/ documents meet the needs of your most important audiences

80/20 rule isn’t quite accurate

Page 16: Site Search Analytics in a Nutshell

(and the tail is quite long)

Page 17: Site Search Analytics in a Nutshell

(and the tail is quite long)

Page 18: Site Search Analytics in a Nutshell

(and the tail is quite long)

Page 19: Site Search Analytics in a Nutshell

(and the tail is quite long)

Page 20: Site Search Analytics in a Nutshell

(and the tail is quite long)The Long Tail is

much longer than you’d suspect

Page 21: Site Search Analytics in a Nutshell

The Zipf Distribution, textually

Page 22: Site Search Analytics in a Nutshell

Some things you can do with SSA

1.Make it harder to get lost in deep content2.Make search smarter3.Reduce jargon4.Learn how your audiences differ5.Know when to publish what6.Own and enjoy your failures7.Avoid disaster8.Predict the future

Page 23: Site Search Analytics in a Nutshell

#1Make it harder to get lost

Page 24: Site Search Analytics in a Nutshell

Start with basic SSA data: queries and query frequency

Percent: volume of search activity for a unique query during a particular time period

Cumulative Percent: running sum of percentages

Page 25: Site Search Analytics in a Nutshell

Tease out common content types

Page 26: Site Search Analytics in a Nutshell

Tease out common content types

Page 27: Site Search Analytics in a Nutshell

Tease out common content types

Took an hour to...• Analyze top 50 queries (20% of all search activity)

• Ask and iterate: “what kind of content would users be looking for when they searched these terms?”

• Add cumulative percentages

Result: prioritized list of potential content types#1) application: 11.77%

#2) reference: 10.5% #3) instructions: 8.6%

#4) main/navigation pages: 5.91%

#5) contact info: 5.79%

#6) news/announcements: 4.27%

Page 28: Site Search Analytics in a Nutshell

Clear content types lead to better contextual navigation

artist descriptions

album reviews

album pages

artist biosdiscography

TV listings

Page 29: Site Search Analytics in a Nutshell

#2Make search smarter

Page 30: Site Search Analytics in a Nutshell

Clear content types improve search performance

Page 31: Site Search Analytics in a Nutshell

Clear content types improve search performance

Page 32: Site Search Analytics in a Nutshell

Clear content types improve search performance

Content objects related to products

Page 33: Site Search Analytics in a Nutshell

Clear content types improve search performance

Content objects related to products

Raw search results

Page 34: Site Search Analytics in a Nutshell

Contextualizing “advanced” features

Page 35: Site Search Analytics in a Nutshell

Session data suggest progression and context

Page 36: Site Search Analytics in a Nutshell

Session data suggest progression and context

search session patterns1. solar energy2. how solar energy works

Page 37: Site Search Analytics in a Nutshell

Session data suggest progression and context

search session patterns1. solar energy2. how solar energy works

search session patterns1. solar energy2. energy

Page 38: Site Search Analytics in a Nutshell

Session data suggest progression and context

search session patterns1. solar energy2. how solar energy works

search session patterns1. solar energy2. energy

search session patterns1. solar energy2. solar energy charts

Page 39: Site Search Analytics in a Nutshell

Session data suggest progression and context

search session patterns1. solar energy2. how solar energy works

search session patterns1. solar energy2. energy

search session patterns1. solar energy2. solar energy charts

search session patterns1. solar energy2. explain solar energy

Page 40: Site Search Analytics in a Nutshell

Session data suggest progression and context

search session patterns1. solar energy2. how solar energy works

search session patterns1. solar energy2. energy

search session patterns1. solar energy2. solar energy charts

search session patterns1. solar energy2. explain solar energy

search session patterns1. solar energy2. solar energy news

Page 41: Site Search Analytics in a Nutshell

Recognizing proper nouns, dates, and unique ID#s

Page 42: Site Search Analytics in a Nutshell

#3Reduce jargon

Page 43: Site Search Analytics in a Nutshell

Saving the brand by killing jargon at a community collegeJargon related to online education: FlexEd, COD,

College on Demand

Marketing’s solution: expensive campaign to educate public (via posters, brochures)

The Numbers (from SSA):

Result: content relabeled, money saved

query rank query#22 online*#101 COD#259 College on Demand#389 FlexTrack

* “online” part of 213 queries

Page 44: Site Search Analytics in a Nutshell

#4Learn how your audiences differ

Page 45: Site Search Analytics in a Nutshell

Who cares about what?

Page 46: Site Search Analytics in a Nutshell

Who cares about what?

Page 47: Site Search Analytics in a Nutshell

Who cares about what?

Page 48: Site Search Analytics in a Nutshell

Who cares about what?

Page 49: Site Search Analytics in a Nutshell

Why analyze queries by audience?

Fortify your personas with dataLearn about differences between audiences

• Open University “Enquirers”: 16 of 25 queries are for subjects not taught at OU

• Open University Students: search for course codes, topics dealing with completing program

Determine what’s commonly important to all audiences (these queries better work well)

Page 50: Site Search Analytics in a Nutshell

#5Know when to publish what

Page 51: Site Search Analytics in a Nutshell
Page 52: Site Search Analytics in a Nutshell

Interest in the football team:

going...

Page 53: Site Search Analytics in a Nutshell

Interest in the football team:

going...

...going...

Page 54: Site Search Analytics in a Nutshell

Interest in the football team:

going...

...going...

gone

Page 55: Site Search Analytics in a Nutshell

Interest in the football team:

going...

...going...

gone

Time to study!

Page 56: Site Search Analytics in a Nutshell
Page 57: Site Search Analytics in a Nutshell

Before Tax Day

Page 58: Site Search Analytics in a Nutshell
Page 59: Site Search Analytics in a Nutshell

After Tax Day

Page 60: Site Search Analytics in a Nutshell

#6Own and enjoy your failures

Page 61: Site Search Analytics in a Nutshell

Failed navigation?Examining unexpected searching

Look for places searches happen beyond main page

What’s going on?

• Navigational failure?

• Content failure?

• Something else?

Page 62: Site Search Analytics in a Nutshell

Where navigation is failing (“Professional Resources” page)

Do users and AIGA mean different things by “Professional Resources”?

Page 63: Site Search Analytics in a Nutshell

Comparing what users findand what they want

Page 64: Site Search Analytics in a Nutshell

Comparing what users findand what they want

Page 65: Site Search Analytics in a Nutshell

Failed business goals?Developing custom metrics

Netflix asks

1. Which movies most frequently searched? (query count)

2. Which of them most frequently clicked through? (MDP views)

3. Which of them least frequently added to queue? (queue adds)

Page 66: Site Search Analytics in a Nutshell

Failed business goals?Developing custom metrics

Netflix asks

1. Which movies most frequently searched? (query count)

2. Which of them most frequently clicked through? (MDP views)

3. Which of them least frequently added to queue? (queue adds)

Page 67: Site Search Analytics in a Nutshell

Failed business goals?Developing custom metrics

Netflix asks

1. Which movies most frequently searched? (query count)

2. Which of them most frequently clicked through? (MDP views)

3. Which of them least frequently added to queue? (queue adds)

Page 68: Site Search Analytics in a Nutshell

#7Avoid disasters

Page 69: Site Search Analytics in a Nutshell

The new and improved search engine that wasn’t

Vanguard used SSA to help benchmark existing search engine’s performance and help select new engine

New search engine “performed” poorlyBut IT needed

convincing to delay launch

Information Architect &

Dev Team Meeting

Search seems to have a few

problems… Nah

.

Where’s the

proof?

You can’t tell

for sure.

Page 70: Site Search Analytics in a Nutshell

What to do? Test performance of common queries

“Before and after” testing using two sets of metrics1.Relevance: how reliably the search engine

returns the best matches first2.Precision: proportion of relevant results

clustered at the top of the list

Page 71: Site Search Analytics in a Nutshell

Old engine (target) and new compared

Note: low relevance and high precision scores are optimal

More on Vanguard case study: http://bit.ly/D3B8c

Page 72: Site Search Analytics in a Nutshell

Old engine (target) and new compared

Note: low relevance and high precision scores are optimal

More on Vanguard case study: http://bit.ly/D3B8c

uh-oh

Page 73: Site Search Analytics in a Nutshell

Old engine (target) and new compared

Note: low relevance and high precision scores are optimal

More on Vanguard case study: http://bit.ly/D3B8c

uh-oh better

Page 74: Site Search Analytics in a Nutshell

#8Predict the future

Page 75: Site Search Analytics in a Nutshell

Shaping the Financial Times’ editorial agendaFT compares these

• Spiking queries for proper nouns (i.e., people and companies)

• Recent editorial coverage of people and companies

Discrepancy? • Breaking story?!

• Let the editors know!Seed your

Page 76: Site Search Analytics in a Nutshell

Can SSA bring us together?

Page 77: Site Search Analytics in a Nutshell

Lou’s TABLE OF OVERGENERALIZED

DICHOTOMIESWeb Analytics User Experience

What they analyze Users' behaviors (what's happening)

Users' intentions and motives (why those things happen)

What methods they employ

Quantitative methods to determine what's happening

Qualitative methods for explaining why things happen

What they're trying to achieve

Helps the organization meet goals (expressed as KPI)

Helps users achieve goals (expressed as tasks or topics of interest)

How they use data Measure performance (goal-driven analysis)

Uncover patterns and surprises (emergent analysis)

What kind of data they use

Statistical data ("real" data in large volumes, full of errors)

Descriptive data (in small volumes, generated in lab environment, full of errors)

Page 78: Site Search Analytics in a Nutshell
Page 79: Site Search Analytics in a Nutshell
Page 80: Site Search Analytics in a Nutshell

Lands End and SKUs

Page 81: Site Search Analytics in a Nutshell

Lands End and SKUs

SKU: # 39072-2AH1

Page 82: Site Search Analytics in a Nutshell

Use SSA to start work on a site report card

Page 83: Site Search Analytics in a Nutshell

Use SSA to start work on a site report card

SSA helps determine common information needs

Page 84: Site Search Analytics in a Nutshell

Read this

Search Analytics for Your Site: Conversations with Your Customers by Louis Rosenfeld (Rosenfeld Media, 2011)

www.rosenfeldmedia.com

Use code WEBDAGENE2013

for 20% off allRosenfeld Media books