Site Search Analytics

©2008 Louis Rosenfeld, LLC (www.louisrosenfeld.com). All rights reserved.

Site Search Analytics: Conversations with your customers

Louis Rosenfeld University of Illinois 9th Annual Webmasters Forum April 30, 2008


About me

www.louisrosenfeld.com www.rosenfeldmedia.com


Anatomy of a search log (from Google Search Appliance) Critical elements in bold: IP address, time/date stamp, query, and # of results:

XXX.XXX.X.104 - - [10/Jul/2006:10:25:46 -0800] "GET /search?access=p&entqr=0&output=xml_no_dtd&sort=date%3AD%3AL%3Ad1&ud=1&site=AllSites&ie=UTF-8&client=www&oe=UTF-8&proxystylesheet=www&q=lincense+plate&ip=XXX.XXX.X.104 HTTP/1.1" 200 971 0 0.02

XXX.XXX.X.104 - - [10/Jul/2006:10:25:48 -0800] "GET /search?access=p&entqr=0&output=xml_no_dtd&sort=date%3AD%3AL%3Ad1&ie=UTF-8&client=www&q=license+plate &ud=1&site=AllSites&spell=1&oe=UTF-8&proxystylesheet=www&ip=XXX.XXX.X.104 HTTP/1.1" 200 8283 146 0.16

XXX.XXX.XX.130 - - [10/Jul/2006:10:24:38 -0800] "GET /search?access=p&entqr=0&output=xml_no_dtd&sort=date%3AD%3AL%3Ad1&ud=1&site=AllSites&ie=UTF-8&client=www&oe=UTF-8&proxystylesheet=www&q=regional+transportation+governance+commission&ip=XXX.XXX.X.130 HTTP/1.1" 200 9718 62 0.17




Querying the queries: Generic questions to help get started

1. What are the most frequent unique queries? 2. Are frequent queries retrieving quality results? 3. Click-through rates per frequent query? 4. Most frequently clicked result per query? 5. Which frequent queries retrieve zero results? 6. What are the referrer pages for frequent queries? 7. Which queries retrieve popular documents? 8. What interesting patterns emerge in general?


Improving metadata and navigation: How SSA can help

1. Analyze for metadata value tone, granularity, and trends

2. Determine (and invest in) important metadata attributes

3. Study queries that follow navigation failures/changes to create better contextual navigation

4. Look to improve “top-down” navigation (e.g., indices, taxonomy, main page)


Determining metadata values through clustering

One week’s top 50 queries, clustered by topic

Represents 20% of all search activity

< 1 hour of work Can help suggest

preferred terms


Determining metadata values through similar queries

BBC explores similar queries (i.e., collaborative filtering)

Helps with misspellings, synonyms, best bets


Determining metadata attributes (AKA types, fields) Top queries

clustered and prioritized by attributes: Place Dept./

Program Service Task

1-2 hours work


Navigation failures/goal changes: (from Google Analytics)


Supporting contextual navigation through embedded queries

SLI Systems can display other queries related to a particular query…

…and queries related to a particular document


Hybrid A-Z index example: Michigan State University


Improving content: How SSA can help

1. Verify content spidering 2. Plug content gaps

(placeholders, new content, appropriate types, best bets)

3. Improve content positioning (highlighting, demotion, deletion)

4. Prioritize content development (invest in appropriate topics and types, more efficient migration and inventory)


Developing new content: Addressing emergent content needs Using SSA to plug gaps

Failure analysis suggests unmet content needs by topic (e.g., frequent queries for “iPod wristband” and “iPod holder for jogging”)

Content typing (covered later) suggests what kind of content to create (e.g., “product comparison” and “specifications”)

0 results report (from behaviortracking.com)


Developing new content: Addressing emergent content needs

Use SSA to develop editorial agenda; determine relative “demand” for content topics and document types

Available content (supply) Content by query (demand)


Best bets: HP example (exposes content model)


Monitoring trends: Tuning content to meet demand

How might such trends impact your content? Search engine? Navigation?

From behaviortracking.com



Monitoring trends

Short head and long tail generally predictable

“Middle torso” is interesting area to monitor


Comparing short head and long tail queries (from Vanguard)

Short head: common queries Long tail: common queries

Beneficiary form 401(k) beneficiary career forms amt money market location loans calculator

403(b)(7) account asset transfer authorization automatic investing Wire transfer instructions adoption agreement international wire transfers socially responsible investing Vanguard tax identification number IRA Asset Transfer form fdic insured account early withdrawal penalties


The shape of your content: Determining content types

Top queries clustered and prioritized by content types: Application News/

Announce-ments

Main page Contact

info Instructions

1-2 hours work

“What kind of page would users want when they searched this term?”


Content types help expose content models, improve navigation

23

artist descriptions

album reviews

album pages

artist bios discography

TV listings


Engaging content owners: Showing how users find their content Connecting pages that are found through search…


Engaging content owners: Showing how users find their content …with how those pages were found.


Engaging content owners: Comparing time periods

Showing trends

(from behaviortracking.com)


Improving search: How SSA can help 1. Understand how search works (and what can

be changed/fixed) 2. Validate “the box” UI design 3. Look for opportunities to “soup up” queries

(e.g., syntax, spell-checking) 4. Design better search results (individual and

groups) 5. Blow up “advanced search” and contextualize

functions 6. Buy a better engine—or improve your own


Improving “the box”: Initial query entry UI

Validate width of query entry field Use query data to calculate “high

number” (e.g., 90th percentile) for query length in characters

Does your entry field accommodate that length?

Amazon: 65 characters

Microsoft: 25 characters


Improving “the box”: Search suggestions Populate query entry with common queries

(Ask) Alternative: use controlled vocabulary

(Netflix movie titles)


Query enhancement: Handling unique query types

Little fixes can add up Spell-checking Proper names (people, places) Unique identifiers (e.g., SKUs, ISBNs)


Query enhancement: Handling unique query types

Little fixes can add up Acronyms (glossary lookup)


Search results design: What to show per individual result

…more descriptive information for open-ended searchers

…less information for known-item searchers


Search results design: SSA helps determine sorting

Financial Times found a high level of dates within queries

Solution: support chronological sorting and filtering


Search results design: SSA helps determine typing Content typing can enable:

Filtering by exposing types (HP)

Faceted classification/navigation (WebMD)


Search results design: Which results where? #10 result is clicked through more often than #s 6, 7, 8,

and 9 (ten results per page)

From SLI Systems (www.sli-systems.com)


Rethinking advanced search: Which functions go where (or away)?


Rethinking advanced search: “Revise your Search” in action


How we determine what users want

Qualitative research (observation) Field studies Focus groups Card sorting Usability testing and task analysis

Quantitative research (data-driven) Help/reference desk/switchboard Web analytics (SEO/SEM, clickstream, SSA)


SSA as a user research method: How is it different? Data that’s:

Real High volume Readily available

Analysis that’s: Quantitative Inexpensive Scalable Complementary to other user research

methods


Comparing SSA with other quantitative user research methods Data-driven methods

Clickstream/server log analysis: tells you where more than what users wants (complementary)

Event logging: comparable but often less data and usually at a different point in customer/user lifecycle

SEO/SEM: Different query types (finding where the answer is versus finding the answer); often focused on commerce conversions


Comparing referral queries with local queries


SSA and qualitative user research methods

What questions lead to Why questions Examples

Personas: SSA supplements fictitious construct with real data (i.e., likely queries a persona might have)

Task analysis: SSA helps identify common tasks to evaluate

Field studies: SSA suggests issues to monitor in the field


Augmenting personas and audience profiles with frequent queries

Persona example (from Adaptive Path)

Frequent queries added (in green)


Using common queries to enhance/fuel task analyses

Use for tasks that are more what than how (i.e., testing information needs rather than functions)

Examples • Can you find a map of the campus? • What study abroad options are available to MSU students? • When is the last home football game of the season?


Identifying trends to monitor during field studies Example: catalog clothing retailer SSA showed:

Preponderance of SKUs, even though they were not displayed on the site

What were they doing there? Subsequent field study showed:

Customers preferred print catalog to web catalog to identify products

Customers preferred web catalog for ordering Action: add SKUs to web catalog


Combining quantitative and qualitative

Exploring the needs of a particular audience: users of BBC children’s content


The power of data: Why UX needs SSA 1. Balance: UX methodologies/

practitioners are often quantitatively weak 2. Legitimacy: Data-driven analyses make

an impression on people hard to impress 3. Cost: The data is available; the analysis

scales well with available time 4. Fidelity: The data is behavioral, real,

and voluminous 5. Comprehensiveness: Quantitative

methods complement and help improve qualitative methods (and vice versa)


Please share your knowledge: Visit our book in progress site Search Analytics for Your Site:

Conversations with your Customers by Louis Rosenfeld and Richard Wiggins (Rosenfeld Media, 2008)

Site URL: www.rosenfeldmedia.com/books/searchanalytics/

Feed URL: feeds.rosenfeldmedia.com/searchanalytics/


Contact Information

Louis Rosenfeld Rosenfeld Media, LLC 705 Carroll Street, #2L Brooklyn, NY 11215 USA

[email protected]

www.louisrosenfeld.com www.rosenfeldmedia.com http://www.slideshare.net/lrosenfeld (these slides)

Business

Site Search Analytics