View
751
Download
0
Embed Size (px)
DESCRIPTION
Citation preview
ionSearch 2012
Roland Dunn
Partner – Refined Practice
Searchbots: Lost Children ... or ... Hungry Psychopaths? What Do Searchbots Actually Do? (and why it matters)
© 2007-2012 Roland Dunn
© 2007-2012 Roland Dunn
© 2007-2012 Roland Dunn
Straightforward for humans and searchbots to find your content?
© 2007-2012 Roland Dunn
Straightforward for humans and searchbots to find your content?
Do you provide what your potential customers want?
© 2007-2012 Roland Dunn
Straightforward for humans and searchbots to find your content?
Do you provide what your potential customers want?
Do you stand out? Are you compelling, engaging, useful, unique, the best?
© 2007-2012 Roland Dunn
Straightforward for humans and searchbots to find your content?
Do you provide what your potential customers want?
Do you stand out? Are you compelling, engaging, useful, unique, the best?
© 2007-2012 Roland Dunn
“Googlebot ... may be unable to completely index all the content on your site”
© 2007-2012 Roland Dunn
© 2007-2012 Roland Dunn
© 2007-2012 Roland Dunn
© 2007-2012 Roland Dunn
© 2007-2012 Roland Dunn
Doesn’t tell us what searchbots index
© 2007-2012 Roland Dunn
© 2007-2012 Roland Dunn
© 2007-2012 Roland Dunn
© 2007-2012 Roland Dunn
© 2007-2012 Roland Dunn
Webserver Logfiles ....
© 2007-2012 Roland Dunn
Webserver Logfiles ....
© 2007-2012 Roland Dunn
Webserver Logfiles ....
© 2007-2012 Roland Dunn
© 2007-2012 Roland Dunn
Over 2 months, 40% of all Googlebot’s requests were to …. just 2 URL’s (on-site search URLs)
© 2007-2012 Roland Dunn
Over 2 months, 40% of all Googlebot’s requests were to …. just 2 URL’s (on-site search URLs) On a website with approximately 40,000 URLs in Google’s index (allegedly, according to site:…)
© 2007-2012 Roland Dunn
Over 2 months, 40% of all Googlebot’s requests were to …. just 2 URL’s (on-site search URLs) On a website with approximately 40,000 URLs in Google’s index (allegedly, according to site:…) On a website serving approx. 150-200K unique natural search visits/month
© 2007-2012 Roland Dunn
Over 2 months, 40% of all Googlebot’s requests were to …. 2 URL’s. On a website with approximately 40,000 URLs in Google’s index. On a website serving approx. 150-200K unique natural search visits/month
© 2007-2012 Roland Dunn
Over 2 months, 40% of all Googlebot’s requests were to …. 2 URL’s. On a website with approximately 40,000 URLs in Google’s index. On a website serving approx. 150-200K unique natural search visits/month
© 2007-2012 Roland Dunn
© 2007-2012 Roland Dunn
© 2007-2012 Roland Dunn
© 2007-2012 Roland Dunn
Categorise URLs by “top level section” – perhaps more useful than just URLs ...
© 2007-2012 Roland Dunn
Table of “top level sections” requested by Googlebot
© 2007-2012 Roland Dunn
© 2007-2012 Roland Dunn
Chart of “top level sections” requested by Gbot – 6 months
© 2007-2012 Roland Dunn
Does Googlebot request all URLs served by a website?
© 2007-2012 Roland Dunn
Does Googlebot request all URLs served by a website?
No!
© 2007-2012 Roland Dunn
© 2007-2012 Roland Dunn
How Does Googlebot Spend Its Time? What Does it Request?
Does Googlebot spend its time cost-effectively?
© 2007-2012 Roland Dunn
Does Googlebot spend its time cost-effectively?
© 2007-2012 Roland Dunn
Not always.
© 2007-2012 Roland Dunn
Googlebot does not always request all content
© 2007-2012 Roland Dunn
Googlebot can become distracted, obsessed, or even lost:
© 2007-2012 Roland Dunn
Googlebot can become distracted, obsessed, or even lost:
• On-site search
• Additive filters/faceted URLs e.g. “shoes?size=3&colour=green&price=10&brand=smith”
• Sections with thin and/or very similar content
© 2007-2012 Roland Dunn
Googlebot may not spend its time efficiently
It needs help focusing on what we value
© 2007-2012 Roland Dunn
© 2007-2012 Roland Dunn
SMX West, March 2011, Matt Cutts: (http://goo.gl/ZZz7E):
“... if Google determines a site isn’t as useful to users, they may not crawl it as frequently"
© 2007-2012 Roland Dunn
© 2007-2012 Roland Dunn
Suggestions:
• Embrace logfiles – full of useful information
© 2007-2012 Roland Dunn
Suggestions:
• Embrace logfiles – full of useful information
• Periodically check your searchbot behaviour
© 2007-2012 Roland Dunn
Suggestions:
• Embrace logfiles – full of useful information
• Periodically check your searchbot behaviour
• Especially if the “Googlebot found a high number” message keeps appearing
© 2007-2012 Roland Dunn
If searchbots appear distracted, unfocused, lost:
© 2007-2012 Roland Dunn
If searchbots appear distracted, unfocused, lost:
• Alter internal navigation & linking (e.g. flatten hierarchy)
© 2007-2012 Roland Dunn
If searchbots appear distracted, unfocused, lost:
• Alter internal navigation & linking (e.g. flatten hierarchy)
• Robots.txt out (blunt approach)
© 2007-2012 Roland Dunn
If searchbots appear distracted, unfocused, lost:
• Alter internal navigation & linking (e.g. flatten hierarchy)
• Robots.txt out (blunt approach)
• Alter URL construction (expensive!)
© 2007-2012 Roland Dunn
If searchbots appear distracted, unfocused, lost:
• Alter internal navigation & linking (e.g. flatten hierarchy)
• Robots.txt out (blunt approach)
• Alter URL construction (expensive!)
• Check “sitemaps” – all URLs? What priorities expressed?
© 2007-2012 Roland Dunn
© 2007-2012 Roland Dunn
Lost Children ... or ... Hungry Psychopaths?
© 2007-2012 Roland Dunn
Or .... Distracted Teenager?
© 2007-2012 Roland Dunn
© 2007-2012 Roland Dunn
© 2007-2012 Roland Dunn
Sonification: What do Search Engines Requests Sound Like?
© 2007-2012 Roland Dunn
Sonification: What do Search Engines Requests Sound Like?
© 2007-2012 Roland Dunn
Online Branding:
http://www.refinedpractice.com/
T: @RefinedPractice
© 2007-2012 Roland Dunn
Natural Search:
http://www.cloudshapes.co.uk/
T: @roland_dunn
Slides Available At:
http://www.cloudshapes.co.uk/talks/