SEO Skeptic

Was crawling around today (er, maybe that doesn't sound too good) and had forgotten to change my user agent switcher from Googlebot.  I landed on Amazon.com and navigated to their book section.  Hmm, I thought, something's missing here.

Let me work backwards.  Here's what I saw after I had switched my user agent back to the default browser (by the way, all these screenshots were generated after I had cleared my cookies and stopped accepting them).

All fine and well, the Amazon we all know and love.  Note that in addition to the ubiquitous search form, Amazon also has a mini drop-down menu.

But this is what I saw when I first cruised in as Googlebot.  I've given you enough clues (um, such as the post title) that you should be able to tell what's missing, but it took me a moment of head-scratching.

That's right, gentle readers, there is no search form for Googlebot.  And there is no drop down menu supplied to Google either.  Google, feel free to use the primary navigation, but don't go acting like some user.

I think there's an excellent reason for this.  Just over a year ago, Google announced that it was going to start crawling HTML forms.  As much as this might aid some sites for discoverability, it can also cause havoc on the indexing front.  If you have a form - especially with multiple inputs, the resulting URLs placed in index can end up as a dog's breakfast of duplicate content and highly unfriendly parameterized URLs.  And while Googlebot stressed it would respect all indexing exclusion protocols like robots.txt, in practice I think this could be very difficult to work out for many forms.

As the post also states it was restricting the initial crawls to "a small number of particularly useful sites" it makes sense that Amazon would have taken the lead in agent-based form delivery (or rather, non-delivery), doubtlessly being deemed by Google to be one of said "particularly useful sites."  I've seen every indication that they've since extended form crawling to rather less "useful" sites, and I've also seen some real pollution in the index originating from this extension.

Ah, the age-old question:  is it cloaking?  I for one am usually against agent-based delivery just on the grounds that it opens you up to a potential cloaking charge, but I'll also open to the procedure as a technique of last resort.  I don't know Amazon's view on agent-based delivery, but apparently they consider particular instance of agent detection and delivery fair play.  I'll keep an eye out for other agent-based form manipulation (maybe I'll even apply it).

By the way you may have noticed the variant "Mother's Day" messages on the human version.  Googlebot always gets the same message (and same link, as per below - PR5):

• Mother's Day is May 10: Save Now

Humans get the following mix:

• Mother's Day is May 10: Save Now
• Get fast, free shipping for Mother's Day
• Get FREE 2-Day Shipping for Mother's Day
• [No Mother's Day message]

How does this correlate to rankings?  That URL appears on page one in Google for the following:

• mother's day deals
• mothers day savings
• mothers day save
• mothers day gift store

These queries are actually number one:

• mothers day gift sale
• mother's day gift savings
• save on mothers day gifts (2 - indent of 1+2 for Amazon)

No page one love for the real biggies, though - "mothers day gifts" or "mother's day gift."  Nice too see Amazon doesn't rule the entire ecomm universe.

Now is that cloaking?  It's pretty clear they're testing the different messaging to see what's most effective, so one could legitimately argue that it's a test that in which Google need not participate.  On the other hand, a zillion Amazon.com pages linking to a stable URL for Mother's Day right before the event is sure to carry a lot of juice.  Juice enough that the page is ranking well.  Really well.

This entry was posted on Tuesday, April 28th, 2009 at 10:56 am and is filed under Cloaking, Rankings, Crawling. You can follow any responses to this entry through the RSS 2.0 feed. You can leave a response, or trackback from your own site.