Maybe Some Gardens Have Walls for a Good Reason?

Posted by: Rob Hof on January 04, 2006

According to Search Engine Roundtable, the popular online classified-ad site Craigslist may be blocking search engines such as Google from crawling its site. Search experts like John Battelle make a good case that this is not a great idea, since it makes it harder for people to find the listings. Don’t yet know why Craiglist is apparently doing this, though CEO Jim Buckmaster recently listed several reasons the site earlier blocked Oodle from scraping it. But Greg Yardley wonders if maybe Craigslist has a point: “As Google morphs into a portal, why should Craigslist allow it to rip-off and monetize their content?” He also wonders:

This does raise interesting questions, though - since Craiglist’s voluntary absence from the search engines makes them all a little bit less useful. What if Craigslist is starting a trend? How many key sites have to block search engine traffic before the engines’ relevance starts to sink, and what could that do to paid search? In an atmosphere of site defection and declining search engine relevance, could a site like Craigslist suddenly allow one and only one search engine to spider its site - for a fee?

I tend to side with John on this, but it will be interesting to hear what Craig Newmark himself has to say.

UPDATE: Danny Sullivan at SearchEngineWatch says Craigslist in fact is not blocking search engines, just blocking duplicate pathways to the listings. As Emily Litella would say….

TrackBack URL for this entry: http://blogs.businessweek.com/mt/mt-tb.cgi/

Reader Comments

Eddie

January 4, 2006 09:21 AM

I also wonder the same thing. Especially when Yahoo! Local results are so much more relevant than Google search results, but Google searches turn up Yahoo! data as its first hit... many times including the phone number you need, without ever having to go to Yahoo!. I wouldn't blame Yahoo! for being upset.

AHCB

January 4, 2006 11:00 AM

In the past, I've noticed that Craigslist postings turn up in searches on Feedster.

Heather Green

January 4, 2006 03:34 PM

Well, it begs the question, does Google let other companies crawl its data?

saleem moonbeemer

January 4, 2006 05:29 PM

>>Well, it begs the question, does Google let
>>other companies crawl its data?

In a sense, yes, there is a way to grab the keywords queries and the frequency of them. As for the index data, why? Anyone can start a crawler, all it takes is a simple script, a huge pipe and lots of disk space and horsepower(if you are a success).

The Guide to Getting More Out of Travel

January 4, 2006 05:29 PM

interesting to see if this will affect google's stock.

Michael Zarb

January 5, 2006 06:38 AM

So in a sense all Microsoft has to do to win the search war is to hand out enough monetary compensation to top site in order to get them to be only indexed by MSN.

However I doubt even Microsoft would have enough resources for such a strategy, and I doubt the Search industry would remain profitable if such tactics where employed especially if Google had to fight back in the same way.

Post a comment

 

About

BusinessWeek writers Peter Burrows, Cliff Edwards, Olga Kharif, Aaron Ricadela, Douglas MacMillan, and Spencer Ante dig behind the headlines to analyze what’s really happening throughout the world of technology. One of the first mainstream media tech blogs, Tech Beat covers everything from tech bellwethers like Apple, Google, and Intel and emerging new leaders such as Facebook to new technologies, trends, and controversies.

Categories

 

BW Mall - Sponsored Links

Buy a link now!