Bloomberg Anywhere Remote Login Bloomberg Terminal Demo Request


Connecting decision makers to a dynamic network of information, people and ideas, Bloomberg quickly and accurately delivers business and financial information, news and insight around the world.


Financial Products

Enterprise Products


Customer Support

  • Americas

    +1 212 318 2000

  • Europe, Middle East, & Africa

    +44 20 7330 7500

  • Asia Pacific

    +65 6212 1000


Industry Products

Media Services

Follow Us

Bloomberg Customers

Maybe Some Gardens Have Walls for a Good Reason?

Posted by: Rob Hof on January 4, 2006

According to Search Engine Roundtable, the popular online classified-ad site Craigslist may be blocking search engines such as Google from crawling its site. Search experts like John Battelle make a good case that this is not a great idea, since it makes it harder for people to find the listings. Don’t yet know why Craiglist is apparently doing this, though CEO Jim Buckmaster recently listed several reasons the site earlier blocked Oodle from scraping it. But Greg Yardley wonders if maybe Craigslist has a point: “As Google morphs into a portal, why should Craigslist allow it to rip-off and monetize their content?” He also wonders:

This does raise interesting questions, though - since Craiglist’s voluntary absence from the search engines makes them all a little bit less useful. What if Craigslist is starting a trend? How many key sites have to block search engine traffic before the engines’ relevance starts to sink, and what could that do to paid search? In an atmosphere of site defection and declining search engine relevance, could a site like Craigslist suddenly allow one and only one search engine to spider its site - for a fee?

I tend to side with John on this, but it will be interesting to hear what Craig Newmark himself has to say.

UPDATE: Danny Sullivan at SearchEngineWatch says Craigslist in fact is not blocking search engines, just blocking duplicate pathways to the listings. As Emily Litella would say….

Reader Comments


January 4, 2006 9:21 AM

I also wonder the same thing. Especially when Yahoo! Local results are so much more relevant than Google search results, but Google searches turn up Yahoo! data as its first hit... many times including the phone number you need, without ever having to go to Yahoo!. I wouldn't blame Yahoo! for being upset.


January 4, 2006 11:00 AM

In the past, I've noticed that Craigslist postings turn up in searches on Feedster.

Heather Green

January 4, 2006 3:34 PM

Well, it begs the question, does Google let other companies crawl its data?

saleem moonbeemer

January 4, 2006 5:29 PM

>>Well, it begs the question, does Google let
>>other companies crawl its data?

In a sense, yes, there is a way to grab the keywords queries and the frequency of them. As for the index data, why? Anyone can start a crawler, all it takes is a simple script, a huge pipe and lots of disk space and horsepower(if you are a success).

The Guide to Getting More Out of Travel

January 4, 2006 5:29 PM

interesting to see if this will affect google's stock.

Michael Zarb

January 5, 2006 6:38 AM

So in a sense all Microsoft has to do to win the search war is to hand out enough monetary compensation to top site in order to get them to be only indexed by MSN.

However I doubt even Microsoft would have enough resources for such a strategy, and I doubt the Search industry would remain profitable if such tactics where employed especially if Google had to fight back in the same way.

Post a comment



Bloomberg Businessweek writers Peter Burrows, Cliff Edwards, Olga Kharif, Aaron Ricadela, and Douglas MacMillan, dig behind the headlines to analyze what’s really happening throughout the world of technology. Tech Beat covers everything from tech bellwethers like Apple, Google, and Intel and emerging new leaders such as Facebook to new technologies, trends, and controversies.



BW Mall - Sponsored Links

Buy a link now!