Posted by: Rob Hof on January 4, 2006
According to Search Engine Roundtable, the popular online classified-ad site Craigslist may be blocking search engines such as Google from crawling its site. Search experts like John Battelle make a good case that this is not a great idea, since it makes it harder for people to find the listings. Don’t yet know why Craiglist is apparently doing this, though CEO Jim Buckmaster recently listed several reasons the site earlier blocked Oodle from scraping it. But Greg Yardley wonders if maybe Craigslist has a point: “As Google morphs into a portal, why should Craigslist allow it to rip-off and monetize their content?” He also wonders:
This does raise interesting questions, though - since Craiglist’s voluntary absence from the search engines makes them all a little bit less useful. What if Craigslist is starting a trend? How many key sites have to block search engine traffic before the engines’ relevance starts to sink, and what could that do to paid search? In an atmosphere of site defection and declining search engine relevance, could a site like Craigslist suddenly allow one and only one search engine to spider its site - for a fee?
I tend to side with John on this, but it will be interesting to hear what Craig Newmark himself has to say.