Technorati allows users to search blogs for specific content and monitor which blogs are the most popular. But with rumors that big boys Google (GOOG
), Yahoo! (YHOO
), and MSN (MSFT
) are soon to enter blog search, the blogosphere is abuzz with speculation that Technorati can't compete. Some users find the search engine taking as much as 30 seconds to spit back results, and when millions of Web searchers flocked to Technorati following the July 7 London bombings, the site had difficulty handling all the traffic.
Sifry maintains these challenges are inherent to blog search, not indicative of his small company's inability to keep pace. He recently spoke with BusinessWeek intern Dana Goldstein. Following are edited excerpts of their conversation.
Q: You say you welcome competition from Google, Yahoo, and MSN, should they decide to offer blog search. Why would you welcome such Net heavyweights as rivals?
A: The larger question is, is it really competition? I look at what Google and Yahoo and other companies in this space are doing, and they're really fantastic at helping you pick out what's the best reference site for something. You go to Google and type in wine, and it will tell you the best places to buy wine. But if you really want to find out what the world's leading wine experts are talking about, Google isn't really built to do that.
Q: Why can't they build themselves up to do that with a blog search engine?
A: Well, good luck. We've been doing it now for almost three years, and it's a lot harder than you think. Doing it on a small scale is not terribly difficult. Doing it to scale becomes pretty hard, and every day the blogosphere is growing by leaps and bounds.
The blogosphere today is about 30 times as big as it was three years ago. So just to give you some ideas on what that means: Every single day we're seeing about 80,000 new people who are starting blogs. And we're seeing about 900,000 new posts every single day. So that's about 11 posts every single second that you've got to now index, you've got to score it, you've got to make sense out of all of its relevance, and you've got to push it to your servers really, really fast so people can stay up to date with what's going on.
Q: Does Technorati have the technological capability to push blog searches through to servers fast enough to keep users happy? Users are used to doing incredibly fast Web searches through traditional search engines.
A: We've built what's called a streaming architecture, as opposed to a polling architecture. We don't actually go out and crawl the entire Internet. We're built into every single publishing platform. So instead of crawling the Web, we literally get notified when new stuff is created or when stuff is updated or deleted. We have an entire search scheme that takes under seven minutes to complete.
Q: You say Google can't tell you what wine experts are talking about, but Technorati can. How do you do that?
A: When you think about the words that we use when we talk about the Web, we talk about pages, we talk about documents, we talk about directories. What does that mean, the language, the metaphor we use when we think about the Web today?
Q: It's a print culture metaphor.
A: Exactly. But there's a long way to go. We're really trying to do something a little different from that. What Technorati is trying to do is looking at the Web in a different way. And the way that I like to think of it is, it's like this big river, it's like this conversation flow. It's about people and conversations.
Just as Google invented page rank, reordering the way that we sort the Web, what we did was say, "O.K., why don't we take the same idea and apply it to people." So the way that Technorati calculates what we call "Net attention" is we look at how many people are linking to you.
Q: What growing pains are you experiencing as blogging becomes more popular?
A: Well first off is just the sheer amount of traffic and attention. When we put up our page about the London bombings, there were so many people, so many bloggers who were posting pictures.
They were doing news reporting essentially. You would be amazed how many people were blogging just to say "I'm O.K." There were people here in the States and in other countries saying, "We feel for you, London." That kind of outpouring of emotion and support is half the equation. Just keeping up with this incredible growth is a challenge.
The second half is that there were so many people who wanted to get up-to-the-minute reports from the street. What's going on? What did people see? It's not just about blogs. You go to Technorati today and you're seeing pictures, you're seeing links, you're getting a multimedia experience.
And you're getting it not just from CBS, ABC, and NBC. You're getting it from millions of people who are on the ground. The challenge and the opportunity that Technorati has is: How do we keep doing great stuff for these bloggers, how do we keep providing them with the best tools for themselves? And how do we also work with the people who aren't bloggers, the people who are just readers? And that's a challenge and a responsibility that we take very, very seriously.
Q: When I used to be a blogger, I thought Technorati was quite addictive. My favorite tool was just searching to see what other bloggers were saying about me and what I wrote.
A: Well, thank you! That is the highest compliment that anybody can pay, that you're addicted, that you love it, that you come back again and again, that we're providing you with great information you can't get anywhere else. That's job one. And I never, ever, ever want to lose that because bloggers are essentially putting food on the table.
To me, what's the big opportunity here? To not only build something interesting that people like, but frankly, to do something that's good for society and good for people. And if you'll allow me to get corny for a moment, the thing that I feel so passionate about, the thing that I feel there is such an opportunity for with blogging -- and you kind of see the nascent pieces of it -- is an opportunity for the reinvigoration of civics. People getting reengaged, so they don't have to feel powerless.
Q: If another company can match what you do, will bloggers stay with Technorati?
A: If Technorati is being of service to bloggers and readers, we're going to do just fine. Our corporate mantra is "Be of Service."
Hey we're a Web site, [but] we also have to be providing some real value to people, because that's what being of service is really about. And that doesn't mean controlling or owning, because when you're of service, you're supporting. If you're providing people with something that they like and that they value, with that value comes money.
So I'm not worried about us in terms of business. Keep your customers and users happy, and everything else follows. We've had four months, month on month, with greater than 40% traffic growth from month to month.
Q: What are you working on that business users might be excited about?
A: First, there's advertising and sponsorship. Business users can advertise on search results that have something to do with their company.
Second, we're unveiling a new service in August that's currently in beta testing that's geared toward professionals -- people who need a deeper view of a company or its products, such as PR people, people in marketing or advertising, financial analysts. [Basically,] people who need to track buzz, how it changes over time, who are the influencers who is talking about their company or their product. These will be subscription products.