The Continuing Saga of Rocketboom Numbers and Yanking Our Estimate

Posted by: Heather Green on November 16

Almost three weeks ago, Rob Walch (a podcasting consultant) and I jumped into the debate that Ze Frank kicked up over Rocketboom’s numbers. This nerdfight was capped off by Ze using an Alexa graph that showed that his show was more popular than Rocketboom—although Rocketboom’s Andrew Baron claimed more than 300,000 daily downloads vs. Ze’s 30,0000 viewers.

Since there were no third party metrics, Rob and I figured that by looking at Rocketboom’s logs, we could get the best approximate sense of whether Ze was right or not. In our initial post on Oct. 27, we found that Rocketboom had around 211,000 daily downloads—fewer then Andrew had claimed, but still a high number.

People reading the post, though, pointed out a problem in our analysis. So I contacted Andrew on Nov. 3 to take another look at the logs and figure out how or whether to correct the estimate.

As a result of this scrutiny, a blog post that Andrew published today shows that there was in fact an error in our original estimate. This error cut in half the original estimate from Rocketboom servers to 78,000 downloads from 156,000. This is exactly the number of downloads that Andrew now claims that his logs serve each day on average.

But, here’s where things get even trickier. Our original 211,000 estimate also included 55,000 downloads from outside services and sites,including Tivo and Rocketboom Japan, that distribute Rocketboom. We got this estimate directly from Andrew and not from the logs. So, based on this correction and the original outside download estimate, Rocketboom would have 133,000 daily downloads.

However, Andrew in his post now says that after a lot of research, he has pinpointed more sources for outside downloads. Instead of 55,000 outside downloads, he’s claiming 129,000 so that overall Rocketboom has 207,000 daily downloads.

The debate over these stats has been hot and this is only going to make things more contentious. Andrew initially let Rob and I look at the numbers because of this. And we waited to figure out how to make a correction assuming he would let us go to the source.

Now the numbers look muddier than ever. And Andrew says that because he’s getting too many similar requests and because of NDAs, he isn’t going to let any third party, including Rob and me, verify these stats. At this point, he says it’s an issue between advertisers and himself. I think that’s wishful thinking at this point and a mistake.

For people interested in nitty gritty of the error on Blogspotting, here’s some more detail:

For the intial post on Oct. 27, Rob and I looked at the data from one log file and multipled that number by four, since Andrew said he mirrored the files across four servers. People who read the post asked us whether we had the IP addresses of the servers and had used them to verify that the servers were serving videos. We had been shown a script that was run showing files being pushed out to four servers. But we had not verified all four from the outside.

In response, Rob got the IP addresses from Andrew for the four servers and did a trace to determine whether they were all serving files. Two weren't when he checked them. That meant that, given the information we had at that point, we could only multiply the numbers we have by two servers, again, cutting in half the original estimate we made from Rocketboom’s servers to 78,500 downloads from 156,000. (And again the original estimate also included another 55,000 downloads from other services and sites.)

In order to figure out how or whether to change the estimate, we needed to verify the mistake by working with Andrew. In response specifically to the server question, Andrew says his hosting company says that he has four servers but that only two run unless there is high demand. But before talking with us about specifics, Andrew said he wanted to work on the stats so that he could then talk to us and post them on his blog.

TrackBack URL for this entry: http://blogs.businessweek.com/mt/mt-tb.cgi/