The Travails of Tracking Web Traffic

The practice of using consumer panels to estimate Web-site traffic is coming under increased scrutiny and fueling calls for industrywide standards

by Catherine Holahan

At first, Digg Chief Executive Jay Adelson thought he just couldn't count. By Digg's estimate, 15 million individuals were visiting the news bookmarking site each month. But advertisers said the site wasn't nearly that popular. Other yardsticks, like the one used by comScore, put the numbers between 1 million and 2 million a month. Adelson figured he was misreading his company's server logs. "We thought, 'O.K., maybe it's us,'" says Adelson.

So Digg hired Web analytics firm WebSideStory (WSSI) to audit the traffic. The firm installed unseen tracking tags on Digg's pages that yielded results matching Digg's own. Still, some advertisers were—and remain—unconvinced and point to lower figures as justification for paying lower ad rates.

Such travails of tracking Web traffic are neither new nor unique to Digg. But lately, they've become so pronounced that the Interactive Advertising Bureau (IAB), an association that counts hundreds of Web publishers among its members, has asked two of the most prominent Web measurement firms to submit to an audit by the Media Rating Council (MRC), a media ratings watchdog.

Panels Are Being Questioned

A method coming under particular scrutiny is reliance on panels, large groups of users whose habits are used as a proxy for the behavior of larger groups. Some of the biggest analytics companies, including Nielsen//NetRatings (NTRT) and comScore, use panels ranging in size from hundreds of thousands to more than 2 million.

Web publishers say the technique, long used to measure TV audiences, doesn't accurately reflect their audiences and results in flawed traffic results. Derek Gordon, vice-president of marketing for Technorati, a search engine for blogs and other user-generated content, says panel-based traffic numbers for his company are often off by a factor of 10. "What the site knows to be true vs. what the panel says to be true is off by an order of magnitude," says Gordon.

As part of the audit, the MRC is likely to probe whether panels skew results and could recommend measures to fix the panel composition as well as additional measurement methods to make statistics more accurate. "Panels may not be leveraging the power and interactivity of our media," says Sheryl Draizen, senior vice-president and general manager of the IAB, whose members include Disney's (DIS) ABC News, Time Warner's (TWX) AOL, and BusinessWeek parent The McGraw-Hill Companies (MHP).

Audit Is Welcomed

Young Internet companies, which rely on traffic numbers to get noticed by marketers, are among the most vocal about the measurement problem (see BusinessWeek.com, 10/23/06, "Web Numbers: What's Real?"). For many, traffic numbers decide not just the price of an ad but whether the advertiser inquires about an ad at all. "It's a question of zero revenue or any revenue—do you even capture the attention of a buyer," says Gordon.

Both Nielsen//NetRatings and comScore say they welcome an audit, though they stand by their panel-based methods. NetRatings already completed a pre-audit, a preliminary review by Ernst & Young intended to flag problems that should be fixed before the full review. Nielsen//NetRatings plans to become certified by the MRC. "We have a sense from the pre-audit report that our services are in pretty good shape," says Manish Bhatia, executive vice-president of global operations at NetRatings.

There are benefits to panels. Unlike server logs, which measure individual users by the number of times an unrecognized Internet protocol address visits a site, panels can tell companies more about the audience's demographic composition. This is possible as panel members share such information as age, gender, and income level. Unlike using IP addresses, panels can also help distinguish between visits by different computers, which may have the same user, and visits by actually different people.