Why We Ignore the Margin of Error--And Everyone Else Does, Too

Posted by: Peter Coy on January 25, 2007

A reader named Dave makes an important point in a comment on my item last week about the supposed 4.5% rise in housing starts in December.

Dave says—accurately—that we don’t really know whether housing starts rose at all, because the margin of error in the Census Bureau’s survey was so large. He thinks we should have admitted our ignorance. In Dave’s words: “Such statistically negligent reporting continues to amaze me!”

He's right. I didn't mention the margin of error at all in my blog item, instead treating the "increase" like a settled fact. In my defense, I was hardly alone: just about every journalist and financial analyst and economist who wrote about the Census Bureau report stated that housing starts rose and made no mention of the margin of error.

If we had been scrupulous, we would have written something like this:

Census Bureau Doesn't Know if Housing Starts Rose or Fell in December

Confusion over the state of the housing industry continued today as the Census Bureau once again said that it doesn't know whether the rate of construction of homes rose or fell in the most recent month. The bureau estimated that starts rose 4.5%, but the margin of error was plus or minus 8.8 percentage points. The same thing happened for October, where the estimated increase was 6.7% but the margin of error was plus or minus 10.1 percentage points. On the bright side, we're pretty sure that starts really did increase in September, because the estimated increase then (14.6%) was bigger than the margin of error (plus or minus 7.6 percentage points).

As this exercise goes to show, the strictly scrupulous approach can be tedious. On the other hand, it has the important advantage of sticking to what's actually known. So I agree with Dave. I'm going to make more of an effort in the future to point out when numbers are particularly squishy because of sampling error.

For those who like reading the fine print, here is what the Census Bureau stated in a footnote on the first page of its housing starts news release:

In interpreting changes in the statistics in this release, note that month-to-month changes in seasonally adjusted statistics often show movements which may be irregular. It may take 4 months to establish an underlying trend for building permit authorizations, 5 months for total starts, and 6 months for total completions. The statistics in this release are estimated from sample surveys and are subject to sampling variability as well as nonsampling error including bias and variance from response, nonreporting, and undercoverage. Estimated relative standard errors of the most recent data are shown in the tables. Whenever a statement such as "2.5 percent (plus or minus 3.2%) above" appears in the text, this indicates the range (-0.7 to +5.7 percent) in which the actual percent change is likely to have occurred. All ranges given for percentage changes are 90-percent confidence intervals and account for only sampling variability. If a range does not contain zero, the change is statistically significant. If it does contain zero, the change is not statistically significant; that is, it is uncertain whether there was an increase or a decrease. The same policies apply to the confidence intervals for percentage changes shown in the tables. On average, the preliminary seasonally adjusted estimates of total building permits, housing starts and housing completions are revised about one percent. Explanations of confidence intervals and sampling variability can be found on our web site listed above. [http://www.census.gov/newresconst]

By the way, Census uses 90% confidence intervals. If the bureau used a 95% confidence interval, as many social scientists do, the margins of error would be even greater.

Dave--thanks for the comment.

Reader Comments

Peter Coy

January 25, 2007 4:38 PM

Erica Filipek of the Census Bureau just returned a call and pointed out that the components of the housing starts numbers have even higher margins of error than the overall number. Examples:
--In the Northeast, single-family house construction starts were reported to have fallen 3.5%, but the margin of error was plus or minus 23.7 percentage points.
--Nationwide, starts on buildings with five or more units was reported to have increased 30.6%, but the margin of error was plus or minus 37.4 percentage points. Scary when even an increase that big might not actually be an increase at all.

Why such a big margin of error? Money. If the Census Bureau could afford a bigger survey, the law of large numbers would kick in and the reliability would increase.

As I mentioned in my post, Census advises looking at five consecutive months of starts before trying to call a trend.

Meanwhile, Filipek says, a word to the wise: "it’s probably not a good idea to use the word 'plunged' when we don’t know if it’s even up or down."

lizziebeth

January 26, 2007 10:57 AM

Thanks for clarifying Peter. It is nice to see someone in the media to actually explain the fuzzy numbers.

I question "plunge" when they so clearly stated that we shouldn't use the word plunge when we don't really know if it's up or down. Did he caution us not use the word "Rise" when we didn't really know if it's up or down???????

Mark Norman

January 30, 2007 11:22 AM

If I were a builder/developer who needed to make decisions on this information the extra accuracy would be important. If I were a builder/developer that needed to depend upon the U.S government's info to make building decisions I probably wouldn't last too long.

As a realtor it is simply an indication. If I asked my buyers about the question of this information I would bet that most would laugh in my face.

In a past life I earned a MBA degree in Strategic Planning, my statistical background is fairly strong. As a marketer you can make numbers say almost anything you want (if that is your purpose).

More important is the trend line. It should just give us a "sense" of the market. If next month's number show a drop of several percentage points, then we'll know that December's numbers were an aberration, if the numbers continue to show an increase it would mean that we might be seeing the beginning of an upswing, but before I said that I would want to see several more months of data before making that declaration.

My suggestion to Dave is "to chill a bit".

John L. Wake

January 30, 2007 4:04 PM

A 95% confidence interval isn't needed in business. A 90% confidence interval is plenty high enough to make informed business decisions.

Anyway, a narrow 95% confidence interval would be using statistics to measure the obvious.

Post a comment

 

About

BusinessWeek editors Chris Palmeri, Prashant Gopal and Peter Coy chronicle the highs and lows of the housing and mortgage markets on their Hot Property blog. In print and online, the Hot Property team first wrote about the potential downside of lenders pushing riskier, "option ARM" mortgages and the rise in mortgage fraud back in 2005—well ahead of many other media outlets. In 2008, Hot Property bloggers finished #1 in a ranking of the world's top 100 "most powerful property people" by the British real estate website Global edge. Hot Property was named among the 25 most influential real estate blogs of 2007 by Inman News.

BW Mall - Sponsored Links

Buy a link now!