Debates over what methods result in the best software often pit those who favor an open-source approach against proponents of proprietary, or closed-source, development. Conventional wisdom holds that open-source software should have fewer security flaws than proprietary software. With more eyes able to look at the underlying source code, bugs should be found and squashed much faster.

But when software security and quality really matter—like crossing the Atlantic on a jet airliner—trust me, you want to fly proprietary.

That conclusion is based on my company's involvement in testing of software security and quality. Working with the Homeland Security Dept. and Stanford University, my firm, Coverity, has closely analyzed 50 of the most popular open-source software projects, scanning more than 20 million lines of code daily. We publish those results here on the Web.

BUG TO BUG COMPARISON. For the first time, we've compared those results with proprietary software from more than 100 different companies, including some of the best-known names in aerospace, financial services, software, and telecommunications—more than 60 million lines of code all together.

In our research using automatic bug-hunting technology, no open-source project we analyzed had fewer software defects (per thousand lines of code) than the top-of-the-line closed-source application. That proprietary code, written for an aerospace company, is better than the best in open source—more than five times better, in fact. That company's software won't let you down when you're flying from New York to London.

Of the more than 150 open-source and proprietary software applications that we have analyzed in this study, closed-source software code grabbed 11 of the top 15 spots for the highest quality and security.

Surprised? We all know that a lot of software stinks. And most software is still closed source, hiding a lot of bugs. There is ample evidence, beyond our own bad personal experiences with blue screens of death or frozen cell phones. A frightening 2002 study by the National Institute of Standards and Technology (NIST)—the most recent comprehensive data available—estimated that software bugs cost the American economy $60 billion annually. It's probably safe to double that amount for a worldwide estimate.

So how come open-source software isn't saving the day?

BETTER ISN'T BEST. The answer is that it can—just not yet. The open-source development community must first graduate from Lake Wobegon University, where all of their software is just above average. They need to take lessons from the all-stars in closed-source software development. Likely, this means more thorough and rigorous end-to-end quality testing. Open source developers have a lot of pride in the quality of their individual contributions to code, but many proprietary organizations understand how to (and are required to) vouch for the high quality of the software system as a whole.

The irony is that our research shows that on average, open-source software is of higher quality than proprietary software. Indeed, open-source projects tend to clump together in the higher-quality range. Proprietary software applications scatter across the quality continuum, but the best ones tend to be considerably better than open source, and customers don't choose software based on industry averages.

The best of closed-source software is found in so-called mission-critical applications: things like jet engines, nuclear power plants, telephone systems, medical devices. These are things that simply can't fail, or people may die.

TESTING, TESTING. It's apparent from our research that the masters of proprietary software development know a few things—or simply utilize strict development practices—that could benefit the oftentimes brilliant (but perhaps less disciplined) open-source community.

This matters, since software is increasingly pervasive in business and government. And more and more of it is open source. As the lines of code pile up, applications become more complicated, more difficult to troubleshoot, and unfortunately, more likely to crash.

We see two trends in proprietary development that promise to make open-source software better, too. First, companies are paying a lot more attention to the lifecycle of their software. That means better end-to-end quality measurement that helps ensure the entire software system delivers on its promise. Second, there's a trend toward automated testing. As software gets more complicated, testing has to be done by machines as well as people. It's just too much for humans. The 2002 NIST study estimated that $22 billion of the cost of buggy software would vanish with better testing.

GOING MAINSTREAM? Because of the higher average quality of open-source software revealed by our research, we strongly believe it can cross the chasm into mainstream use. It offers too many advantages for both developers and consumers.

But in order for open-source software to become more prevalent in mission-critical applications, the open-source community must put more emphasis on industry best practices. We challenge this community to take a closer look at how the best proprietary software gets built and learn from that. Software quality and security are the most important factors in the choices that developers and companies make— not open-source vs. proprietary.