Reliability vs. Validity


A couple of weeks ago in Detroit, I had the delight of sitting in the cockpit of a brand new Pontiac (GM) Solstice. What a beauty: a drop-dead gorgeous convertible roadster listing at a mere $19,995.

The Solstice is destined to be a hit. When my mind came back from the imaginary thrill of driving up to our cottage with this little baby's roof down, I realized that the Solstice brought into high relief a burning question concerning design: If you can have great design without a significant cost penalty, why do so few companies use design to win?

It has nothing to do with a trade-off between great design and cost effectiveness. Rather, it stems from the largely hidden trade-off that every CEO should address -- the one between reliability and validity.

The Trade-off between Reliability and Validity

Reliability is the result of a process that produces a consistent and predictable result over and over. To enhance the reliability of any process, one has to reduce the number of variables considered and use quantitative, bias-free measurement. IQ testing is an example of a highly reliable process with these characteristics. If you take the Stanford-Binet IQ test over and over, you will score a nearly identical result each time.

The test achieves reliability by defining intelligence very narrowly, as the ability to solve simple analytical problems. It measures intelligence via a multiple-choice test that can be evaluated with no possibility of bias in measurement or judgment.

The problem is that IQ doesn't serve as a particularly great predictor of anything. In his best-selling book Emotional Intelligence, Daniel Goleman argues that EQ makes for a much better predictor of success in life than IQ. In essence, he argues that while IQ may demonstrate high reliability, it has modest validity.

To increase the validity of any process, one must consider a wide array of relevant variables. Goleman's EQ, for instance, builds on more qualitative considerations and judgment to produce what he argues has higher validity.

Of course, we would like a process that has both high validity and reliability. Up to a point, it's possible to get more of both, simply by being more thoughtful and less sloppy. But ultimately, more reliability requires fewer variables and therefore less validity, and vice versa. Reliability and validity seem to conflict.

Challenge for Design

How does this trade-off relate to design? And why should it matter to top execs? Design posseses an inherent bias toward validity. Great designers seek deep understanding of the user and the context, which entails consideration of many variables.

They don't limit their considerations to aspects that can be thoroughly quantified. They worry less about whether they can replicate a particular process -- and more about producing a valid solution to the problem before them.

Entrepreneurs -- being essentially designers of business models -- have a similar approach. They start out with new-to-the-world ideas they believe in but typically can't prove. They value judgment, experience, and gut instinct -- in this way, they're highly validity-oriented.

But as a successful entrepreneurial venture grows, it acquires outside investors and a board of directors, and it begins listing toward reliability. With ever bigger bucks at stake and more scrutiny of investment decisions, the growing corporation increasingly values processes that are quantitative, analytical, and bias-free.

The successful entrepreneurial venture wants "proof" -- facts from the past that can demonstrate that something will be true for the future. Variables difficult to measure quantitatively -- feelings, relationships -- get dropped. And the narrow set of variables creates the "proof" of the sort that brought the world the disastrous Aztec, ironically from the same Pontiac division that now brings us the Solstice.

It's not that corporations don't like or want great design. It's just that when a validity-oriented design comes to an important corporate decision gate, the reliability-oriented question inevitably gets asked: "But can we prove this will work?" Or, "How can we be sure this will work?"

Typically the answer is no, it can't be proven, and we can't be sure. Nobody could prove before Herman Miller launched the Aeron chair that it would succeed at all let alone become the most successful office chair of all time. And so design often gets undermined or subdued or killed without explicit intent, a victim of the corporate bias toward reliability.

Safeguarding Validity

If a corporation wants to enjoy the benefits of design in its products, services, processes, or business models, it must go considerably beyond simply hiring designers or declaring itself design-oriented. The CEO must take responsibility for safeguarding validity. If the CEO doesn't, the corporation's natural inclination toward reliability will win out.

So the CEO's behavior is crucial. The questions she asks, the proof she demands, the way she treats failure will send signals that design thinking is safe or not. If the questions are all about the most readily quantifiable numbers, and if the standard of proof is high and numerically driven, and if failure is treated as indicative of incompetence, the organization will understand that its leader values reliability over validity. And it will give her reliability to the best of its ability.

If, however, her questions tease out the trickier qualitative aspects of a decision along with the hard numbers, and if she utilizes a balanced standard of proof that takes into account the complexity of the issue at hand, and if she treats failure as an unfortunate consequence of living in a risk-filled world, then she'll signal that she balances the need for reliability with the desire for validity. And she'll get design thinking.

The marketplace tends to discipline corporations so biased toward reliability that they produce mediocre products or services -- buyers choose not to buy the proverbial Aztec. This keeps reliability and validity at least reasonably in balance.

However, certain corporate divisions -- including powerful ones like finance -- are more insulated from direct market pressures and can more easily slide into deep reliability. Strict numerical proof is required before anything can happen. Finance provides the templates for analysis, finance sets the burden of proof, and anything that can't be strictly quantified is unnecessarily risky.

Every CEO needs a strong finance function -- and human resources, product development, legal, etc. -- but he needs to understand that he can't let finance or any other division run roughshod over validity, or he'll unknowingly drive design thinking completely out of his corporation. That's why an additional task for the CEO is to act as the CVO -- chief validity officer -- in order to protect and nurture a design culture.

READER COMMENTS


Burger King's Young Buns
LIMITED-TIME OFFER SUBSCRIBE NOW

(enter your email)
(enter up to 5 email addresses, separated by commas)

Max 250 characters

 
blog comments powered by Disqus