Executives Embrace Experimentation

Finally, senior management is conquering its fear of going with the gut over the data, according to H. James Wilson and Kevin Desouza

By H. James Wilson and Kevin Desouza

Posted on Harvard Business Review: December 27, 2010 10:00 AM

While he was at Amazon from 1997-2002, Greg Linden prototyped a system for making personal recommendations for other products to customers as they checked out.

Problem was, a marketing Senior Vice President thought these would distract customers from completing their purchase and didn't think Linden had proof it would work. A business drama ensued; classic "gut versus data" stuff. "I heard the SVP was angry when he discovered I was pushing out a test," recalled Linden in his blog. "But, even for top executives, it was hard to block a test. Measurement is good. The only good argument against testing would be that the negative impact might be so severe that Amazon couldn't afford it. A difficult claim to make."

Linden's experiment ultimately showed that the customers really liked the feature, and recommendations have become a signature design feature for Amazon.

These days, the aspiring Greg Lindens of the world might expect a bit less resistance from business leaders when it comes to experimentation, but they may be surprised that it's only now, in 2010, that a majority report experimentation is the preferred approach. In our recent sample of 149 senior executives in the Babson Executive Education survey of global respondents, December 2010, 51% say that experimentation is now their organization's preferred approach to understanding and acting on potential opportunity. Interestingly, for the subset of executives leading the highest growth firms (more than 20% revenue growth last year), the percentage encouraging experimentation is 10 % higher than that (61%). The practice of experimentation is widespread, and seems to be linked to higher revenue performance as well, our data suggest.

Would-be experimenters have another advantage nowadays too: it's becoming easier. Experimentation tools are coming out of R&D labs into the mainstream, and can be made available for anyone's workspace or department. Below we describe three of the most accessible types of experimentation we have noted in our research:

Prototyping involves the process of trial-and-error, where numerous iterations of an idea are created and tested to see whether it fits the environment and provides the desired results. Prototyping used to be complicated, but now can be done inexpensively by using rapid prototyping methods (RP) to quickly convert computer graphics (CG) designs into 3-D prototypes, for example, or employing low-cost tools like Google's Sketchup.

As a result, hands-on modeling has been pushed up earlier in the product development cycle. For example, "Sony Ericsson has a policy where they don't want their designers to work more than a few hours before they build their very first physical model," said one respondent. The basic steps for prototyping are:

1. create a concept/prototype
2. document it
3. experiment with it, generate and analyze results
4. refine the concept or the experiment, if needed, and
5. repeat until satisfactory outcomes are reached.

Simulation of a product or service is another form of experimentation that has come a long way in the past few decades, resulting in more efficient team decision making and better productivity.

Business Exchange

Track and share business topics across the Web.