Last night Netflix hosted a gathering at its spa-like headquarters for 200 or so members of the Clouderati—the engineers building out the computing infrastructure on which businesses will someday run. Netflix (NFLX) has a reputation for pushing the limits of cloud computing, running much of its movie streaming business on Amazon.com’s cloud rental system. And the engineers from around Silicon Valley came to hear about the latest and greatest tools Netflix’s technical staff had concocted.
When it comes to infrastructure technology, we live in absurd times. Netflix—like Facebook (FB), LinkedIn (LNKD), Twitter, Yahoo (YHOO), and other Web celebs—open sources much of the software that underlies its operations. Put more bluntly, they fight for the right to heap money on the smartest engineers, then give away their work so that others can build on top of it. Together, all these companies are forging the cutting-edge cloud computing technology that mainstream companies will use in the years to come.
At its event, Netflix looked to turbocharge the process. The company announced $100,000 in prizes—$10,000 for 10 different awards—for volunteer coders who can develop interesting tools based on Netflix’s open-source code over the next six months. (Rules here.) The revelation of these prizes was met with great applause. Now the race is on for people who don’t work at Netflix to improve the company’s infrastructure.
By rights, none of this should be happening. Great computing giants, such as IBM (IBM), Sun Microsystems, and Hewlett-Packard (HPQ), were supposed to be setting the course for the future of computing. Just a few years ago, executives from these companies could hold an event and make some broad proclamation about a new technology wave—utility computing, service-oriented architectures, automated management—and people would take them seriously and get to work. These companies, though, moved too slow and thought too small. They had an inkling about what the next generation of computing would look like but couldn’t build it quite right or on time.
The result is that a retailer (Amazon (AMZN)), an entertainment company (Netflix), an advertiser (Google (GOOG)), a résumé site (LinkedIn), and an address book (Facebook) have ended up shaping the future of infrastructure. This is not to say that every mainstream business down the road will need the tools to operate a massive website. It’s more that Netflix, for example, is pounding on Amazon.com’s cloud computing service to figure out where its weak spots are and what technology needs to be improved. Or it’s LinkedIn developing automated tools for checking the health of software code. All this work makes sturdy cloud computing services for the masses possible at a later date.
Giving away its technical smarts for free and paying volunteer coders makes perfect sense for Netflix, says Adrian Cockcroft, a former Sun Microsystems engineer who is now the director of cloud computing architecture at Netflix. “We end up with better-quality code,” he says. “And, when it comes to hiring people, we can see people who already know our code base and are doing interesting things.”
Netflix has built out a great arsenal of open-source cloud computing technology. It has tools for installing preset packages of software on Amazon.com’s servers, reconfiguring them, and testing them on the fly. It has other applications that purposefully try to wreak havoc on its infrastructure in a bid to find holes and performance problems. And it has yet more code that pushes the latest and greatest database technology to the limits. Organizations as varied as the Obama administration and Intel (INTC) have grabbed these packages for use in their own operations.
Eventually, Netflix wants to open source an entire “platform.” Instead of releasing cool but disparate projects, the company wants to put together an entire cloud management system that software developers can poke and prod and advance. “What we are hoping is that people start feeding off each other,” says Cockcroft. “We are all building code in public here.”