Go To Businessweek.com

BW Mall - Sponsored Links

Buy a link now!

text size: T T GigaOm November 09, 2011, 3:06 PM EST

Big Data: Science's Microscope of the 21st Century

Coping with future avalanches of data in terabytes and petabytes will lead to advances in astronomy and physics

By

Johns Hopkins is taking a $1.2 million grant from the National Science Foundation to build a 100 gigabit-per-second network to shuttle data from the campus to other large computing centers at national labs and even to Google. The network will be capable of transferring an amount of data equivalent to 80 million file cabinets filled with text each day.

The head of the project, Dr. Alex Szalay, details the plans, which include networking gear from Cisco, Arista, and Solarflare; Nvidia GPUs; and 66,000 x86 cores. That’s on top of the actual fiber that will connect a new, 1 megawatt data center inside the physics building to the regional Mid-Atlantic Crossroads research and engineering network at the University of Maryland.

That connection will be the 100 Gbps element funded by the NSF, and the Mid-Atlantic Crossroads network connects to Pittsburgh and on to Chicago via other 100 Gbps networks that are growing in number across the country. Inside the campus, Szalay, who is the alumni centennial chairman in physics and astronomy at Johns Hopkins, is setting up a 40 Gbps network between buildings that deals with lots of data, such as the medical and computer science hubs. “To keep looking at big data sets, we have to move the big data to a location where we can analyze it, and the stumbling block is [data sets of more than] 100 terabytes because of the speed of the network,” Szalay says.

He ascribes this massive amount of data to the emergence of cheap computing, better imaging, and more information and calls it a new way of doing science. “In every area of science we are generating a petabyte of data, and unless we have the equivalent of the 21st century microscope, with faster networks and the corresponding computing, we are stuck,” Szalay says.

SPARKING OTHER ADVANCES

In his mind, the new way of using massive processing power to filter through petabytes of data is an entirely new type of computing that will lead to advances in astronomy and physics, much as how the microscope’s creation in the 17th century led to advances in biology and chemistry. In that light, the creation of a 100 gigabit-per-second research network at Johns Hopkins becomes not just a fast network but also an essential tool for research and discovery, a basic component of the 21st century microscope.

For example, he described trying to send a 150 terabyte chunk of astronomy data for analysis to Oak Ridge National Lab in Tennessee as “painful” because of the limits the 10 gigabit connection presents between the university and the national lab. When he looks ahead 10 years and anticipates a colleague’s next-generation astronomy project currently underway that Google is supporting with 14 million compute hours, he believes it could generate 100 petabytes of data.

If that kind of data avalanche is a mere decade away, it appears our faster networks can’t come soon enough.

Also from GigaOM:

Archiving Our Cultural Heritage in the Cloud (subscription required)

Coming to America: China Telecom Launches U.S. Service

Fisker Won’t Ramp Up Volume of Karmas until Q2 2012

Amazon’s Oregon Cloud Region Open for Business

America! Stop Whining and Get Online!

Provided by GigaOm

READER DISCUSSION