Jobs centered on data have been falling into Ana Bertran Ortiz’s lap since she finished her electrical engineering Ph.D. in 2007.
The jobs all come from her command of statistics, making her a beneficiary of the growing demand by U.S. employers for so-called data scientists who can analyze and manipulate the mountains of information generated and stored in the Internet age. Harvard Business Review last year called this profession “the sexiest job of the 21st century.”
One measure of demand: Hours billed for work in statistical analysis grew by 522 percent in the first quarter compared with the same period in 2011, according to data compiled for Bloomberg by oDesk Corp., which runs an online service connecting employers with remote freelancers. Time billed on oDesk for all categories of work in the same time span grew by 135 percent.
“In most areas of the modern economy, math and statistics have never been more important,” said Susan Athey, an economics professor at the Stanford Graduate School of Business near Palo Alto, California. “As firms get more and more data-driven, there become fewer and fewer careers that don’t require those skills.”
Bertran Ortiz got her start designing algorithms for a NASA lab to measure sea levels from outer space and orchestrate the landing of the Mars rover. She’s researched how to design flight paths to get more information from radar signals, and helped hone a mobile application that forecasts weather in 10-minute increments. She’s now working on software that automatically diagnoses glitches in the networks that house the world’s ever-expanding trove of information.
“I knew that in electrical engineering, it was very important to understand the randomness of data,” Bertran Ortiz said. “But I didn’t think it would become so important outside of my field.”
Unlike statisticians of a previous generation, data scientists work with information sets so big -- far too large and unwieldy to fit into an Excel spreadsheet -- that they need to write extensive computer code to extract the right segments.
Often, this data is on a scale that requires multiple servers to even access the numbers. After that, the analysts run calculations -- correlations, regressions, t-tests, machine learning algorithms -- to discover the patterns they’re looking for.
The scope of data collection is widening in the private and public sectors, a shift that was highlighted recently when the Guardian and Washington Post disclosed the existence of secret U.S. government programs that collect data on U.S. residents’ telephone calls and foreign nationals’ Internet activity. James R. Clapper, the director of national intelligence, subsequently confirmed the existence of the programs.
The national security industry is among the biggest employers of big-data professionals, according to an analysis from Burning Glass, a Boston-based job-matching company. One of the best-known companies specializing in big-data analysis is Palantir Technologies Inc., which made its name offering terrorism analysis software to the Pentagon and the Central Intelligence Agency.
Douglas Puett runs pattern analyses for Pulse, a news aggregator that was acquired by Mountain View, California-based LinkedIn Corp. (LNKD:US) in April. Every day, the 25-year-old peers into the mobile application’s logs to track figures, including which news outlets are keeping readers most engaged.
Armed with those numbers, Puett helps Pulse’s product team make decisions such as how to tweak the algorithms that suggest news stories. He also helps design, run and analyze experiments to make sure product changes lead to positive results.
He didn’t even know a job like this existed until he was halfway into his master’s program at Columbia University in New York in applied statistics, a degree which he completed in 2012. Today, data analysts are among the most commonly advertised positions in the technology sector, according to an analysis of online job postings from May this year by San Francisco-based Bright, a website that matches employers with workers.
“I was thinking I’d want to do research in sociology or political science,” said Puett, who said he chose to study history over math or physics for his undergraduate degree because he became “disengaged” with the abstractions of the quantitative fields. Now at Pulse, “the patterns I see will inform the decisions we make that will get into the product, and millions people are affected by it.”
The challenge for employers is that there aren’t enough Douglas Puetts out there, with the multiplying trove of information likely to further exacerbate the shortage of these analysts. By 2020, all the digital data created, replicated and consumed in a single year will grow to 40,000 exabytes, or 40 trillion gigabytes, according to a December study by technology research firm IDC. That’s a 300-fold increase from the 130 exabytes in 2005.
By 2018, the U.S. may face a shortage of as many as 190,000 people with deep analytical skills and 1.5 million managers and analysts who know how to use big data to make decisions, McKinsey Global Institute said in a report in 2011.
“It’s so cross-functional and you need multiple skills -- you need programming, you need statistics, you need visualization, you need database skills,” said Harpinder Singh Madan, co-founder and head of product and marketing at Slice, a Palo Alto-based startup that helps consumers track and analyze their e-mailed receipts. “The bottom line is that there’s no institution that trains for this.”
Businesses are improvising, pulling people from all kinds of backgrounds that require an understanding of statistics. They’re former nuclear physicists to neurosurgeons and marine biologists, many of whom hold doctorates in their previous domains.
The educational fields of freelancers who listed data analysis as a skill ranged from computer science -- the most common at 11 percent -- to business, math, economics, industrial engineering and psychology, figures from Redwood City, California-based oDesk show.
“There are a lot of people who have experience in statistics, and people need to look outside the traditional places” to hire more analysts, said Bertran Ortiz, the electrical engineering Ph.D. who earns more than $100,000 at Virtual Instruments, a San Jose-based company that monitors the health of data storage networks. “You need some theoretical background but you also need to be able to apply it” in “the real world,” she said.
Booming demand allows Aaron Merlob to now bill $100 an hour for his contracting side-job on oDesk’s online freelancing platform. The 27-year-old has a full-time position as director of data analytics at Activate Networks.
At the Newton, Massachusetts-based company, he determines which physicians are the most influential in their communities, crunching records pulled from medical and pharmaceutical claims to see which high-prescribing doctors are the most likely to interact with each other. When promoting their products, drug makers use that information to prioritize physicians.
“I definitely feel very well-positioned in today’s job market,” said Merlob, who in May 2009 received a combined bachelor’s and master’s in finance and a minor in statistics. “The big companies started this, but now the little companies are starting to think they need to do this too or they’ll fall behind.”
To contact the reporter on this story: Aki Ito in San Francisco at firstname.lastname@example.org
To contact the editor responsible for this story: Chris Wellisz at email@example.com