Big Data: Rise of the Machines

For a column that laid out some second thoughts on Big Data, one of the people I talked to was Thomas H. Davenport, who has worked in the fields of knowledge management and analytics for 15 years. Data analytics is the predecessor to Big Data. He knows the context - what's new and what's not with Big Data - as well as anyone.

Mr. Davenport, a visiting professor at the Harvard Business School (on leave from Babson College), has authored and co-authored several books on analytics, including “Competing on Analytics: The New Science of Winning” (with Jeanne G. Harris, Harvard Business School Press, 2007). Shortly after the Big Data phenomenon took off, Mr. Davenport said, only half-joking, that he considered simply substituting the term “Big Data” for “analytics” for updated versions of his books.

But as he looked more deeply, there really was something different in Big Data. Data volumes have been steadily increasing for decades, Mr. Davenport noted, though the pace has accelerated sharply in the Internet age. “More than the amount of data itself, the unstructured data from the Web and sensors is a much more salient feature of what is being called Big Data,” he said.

I also asked David B. Yoffie, a technology and competitive strategy expert at Harvard, who is not part of the Big Data crowd, what he thought. The Internet, he observed, has been a mainstream technology for 15 years, and so has the ability to monitor and mine Web browsing behavior and online communications, even if those skills are much improved now.

Still, Mr. Yoffie is most impressed by the rapid spread of low-cost sensors that make it possible to monitor all kinds of physical objects, from fruit shipments (sniffing for signs of spoilage) to jet engines (tracking wear to predict when maintenance is needed).

“The ubiquity of sensors is new,” Mr. Yoffie said. “The sensors make it possible to get data we never had before.”

Machine-generated sensor data will be become a far larger portion of the Big Data world, according to a recent report by IDC. The research report, “The Digital Universe in 2020,” published in December, traces data trends from 2005-20. One of its forecasts is that machine-generated data will increase to 42 percent of all data by 2020, up from 11 percent in 2005.

“It's all those sensors, the Internet of Things data,” said Jeremy Burton, an executive vice president at EMC, which sponsored the IDC report.

The implication is that Big Data technology will steadily move beyond the consumer Internet. Industrial companies like General Electric are a lready making big bets on the payoff. The IDC forecast also suggests that there is a lot of substance to the vision of machine-to-machine communication and intelligence that W. Brian Arthur terms “the second economy.”