There are lots of reasons for the current boom (some would say bubble) in data, including cheap computing, sensors everywhere and lots of new algorithms.
To all of those, perhaps we should add the rising likelihood of failure, both the expensive kind and the cheap kind.
The expensive kind is when your business or employer gets wiped out. That is happening with greater frequency. According to Richard N. Foster of Yale University, the average tenure of a company in the Standard & Poor's 500 is now about 16 years, down from 60 years in 1959.
“The duration of your working life is now almost certainly greater than the lifespan of a company,†said Alistair Croll, an entrepreneur and author. “That makes everyone more willing to accept that they will be disrupted.†When people know there are reasons to think their business will be blown up by new market developments, he said, they're more likely to seek data that helps them respond.
Mr. Croll is the author of the book “Lean Analytics,†which is about how companies, and start-ups in particular, can better focus on change by working with lots of different data sources.
More than big computers or huge databases, diversity of information is at the heart of what is called big data. That term may be somewhat hyped, but there is no doubt that analysis of standardized information is becoming the norm in more of our lives, from personal medicine to real-time analytics of big industrial machines.
It is also cheaper to take risks and fail, both for start-ups and corporate divisions. Many costs that existed even a decade ago have fallen sharply. Computer hardware and software are now rented through cloud computing, social media is a proxy for much of marketing and a burgeoning number of business applications are sold cheaply in Google's Android and Apple's iOS stores.
“When things like that happen,†Mr. Croll says, “companies focus less on costs, and more on experimentation about what is going to make their original idea work. There is more desire to experiment.â€
Experimentation, of course, involves a lot of failure, as failure is where most learning takes place. Data around the failures of others are collected and studied as part of the overall process now. Data on failure is cheaper to create, and cheaper to come by. That is another way of saying that people are more likely to make new and interesting mistakes, instead of the same old ones, which is probably a good thing.
One big result of this failure-driven world, Mr. Croll says, is that organizational leadership is changing toward a more structured learning environment. “In the past, a leader was someone who could get you to do stuff in the absence of information,†he says. “Now it's the person who can ask the best question about what's going on, and find an answer.â€
