Intelligence Agencies and the Data Deluge

Political leaders including President Obama and Senator Dianne Feinstein in recent days have defended snooping on private communications by noting that intelligence agencies were not reading personal messages, but rather information about the messages. “This is just metadata,” Senator Feinstein said at a news conference. “There is no content involved.”

In fact, say researchers in the field of data analysis, the metadata, or the information about such things as where a message came from and when it was sent, is frequently more valuable to security officials than the content of the messages. It provides dense and useful information to agencies increasingly swamped by the global computing onslaught.

A study published in Nature last March demonstrated that just four data points about the location and time of a mobile phone call made it possible to identify the sender 95 percent of the time. Using just two randomly chosen points, it was possible to identify half of users. Such information can come not just from phone companies, but also from smartphone applications or Wi-Fi hot spots.

Metadata is also useful because national security is now dealing with a deluge of information. Researchers at IDC have determined that the amount of global digital information, like e-mail, Twitter posts and digital photos, has risen from about 500 billion gigabytes in 2008 to almost four trillion gigabytes this year. By 2015, they estimate, there will be eight trillion gigabytes of material to go through, much of it from fast-growing countries with young populations, like China and Indonesia.

Unlike e-mails written in different languages or with personal touches, metadata about who sent and received a message, when it was sent and from where, always looks the same. Besides cutting down on the absolute amount of traffic to examine, metadata makes it easy to organize information and search for patterns, establishing social networks from individuals.

For some communications, metadata matters more than content. “A call to a suicide hot line, Alcoholics Anonymous, or a gay sex chat room at 2 a.m. are all more sensitive” than the actual message, said Christopher Soghoian, principal technologist at the American Civil Liberties Union. “You can text political donations. The metadata shows your political leanings, the content just shows the amount you gave. Calling a cell tower away from my house in the middle of the night indicates I’m not sleeping at home.”

“Metadata is the least protected form of communications information, and that is a shame,” he said. “You just have to say it’s important to an ongoing investigation.”

In the past decade, commercial cloud computing has grown powerful faster than the rate at which highly engineered supercomputers have improved, creating more problems for government spies. “They have better stuff than what you can get commercially, but they’re not five or six generations ahead anymore,” said Jonathan Schwartz, the former head of Sun Microsystems, which sold the government some of its most advanced computers. “You can look at where Amazon or Google will be in two years, and get a pretty good idea of where they are now.”

To cope with its challenges, the government has put on a more open public face, while privately making even more intense efforts. In 1999 the C.I.A. created In-Q-Tel, an independent nonprofit venture capital arm, which it uses to get a window on early-stage technologies. Contests like Virginia’s Cyber Challenge are modeled on talent-spotting contests that the People’s Liberation Army of China uses to spot young hackers.

Even the secretive National Security Agency has tips on improving K-12 education for future code crackers, and offers high school students internships through its Web site. The agency also speaks at open-source conferences, as seen in this unusual video.

The N.S.A. maintains a museum in Fort Meade, Md., that shows off past supercomputers and secure smartphones, as well as older machines and storied tales, like the coded messages hobos left for one another. It is a popular stop for tech executives after sales calls.

Privately, the N.S.A. has played hard with Internet technology companies, seeking secret product modifications to make it easier to spy. “The pitch they give you is not subtle,” said a former executive who was involved in these discussions. “They tell you that somewhere there is an American who is going to be blown up; the only thing that stands between that and him living is you.”

The executive, who requested anonymity to preserve his professional relationships, said his company debated cooperating. “The first question we asked was how big a customer they were,” he said. “Then, what it would do to our foreign sales if it got found out. Then civil liberties.” When he balked, he said, N.S.A. agents “told me they were doing ‘a standard counterterrorism assessment’ on me.”

His company declined to help, he said, partly because it was convinced it would not matter. “We had former N.S.A. people working for us. They told us: ‘You don’t understand. They have so many resources that they can just get what they want. They’re only asking you to make it easier for them.’”