Defense, Intel communities wrestle with the promise and problems of 'big data'.

• Author: Magnuson, Stew

For some intelligence analysts, the glut of data coming from multiple sources is an overwhelming problem. There just aren't enough hours in the day to sift through all the potentially valuable information.

For others, this is a potential boon. Never before has there been so much data available. If they could somehow efficiently pick the best pieces of information, fuse them together with other sources, then they could put together a clearer picture of what is happening in the world.

This all falls under the latest industry buzzword, "big data." It is a term with different definitions, depending on who is describing it. But one that has the potential to revolutionize the way the defense and intelligence community collects and interprets information, experts told National Defense.

"Right now, this is like the Dark Ages," said James Canton, a futurist who consults with intelligence agencies, the military and the commercial sector.

"We have a long way to go in the I.C. [intelligence community] and defense community when it comes to modernizing--let alone thinking about the future of information technology--to be able to do what even the private sector can do," said Canton, who is CEO and chairman of the Institute for Global Futures in San Francisco.

The potential of big data is a world where computers automatically sort through several different information sources--sensors aboard unmanned aerial vehicles, social media like Twitter, shipping manifests, and so on, pick out the important data, piece them together in real time, and send them to an analyst's smartphone in an easy to comprehend report.

Canton specializes in what he calls, "the extreme future." This is a vision. It could happen, but there is a possibility that it won't. The main reasons that it may not come to pass wouldn't be surprising to anyone who follows the federal bureaucracy.

Databases are kept in so-called "silos," unconnected to each other, and operated by bureaucrats who don't want to share what they know. The systems are old, and the acquisition system to refresh outdated government IT is slow, Canton said.

"Unless we accelerate this big data strategy and bring it to the big table, it's not going to happen. You're going to have missed opportunities like we have had in the past," Canton said. He was reluctant to share what some of the "three-letter agencies" were currently doing because he currently acts as a consultant to them. Some like the Central Intelligence Agency and the National Reconnaissance Office are doing some good preliminary work, he said.

But none are as agile as the private sector, which has used big data techniques for years to understand its customers better, and to maximize profits. Companies are much more...