Transformational 'big date' tools.

Director of National Intelligence James R. Clapper has asked the government's tech gurus and the private sector to "help us find the needles without having the haystacks."

Clapper's clarion call comes at a time of unprecedented demand for data-intensive products and services at all levels of the U.S. national security apparatus. The task of filtering and sorting through massive loads of data is only going to get bigger as the military and intelligence agencies collect more information than they can handle. There are more drones and satellites collecting video and imagery than ever before, and human analysts desperately need automated tools to find those needles in ever-expanding haystacks.

"Our next big investment is big data," says Dawn Meyerriedcs, deputy director of the CIA's directorate of science and technology. The challenge for data scientists is "figuring out how we deal with high volume intelligence."

Government agencies find that software tools that can parse huge loads of information into actionable information are becoming increasingly more sophisticated, but there are still many gaps to be filled.

As the United States steps up the fight against elusive extremist groups, the traditional methods of finding and tracking targets are inadequate. The amount of data being collected has made it nearly impossible to track and identify suspicious activities and potential security threats solely through human analytical processes.

The intelligence community sees its future in "activity based intelligence," which is computer-assisted problem solving to help understand how enemy networks operate by following their movements and financial transactions.

The government's gargantuan appetite for data has spurred an arms race within the tech industry. Much of the innovation these days comes from Silicon Valley, where there is a burgeoning crop of firms that are jumping in to fill big data needs.

"When the agencies first saw our software, they didn't know software could do what our software did," says Sean Varah, CEO of MotionDSP. The company's image processing software initially was created to clean up grainy cell phone videos from the pre-iF'hone days. U.S. military and intelligence analysts now use it in the war against the Islamic State. Agencies have rooms full of people who manually, frame by frame, clean up images that may be hard to see, or are clouded by bad weather or smoke. That typically takes weeks, says Varah, whereas the software...