Data-mining initiatives.

• Author: Swartz, Nikki
• Position: Up front: news, trends & analysis

To harness the vast information flow generated each day, computer scientists are developing sophisticated software that can instantly mine streaming data without ever needing to archive it.

According to The Christian Science Monitor, specialists worldwide are working to enable users quickly to navigate and retrieve information nuggets from vast storehouses of data that are growing at unprecedented rates by developing new tools to "mine" digital databases and the streams of information that feed them.

Data storage is growing by leaps and bounds, experts say. For example, each day the U.S. intelligence community collects information equal to all the printed pages in the U.S. Library of Congress. The World Wide Web is growing by more than 1.5 million Web pages daily, taxing the ability of the current generation of search engines to track down the answer to a user's query quickly and accurately. Overall, the average size of a database and the software needed to use it are growing faster than computer-processing speeds, which double about once every 18 months.

Meanwhile, storage capacity has grown even as the cost has plummeted. Four years ago, a credit-card-size hard-disk storage device in a typical consumer laptop computer might have held 6 gigabytes of data--enough space to store six conventional movies. Today, laptops have hard drives capable of holding from 40 to 60 gigabytes and more.

Such trends are prompting researchers to explore more sophisticated data-mining technologies. In addition, recent U.S. government initiatives such as the Homeland Security Act may require an ability to search existing financial, criminal, immigration, or other fixed databases, as well as to monitor streaming video and audio sources for evidence of...