The digital tsunami: a perspective on data storage: to meet demands, organizations will need to increase today's storage offerings 10 times. But how will such dramatic increases be addressed by technology and systems in the next few years?

• Author: Straub, Joe

At the Core

This article

* examines current issues and future possibilities in data storage

* compares alternative technologies for data storage

* identifies existing and emerging technologies for data storage

The rapid emergence of digital video is just one indicator of the growing markets for storage systems with much lower costs and much larger capacities. Other major demands come from the standard business modal for corporate data storage" and the growth of storage' service providers (SSPs) that are accumulating data from small business and consumer markets.

According to "How Much Information," a study by the University of California Berkeley School of Information Management and Systems, to meet escalating requirements, each of these markets will need to improve today's storage offerings by 10 times. By 2010, a 100-fold increase will likely be necessary, the study shows. As records and information managers must deal with increasingly larger volumes of records, keeping up with storage media trends is becoming more critical.

According to International Data Corp., in fiscal year 2003, Fortune 500 companies spent $7 billion on approximately 1,200 terabytes (1 terabyte = 1,000 gigabytes) of data stored on magnetic tape. Additionally, Jim Porter of Disk/Trend Inc. reports that companies have spent $40 billion to store 1,400 million terabytes of data on magnetic disk. Even though storage requirements have now started to increase at more than 100 percent annually, technology is improving in cost/performance at only 35 percent each year.

This creates a significant gap in terms of the cost/performance of storage systems required to meet the growing need. To allow that data to be stored without causing large cost increases, data storage must move from relatively expensive disks to lower-cost media. In addition, the cost/performance of these lower-cost media must continue to improve at an annual rate of nearly 100 percent to meet the continuing data growth.

This growth in data can be felt in personal terms also. In the eddy 1990s analysts were foretelling that each Fortune 500 company would have a terabyte under management in a few years. In 2000, experts such as Patti Tobin predicted that in 2003, thanks to the advent of digital movies on DVD, commercial music collections on CD-ROMs, and personal music or digital photographs stored on CD-Rs or CD-RWs, each computer-savvy home would have the equivalent of a terabyte of data under its roof. By 2006, as wireless and video technologies become ubiquitous, we should reasonably expect that individuals will move from having hundreds of gigabytes to a terabyte of personal storage on their desktop or laptop. The U.S. government--often a "canary in the mine" for new technology--has several data storage systems under development that will provide about 80 petabytes (80,000 terabytes) each.

This explosive growth of data, called the "Digital Tsunami" by Gary Ashton in his 1996 National Media Laboratory Report "Future Trends in Storage, Interconnectivity, and Data transfer," is fundamentally driven by the conversion of analog systems to fully digital systems (e.g., film cameras to digital cameras) and a leveling of computer resources available (e.g., each home computer is now equivalent in performance to early supercomputers).

But how will these dramatic increases ill data storage be addressed by technology and systems over the next several years?

Comparing the Media

The means of storing computer data has changed little over the past few decades and still consists of only about half a dozen different types of media and formats. These include silicon based dynamic and static random access memory (RAM), magnetic hard disk drives, optical disks (including write once read many [WORM], magneto/optic, CD-ROM, and DVD disk, and magnetic tape.

In any specific storage application, the preferred medium is selected because of a particular characteristic that is superior to that offered by the other options. RAM offers high data-transfer rates and high-speed access even though it is the most expensive. Hard-disk drives provide high-capacity (greater than 100 gigabytes) and moderate access times in an affordable package but, like RAM, are neither removable nor archival, even though in recent years the swap-out capability of drives and media has lead to a change in thinking about the removability of hard disk systems. CD-ROM provides a removable disk media of limited capacity (650 megabytes) and the DVD provides 5 gigabytes. Emerging formats of the WORM optical disk from Sony and Plasmon offer 30 gigabytes of archival storage in a 5.25-inch format.

The data-sharing function of the floppy is now facing obsolescence due to its slow data rate and low capacity. It is being replaced in sophisticated installations by portable universal...