NYT : Security Through Dispersion

Brilliant new technique emerging to address a surging demand to store data online cheaply.  Excerpts from the New York Times article entitled “A Move To Secure Data By Scattering The Pieces” -

…..For companies and government agencies trying to secure networked data, Cleversafe offers a simple way to store digital documents and other files in slices that can be reassembled only by the computers that originally created the files.  The idea of distributed data storage is not new.  But Cleversafe is significant because it is an open-source project — that is, the technology will be freely licensed, enabling others to adopt the design to build commercial products.  That approach may contribute to Cleversafe’s potential to lower the cost of reliably storing data on the Internet.  Mr. Gladwin (founder) contends that Cleversafe can store data at a lower cost and make it more secure than current Internet services.  The group is counting on a continuing explosion of consumer digital data of all types, including new generations of high-definition still and video cameras that will create demand for secure and private backup capabilities.  Computer scientists argue that projects like Cleversafe are an indication that the broadband Internet will soon have the same impact on data storage that it has had on computing and communications technologies.  Dozens of commercial Web storage services are already used to back up data safely.  In addition, Amazon’s S3 and other services are intended to enable an array of digital Internet services to operate without any local storage capacity.  But the current design of such services generally involves making as many as five or more complete copies of the original data and storing them at multiple locations to ensure that information is not lost through a drive failure or other catastrophe.  The Cleversafe design will cut the amount of storage space needed for secure backup by more than half.  Mr. Gladwin, 42, said he was deeply influenced by a seminal paper, “How to Share a Secret,” written in 1979 by Adi Shamir, a designer of the encryption algorithm known as public-key cryptography.  The paper describes how a message can be broken into pieces and then reassembled from a subset of those pieces without revealing the message.  Mr. Gladwin developed a set of software routines that would copy the data stored on his PC into a large number of fragments, or slices.  The mathematics of his solution had an additional benefit: the original data could be reconstructed from a majority of the slices.  The design made it possible to retrieve a complete set of his original data even if some of the disks that held portions of the data failed or went offline.

The design of such “distributed file systems” is already a rich area of computer science research, and commercial systems are widely available in the software and data-storage markets.  But Mr. Gladwin argues that his new standard offers security and efficiency features not easily available either to information technology managers or to individual computer users.  The experimental Cleversafe research grid is located at 11 storage sites around the world, but Mr. Gladwin is hoping that a commercial network will evolve, composed of tens of thousands or hundreds of thousands of storage sites that will be accessible at low cost.  The Cleversafe design could lead to a communal Internet storage system that Mr. Patterson called “hippie storage.” The idea is similar to SETI@Home, the shared computing system that allows PC users to contribute idle time on their machines to create a distributed supercomputer.  Today most distributed storage systems work by making multiple copies of data at multiple locations and then using various mechanisms to keep the copies synchronized.  Examples include distributed file systems from Microsoft and Google as well as a system designed by software developers at Stanford known as Lockss — Lots of Copies Keep Stuff Safe — that is used to preserve the digital versions of academic journals.  The Cleversafe project uses a different approach based on dispersing data in encrypted slices rather than copying it.  That approach shares some design similarities with a Berkeley research project known as OceanStore, which is also intended to create a globally distributed computer storage system.  Data storage on the Internet is one of the most brutally competitive markets in the world. But nobody is using this architecture, and the logical benefits of this are remarkable…..

Reference : http://www.nytimes.com/2006/08/21/technology/21storage.html?_r=1&oref=login