It’s rare that you have exactly the amount of storage that you need at any one time. Growing enterprises frequently need to purchase more capacity, while storage providers can experience the opposite issue. Most data centers (and even individual businesses and consumers) have some amount of unused storage capacity, although virtualization and storage enhancements have dramatically improved efficiency. At present, this spare capacity is effectively wasted, but there could be an opportunity for organizations to sell – and buy – excess capacity on a storage marketplace.
This would work in effect like the electrical grid. Power companies buy and sell electricity to each other constantly to match supply and demand, both making up for shortfalls and monetizing their excess. While at first there were a few main power distribution centers, now there is a huge range of suppliers on a local and national level. Ultimately the end user has no idea where their power is coming from, just that their lights are on.
A marketplace for storage would operate under exactly the same principles: a community storage grid, with both storage providers and consumers, all underpinned by blockchain. Organizations with excess capacity could act as storage providers for those with local storage requirements. When these providers needed to reclaim this excess capacity, blockchain could enable a seamless transition for those customers’ data to the most appropriate location, based on rules like performance, policy and service level agreements. This would be a new, distributed model for storage.
Why not just use the cloud?
The idea of flexible storage certainly isn’t new, and many organizations are already using cloud providers to flex their storage up and down. Given the size and scale of the public cloud, you’d be forgiven for asking why you’d want to use the distributed model instead.
There are, however, a number of important limitations to the cloud, not least bandwidth and latency or in effect speed of service. While hyperscale public cloud is useful when bandwidth and latency aren’t important, these factors can be very limiting. And importantly, cloud service providers are certainly not immune to factors like power outages and malicious actors, which can severely impact businesses.
By contrast, under the distributed storage model bandwidth and latency are no longer an issue. Organizations will never run out of capacity, as they can unlock excess from the surrounding area; this means IT teams no longer need to purchase more storage than they need at any given point in time. Meanwhile, businesses that have excess storage can monetize it. This commoditization of disk space will enable businesses to operate much more efficiently, with a clear return on their storage investments.
How to build a marketplace in three steps
So, what will be needed to create the storage marketplace? It comes down to three essential elements, the first of which is distribution. This is perhaps the simplest issue to address. It’s possible to accomplish high performance distribution across a multi-node environment with peer-to-peer content distribution. This method of content delivery is already in use at organizations like Microsoft, where it lessens the stress on central distribution hubs. Distribution can be even further accelerated by chunking files or objects into standard or adaptive file sizes as appropriate.
The second requirement is of course a critical consideration for any business outsourcing their storage: security, including confidentiality, availability and integrity. Confidentiality can be achieved with known techniques for encryption and data protection, with effective key management ensuring that data components are protected from prying eyes.
Availability is equally important to ensure that data is not lost and remains accessible when either nodes go offline or storage providers reclaim their capacity. This can be achieved by ensuring that every piece of data is stored in multiple and redundant locations.
Integrity means preventing any interference with data as it’s distributed, stored and received. This can be enforced through strong user access controls, leveraging checksums and hash tables shared across the community storage grid.
The third and final requirement is of course the marketplace itself. For the distribution model to work, the marketplace must track every sale and purchase of capacity. Each transaction has to be immutable, so that all payments can be managed effectively and both the provider and consumer can benefit from the exchange.
The importance of blockchain
Blockchain will be fundamental to both the security of this distributed model and the operations of the marketplace itself. Blockchain technology can guarantee that every action is recorded as data is segmented and distributed across the grid. The chain will ensure that nothing occurs outside of the known ledger and create immutability, effectively eliminating any blind spots. Blockchain also supports improved availability, by ensuring that any storage reclaimed by a data reclaimer would trigger the copying of that data to elsewhere.
Finally, blockchain will provide proof that a contract exists between the owner of the data and the provider of the storage – and, indeed, that the data itself exists. By providing clear evidence of transactions, blockchain can eliminate the need for businesses to track exchanges manually, creating greater confidence and convenience in the whole system.
A new storage mindset
A marketplace for data storage seems like a natural progression, in a world that has already been partly commoditized by the advent of the cloud. However, adopting a new distributed model will clearly require a significant change in mindset for businesses. There has been a long-held assumption that data needs to be stored centrally to be trusted. Today data is businesses’ most valuable asset, making data privacy and security acute concerns. For that reason, it will take time to shift that view.
But there could be incredible opportunities for businesses willing to make the switch. Distributed storage can create faster networks, reduce latency and greater efficiency for any users that choose to participate. There’s even the opportunity to realize untapped sources of revenue. We aren’t far away from the marketplace model, as in reality all the technology needed exists today. Blockchain is presenting an exciting new storage paradigm; it will be fascinating to see which organizations will embrace it.