In the next few years enterprises will consume storage capacity faster than disk manufacturers can produce it, according to some forecasts. If something doesn't change that means the word is heading for a "storage gap": there won't be enough hard drives to accommodate all the data that enterprises want to store.
The reasons for the exploding demand for enterprise storage are well known and include the rise of Big Data implementations and the desire to "store everything" for later analysis. In the near future Internet of Things projects that generate vast quantities of data are also likely to require huge amounts of storage. In 2015 a total of about 4 zettabytes (about 4 billion terabytes) of data will be generated, but in five years that volume is likely to have grown ten-fold to about 40 zettabytes, according to an IDC study.
So in the short to medium term at least the storage market is likely to be disrupted, according to Gene Ruth, a research director at Gartner. "Growth in capacity production is no longer accelerating, and is actually slowing down," he confirms. "It is not clear that it will be possible for capacity to match requirements, so in the next few years there are the conditions for a shortage in drives to put data on. For the next few years there could be a limitation."
In the past, disk manufacturers have introduced new technologies that allowed them to make disks with capacities that increased in line with the demand for storage while the cost per gigabyte actually fell.
But the predicted growth in demand for storage is so high that the manufacturers can't keep up, according to Mark Whitby, a vice president at California-based disk vendor Seagate. "By 2020 companies will be producing and storing ten times more data than today but we are bound by the laws of physics. Innovation has allowed us to keep the economics (of hard drives) very attractive, but we can’t keep up with this ten times growth over the next five years," he says.
Rather than making a similar number of hard disk drives at similar prices with capacities that grow in line with the growth in demand, in the future companies like Seagate will have to make more drives, and these will have to be bigger, with more spinning platters in them, Whitby believes. And inevitably, the cost per gigabytes will rise.
That's unless and until manufacturers can invent new technologies to cram an order of magnitude more data onto drives and produce them at a price per gigabyte comparable to today's prices.
The problem is that neither of these two options – making more drives or making higher capacity drives with new technology – are things that can be done in the short term. "There is a two year lead time at least to bring a new manufacturing facility online," explains Whitby, "and the reality is that it isn't just one factory that's required. Different parts are made in different factories."
A new factory costs about $1.5 billion to get up and running, and that's a significant investment, he adds. "We have to decide when to make an investment, because the industry's history is one of over-capacity and then under-capacity," he says. "But we also have to decide in which area to invest: more disks, or innovations that cram more data on."
In many ways the hard drive market is very similar to the oil market. At various times over the last fifty years or so there have been warnings that oil would run out or that increasing demand would outstrip supply – leading to serious shortages.
But in every case these scenarios have been avoided thanks to the power of market forces. The price mechanism has come to the rescue, altering the behaviour of oil consumers and producers in such a way as to stimulate supply and reduce demand. The price of oil has gone up until demand has subsided (due to consumers switching to cheaper alternative energy sources or by adopting new technologies such as more fuel efficient automobile engines). Or supply has increased using previously uneconomic hydrocarbon extraction techniques, or completely new ones such as hydraulic fracturing, or fracking.
What we're likely to see in the storage market is something similar. The current period of low and falling hard drive storage costs may be about to end, replaced by a period of rising costs per gigabyte of data stored. The obvious questions then are how will this influence storage consumers' and producers' behaviours?
On the demand side, rising hard drive costs (per gigabyte stored) will make buying new hard drive storage capacity relatively less attractive compared to alternatives. So what are the alternatives?
One obvious alternative is flash storage, but this medium is far more expensive per gigabyte than hard drive storage.
However, enterprises do use flash storage when its higher cost can be justified by the performance benefits that it offers over hard drive based storage. This suggests that if the price differential decreases due to rising hard drive costs then at the margin it will make economic sense for some enterprises to buy flash storage for its performance benefits where before it didn't.
But flash storage usage is just a drop in the ocean compared to the amount of disk-based storage in use, so the effect of this substitution is likely to be negligible.
A far more likely scenario is that enterprises will look at the increased price differential between disk drives and low cost tape storage and decide that the drawbacks of tape – such as poorer performance – are worth swallowing in an increasing number of circumstances. "The available tape storage capacity is mind blowing, so we could see companies tiering off to tape, and accepting slower recovery speeds," says Mark Peters, a senior analyst at Enterprise Strategy Group.
He also suggests that existing capacity will have to be used more effectively. "We have become profligate with storage because it is cheap. We will have to find ways to be smarter."
For example, he suggests companies could make use of sliced storage systems that store encrypted chunks of their data on storage systems belonging to companies around the world that have excess, unused capacity.