Innovators' Challenge - Normalizing the Clouds
The Old Lock-in Problem
Before VMware, there used to be three or four large block storage companies: IBM, EMC, HP, etc., and each produced their proprietary storage systems.
Each of these companies maintained large farms of compatibility testing to ensure that their storage systems worked with all kinds of different versions of client operating systems (like Solaris v1, v2, etc., Windows v1, v2, etc.) and their respective software drivers and hardware. This vast compatibility testing matrix used to be a substantial barrier for new innovative storage companies entering the market.
It was also nontrivial to transfer data from one vendor to another, making that the era of total data repositories lock-in.
The 2000s
VMware came along with the server virtualization software, and their observation was that most servers were IDLE, and possibly 2% busy on average.
Virtualization technology helped organizations to consolidated servers by 10 to 20x, and those were easy savings that CIOs could easily understand; additionally, VMware provided management software that made it easy to manage a collection of servers. 10x cost savings with easy infrastructure management was a swift game-changer for the industry.
The other important thing VMware did was to normalize all the prominent storage vendors. VMware abstracted away all the storage nuances of the hundred different client operating systems and their different versions. VMware provided a straightforward way to plug in storage systems to the virtualized workloads, and every storage vendor had to adapt to a VMware supported protocol, and that was enough to test.
This significantly reduced the barrier for new innovative storage companies to enter the market, but it also removed data repositories lock-in.
This change enabled a significant number of new storage companies to flourish and bring innovations to the market (companies like Nutanix, Datrium, etc.). It was a win-win for the industry and for customers.
Customers could now move from one storage system to another with the push of a button, and it was the end of the storage vendor lock-in.
The New Lock-in Problem -- deja vu
We are at a similar crossroads today with the three or four big public clouds. AWS, Azure, and GCP each offer different compute and storage services - there is no consistency across them, and there is no integration between them.
A customer using AWS cannot promptly move their business applications to another cloud vendor, and vice versa. Further, mobility across clouds requires some rewrite of applications. This is lock-in 2.0.
Enter VMware again with the recent announcement that they can run their VMware Cloud Platform (VMC) in all public clouds. This enables uniform compute management across clouds, providing customers with a consistent experience. This is a victory for customers who want to shutdown private data centers and leverage public clouds without having to rewrite their applications or leverage public clouds as a natural extension of their existing data centers.
This uniform compute service across clouds is excellent, but there are still significant challenges with regard to storage services.
The Shifting Of Dollars
For every $1 spent on VMware’s vSphere in the datacenters, there are $10 spent on compute servers, $7 on networking gear, and $5 on storage systems. There are about 100 million VMware virtual machines deployed across enterprises today, and VMware has an approximately $6B vSphere annual revenue with 82% market share. Further, about $30B is spent on storage systems.
As enterprise customers start adopting VMware Cloud on AWS, GCP, and Azure, all compute, networking and storage dollars will shift directly into the coffers of public cloud vendors.
Now you can understand the motivation of the public cloud vendors to attract VMware customers to run their applications on their cloud by offering VMware services on their platforms (even Oracle announced VMware on their cloud).
This is the most natural path for customers to adopt the public cloud without having to retool and rewrite applications.
The question then is, what happens to all the infrastructure vendors in the private data center space? With a change of this magnitude, there is a massive opportunity for infrastructure vendors able to adapt and thrive in this new cloud model.
The Storage Problem
It is easy to assume that public cloud vendors have many different types of storage services, and hence there is no room for traditional datacenter storage vendors to play in this space.
Beyond the lock-in, the real problem is that storage services in the cloud are not as enterprise-class as one would expect, and this is recognized by the cloud vendors themselves.
Let us take AWS as an example. AWS has S3 for cheap and deep storage with very high durability, but it has very restrictive I/O behavior with slow performance, and you cannot really use it to run your virtual machines directly off that. AWS also offers EBS, which is All-Flash with good performance but very expensive. Additionally, EBS has a high probability of data loss (based on AWS documentation). Finally, AWS Glacier that is exceptionally cheap, but behaves like a tape library in terms of performance.
Enterprises are used to getting high quality, high performance, and highly resilient storage systems for their data center. Further, AWS zones can go down, and customers need to promptly recover at a different zone, or at even a different cloud provider.
Finally, you cannot move to a different cloud if you are locked into a specific provider.
New Storage Paradigm – Enormous Opportunity
Enterprises demand a storage solution that provides services at the cost of AWS Glacier with the durability of AWS S3, but with the performance of AWS EBS. In addition, this new storage must be able to work across clouds to avoid lock-in and efficiently move across clouds. This data movement enables enterprises to prepare for a disaster when one cloud is down, or the business decides to change directions.
However, this is a nontrivial problem in storage system design.
Existing storage vendors will have a remarkably hard time to pivot their software to work in the cloud because storage system software designs are like database schemas, which are very hard to change once you have created the initial architecture. Hence, the cloud offers a totally new space to explore for new challengers, without the burden of competition from the legacy incumbents.
Legacy storage vendors have been fast to change their marketing materials to say “multi-cloud,” but what they mean is putting their legacy physical storage boxes in colocation facilities, with expensive network links to the clouds. Customers can see through this marketing spin.
Multicloud Compute & Data Planes
VMware now provides uniform compute services across clouds. Kubernetes also does that, and VMware realizes that hence, Project Pacific.
However, the missing piece is a uniform data plane that provides a consistent and resilient data experience across clouds. Snowflake (a new data warehousing vendor) is doing something similar for OLAP datasets, but organizations need similar services for VMware VMs & Container data.
Our team at Datrium is focused on solving this multi-cloud data plane problem and we have a real opportunity to help organizations to start bridging the multi-cloud chaos today.
– By Andre Leibovici & Sazzala Reddy