VDI and Storage De-Duplication: Good or Bad marriage?

In my opinion if you are doing VDI with Persistent Desktops using Full Clones you are rowing against the current.

I have previously stated that Floating Pools and Linked Clones (or similar technology) is the way I believe every VDI deployment should be done (either using VMware View, XenDesktop or any other broker). Persistent Desktops should be treated as exceptions to address specific user or application needs. You will find reasoning in my article Floating Pools are the way to go…

The idea of data de-duplication to provide storage savings does not make sense for VDI. The real cost of a VDI solution is on the number of IO operations you can get per $ spent, not on the usable disk space you get per $.


There is need to divide VDI in two large buckets: Full Clones and Linked Clones.


De-dupe with Linked Clones

Linked Cloning technology utilizes a single image to provide a unique system disk to all virtual machines running in a View cluster (if you are interested in a deep dive on how Linked Cloning works please read VMware View 4.5 Linked Cloning explained). This single image called Replica will be eventually serving hundreds of clone VMs and this data will often be served from your fastest pool of disks, most likely SSD. These Replicas have a very small storage footprint – often around few GB. De-dupe is not useful for this little amount of data that will be kept as hot blocks for 100% of the time.



In View 4.5 VMware introduced the concept of Disposable disks. These are .vmdk files created to host Windows temporary files such as log, Internet cache and Windows swap file. Each individual VM will have data inherent only to that particular VM and not necessarily common blocks to be de-duped. If disposable disks are used then a de-dupe solution that is not inline (I am not aware of any storage vendor providing inline de-dupe for primary storage) will not provide any benefit. Either way, just to close this thought – disposable disks are deleted with every VM power off.

It is also possible to have Persistent VMs using Linked Clones. This scenario gives you great operational flexibility, allowing use of Refresh and Recompose operations. For this end it is common to make use of Persistent Disks. Those are disks created to host user’s personal settings and sometimes user data, such as My Documents. These disks are better served from a NAS instead of primary storage array as they do not require the same performance. If there is an opportunity for de-dupe this is the place, however If you offload the data to the NAS the persistent disk will only be couple MB, hosting user and computer registries.


De-dupe with Full Clones

Full Clones are “full fat” VMs that will persist across sessions. Full Clones can be created and managed by the connection manager (eq. VMware View Manager) or created and managed by an external entity. If VMs are created by an external entity they are then added to a Manual Pool trough available APIs and/or CLI’s.

The first point about full clones that I would like to create awareness is that Full Clones do not offer a supported and easy way to provide DR. If you are interested in a deeper read on DR for VDI you may go on and read my article VMware View Disaster Recovery Scenarios & Options.

Full Clones are the only pool mode that would make use of de-duplication since all VMs will have exactly the same data in storage. However, most of the block communalities exist only up to the moment the user starts to use the VM. When the VMs are in use they will write their own pertinent data, logging files and Windows Page Files. There are block communalities; however as memory starts to swap to disk this communality will be much lesser.

In my opinion, it doesn’t make sense to have a de-dupe engine running in the background or scheduled to run out of business hours when it’s more important and expensive to buy IOPs than usable disk space for VDI solutions.

One of my customers had offline de-duplication scheduled to run overnight and the task was running into business hours affecting user’s ability to perform their work.

Either way you know my take, Floating Pools are the way to go….


Reduce your storage footprint

I am not saying that reducing storage footprint is not important, but there are other ways to achieve that without having to sacrifice your VDI infrastructure.

Increasing VM memory reservation is one of the easiest ways to reduce storage footprint. As an example, a VM with 2GB RAM will often utilize 2GB of disk space for the VM swap file. Setting the Parent VM with a 50% memory reservation in vSphere will make the storage footprint drop to 1GB. This little teak could help you save 1 terabyte of storage in a VDI solution with 1,000 desktops.

Another powerful option is the Windows profiling and customization. I recommend reading Mastering VDI Templates updated for Windows7 and PCoIP and VDI Base Image: The Missing Step for additional information on how to profile your guest OS.


My personal take is that

  • De-dupe does not offer practical nor operational benefits when using Floating or Persistent Pools with Linked Clones.
  • De-dupe may be beneficial in environments that use Persistent Full Clones 100%, however Full Clones and Manual Pools are not the way to move forward with your VDI strategy, specially in DR situations.
  • $ per IO is far more important than $ per GB for VDI deployments.
  • Finally, I believe on VMware Linked Cloning and Citrix MCS technologies.


2 pings

Skip to comment form

  1. Andre,
    looks like you’re enjoying your new job, and the perspectives that its bringing. You article is well constructed and interesting, but in my opinion your argument is flawed when you consider how deduplication is implemented on a NetApp array (which is pretty much the only game in town with deduplication for VDI). The flaw comes because you ignore the read IOPS benefit you get from vastly better cache hits in deduplicated environments. Even without flashcache, the benefits are impressive as the very large percentage of a 4GB O/S image will pretty much sit in cache of even the smallest FAS array during bootstorms regardless of how many clones you see.

    Secondly, by deduplicating the image you make more freespace available in the netapp aggregate which makes the write allocators job easier which speeds up writes. (check my blog posts again if you need more details on how that works)

    Lastly, with View linked clones a lot of the write activity is partial writes because of the way differencing disks work. Partial writes are evil to most arrays because of the read/modify/write cycle they impose, whereas this pretty much doesn’t happen at all with VDI images created with NetApp’s rapid cloning utilities (which are based on the same WAFL block sharing capabilities as our deduplication technology).

    Overall from a performance perspective theory tells us that in systems that follow best practice, deduplication actually allows better IOPS/spindle than non deduplicated environments, and not unusually for NetApp, this is exactly what our customers see.

    This isn’t to say that linked clones and MCS technologies don’t have a management edge over deduplicated persistent desktops, they absolutely do, and for this reason I strongly reccomend using non persistent desktops wherever practical asked. The benefits however have very little to do with performance.

    The unfortunate fact of the matter is that persistent desktops are a long term fact of life for a reasonably large percentage of VDI environments and NetApp’s dedup technology is still the best alternative for reducing the storage costs while improving the performance and reliability of these environments. Deduplication is also one of the important storage efficiency techniques that helps keep user data storage growth down to a minimum

    I hope your new job is working out OK for you, and If you’d like me to clarify this further let me know, you’ve got my details 🙂

  2. @John Martin
    Thanks for your coments. I knew I would hear from you 🙂

    I am glad that we both agree that Persistent Desktops using Full Clones is a NO GO area for any organization, unless they have very specific requirements. We also agree that Linked Clones is the way to go; not because of performance but because of the operational benefits.

    I actually have seen the pain of customers who went with Full Clones because it was simpler to deploy and later spent aditional money with consulting to move users to Floating mode. I personally would find hard to justify aditional VDI expenditure after the full roll-out.

    Please note that I didn’t want to mention vendors or tools in my article for obvious reasons. However you having mentioned RCU (Rapid Cloning Utility), I honestly do not recomend organisations to use it for the obvious reasons around Manual Pools. (ok, we both agree on Manual Pools)

    I really don’t want to enter into this vendor discussion with you because all products have their strong and weak points, as we both know. Every vendor has diferent methods to deal with performance and this is not why de-dupe was introduced and is used in first place.

    There are no advantages in using primary storage de-dupe for VDI environments, specially because Full Clones is a no go area for most of my VDI designs.

  3. Andre – great points. I completely agree with everything you said and I was nodding reading the entire article until literally the last two words in the last sentence. I realize you don’t want to get into a discussion of features between vendors but in this case since you brought up the value of IOPS in VDI I thought it made sense mentioning.

    I think that XenDesktop MCS is really only appropriate for small deployments. Although the technology is very similar to the way VMware’s Linked Clones work in reality the VMware implementation is far and away better than XenDesktop MCS. I believe that if you’re going to deploy XenDesktop on any decent size deployment you shouldn’t use MCS but instead use Provisioning Server (PVS). PVS offers ways to greatly reduce read and write IOPS regardless of your SAN array, local storage, etc.

    I think we all agree that “full fat” images reduce some of the management benefits you gain from deploying VDI in the first place. I’m fully in the non-persistent/floating pools camp with you, though I do think that there will always be use cases for persistent desktops for some users.


  4. @Matt Liebowitz
    Thanks for the comment and support.

    I agree with you in regards to PVS against MCS, and also agree that there will always be use cases for persistent desktops. Myself would probably be one of them :-). However, they should always be treated as exceptions.

    Thanks Again,

    • VDI Storage and De-duplication | Wall's View on 06/09/2011 at 2:30 pm

    […] Andre has a great write up on VDI with full clones and dee-dup on the storage on his blog (http://myvirtualcloud.net/?p=1916). This entry was posted in Uncategorized. Bookmark the permalink. ← Make your ThinApp […]

Leave a Reply