Here is why I copied the StorageReview Performance Benchmark for MySQL on Datrium… but there was a Gotcha!

  • 01/20/2020

If you want to benchmark a storage subsystem, it’s best to use pre-existing benchmarks to be able to compare results. Instead of reinventing the wheel, I just copied the configuration and tests performed by our good friends at StorageReview. (https://www.storagereview.com/node/3598)

I am also writing a detailed white-paper to describe what and how the tests have been performed. I also cover test results with Zero RTO restores against Ransomware and orchestrated Disaster Recovery as a Service to the VMware Cloud on AWS. The marketing team is now doing their thing tio make it into a customer facing document.

This test was created to benchmark a real database performance. We focus on the OLTP-like database benchmark using a script that simulates read and write mixed transactions using MySQL InnoDB.

MySQL was configured with 100 tables and each table with 10,000,000 entries and 24MB caching (innodb_buffer_pool_size). This configuration created a database of approximately 230GB.

The Gotcha!

As you will read these benchmarks you will notice that Datrium outperforms other Hyper-Converged systems with more than double the number of hosts, and I was really happy with the result. But that was just until I found out that StorageReview was using the test script with a modified variable that generated about 75% reads and 25% writes, while my I/O ratio had been about 60% reads and 40% writes all along.

Random Writes is What Separates the Men from Boys in the storage Industry.

Anyone in the storage industry knows that there are many ways to game benchmarks. But ask any engineer or SE about what’s a truly challenging workload, and they will tell you “random writes over a large address space”.

Basically I failed in copying our friends at StorageReview because my benchmark would be way more challenging than what was performed with other vendors. I marched forward with my numbers anyway and what you will see next is incredible!!

To determine how the Datrium responds to MySQL workloads I scaled Sysbench VM benchmark up to 2 total VMs (1 per host). As the Sysbench OLTP workloads are scaled, they measured a performance increase from 248.72 TPS with a single VM and 4 Threads (pic. 1) up to 2043.73 TPS with 2 VMs and 128 Threads (pic. 2), while latencies were nearly the same across both tests.

This works out to be a 94% jump in performance with the workload footprint doubling. This result is similar or higher than other Hyper-Converged solutions with 4 MySQL VMs and 4 hosts.

In the example above, just 2 MySQL VMs delivered an impressive 2042.73 TPS on VMs with 32GB RAM, 16 vCPU, 2 consumer-grade SSDs per host, and yet delivering Always-on compression, deduplication, erasure coding (RF3) and end-to-end FIPS compliant encryption.

When testing OLTP workloads with Sysbench, it is vital to put the results in context to what has been tested to be able to compare apples-to-apples. To make comparisons with other Sysbench OLTP workloads, both the database and the VM configuration must be the same.

The graph below from the Datrium GUI focuses on VM level metrics, and it demonstrates that at peak performance with 2 VMs running Sysbench at 128 Threads the average storage latency is only ~2.7ms. The higher Sysbench latency displayed in the previous two graphs is an aggregation of all system latencies, including vCPU and the MySQL InnoDB itself.

It is important to recall that this test is being executed on just 2 hosts with 2 consumer-grade SSDs. Some customers may opt for more flash devices for business-critical applications, or even NVMe devices for even lower latencies.

Proof Point

To prove my point, I ran the same benchmark with 1 VM and 1 host while increasing vCPU to 56 and RAM to 500GB. With these resources available to MySQL InnoDB, the number of Transactions Per Second rose to 8226.99, a 760% jump in the number of transactions. The Sysbench latency also dropped considerably, and the average storage latency, as seen by the VM, was never above 1.9 ms.

If we compare these results with the ones listed by StorageReview (here) with an easier read-centric workload, we notice that Datrium outperforms other benchmarks in terms of performance while having all data services enabled. Further, we will soon also see the results with Datrium SaaS features, such as Backup-as-a-Service and DR-as-a-Service to VMware Cloud.

The paper will be available soon and I’ll publish it here.

This article was first published by Andre Leibovici (@andreleibovici) at myvirtualcloud.net

I solved Natural Hazard DR with Python and Datrium

  • 01/01/2020

I solved Natural Hazard DR with Python and Datrium.

One of the critical issues with disaster recovery, in general, is that you only want to trigger DR when the disaster strikes.

As much as I would love to say that, ideally, applications should move around freely, the reality is that there are too many interdependencies to make us want to migrate applications if a natural disaster has not happened yet, or if it is uncertain it is going to happen.

Some of the new app development architectures are trying to solve this problem with container infrastructures. However, for stateful datasets, this is still a problem if application-level coherence isn’t available across zones and regions.

Organizations may have their replication and DR plans in place, but the missing component is the intelligence to deal with natural disaster events as they unfold, without relying on human interaction and without triggering DR if not needed.

The solution to the problem is to be as prepared as possible in case of a DR event becomes necessary, and that means having the most recent data available in the target DR datacenter. Hence, if the disaster strikes, you now have the shortest Recovery Point Objective (RPO) as possible.

One way to solve the problem is to have all your core applications being synchronously replicated between datacenters. While this seems logical, it is extremely expensive and will not withstand a ransomware attack or corruption since all data is automatically replicated across source and target data centers.

This post assumes some prior knowledge of how Datrium snapshot and replication work.

The Python tool

The reason I built this Python tool for the Datrium DVX is to enable real-time weather and geological monitoring using real-time data from the U.S. Geological Survey and to trigger actions under certain conditions. The tool uses Datrium API SDK for Python.

These conditions may be high winds or temperatures above the defined limits or a high magnitude earthquake that just hit the region and has the potential to cripple the public infrastructure in the next few minutes or hours.

The tool can be configured to monitor natural hazard events in a particular location every few minutes, and if conditions are satisfied, it will snapshot applications and force replication to the DR site.

If you are curious about Datrium Zero RTO I recommend this reading.

This solution is generally better than having all your VMs snapshotting and replicating every few minutes due to resource utilization and cost. Typically, organizations have well-defined SLAs, but if they need to increase snap and replication of specific applications under certain natural hazard conditions, they can you this tool.

Note that Datrium also provides orchestrated plans for DR that can be triggered if necessary, and when using in conjunction with this tool, you will always have the latest data available when needed.

Of course, this tool can also be repurposed for different infrastructures, but Datrium offers the perfect platform for Site DR, Cloud DR, and Ransomware protection.

I have posted the Python source code under Apache License, Version 2.0  licensing on my GitHub repository here.
Natural Hazard monitoring

This article was first published by Andre Leibovici (@andreleibovici) at myvirtualcloud.net

Backups – Feasibility vs. Requirements

  • 12/17/2019

A recent conversation regarding application protection triggered me to think about feasibility vs. business requirements. This topic applies to pretty much any enterprise technology domain, but in this case, I want to focus on backup and snapshot technologies. I’ll use Datrium as an example given its high degree of flexibility.

First some key technology fundamentals.

  1. Datrium provides native VM-centric snapshots
  2. Snapshots are non-intrusive and offer zero performance impact
  3. Snapshots may be configured as often each minute, allowing 1 minute RPO restores.
  4. 1.2 Million snapshots are supported
  5. 2,000 snapshots per VM are supported
  6. Entire datastores can be snapshotted at once
  7. Snapshot retention can be of any desired period
  8. Local snapshots enable Zero RTO restores

If the fundamentals above are presented to application owners as an elective SLA, they will likely choose the SLA that minimizes their downtime exposure. Every 60 minutes seem to be the response I hear the most, but 30 and 15 minutes RPO aren’t uncommon.

Compared to current backup strategies that offer backups once every 24 hours, this is already a radical change towards improving application SLAs.

The next topic relates to the snapshot retention period, or how long the application owner would like to keep point-in-time copies for – and the answer is regularly anything between one and seven years.

Next, we start having a conversation about snapshots for site recovery vs. snapshots for ransomware recovery. Yes, those have very different requirements.

As an example, you may want applications snapshotted every 10 minutes and available onsite for the purpose of recovering individual applications from a ransomware attack keeping these for 24 hours. However, you may likewise want to have another time-series snapshot that happens every hour, is kept for 1 month and at the same replicated offsite for the purpose of disaster recovery.

The application X data protection schema could look like this:

  • Time-Series 1 – 10 minutes RPO – 24 hours retention
  • Time-Series 2 – 1 hour RPO – 30 days retention > offsite replication
  • Time-Series 3 – 24hs RPO – 7 years retention > cloud replication

In this scenario, application X can be restored back to a known-good-state every 10 minutes for the first 24 hours, every 1 hour for 30 days, and every 24hs for 7 years. It seems like a good protection plan to me, but the application SLA is what defines if the plan is good enough.

Datrium offers metadata-based snapshots with compression and deduplication, and that certainly helps with data reduction across common datasets. However, if the data being created is not really de-dupable it will generate overhead that could use a reasonable amount of storage capacity. Further, it also comes down to the application data change rate – unique data being created.

Storage Capacity

Let’s look at the same scenario from a storage consumption angle using Datrium Global De-Dupe Index across all customers (4.4x), but please note that your mileage may vary according to the data. Moreover, let’s use an initial 100TB dataset to make the math easier, and for change rate let’s assume 0.20% for 10 min, 0,4% for 1 hour and 1% for daily.

I’m not going to get into the calculation details in this post, but the total capacity required in this case with the snapshot frequency and retention period mentioned would be ~667 TB.

So it is important to ensure you have the adequate usable capacity or have plans to add capacity in the future since 668 TB will be the total capacity required in year 7. Datrium current upper capacity is 4.4 Petabytes, but insiders are already taking in Exabyte terms.

Optionally, you may prefer to have longer retention policies in a DR or Cloud site then on-premises.

(Protection Scheduled – click to make it bigger)

Snapshot Counting

Another layer for decision making is the total amount of snapshots. Assuming we have 100 VMs backed up according to the schedule above we would have 352,000 snapshots after 7 years. However, if we are backing up 1,000 VMs with this same schedule there would be 3.5M snapshots, a number above systems limits today.

The engineering and QA teams are always improving system limits, but for production environments, the solution should be architected with current limits in mind.

Conclusion

As you can see there is tremendous flexibility in the way Datrium enables organizations to protect applications, but with power comes responsibility and IT must work with application owners to achieve the expected SLA within the existing constraints.

Just because it is possible, doesn’t mean you should do it.

A more common approach nowadays is the creation of SLA tiers, such as bronze, silver, and gold that are then offered to the business as alternatives, eliminating complexity and ensuring the platform correctly sized.

This article was first published by Andre Leibovici (@andreleibovici) at myvirtualcloud.net

Load more