Datrium Proactive Protection for Natural Hazards is now on Docker

I recently released a set of Python components that make Datrium take pro-active virtual machine snapshots and replicate them upon detection of high winds, high temperature, or earthquakes in a defined region.

The reason I built the tool is to enable real-time weather and geological monitoring using real-time data from the U.S. Geological Survey and to trigger actions under certain conditions.

While it is simple to use the tool, it does require Python knowledge and the creation of an environment to run the scripts and modules.

For this reason, I converted the tool into a Docker container that can be instantiated and updated on-demand without the need to know anything about Python.

The container can be downloaded from here.

Before starting, make sure Docker and Docker-Compose are both installed in your system.

mkdir datrium_natural_hazard_protection
cd datrium_natural_hazard_protection
touch docker-compose.yml
vi docker-compose.yml

Copy/Paste the content below into docker-compose.yml, replacing and adding values where necessary. You must register for a free API key with OpenWeather at https://home.openweathermap.org/users/sign_up and add to the pyowm_api_key value.

version: "3"

services:

  com.myvirtualcloud.dvx.automation:

    image: andreleibovici/datrium_natural_hazard_protection:latest

    environment:

      # Location (City)

     - prod_location_city=Auckland

     # Country Abbreviation (2-digit)

     - prod_location_country=NZ

     # DVX Fully-Qualified-Domain-Name

     - prod_dvx_fqdn=DVX_FQDN

     # DVX Admin username

     - prod_dvx_username=admin

     # DVX Admin Password

     - prod_dvx_password=PASSWORD

     # Choose true or false to enable snapshots replication

     - repl_replicate=false

     # Replication site target as configures in DVX

     - repl_location=DVX_NAME

     # Retain snapshot for the specified amount of time (in seconds) from current time.

     # Use 'forever' to retain the snapshot forever.

     - snapshot_retention=3600

     # Filename containing list of vms to protect

     - vms_to_protect_filename=vms_to_protect.txt

     # Frequency to check for natural hazard events (seconds) and snapshot/replicate

     - frequency_check=300

     # Km/h

     - wind_speed_max=40

     # Celsius

     - temperature_max=45

     # Richter scale minimum earthquaque magnitude

     - quake_magnitude_min=4

     # "hour", "day", "week", "month"

     - quake_period=hour

     # You MUST provide a valid public API key from pyowm

     - pyowm_api_key=YOUR_PYOWM_API_KEY

    volumes:

      - /path/vms_to_protect.txt:/vms_to_protect.txt

You will need to make sure a file name vms_to_protect.txt is available in the directory, and this file must have the VMs to be protected. Here is an example:

Citrix-01
MySQL-Linux-01
MySQL-Linux-02
epic-cache01
epic-clarity01
pgress-01
(EOF)

Download and start the Docker image using docker-compose:

docker pull andreleibovici/datrium_natural_hazard_protection
docker-compose up -d

To the application check logs, use:

 docker-compose logs -f

This article was first published by Andre Leibovici (@andreleibovici) at myvirtualcloud.net

I saved my datacenter in 4 minutes

Disaster Recovery is one of the most complex exercises IT will endeavor. From infra and data replication to applications and orchestration, it is hard! See how I executed DR for a couple of VMware VMs to the VMware Cloud in 4 minutes.

I’m not saving an entire datacenter, but the video gives you a good idea of how this can be easily accomplished.

I saved my datacenter in 4 minutes

This article was first published by Andre Leibovici (@andreleibovici) at myvirtualcloud.net

Here is why I copied the StorageReview Performance Benchmark for MySQL on Datrium… but there was a Gotcha!

If you want to benchmark a storage subsystem, it’s best to use pre-existing benchmarks to be able to compare results. Instead of reinventing the wheel, I just copied the configuration and tests performed by our good friends at StorageReview. (https://www.storagereview.com/node/3598)

I am also writing a detailed white-paper to describe what and how the tests have been performed. I also cover test results with Zero RTO restores against Ransomware and orchestrated Disaster Recovery as a Service to the VMware Cloud on AWS. The marketing team is now doing their thing tio make it into a customer facing document.

This test was created to benchmark a real database performance. We focus on the OLTP-like database benchmark using a script that simulates read and write mixed transactions using MySQL InnoDB.

MySQL was configured with 100 tables and each table with 10,000,000 entries and 24MB caching (innodb_buffer_pool_size). This configuration created a database of approximately 230GB.

The Gotcha!

As you will read these benchmarks you will notice that Datrium outperforms other Hyper-Converged systems with more than double the number of hosts, and I was really happy with the result. But that was just until I found out that StorageReview was using the test script with a modified variable that generated about 75% reads and 25% writes, while my I/O ratio had been about 60% reads and 40% writes all along.

Random Writes is What Separates the Men from Boys in the storage Industry.

Anyone in the storage industry knows that there are many ways to game benchmarks. But ask any engineer or SE about what’s a truly challenging workload, and they will tell you “random writes over a large address space”.

Basically I failed in copying our friends at StorageReview because my benchmark would be way more challenging than what was performed with other vendors. I marched forward with my numbers anyway and what you will see next is incredible!!

To determine how the Datrium responds to MySQL workloads I scaled Sysbench VM benchmark up to 2 total VMs (1 per host). As the Sysbench OLTP workloads are scaled, they measured a performance increase from 248.72 TPS with a single VM and 4 Threads (pic. 1) up to 2043.73 TPS with 2 VMs and 128 Threads (pic. 2), while latencies were nearly the same across both tests.

This works out to be a 94% jump in performance with the workload footprint doubling. This result is similar or higher than other Hyper-Converged solutions with 4 MySQL VMs and 4 hosts.

In the example above, just 2 MySQL VMs delivered an impressive 2042.73 TPS on VMs with 32GB RAM, 16 vCPU, 2 consumer-grade SSDs per host, and yet delivering Always-on compression, deduplication, erasure coding (RF3) and end-to-end FIPS compliant encryption.

When testing OLTP workloads with Sysbench, it is vital to put the results in context to what has been tested to be able to compare apples-to-apples. To make comparisons with other Sysbench OLTP workloads, both the database and the VM configuration must be the same.

The graph below from the Datrium GUI focuses on VM level metrics, and it demonstrates that at peak performance with 2 VMs running Sysbench at 128 Threads the average storage latency is only ~2.7ms. The higher Sysbench latency displayed in the previous two graphs is an aggregation of all system latencies, including vCPU and the MySQL InnoDB itself.

It is important to recall that this test is being executed on just 2 hosts with 2 consumer-grade SSDs. Some customers may opt for more flash devices for business-critical applications, or even NVMe devices for even lower latencies.

Proof Point

To prove my point, I ran the same benchmark with 1 VM and 1 host while increasing vCPU to 56 and RAM to 500GB. With these resources available to MySQL InnoDB, the number of Transactions Per Second rose to 8226.99, a 760% jump in the number of transactions. The Sysbench latency also dropped considerably, and the average storage latency, as seen by the VM, was never above 1.9 ms.

If we compare these results with the ones listed by StorageReview (here) with an easier read-centric workload, we notice that Datrium outperforms other benchmarks in terms of performance while having all data services enabled. Further, we will soon also see the results with Datrium SaaS features, such as Backup-as-a-Service and DR-as-a-Service to VMware Cloud.

The paper will be available soon and I’ll publish it here.

This article was first published by Andre Leibovici (@andreleibovici) at myvirtualcloud.net

Load more