vMotion Error – Failed to receive migration

I recently ran into a situation, when after adding a new ESXi Host into a vSphere Cluster that will be used for Nested ESXi, I was unable to vMotion live VM’s onto the new Host. The error message I was getting was ‘Failed to receive migration’

A quick Google search didn’t yield any results, so I had to resort to reading the logs. In the Virtual Machine log file (vmware.log) I noticed this error message: (Scroll to the right)

2019-01-10T20:31:06.254Z| vmx| I125: Msg_Post: Error
2019-01-10T20:31:06.254Z| vmx| I125: [msg.cpuid.vhv.enablemismatch] Configuration mismatch: The virtual machine cannot be restored because the snapshot was taken with VHV enabled. To restore, set vhv.enable to true.

Doing a quick search of the term: vhv.enable showed me that this is required to be set on hosts that are being used for Nested ESXi. Thanks William Lam (https://www.virtuallyghetto.com/2012/08/how-to-enable-nested-esxi-other.html)

So I ran the following command on the new ESXi Host:

echo 'vhv.enable = "TRUE"' >> /etc/vmware/config

After that configuration was added to the config file, vMotions began to function as expected.

Quick And Easy Replication For VDI Golden Images

Continuing on from my previous post; Backup, Restore And Replicate App Volumes, AppStacks And Writable Volumes another large challenge I faced running a Global Enterprise VDI environment was managing the Golden Images.

Replicating VDI Golden Images

The Challenge of Golden Image Replication

For those of you who might not be familiar with VDI environments, Golden Images are the virtual machines that virtual desktop pools are created from. If you need to add an application or make an OS customization for all of your end users, typically you will make the change in the Golden Image and then push that image out to the virtual desktops.

When a VDI environment consists of multiple sites, each separate site usually has its own instance of VMware Horizon or Citrix XenDesktop/XenApp. Each of these separate VDI instances will need their own Golden Images from which virtual desktop pools will be created. In an ideal world, being able to use the same Golden Images for all desktop pools regardless of which site they are in would make the most sense as typically the Pools are the same between sites and this will keep desktop images consistent for all end users. However, until now, this has proven difficult to achieve.

In legacy HCI environments, the replication of virtual machines is very difficult and painfully slow. The main two replication options we see being used are:

  • vSphere Content Library – Using the built-in Content Library service, vCenters can be linked to provide the replication of Golden Images between vCenters
  • Manual Export/Import – Using vCenter to export the Golden Images into an OVA/OVF template, manually copying them to the remote sites and then importing the template into vCenter

There are, however, downsides to both of these methods. In both cases, the whole Golden Image is being replicated each time an update is made. If an image is 60GB in size and there are three remote sites, that’s quite a lot of data that needs to be transferred, this usually is a very slow transfer process and can put a heavy strain on the WAN infrastructure. Additionally, when these images are copied between sites, the vSphere Snapshots are lost in the process making it very difficult to roll changes back if there are issues as all snapshot history is lost.

Golden Image Sprawl

Because of these replication challenges, what we end up seeing is Golden Image sprawl, configuration drift, and an inconsistent-user user experience. Each site has its own set of Golden Images, which are similar to the other sites, but over time the configuration of these images deviates from the other sites. These additional Golden Images also add operational overhead. Each image will need to be patched and kept up to date. The more images you have, the harder it is to keep track of them and the longer updates take to roll out.

Quick And Easy Replication using Datrium DVX

It shouldn’t be this painful. It doesn’t need to be this painful! Using Datrium DVX it only takes a few clicks to replicate Golden Images between multiple sites. The following video (5 minutes) is a brief demonstration of how simple it is to replicate a new snapshot on a Golden Image to a second site.

Some of the key benefits to understand are:

Deduplication and Compression – DVX uses the always-on global deduplication and compression service to only send the changes made to the Golden Image. Rather than sending the whole virtual machine, DVX will only send the data that has changed. This dramatically improves replication times and reduces the load on the WAN infrastructure.

Persistent vSphere Snapshots – When Golden Images are replicated between sites, so are vSphere snapshots. So each Golden Image will show the same snapshot history, regardless of the site they are in. This makes it a lot easier to rollback and forward changes.

Backup, Restore And Replicate App Volumes, AppStacks And Writable Volumes

no problemOne of the biggest challenges I faced running VMware’s internal Horizon environment, was being able to backup, restore and replicate Writeable Volumes and AppStacks. Not being able to backup and replicate Writeable Volumes meant were unable to use Writables to capture our user’s data as we were unable to copy the data to an offline backup array for archive or replicate the data to our other data centers to be used in the event of a site failure.

This is no longer a challenge using Datrium DVX.

App Volumes

App Volumes is a product offered by VMware that has two main features:

  • AppStacks – Packaged applications that get connected/disconnect to a user’s virtual desktop during login/logoff.
  • Writable Volumes – Virtual disks that get connected/disconnect to a user’s virtual desktop during login/logoff. These virtual disks capture any changes made to the virtual desktop, giving a persistent user experience in a non-persistent VDI environment.

If you are not familiar with App Volumes, have a look at this great video – VMware App Volumes Technical Overview

Both AppStacks and Writable Volumes live their lives as VMDK files. This makes it very easy to connect/disconnect them to virtual machines during the login/logoff process. However, there is a massive downside to using VMDK files. AppStacks and Writeable Volume VMDK’s are not permanently associated with a virtual machine, which makes it is almost impossible to back them up using traditional vSphere backup software. Due to limitations in the vSphere API, traditional backup solutions can only back up virtual machines and virtual disks connected to the virtual machines. If a VMDK is not connected to a virtual desktop, it cannot be backed up at scale…. Until now.

Datrium DVX

Backup and Restore Writable Volumes and AppStacks

Running your Horizon environment on a DVX platform makes backing up and restoring App Volumes AppStacks and Writable Volumes almost a single click process. Don’t believe me? Check out this video I recently captured showing the backup and restore of a Writable Volume.

Replicate Writable Volumes and AppStacks

Replicating Writables and AppStacks is also just as easy as backing up and restoring them with Datrium DVX. I’ve also created another video below to show you how simple this process is.

Summary

Using the built-in Snapshot and Replication functions of Datrium DVX will make the management of your App Volumes deployment extremely simple and straightforward. I really wish I had this technology available to me when I was running VMware’s global VDI environment as I’m pretty sure I’d have a few less gray hairs than I do now 😉

Introducing Datrium CloudShift

Hey guys, I have some exciting news for you. Today, we (Datrium), announced CloudShift. Below I’ve summarized some of the things that really excite me about CloudShift. However, before I begin, I think it would be a good idea to quickly go over how Datrium DVX handles Backups and Replication for those of you who might be new to this solution.

Datrium DVX Backup and Replication

As mentioned in my previous post; What is Datrium DVX?, DVX is a Primary storage solution that provides extremely fast storage performance to virtual machines and applications running on DVX Compute Nodes. This is achieved by caching all required data locally on SSD devices installed on the Compute Nodes. DVX then adds a layer of protection by replicating all write IO’s over to the DVX Data Node which acts as a mirror copy for all of the data living in the SSDs on the Compute Nodes.

The Datrium On-Prem DVX system provides a built-in backup mechanism for protecting and restoring data. Protection Groups can be used to group a set of workloads together. Protection schedules are then be applied to the Protection Group based on how regularly data snapshots should be taken and how long they should be kept for. These Snapshots can then be replicated either to another On-Prem DVX system or to a Cloud DVX instance running in AWS.

Hopefully, that gives you enough background for the next section.

Datrium CloudShift

Datrium CloudShift is a SaaS-Based Disaster Recovery (DR) and Mobility orchestration solution, hosted in AWS, that can orchestrate a variety of DR scenarios;

  1. On-Prem (primary) failover to On-Prem (secondary) using backups that are stored at the On-Prem (secondary) site.
  2. On-Prem failover to Public Cloud using backups that are stored in the public cloud
  3. On-Prem failover to On-Prem & Cloud using backups that are either stored at the On-Prem (secondary) site or in the public cloud

I feel this covers the majority of failover scenarios that our customers need to restore their business in the event of a disaster.

I imagine your next question is, “If this a SaaS-based solution, how do I get my data from one data center to another, or to the Cloud? Well, this is where Datrium DVX and Cloud DVX come into play.

Scenario 1 – On-Prem > On-Prem

Scenario 1 is most commonly used when a customer has two data centers, both of which have a Datrium DVX. The DVX is configured to continually replicate data using Elastic Replication from the primary over to the secondary data center (and possibly vice versa). In the event of a failure in one of the datacenters, CloudShift will orchestrate the failover of workloads from one data center to the other. Thanks to the replication technology in DVX, Recovery Point Objective (RPO) times can be less than 1 minute.

CloudShift - On-Prem to On-Prem
 

Scenario 2 – On-Prem > Cloud

Read the rest of this entry »

What is Datrium DVX?

Having recently joined Datrium, one of my first tasks is to begin to understand the product Datrium calls DVX. As a way of helping me wrap my head around this technology, I’ve decided to blog about it. I plan on starting at a high-level and then slowly digging deeper under the covers as I learn more.

If you are interested in following my posts, you can subscribe here and follow me on Twitter.

So to begin with, let’s start at the 100 level.

What is Datrium DVX? – 100 Level

Datrium DVX offers Tier-1 Hyper-Converged Infrastructure along with Cloud Mobility. Yep, lots of buzzwords there! So what does that actually mean? Let’s start with the basic question, “What is the Datrium DVX solution made up of?”

Datrium DVX

Read the rest of this entry »

Goodbye VMware, Hello Datrium

datrium

After almost 8 years at VMware, I am excited to announce the next chapter in my career. Today, I’ll be joining Datrium as a Sr. Solutions Architect.

“Datrium?” I hear you say? “Yes, Datrium!”.

If you are not yet familiar with Datrium, you soon will be. As I begin to learn and understand the technologies that Datrium offers, I’ll be blogging them as a way to help myself and my readers learn more.

I’d just like to thank of my friends and colleagues that I leave behind at VMware. It’s been an amazing place to work and I will certainly miss everyone. Even though I am now longer at the Mothership, you’ll most certainly still see me around. Sometimes you have to take a leap of faith and put yourself into situations that you can learn and grow from.

Watch this space.

 
Get Adobe Flash player