I recently ran into a situation, when after adding a new ESXi Host into a vSphere Cluster that will be used for Nested ESXi, I was unable to vMotion live VM’s onto the new Host. The error message I was getting was ‘Failed to receive migration’

A quick Google search didn’t yield any results, so I had to resort to reading the logs. In the Virtual Machine log file (vmware.log) I noticed this error message: (Scroll to the right)

2019-01-10T20:31:06.254Z| vmx| I125: Msg_Post: Error
2019-01-10T20:31:06.254Z| vmx| I125: [msg.cpuid.vhv.enablemismatch] Configuration mismatch: The virtual machine cannot be restored because the snapshot was taken with VHV enabled. To restore, set vhv.enable to true.

Doing a quick search of the term: vhv.enable showed me that this is required to be set on hosts that are being used for Nested ESXi. Thanks William Lam (https://www.virtuallyghetto.com/2012/08/how-to-enable-nested-esxi-other.html)

So I ran the following command on the new ESXi Host:

echo 'vhv.enable = "TRUE"' >> /etc/vmware/config

After that configuration was added to the config file, vMotions began to function as expected.

Continuing on from my previous post; Backup, Restore And Replicate App Volumes, AppStacks And Writable Volumes another large challenge I faced running a Global Enterprise VDI environment was managing the Golden Images.

The Challenge of Golden Image Replication

For those of you who might not be familiar with VDI environments, Golden Images are the virtual machines that virtual desktop pools are created from. If you need to add an application or make an OS customization for all of your end users, typically you will make the change in the Golden Image and then push that image out to the virtual desktops.

When a VDI environment consists of multiple sites, each separate site usually has its own instance of VMware Horizon or Citrix XenDesktop/XenApp. Each of these separate VDI instances will need their own Golden Images from which virtual desktop pools will be created. In an ideal world, being able to use the same Golden Images for all desktop pools regardless of which site they are in would make the most sense as typically the Pools are the same between sites and this will keep desktop images consistent for all end users. However, until now, this has proven difficult to achieve.

In legacy HCI environments, the replication of virtual machines is very difficult and painfully slow. The main two replication options we see being used are:

• vSphere Content Library – Using the built-in Content Library service, vCenters can be linked to provide the replication of Golden Images between vCenters
• Manual Export/Import – Using vCenter to export the Golden Images into an OVA/OVF template, manually copying them to the remote sites and then importing the template into vCenter

There are, however, downsides to both of these methods. In both cases, the whole Golden Image is being replicated each time an update is made. If an image is 60GB in size and there are three remote sites, that’s quite a lot of data that needs to be transferred, this usually is a very slow transfer process and can put a heavy strain on the WAN infrastructure. Additionally, when these images are copied between sites, the vSphere Snapshots are lost in the process making it very difficult to roll changes back if there are issues as all snapshot history is lost.

Golden Image Sprawl

Because of these replication challenges, what we end up seeing is Golden Image sprawl, configuration drift, and an inconsistent-user user experience. Each site has its own set of Golden Images, which are similar to the other sites, but over time the configuration of these images deviates from the other sites. These additional Golden Images also add operational overhead. Each image will need to be patched and kept up to date. The more images you have, the harder it is to keep track of them and the longer updates take to roll out.

Quick And Easy Replication using Datrium DVX

It shouldn’t be this painful. It doesn’t need to be this painful! Using Datrium DVX it only takes a few clicks to replicate Golden Images between multiple sites. The following video (5 minutes) is a brief demonstration of how simple it is to replicate a new snapshot on a Golden Image to a second site.

Some of the key benefits to understand are:

Deduplication and Compression – DVX uses the always-on global deduplication and compression service to only send the changes made to the Golden Image. Rather than sending the whole virtual machine, DVX will only send the data that has changed. This dramatically improves replication times and reduces the load on the WAN infrastructure.

Persistent vSphere Snapshots – When Golden Images are replicated between sites, so are vSphere snapshots. So each Golden Image will show the same snapshot history, regardless of the site they are in. This makes it a lot easier to rollback and forward changes.

One of the biggest challenges I faced running VMware’s internal Horizon environment, was being able to backup, restore and replicate Writeable Volumes and AppStacks. Not being able to backup and replicate Writeable Volumes meant were unable to use Writables to capture our user’s data as we were unable to copy the data to an offline backup array for archive or replicate the data to our other data centers to be used in the event of a site failure.

This is no longer a challenge using Datrium DVX.

App Volumes

App Volumes is a product offered by VMware that has two main features:

• AppStacks – Packaged applications that get connected/disconnect to a user’s virtual desktop during login/logoff.
• Writable Volumes – Virtual disks that get connected/disconnect to a user’s virtual desktop during login/logoff. These virtual disks capture any changes made to the virtual desktop, giving a persistent user experience in a non-persistent VDI environment.

If you are not familiar with App Volumes, have a look at this great video – VMware App Volumes Technical Overview

Both AppStacks and Writable Volumes live their lives as VMDK files. This makes it very easy to connect/disconnect them to virtual machines during the login/logoff process. However, there is a massive downside to using VMDK files. AppStacks and Writeable Volume VMDK’s are not permanently associated with a virtual machine, which makes it is almost impossible to back them up using traditional vSphere backup software. Due to limitations in the vSphere API, traditional backup solutions can only back up virtual machines and virtual disks connected to the virtual machines. If a VMDK is not connected to a virtual desktop, it cannot be backed up at scale…. Until now.

Datrium DVX

Backup and Restore Writable Volumes and AppStacks

Running your Horizon environment on a DVX platform makes backing up and restoring App Volumes AppStacks and Writable Volumes almost a single click process. Don’t believe me? Check out this video I recently captured showing the backup and restore of a Writable Volume.

Replicate Writable Volumes and AppStacks

Replicating Writables and AppStacks is also just as easy as backing up and restoring them with Datrium DVX. I’ve also created another video below to show you how simple this process is.

Summary

Using the built-in Snapshot and Replication functions of Datrium DVX will make the management of your App Volumes deployment extremely simple and straightforward. I really wish I had this technology available to me when I was running VMware’s global VDI environment as I’m pretty sure I’d have a few less gray hairs than I do now 😉

Over the past 10 years, I’ve been a part of this amazing community. Without this communities support, I wouldn’t have landed my dream job as a consultant within VMware, nor become a Double VCDX. At VMworld 2018 this year, I was lucky enough to spend 20 mins talking to the vCommunity about how they can harness the vCommunity to help further their career. This session was recorded and made available online. In this session, I will share my story and highlight ways that you too can leverage our community to help you reach your career goals and aspirations.

I hope you find this useful. Feel free to message me on Twitter if you have any questions on the content or want to discuss things further.

In an attempt to try and clean up my Twitter feed to make Twitter conversations and tweets a little easier to notice and follow, I followed the recommendation given to me by my followers and decided to split the Twitter accounts I follow into Lists, so they are easier to sort and categorize. One of the lists I started to put together was a vCommunity Twitter List. This list contains many of the people who make up the vCommunity. One of the cool things about Twitter Lists is you can see people who have been added to the lists and you can follow them on your own account. It’s a great way to find people with the same interests. As you can imagine, the vCommunity Twitter List is designed to include people who mostly tweet about Virtualization topics.

Tweet Criteria

Although this is a very informal list, as I mentioned earlier, the whole aim of this list is to remove unnecessary noise from my Twitter feed. So I need to try and put some restrictions in place.

What to Tweet

You can tweet about whatever you are interested in, I’m not going to become the Twitter Police. But ideally;

• Virtualization topics
• Technology topics – We are all geeks at heart

What not to Tweet

I understand that from time to time we all tweet about a wide variety of topics, try and limit the following, if possible.

• Politics …zzzZZZ
• Cat pictures
• Endless Retweets – We want your thoughts, not just the thoughts of others.

Getting added to the list

The vCommunity Twitter List is still a work in progress and always will be. My intention is to keep this Twitter List dynamic. As new people join Twitter and start to tweet about relevant Virtualization topics, they’ll be added. And on the flip side, as people’s interests change and maybe their tweets are no longer focused on the topic of Virtualization, they will be removed.

If you want to be added to the list, please message Follow Me and message me on Twitter.

In a previous post, I talked about ‘What is Datrium DVX?’.
If you are not familiar with Datrium DVX, I’d recommend reading that post first as Cloud DVX integrates with on-premises DVX.

Datrium Cloud DVX

Cloud DVX is a SaaS backup service offered by Datrium that enables on-premises vSphere customers to replicate backup snapshots of their virtual machines and files into public cloud storage (AWS).

By replicating snapshots of virtual machines and files to Cloud DVX, customers are ensuring their data is safely stored offsite should there be a disaster within their own private datacenter. It’s a similar concept to the good old days of tape backups; backing up data to tape drives and sending them to a 3rd party to store offsite. Except of course with Datrium, everything is done automatically for you and the recovery of your data can be almost instantaneous. Read the rest of this entry »

Hey guys, I have some exciting news for you. Today, we (Datrium), announced CloudShift. Below I’ve summarized some of the things that really excite me about CloudShift. However, before I begin, I think it would be a good idea to quickly go over how Datrium DVX handles Backups and Replication for those of you who might be new to this solution.

Datrium DVX Backup and Replication

As mentioned in my previous post; What is Datrium DVX?, DVX is a Primary storage solution that provides extremely fast storage performance to virtual machines and applications running on DVX Compute Nodes. This is achieved by caching all required data locally on SSD devices installed on the Compute Nodes. DVX then adds a layer of protection by replicating all write IO’s over to the DVX Data Node which acts as a mirror copy for all of the data living in the SSDs on the Compute Nodes.

The Datrium On-Prem DVX system provides a built-in backup mechanism for protecting and restoring data. Protection Groups can be used to group a set of workloads together. Protection schedules are then be applied to the Protection Group based on how regularly data snapshots should be taken and how long they should be kept for. These Snapshots can then be replicated either to another On-Prem DVX system or to a Cloud DVX instance running in AWS.

Hopefully, that gives you enough background for the next section.

Datrium CloudShift

Datrium CloudShift is a SaaS-Based Disaster Recovery (DR) and Mobility orchestration solution, hosted in AWS, that can orchestrate a variety of DR scenarios;

1. On-Prem (primary) failover to On-Prem (secondary) using backups that are stored at the On-Prem (secondary) site.
2. On-Prem failover to Public Cloud using backups that are stored in the public cloud
3. On-Prem failover to On-Prem & Cloud using backups that are either stored at the On-Prem (secondary) site or in the public cloud

I feel this covers the majority of failover scenarios that our customers need to restore their business in the event of a disaster.

I imagine your next question is, “If this a SaaS-based solution, how do I get my data from one data center to another, or to the Cloud? Well, this is where Datrium DVX and Cloud DVX come into play.

Scenario 1 – On-Prem > On-Prem

Scenario 1 is most commonly used when a customer has two data centers, both of which have a Datrium DVX. The DVX is configured to continually replicate data using Elastic Replication from the primary over to the secondary data center (and possibly vice versa). In the event of a failure in one of the datacenters, CloudShift will orchestrate the failover of workloads from one data center to the other. Thanks to the replication technology in DVX, Recovery Point Objective (RPO) times can be less than 1 minute.

Scenario 2 – On-Prem > Cloud

Read the rest of this entry »

VMware just released Security Advisory VMSA-2018-0019 for customers who are using VMware Horizon v6 or v7 and the Horizon Client for Windows. As many of my readers are VMware Horizon customers, I thought I’d bring this to your attention in case you missed the official notifications from VMware.

Advisory Summary

Out-of-bounds read vulnerability in the Message Framework library.

Horizon 6, 7, and Horizon Client for Windows contains an out-of-bounds read vulnerability in the Message Framework library. Successfully exploiting this issue may allow a less-privileged user to leak information from a privileged process running on a system where Horizon Connection Server, Horizon Agent or Horizon Client are installed.

Note: This issue doesn’t apply to Horizon 6, 7 Agents installed on Linux systems or Horizon Clients installed on non-Windows systems.

Remediation Steps

• VMware Horizon 7.x.x customer should update to Horizon 7.5.1
• VMware Horizon 6.x.x customer should update to Horizon 6.2.7
• Horizon Client for Windows users should update to 4.8.1

For further information and direct links to the patch downloads, visit the Security Advisory VMSA-2018-0019 page.

Hey, another VMworld is almost upon us. I am fortunate enough to be talking again this year. However, this will be my first VMworld where I won’t be speaking as VMware employee. Below you’ll find the information about my session. Be sure to register if you are interested in attending.

VMTN TechTalk: Harnessing the vCommunity to further your career [VMTN5508U]

Tuesday, Aug 28, 2:30 p.m. – 3:00 p.m.

Session Summary: Over the past 10 years, I’ve been a part of this amazing community. Without this communities support, I wouldn’t have landed my dream job as a consultant within VMware, nor become a Double VCDX. In this session, I will share my story and highlight ways that you too can leverage our community to help you reach your career goals and aspirations.

Having recently joined Datrium, one of my first tasks is to begin to understand the product Datrium calls DVX. As a way of helping me wrap my head around this technology, I’ve decided to blog about it. I plan on starting at a high-level and then slowly digging deeper under the covers as I learn more.

If you are interested in following my posts, you can subscribe here and follow me on Twitter.

So to begin with, let’s start at the 100 level.

What is Datrium DVX? – 100 Level

Datrium DVX offers Disaggregated Hyper-Converged Infrastructure (DHCI) along with Cloud Mobility. Yep, lots of buzzwords there! So what does that actually mean? Let’s start with the basic question, “What is the Datrium DVX solution made up of?”