Building a Hybrid Cloud on Nutanix Community Edition

0 Shares

In this deep-dive guide, I’ll show you how to set up a Hybrid Cloud for a real company. I’m going to use my xenappblog LLC as an example and record training videos for the Automation Framework Community Edition at the same time. It’s a long article so the Table of Contacts on the right will definitely come in handy 🙂

Hardware and Prerequisites

The hardware is Supermicro E200-8D with 128 GB, 32GB SATADOM and two Samsung 970 EVO Plus SSD 1TB – M.2 NVMe (adapter needed). The price is $1650 without storage. For the hypervisor, I’m using Nutanix Community Edition 2019.11.22 which is the first version to support All Flash.

I’m not going to show you the Nutanix CE setup, that has already been covered by fellow Nutanix NTC Patrick Damen in this amazing blog post series. He is using the same hardware as me and goes really deep dive.

The prerequisite going forward is a VM running with my Automation Framework Community Edition (AFCE) which has FREE Training Videos.

For this environment I’m going to leverage Server Core for as many supported services as possible. First out is two Domain Controllers which will also be running DHCP Failure. The hardware configuration is 2 vCPU / 2GB / 40 GB. If you want to go the same route, remember to change the Operating System to Server Core.

Setting up Domain Controllers when building a Hybrid Cloud on Nutanix

AFCE automatically let you deploy Domain Controller and Additional Domain Controllers fully automated. It even setup PKI and export the domain wildcard certificate to pfx, crt, pem and key file format.

Setting Up Remote Access with 2-factor authentication and DHCP Failover

With the Domain Controllers running on Server Core and a temporary Hydration server for deployment only, we’re going to setup Remote Access first using Parallels Remote Application Server (PRAS).

Setup Remote Access first using Parallels Remote Application Server (PRAS).

Don’t worry, we will be setting up a full Citrix Hybrid Cloud solution, but I need something for Remote Access today and I also like to have a fall back solution.

The PRAS infrastructure (PRAS-01) will be running on Server Core with 2 vCPU / 2GB / 40 GB. The RDSH Worker (PRAS-WS19-01) will be running Server with Desktop Experience with 2 vCPU / 4GB / 60 GB.

From the Hydration Server make changes to the configuration file located in C:\Source\Examples\Parallels to reflect your environment. For more information check the blog post Prevent RDP Hacking in 2 Minutes with OTP.

Open a RDP session to PRAS-01 and type WhoAmI to make sure you’re logged in as Domain Administrator. Type PowerShell.

This image has an empty alt attribute; its file name is image-6.png

Now copy & paste the customized Parallels configuration above.

This image has an empty alt attribute; its file name is image-7-1024x623.png

To logoff the Server Core session type sconfig and then 12.

Finally download, install and open the Parallels Client to scan the QR code into Google or Microsoft Authenticator (after authentication).

If you’re so lucky to have a UniFi Secure Gateway (USG) simply head over to Settings – Routing & Firewall – Port Forwarding. It’s that easy.

setup Remote Access first using Parallels Remote Application Server (PRAS).

I found that the USG DDNS reports the internal IP address of the ISP modem, so I use Dyn.com and their Dyn Update client running on my Office PC to always update my DNS record with my external IP address.

The final part is to configure DHCP Failover. Open a RDP connection to PRAS-WS19-01 and install DNS RSAT Tools and connect to DC-01 and DC-02.

This image has an empty alt attribute; its file name is image-11-1024x399.png

First of we need to right click DC-02 and Authorize it. Second expand IPv4 on DC-01 and click Scope Options. Double click 006 DNS Servers to add the 2nd DNS server.

This image has an empty alt attribute; its file name is image-12.png

Right click IPv4 on DC-01 and select Configure Failover.

This image has an empty alt attribute; its file name is image-13.png

Click Next

This image has an empty alt attribute; its file name is image-14.png

Add DC-02 and click Next.

This image has an empty alt attribute; its file name is image-16.png

Set the Mode to Hot standbyStandby and type in a Shared Secret.

  • Load balance mode (Both DHCP servers are active and the load is shared among the two).
  • Hot standby mode (One of the DHCP servers is active and the other is passive. When active goes down, passive takes over and becomes active).
This image has an empty alt attribute; its file name is image-17.png

Click Next – Finish.

This image has an empty alt attribute; its file name is image-18.png

You’ve now successfully setup a DHCP Failover.

This image has an empty alt attribute; its file name is image-19.png

You can verify that it’s working by selection Address Leases on DC-02.

This image has an empty alt attribute; its file name is image-20-1024x399.png

Group Policy Central Store, WinRM, ControlUp and LAPS 

Setting up Group Policy Central Store, enable Windows Remote Management (WinRM), install and configure ControlUp and Local Administrator Password Solution (LAPS) when building a Hybrid Cloud. 

To take advantage of the benefits of .admx files, you must create a Group Policy Central Store in the SYSVOL folder on a Windows domain controller. The Central Store is a file location that is checked by the Group Policy tools by default. The Group Policy tools use all .admx files that are in the Central Store. The files that are in the Central Store are replicated to all domain controllers in the domain.

Copy the folder C:\Windows\PolicyDefinitions to \\FQDN\sysvol\FQDN\Policies. You can verify it working by opening the Group Policy Management Console.

Next we’re going to create a Group Policy to allow for Windows Remote Management. I’m going to name it Global CC – WinRM.

Go to Computer Configuration – Policies – Administrative Templates – Windows Components – Windows Remote Management (WinRM) – WinRM Service. Double-click Allow remote server management through WinRM and set it to Enabled as shown below.

Go to Computer Configuration – Preferences – Control Panel SettingsServices and choose New – Service. Choose Automatic (Delayed Start) as startup type, pick WinRM as the service name, set Start service as the action.

Go to Computer Configuration – Windows Settings – Security Settings – Windows Defender Firewall with Advanced Security and browse down to Inboud Rules. Right-click to create a New Rule.

Next up is the ControlUp Console which you can automatically download and install using my Github code.

Start .\ControlUpConsole.exe and log on or create a new Account.

Click Add Machines.

This will automatically install the ControlUp agent on all selected Servers.

Next I’m going to create a Service Account for ControlUp Monitor called svc-controlup with a complex password set to never expires.

This account needs Allow log on Locally. Create a new GPO called Global CC – ControlUp and go to Computer Configuration – Windows Settings – Security Settings – Local Policies – User Rights Assignment. Be aware that you NEED to add Administrators as well.

In the same policy also add the firewall port used by ControlUp.

Head back to ControlUp – Add Monitor.

Add the Service Account credentials and click Shared to be able to reuse.

The rest is straight forward or consult ControlUp documentation for more details. I’m going to install the Monitoring Agent on my MDT server. In Larger Enterprises you would setup multiple dedicated servers for this purpose only.

The last part of this section in building a hybrid cloud is deploying and configuring LAPS. First we need to create a Share where we’ll host the MSI we’re going to deploy through Group Policy. YEAH I’m not a big fan either, but that’s the fastest and easiest way to get this up and running as quick as possible. RDP into your Domain Controller or File Share where you want to host the Share.

Download Local Administrator Password Solution (LAPS) into a sub folder called LAPS\Version on that new share created above.

Create a new Group Policy at Domain Root called Global CC – LAPS and go to Computer Configuration – Policies – Software Settings – Software Installation and right click to Add New Package. Navigate to the MSI using FQDN and select Assigned.

Now reboot all your VMs for the software to be installed. One of the many amazing features of ControlUp is Manage Programs and Updates which allows you to see where LAPS are installed.

On your Management Server install all the LAPS components.

Last step is to copy the AdmPwd.admx and AdmPwd.adml located in C:\Windows\PolicyDefinitions to your Central Group Policy Store.

Edit the policy Global CC – LAPS and go to Computer Configuration – Policies – Administrative Templates – LAPS

Finally open PowerShell as Administrator and run the following commands to the extend AD Schema. The last line is for which OU you want to enable LAPS.

MAKE SURE YOU HAVE DOMAIN ADMIN ACCOUNT WHICH IS NOT THE DEFAULT ADMINISTRATOR OR YOU WILL LOCK YOUR SELF OUT. HAPPENED TO ME AND TRYING TO RECOVER NOW!!!! OR SIMPLY DO NOT ADD THE GPO AT ROOT LEVEL SO IT DOES NOT APPLY TO DOMAIN CONTROLLERS.

DNS and Citrix Cloud Connector for Your Hybrid Cloud

Configuring DNS correctly, setting up a Group Policy for Authoritive Time Server automation and Citrix Cloud Connector with the Nutanix Plugin.

DNS is critical in any environment and especially so when building our hybrid cloud, so we’re going to cover the basics here. If you want to learn more I HIGHLY recommend Carl Websters Presentations on the topic.

The number one recommendation is to be consistent in how domain controllers have the DNS servers configured. There is not just one way to do DNS configurations for domain controllers. Rule #1 is to pick a standard and be consistent in the application of that standard.

I’m going to set my primary DNS entry to the Domain (PDCe role holder). To find it run the following command.

Next up is setting DNS Scavenging server for all AD-Integrated Zones, except Trust Anchors. We’re going to use Carl Websters script for this.

You know the saying It’s DNS. It’s always DNS? It normally is, but when it’s not it’s NTP. So again we’re going to use Carl Websters script and post Creating a Group Policy using Microsoft PowerShell to Configure the Authoritative Time Server to fix this.

I would also recommend this post Configuring the time zone and code page with Group Policy.

Next up is setting up Citrix Cloud Connectors. The recommended leading practice are now 3 Cloud Connectors per site.

Setting up Citrix Cloud Connectors when building a Hybrid Cloud.

I’m running the Citrix Cloud Connectors on Windows 2019 Server with 2 vCPU and 3 GB of memory. This is NOT a recommended hardware configuration in Production, but I’m keeping an eye on ControlUp Insigts and real time stats.

To automatically install the Citrix Cloud Connector you can use the code below. Please note that you need to create your API key at Identity and Access Management – API Access.

Warning! Citrix has stripped down the size of the Cloud Connector and are synchronizing the rest from the Cloud. If you run the Nutanix Plugin setup straight after it will fail. The current workaround is to wait for these Files and Folders to show up before installing the Nutanix Plugin.

If you want to automate make sure to also download my custom MST file. My setup took 20 minutes for the Citrix Cloud Connector to sync up.

Without the Nutanix Plugin installed on ALL your Citrix Cloud Connectors you won’t see Nutanix AHV in the Connection type list.

The Nutanix Plugin for building a Hybrid Cloud.

Certificates, AD Logon, Windows Admin Center and Admin Portal

Changing the self-sign Nutanix Cluster certificate and enabling Active Directory Log on. We’re also going to setup Windows Admin Center and Admin Portal.

Security should be on anybody’s minds these days and friends don’t let friends use Self Signed Certificates! While wildcard certificates is not the perfect solution, I find them very easy to setup, use and implement. After all they’re a lot more secure than the Out of the Box Self Signed Certificates.

If you used my Automation Framework Community Edition for your Domain Controller setup you automatically got PKI installed, wildcard certificate created and exported to the various file formats. In case you didn’t you can download the scripts from my Github repo (remember to edit the ssl.ini file).

Before we continue I’m going to configure my Nutanix Cluster to use my wildcard certificate and enable Active Directory integration for logon.

Head over to your Nutanix Cluster and click the Gear – SSL Settings – Replace Certificate.

Select Import Key and Certificate.

Browse to the private key and certificate. The internal domain CA certificate chain has to be exported manually though, through MMC – Certificates – Trusted Root Certificate Authority Store.

After you’ve imported the files you have successfully enabled SSL for your Nutanix Cluster.

To test that LDAPS works in your domain start ldp.exe from your Domain Controller. Set the port to 636 and select SSL.

Click OK. The following shows a successful test.

Next up is Active Directory integration. Select Gear – Authentication and fill in your LDAP information. Remember to use LDAPS and use a dedicated Service Account with a strong password set to never expires.

When you click Save you’ll need to Create Role Mapping.

Finally test login with Active Directory.

I also did the same for my VMware Cluster using the instruction in this post How to replace default vCenter VMCA certificate with Microsoft CA signed certificate.

For Windows Admin Center I’m using a Windows 2019 Server Core with 1 vCPU and 1 GB of memory. The first step is to install the wildcard domain certificate.

Then download Windows Admin Center and run the setup. We’re going to use port 443 and our wildcard certificate. You’ll find the thumbprint in the wildcard.txt file.

To enable Single Sign On you need to add the Server to the Local Intranet. You should use Group Policy for this. Go to User Configuration – Policies – Administrative Templates – Windows Components – Internet Explorer – Internet Control Panel – Security Page – Site to Zone Assignment List.

Click Add to add your resources.

When Searching Active Directory you can use a wildcard.

To enable Single Sign On we need to enable Kerberos Constrained Delegation. Run the following Powershell code on the WAC server to automate this. Make sure to edit Line 3 to reflect your Domain.

During the import of the wildcard certificate we also setup a basic IIS server. With port 80 available on the WAC server, this is the perfect place for the Admin Portal, because its just favourites. The code can be downloaded here.

This image has an empty alt attribute; its file name is image-65-1024x555.png

Place the code in C:\inetpub\wwwroot and run the following command.

Make sure to delete Default.htm and iisstart.htm that way you get index.html loaded by default.

Backing up the Cluster Using Nakivo Backup

Setting up a Nakivo backup solution for our Nutanix & VMware Cluster. We now have fully functional infrastructure, so it would be a PITA to lose it all in case anything goes wrong.

Currently, Nakivo does not run Director on AHV, only Transporter. So you’ll need another Hypervisor, NAS (Synology, QNAP, WD, Netgear) or even a Physical Machine to control everything. In this post I’m going to use my VMware Cluster (Director and Repository) to backup my Nutanix Cluster (via Transporter).

Download the Full Solution Nakivo Backup and Replication OVA and deploy it through vCenter. After the VM has been created I’m adding a 2nd disk which is going to be my Backup Repository.

Backing up the cluster using Nakivo for your Hybrid Cloud.

Start the VM and launch the Web Console. Select Manage NAKIVO services – Onboard repository storage. Select the new Storage – Use this disk as backup storage.

Select Network settings to find the DHCP address or manually set a static address. You also might want to change the Time Zone. I’m adding a DNS record so I can start the console through the Admin Portal.

This image has an empty alt attribute; its file name is image-65-1024x555.png

Create an User Account and click Proceed.

Next we’re going to create an Inventory. I’m going to create one for my VMware Cluster and another one for my Nutanix Cluster.

Then I’m going to select Deploy New Transporter to the Nutanix Cluster. The Onboard transporter is the one used by VMware.

Click Add Backup Repository – Create new backup repository.

Click Finish at the bottom of the page.

Next click Create – Nutanix AHV backup job. On the Options tab make sure to set App-aware mode to Disabled. That way you don’t need to install the Nutanix Guest Tools on every VM.

The rest is straight forward. At the end click Finish & Run.

If you use my Automation Framework Community Edition to deploy the Domain Controller you already have the wildcard certificates exported to the various format, see also Building Hybrid Cloud on Nutanix CE – Part IV.

Navigate to Configuration – System Settings and scroll to the bottom to select Install new certificate.

Finally you would want to setup email notifications from Configuration – Email Notifications. The following is the required configuration for Office 365.

Building a Master Image

Building our first Master Image semi automatically using the Automation Framework for Microsoft Deployment Toolkit.

The Automation Framework has been configured according to the training videos inside the paid membership area. If you’re not a member you can use the free Automation Framework Community Edition instead, mostly manually though.

Building a master Image for a hybrid cloud on Nutanix.

This will automatically kick off the Master Image build and notify through email and webhooks when completed. The process is more or less the same independent of the Hypervisor and EUC vendor.

Running Automation Framework

After the Master Image has been deployed and optimized with the Base Image Script Framework (BIS-F) it’s time to manually create a Machine Catalog and Delivery Group in Citrix Cloud. We’re going to use the new Web Studio Console.

Machine catalog setup for the Automation Framework

After a couple of minutes the Machine Catalog is ready and it’s time to create the Delivery Group.

I highly recommend against using the Library option. Just don’t do it!

That was the manual part. What if we could do the same 100% automated? Even running on a Schedule? We’ll that’s what you get as a paid member.

The script uses encrypted password files to authenticate to the Hypervisor and Citrix Cloud. When the deployment is finished, notifications are sent via Webhooks to Teams and Slack + email defined in the Global xml file.

A Machine Catalog and Delivery Group are also automatically created in the Citrix Cloud with access permissions defined in the Global xml file.

Setting up the Core Infrastructure on Azure

This section contains PowerShell scripts to both Create and Delete your Azure Lab.

Before you start there’s some essential information required to be able to run the build script. First locate the name of the subscription you want to use.

Setup core hybrid cloud infrastructure in Azure

Next you need do decide which location you want to setup the Resource Group. See the Azure map above to pick a location close to where you live.

Since you’re going to setup a Site 2 Site VPN to Azure you need to specify your Onprem DNS Servers so that we can automatically join VMs in Azure to your Domain.

To be able to setup a Site 2 Site VPN tunnel to Azure, you’ll need to find the external IP address of your Gateway. There are many solutions available, but the only one I recommend is Ubiquiti Unifi Security Gateway (USG) for $131.99.

From the Unifi Dashboard select Devices – USG and expand WAN1. If the IP Address is an internal one, then you’ll need to switch your ISP modem to Bridge Mode. Most modern modems support this, in my case I had to subscribe to a Fixed IP Address plan.

This image has an empty alt attribute; its file name is wan1.png

To be able to reach your Onprem networks from Azure, you need to specify those subnets. In my example its the Server Network, VM Network and my Datacenter in Canada. $LNGIP1 is your external IP address.

Finally you’ll need to set your Shared Secret Key. That’s the password used on both sides to authenticate and establish the VPN Tunnel.

After running the complete script, it will show the public IP address of your VPN Gateway in Azure.

This image has an empty alt attribute; its file name is image-141.png

Go to Settings – Networks – Create New Network. The configuration is straight forward. Remote Subnets are the Subnets you want to reach in Azure from your Onprem environment.

This image has an empty alt attribute; its file name is image-142.png

That’s it! This should get you up and running pretty fast. You might need to run it a couple of times to get it perfect. To completely delete a Resource Group you can run the following command:

Setting up Active Directory in Azure for the Hybrid Cloud

Reduce the latency caused by sending authentication requests from the Cloud back to AD DS running on-premises.

Setup active directory in azure network for hybrid cloud

I recommend starting small to keep the cost as low as possible. I’ve had great success running a Standard_B1ms with 1 vCPU and 2 GB of memory with a Standard HHD. Thanks to feedback from Daniel Viklund I’ve modified the script to use a 32GB disk instead 127GB, why pay for more?

This image has an empty alt attribute; its file name is image-143.png

Below is generic PowerShell script that will automatically build and join a VM to your Active Directory Domain.

  • Line 36-37 = Local Administrator password.
  • Line 42-44 = Domain to Join, OU and Join User account.
  • Line 48 = Password for the Join User account.
  • Line 53 = DNS Servers

When the deployment is finished the PowerShell script will display the IP address which you need to connect using RDP. Make sure you have setup a GPO prior that allow for RDP, if not you’ll probably be unable to connect.

I’m not going to cover how to install and configure Active Directory through Add Roles and Features, the process is the same as on-premises. However I’m going to give you some tips on how to configure after the Domain Controller has rebooted.

Open Active Directory Sites and Services and create all the IP subnets for you network infrastructure. This will prevent e.g. WVD trying to authenticate against my Domain Controller to Brasil.

Rule of thumb, if you don’t configure ALL your subnets, you have NO CONTROL and slow logons will be guaranteed.

active directory in azure site and services subnets

I’ve always learned that you shouldn’t mess with the NTDS Settings and let Active Directory handle those automatically. Unfortunately that’s not the case! You need to manually add connections to the Domain Controllers you need. E.g. in my example only a connection to Canada was created automatically. Well if that site link goes down, there’s no longer a sync between Azure and my Brasil site.

active directory in azure NTDS Settings

I’ve configured my Active Directory to replicate changes instantly. This is very important in my environment. Read how to configure it here. Finally you’ll need to put the IP Address of your Azure Domain Controller on the top of your DNS Server list.

active directory in azure set DNS servers on virtual network

That’s pretty much it for configuring Active Directory in Azure. Below is the command to check replication status.

active directory in azure repadmin

Using Azure Image Builder to Build a Win 10 Enterprise Multi-Session Master Image

This section is based upon the amazing work of Jim Moyle. Make sure to subscribe to his YouTube Channel before you continue!

Using Azure Image Builder when Building a Hybrid Cloud on Nutanix Community Edition

These posts on Windows Virtual Desktop (WVD) would never have seen the light if it wasn’t for Jim Moyle. I’ll reference to his Github for the code I’m using below.

First, as explain in Episode 2 you’ll need to configure and run the script Once Only Setup.ps1. I’ve changed the following parameters to reflect my location (Available for preview in these regions) and naming standard. Learn how to automatically setup an Azure Lab here.

The rest of Episode 2 and 3 are optional, but highly recommended. What you REALLY want is covered in Episode 4. In that episode you’ll learn how to also install applications during the Azure Image Builder process. At the end of the day, what’s an OS without applications!

You’ll need to edit DeployTemplate.ps1 to include your custom parameters from above + $identityName which you’ll find inside the Resource Group.

Azure Image Builder Managed Identity

Next you’ll need to edit template.json, and search / replace westeurope with your preferred location. When I realized I could simple add any raw code from my Github repo, my head was totally blown away!

Below you can see that I’ve added mRemoteNG, uberAgent, AppVentix and also using my own Windows Update script. Jim mentioned in the video that he had had mixed results with his update code, but mine have never failed.

Azure Image Builder template.json

And that’s it. Kick it off and it will use Azure Image Builder to automatically build and optimize a shiny new Master Image. In the next update to this post on building your Hybrid Cloud, we’ll automatically configure a complete WVD solution using this image and leverage Azure Ephemeral Disk. Stay tuned.

0 Shares

Automation Framework Community Edition

The fastest way to build your lab environment.

Virtual Expo

Friday 24th of September 2021

Leave a Comment