Nested Virtualization using VMware Workstation, Esxi and Windows 10 on Linux - Part 1

Happy new year all 🎉, It's been a while since my last post. I am currently enrolled in a full-time course hence the lack of time. However, I'll try to do better in regards to getting a schedule in place.

Introduction

Recently, we were tasked with setting up:

• Domain controller vm.


• Web-server vm.
  both using Windows Server 2016 running on VMWare ESXi. Also:


• Two windows 10 clients.


• All of the above running on VMWare Workstation.

The following were additional requirements for the above project:

1. All the clients enrolled into the Domain.


2. A static site hosted on the web server vm.


3. The static site should be accessible on all clients through a custom URL.


4. Create 4 Organisational Units with two users each able to access the client VMS.


5. DNS server running on the domain controller.

Prerequisites

For anyone looking to replicate this project, the following are required:

• A laptop/desktop with a 64-bit operating system (either Windows or Linux).


• Atleast 16GB of RAM.


• 256GB of storage or more (preferrably an SSD).


• Atleast 4 cores with virtualization enabled(check your BIOS options).


• A browser.

Why not Use QEMU/KVM?

As a fellow Linux enthuasist (I use Arch btw), I can already hear the furious keyboard typing about using QEMU/KVM. I use it to host my Windows 11 VM with GPU Passthrough for gaming and 3D tasks. I might write about it eventually.

While I agree that QEMU/KVM is a great tool, I ran into a major issue after setting up the above project. This relating to networking, I couldn't for the life of me get the networking to work properly inside ESXi. I tried setting up a bridge, setting up a virtual network, and even setting up a virtual switch. Nothing seemed to work. I was about to give up when I decided to use VMware workstation.

Yes, its probably a skill issue on my end but I was able to get everything working with VMware workstation in a matter of hours.If you're able to replicate this setup, kindly let me know. I'd love to know how you did it. All that side, lets get to it.

Project Overview

The diagram below provides an overview of the project:

Here we're setting up a Type 2 Hypervisor (VMware Workstation) on a Linux Host then installing three Virtual Machines:

1. Type 1 HyperVisor(VMWare ESXi) -> inside we'll have 2 VMs:
   a. Domain Controller (Windows Server 2016)
   b. Web Server (Windows Server 2016)


2. Windows 10 client 1


3. Windows 10 client 2

The diagram below showcases a high-level view of the Virtual Machines:

Setting up VMware Workstation on Linux

VMware Workstation is a Type 2 Hypervisor that allows you to run multiple virtual machines on a single host. Following the acquisition by Broadcom, there was alot of uncertainty about the future of VMware Workstation. However, they now offer a free license for Personal use for Workstation.

To get started, follow these steps:

1. Download the VMware Workstation installer from the official VMware website. Check the appropriate version for Linux.


2. Log in to the Linux host with the user account that you plan to use with VMware Workstation.


3. Open a terminal interface. For more information, see Opening a command or shell prompt (1003892).


4. Change to root. For example:
   

su root

Note: The command that you use depends on your Linux distribution and configuration.

1. Change directories to the directory that contains the VMware Workstation bundle installer file. The default location is the Download directory.


2. Check if you have the /etc/init.d directory, if not create it before running the installer.
   

sudo mkdir /etc/init.d

1. 
   

   Run the appropriate Workstation installer file for the host system.

   
   
   

   For example:

   
   
   

   sh VMware-workstation-Full-_xxxx-xxxx_._architecture_.bundle [--_option]_

   
   
   

   Where:

   
   
   
   
   
   • 
     xxxx-xxxx is the version and build numbers
   
   
   • 
     architecture is i386 or x86_64
     
   
   
   • 
     _option_ is a command line option.
   
   
   
   

For further information, check the linux install docs.

Arch-Linux install

VMware relies on compiling it's own kernel modules, due to the bleeding edge nature of Arch -Linux, you may need to compile your own kernel modules. Incase you run into the following error:

This is a well-known issue with running VMware on Linux - compiling your own kernel modules. This isn't much of a problem for seasoned users but for others it might be a deal breaker. This repo will save you hours of searching google.

On Arch you can install VMware Workstation on Arch-Linux using the AUR or Chaotic AUR:

• Install yay
  

  sudo pacman -S yay

• 
  

  Ensure all packages are up too date by running
  
  

  
  

  yay
  
  

  
  
  



• Install VMware Workstation
  
  

  yay -S vmware-workstation

• Once completed launch using your application or run the following command in the terminal:
  

vmware

In-case you have an issue with the networking or get an error pop-up, start as root:

sudo vmware

Running ESXi on VMWare

Download the ESXi iso from the Broadcom website, the latest version is 8. However if your hardware is not compatible with the latest version, you can download version 7 here.

Installation is similar other VMs:

1. Create a New Virtual Machine.


2. Use the Typical Configuration.

1. Select the ESXi iso.

1. Choose the location to save the VM files.

1. Enter the Maximum disk size. In my case I used 60GB (Note the recommended is 142GB).

1. Lets Customize the Hardware options:
   i. Change the allocated memory to 16GB.(Adjust according to your host specifications).
   ii. Change the CPU cores to 12.
   iii. Change the Network Adapter to Custom. I'm using /dev/vmnet8. (This is important for connecting the client VMs)

1. Finish the setup and power on the VM.

Installing ESXi

Once Esxi powers on, you'll be greeted with the following screen:

Press Enter to continue, next, you'll be prompted to accept the EULA, press F11 to accept and continue.

The next screen will prompt you to select the disk to install ESXi on. Select the disk and press Enter.

You'll be prompted to enter a password for the root account. Enter the password and press Enter.

Finally you'll be prompted to press F11 confirm Installation of ESXi. Press F11 to install ESXi.

Once the installation is complete, you'll be prompted to remove the installation media. Press Enter to reboot the system.

Once the system reboots, you'll be greeted with the following screen once booting is complete:

Note: your IP address will be displayed on the screen. You can access the ESXi web interface by entering the IP address in your browser.

Due to SSL issues, you'll get a warning, click on advanced and proceed.

The login screen will be displayed, enter the root username and password you created during installation.

You'll be greeted with the following screen:

Configuring a new datastore

Before we can create new VMs, we need to create a new datastore in Esxi. But we need to first add a storage device to the VM in VMware Workstation.

1. Open VMware Workstation and select the ESXi VM.


2. Click on Edit virtual machine settings.


3. Click on Add.


4. Select Hard Disk and click Next.


5. Select SATA and click Next. If you have m.2 or NVMe drives, select the appropriate option.


6. Set the disk size to 60GB and click Next. (Note: You can adjust the size according to your needs).

Boot the VM and login to the ESXi web interface. Click on the storage tab > Adapters and you'll see the new storage device.

Now we can create a new datastore:

1. Click on the Datastores tab.


2. Click on New Datastore.

1. Select Create a new VMFS datastore and click Next.


2. Select the storage device and click Next.

1. Enter a name for the datastore and click Next.


2. Select Use full disk and click Next.

1. Confirm the settings and click Finish.

1. The new datastore will be created.

Conclusion

In this Part 1 of the series, we've covered:

• Setting up VMware Workstation on Linux.


• Installing ESXi on VMware Workstation.


• Configuring a new datastore in ESXi.

In the next part, we'll cover:

• Creating a new VM in ESXi.


• Installing Windows Server 2016 on the new VM.


• Configuring the Domain Controller and Web Server VMs.


• Setting up the Windows 10 clients.


• Enrolling the clients into the Domain.


• Setting up the DNS server on the Domain Controller.

Stay tuned for Part 2. Feel free to ask any questions or provide feedback. I'll be happy to help. Happy learning!

References

• VMware Workstation Documentation


• VMware ESXi Documentation


• VMware Workstation Free License


• VMware Workstation Linux Install