Virtualization for the win

Hi! I haven't seen much about using VM for GPU workloads, and I haven't seen anything on this forum. So, I though I would write a bit about it here, because I think it's great.

Summary
It works. No performance penalty found, yet.
TL;DR version at the bottom.

About me
I work pretty much every day with designing/installing/maintaining small to almost-enterprise size VMware virtualization systems, using almost exclusively HPE hardware. It's not ALL I do, but it's a major part of my work.
My posts here are my own and are not endorsed, or even known, by my employer, HPE or VMware.

Why virtualize?
(Warning, some VMware marketspeek included)
Virtualization, generally, provides the following benefits:

• Flexibility
  

Virtual guest can be moved around to different hardware and physical locations without downtime
Resources can be assigned from a pool, rather than beeing a fixed size decided at instalation time

• Agility
  

Resources can be quickly changed; CPU/memory can be hot added
Network redesign can be done without affecting the guests
VM can be deployed "instantly", without having to order new hardware

• Availability
  

In case of a host (hardware) failure, guest can be automatically restarted on other hosts
Much reduced hardware in the guests reduces the amount of drivers that can cause problems

In this case, using ESX to host guest running oclHashcat, the following benefits are also realized:

• Hardware utilization
  

Using a physical box just to power GPU is a waste of space, power and CPU resources. Why not use the box for other purposes at the same time? (Like running a hashtopus installation)

• Maintenance made easy
  

It is a known issue of all (?) OS:es that AMD and nVidia drivers cause each other problems. Using virtualization, you can install separate guests for AMD and nVidia GPU, with the right drivers in each without conflicts. Or why not have one guest per GPU?
OS, driver and hashcat updates may cause problems or performance issues. Resolution: power down guest, snapshot, power up and perform update(s), test. If issues occur, revert snapshot. In case of a multi-GPU box, do this for one VM with one GPU. When the update procedure is finalized, repeat it on the other guests; downtime is greatly reduced.

• Different versions
  

Different versions of hashcat may perform better or not at all with some GPU. Install as many guests as you need, with different versions that fits your need. Move the GPU around to the guest that best fits the situation. Run different versions for different GPU, all in the same box.

Why NOT virtualize?
The oldschool way of thinking is that "if you need performance, you need dedicated hardware". This is not true anymore; first, hypervisors have matured and does not cause much overhead. Second, most overhead from virtualization is on the CPU part. CPU today are hardly ever bottlenecks - and if they are, you can add more cores. The only instance you may actually require dedicated hardware is if your application needs very very high single-thread performance. Well that and if your idiot vendor says virtualization is unsupported...

A real reason not to virtualize is increased complexity. Yes, virtualization makes the system more complex, it adds another layer of "stuff" that you need to plan, maintain and troubleshoot. If you are unwilling to learn something new, buy hardware.

Design
The goal here is to install a vShere host with one or more GPU. These GPU are then assigned to VM using PCI passthrough. The operating system is ESXi; it is available for free from vmware.com. You need to register and checkout a license, or it will not power on guests after 60 days. The free license is limited to 8 vCPU/guest and cannot be connected to VMware vCenter (=> no cluster).
ESXi only supports/contains drivers for a subset of hardware. It can be installed on consumer hardware if the hardware is built from supported chips; unsupported hardware may also work with community drivers. Pretty much all ready-made servers today supports ESXi; but a lot of servers to not support high-power GPU, or they have insufficient room for large GPU coolers, or they do not have (enough) PCI-e power.
There are a lot of options for hardware, you need to research a bit to find something that works. I use an HPE Ml350p Gen8: it can fit two 2-slot GPU/CPU but you need a special power cable for PCI-e power. Fortunatelly, the power socket was compatible with Corsair modular PSU power cables.


Preparation
Download ESXi. If you have a HPE/Dell server, download the custom ISO or you may not have the drivers you need. You can add drivers to the installtion CD but you cannot add drivers on the fly during the installation.
Fit the GPU and make sure it has enough power.
When you (later) add the GPU to your VM, it will reserve 100% RAM that you configured to your VM. If you need 4 gig RAM/VM and 8 VM, you need at least 4gig x 8 + (overhead + ESXi) RAM ~= 34gig ram. You are not constrained to whole GB here, you may assign 3750MB RAM instead, if you want to.
(Optional:
I recommend setting the VM advanced option "mem.ShareForceSalting" to 0, this enables me to overallocate RAM better (it enables RAM dedup between guests). It won't make a difference on guest with passthrough, though.
Since these VM will be in an isolated environment (I hope), I would turn off ASLR to increase guest RAM dedup.
Overallocating CPU is OK, but will notice increased latency for certain applications. Web server generally do not perform well on overallocated CPU - you may apply this fix to the VM if you think you need it: http://kb.vmware.com/kb/1018276. But consider the fact that you will be burning away some CPU that may be used for other VM instead. Also, those idle loops are counted towards the guest resources shares - you may end up with using half your CPU for nothing and not be eligible for CPU when you need it. SQL servers are not bothered by CPU latency; performance decreases with less CPU available, but it will use those CPU cycles effectively.
If you REALLY want to reserve an entire core (or several) for your guest, change CPU reserve to 100% and set the advanced option "monitor_control.halt_desched" to false. Your VM will now never be required to share it's core. Also, if you have hyperthreading active (you should), you may ensure that the other thread is not used by setting Hyperthreaded core sharing: none. This ensures that your VM will have 100% of the core cache and CPU time.
)

Installation
Install ESXi.
Connect to ESXi using a web browser and download vSphere Client (the client is version specific and the link is to and Internet URL).
Configure time and date. This is important: when a guest is restarted or a snapshot reverted, the guest time is set to whatever ESXi has. Is is a reoccuring problem that this step is forgotten, and VM start up with the wrong time/date.
In vSphere client, select your host, tab Configuration, option Advanced Settings. Enable PCI passthrough on your GPU. Reboot.
Install a VM to run oclHashcat. I use Ubuntu according to this guide: https://hashcat.net/wiki/doku.php?id=linux_server_howto Thus, I have a 64bit-only installation, version 14.04, with 'fglrx' drivers.
Install open-vm-tools; trust me, you want it. You will then be able to right-click "shutdown" instead of "poweroff" and you will see the IP addresses in vSphere client.
When the VM is installed and updated and works like you want it to, shut it down and edit the virtual hardware.
Add the PCI device (your GPU).
Start your MV, get oclHashcat.
Done!

Benchmarking
This is from my server, running ESXi 5.5 on an HPE ML350p Gen8, and a second-hand Radeon 6970. The host is running a total of 12 VM with total 17 vCPU, on a single 4-core Intel E5-2609 (no hyperthreading). 48gig RAM with 60gig RAM assigned to guests.

CPU usage during benchmark was average ~250MHz, or 10% of core speed. That's a lot of CPU that can be used for other things... It's hash dependant though, some hashes actually used 97% CPU.
Performance of the 6970 I find around the Internets, is MD5=5878MH/s and WPA2≃82kH/s.
I was expecting to get at least 90% of that speed - but the benchmark got _more_ than others have posted for WPA2, and the same (almost) for MD5. It seems performance of hashcat is totally unaffected by beeing virtualized.
My performance is MD5=5698MH/s and WPA2=95kH/s.


Extras
I use hashtopus to control this VM. To start the agent, I use this little script:
The agent execution command line (set in hashtopus agent config) must contain "--gpu-temp-disable" for the fan speed config to work.
Here's a quick reference to the screen command: http://aperiodic.net/screen/quick_reference

You may check GPU temp and fan speed this way:
aticonfig --odgt --pplib-cmd "get fanspeed 0"

Conclusion
Running oclHashcat in a VM instead of on hardware, can be a viable way to optimize hardware usage and ease management, IF you have the knowledge and time to set it up. So far, it seems there is not inpact on performance, though further testing on multiple and more recent GPU are needed to establish this as a fact.


TL;DR:
1. Install ESXi
2. Activate PCI Passthrough, reboot
3. Install a virtual guest as usual - add open-vm-tools
4. Shutdown VM and add GPU to VM
5. Start VM, install oclHashcat
6. ?
7. Profit!

Try it, you'll like it!