In a recent blog entry, VMware’s HPC guru Josh Simons provides an overview of the work that his group has been undertaking in recent months. He presents a sneak peak of some of the most significant HPC-relevant developments, and promises to offer a deeper dive in the weeks ahead.
Last spring, Simons’ group put together a four-node InfiniBand HPC cluster in their Cambridge, Mass., lab. The test system includes four HP DL380p Gen8 servers, each with 128 GB memory and two Intel 3.0GHz E5-2667 eight-core processors, plus Mellanox ConnectX3 cards that support both FDR (56 Gb/s) InfiniBand and 40 Gb/s RoCE. A 12-port Mellanox switch connects the nodes.
The blog details the team’s experiments with InfiniBand, RoCE, and SR-IOV, and Phi. VMware continues to decrease, and even eliminate, the latency hit associated with some types of virtualization. In one set of experiments, using an engineering build of ESX eliminates the 15-20 percent latency overhead for very small messages as compared to using ESX 5.5u1.
The group has also been working with partner Mellanox to evaluate an early version of InfiniBand SR-IOV support for ESX 5.5. “Unlike passthrough mode, which makes an entire device directly visible within a virtual machine, SR-IOV (Single Root I/O Virtualization) allows single hardware devices to appear as multiple, virtual devices — each of which can be shared with a different VM…. The PF (physical function) driver is an early version provided to us by Mellanox and the VF (virtual function) driver is included in the latest releases of the Mellanox OFED distribution,” he writes.
“One of the primary HPC use-cases for SR-IOV is to allow multiple VMs on a host to access high-performance cluster file systems like Lustre and GPFS by sharing the single, physical InfiniBand connection between the host and storage system,” Simons adds. Those interested in test driving this capability, can stop by the EMC booth at SC14 in New Orleans.
For the Phi curious, VMware ran some Intel Xeon Phi performance benchmarks on a test system provided by Intel. Using a prototype ESX build and VM Direct Path I/O (passthrough), they saw nearly identical performance relative to bare metal in both double and single precision runs. The graph below shows virtual and bare metal performance using two different Intel programming models (pragma and native). The team was forced to use an engineering build of ESX because Xeon Phi is not usable with the shipping versions of ESX 5.5 on account of PCI limitations.
Simons reports that he will be attending SC14 in New Orleans, as will his colleagues Andrew Nelson and Matt Herreras. You might find them at the EMC booth demonstrating the SR-IOV use case as well as VMware’s self-provisioning approach to virtual HPC clusters in a vRA private cloud.