Conducting computational experiments often requires extensive but oftentimes sporadic compute power. The need for vast resources for shorter bursts of high activity makes building statistical models of systems and related experiments seem like an ideal fit for cloud computing. However, questions remain about the practicality of using cloud resources, especially public clouds, for such work.

A team from St. Andrews University recently tackled a series of questions about how suitable cloud computing and virtualized hardware is for running time-sensitive experiments. When approaching the matter, Ian Gent and Lars Kotthoff imagined that CPU time would not be reliable and that they would not be able to confirm consistent results. For Kotthoff and Gent, it was especially critical to be able to ensure reproducability, thus this was a primary component of their methodology.

The outcome of their investigation demonstrated that while their suspicions were correct in cases where there were several virtual machines running on the same physical hardware, “there is no inherent variation introduced by using virtualized hardware compared to non-virtualized hardware.”

These findings were worth investigating further, so we spent some time talking with Lars Kotthoff from St. Andrews University. Specifically, we wanted to find out about the reliability and reproducability of results of key experiments and how these were priced, especially against the only other alternative–physical hardware.

Some of Kotthoff’s research into the viability of running time sensitive applications on Amazon’s cloud has come from Amazon Web Services. The company has also backed other research projects, including his work on prototyping a system based on the Minion constraint solver that automatically splits and distributes large problems across different machines.

On that note, it should come as no surprise that Kotthoff and Gent were conducting their “experiment on experiments” using Amazon Web Services.

HPCc: Can you describe what led up to this study? What were some of the perceived challenges for running scientific (and computational) experiments on virtualized hardware and were these expected problems present?

Kotthoff: The initial motivation for this study was that we realized that it took us a long time to set up experiments, run them, evaluate them and make sure that they are reproducible. Especially the last point was a particular headache for us because we use sets of machines with different hardware and configurations. We also observed large variations in runtime of the same experiment on the same
machine because of the way CPU caches are shared among several cores.

The way we tried to tackle both of these problems was by running experiments several times and then using statistical methods to evaluate them. This has two main repercussions. First, we are going to need more resources to run experiments. Second, it becomes less important to run them on a set of machines
with the same hardware and software configuration. This is how we had the idea to run experiments in the cloud.

We had previously observed that the clocks of virtual machines running on desktop virtualisation software such as VMWare or VirtualBox were very unreliable. This is what prompted us to do this investigation.

HPCc: Along those lines, can an you provide something of a quick synopsis of your main findings?

Kotthoff:  We found that only for the small types of virtual machines in the Amazon cloud getting reliable CPU timings was a significant problem. For larger types, the reliability becomes as good as on physical hardware or even better. We do of course not know the actual configuration of the hardware of the cloud computing provider, but it is not hard to guess that for the smaller virtual machine type more are running on the same physical hardware and therefore the virtualization overhead is higher and affects the reported CPU times in the virtual machine. The largest type of virtual machine probably corresponds to a physical machine and introduces no measurable additional variability.

HPCc: Describe the reliability and reproducibility angles to your research—what we the outcomes and what does this mean for researchers considering using virtualized hardware?

Kotthoff: Reliability and reproducibility of results is essential for our research. If people compare different methods, there is almost always an empirical evaluation as well. The justification for new research is almost always that it decreases the time needed to solve a problem. If however the measured times are not reliable, the research findings are jeopardised.

This is especially crucial when implementing complex systems that solve artificial intelligence problems. During the implementation, many design decisions have to be made and algorithms have to be chosen. These choices are usually justified by empirical evaluation – either during the implementation itself or based on results reported in the literature.

Our results show that there is no inherent disadvantage to using virtualized resources for such experiments as long as the results are carefully evaluated and statistical methods used to identify outliers and establish confidence in the conclusions drawn from the results.

We believe that using a cloud to run experiments would, in terms of reproducibility of results, take our research to the next level. We can package the entire environment of the experiment as a virtual machine image, make it available to the public and reference it in publications. This makes reproducing our results much easier for other people.

Currently, we describe the hardware and sometimes the software used in experiments, but unless other people happen to have the same hardware, there is a significant barrier to being able to reproduce the exact experiments we ran.

HPCc: Do you think resources that are tailored for computationally intensive workloads like Amazon’s recent HPC offering, Cluster Compute Instances, will solve some of the problems you discuss?

Kotthoff:  That certainly seems to be case from what we saw in our experiments. It might however be “accidental” as virtual machines tailored for computationally intensive workloads tend to run exclusively on a physical machine. If many of those virtual machines with high specifications would run on for example on a single supercomputer, the original problem of a large variability in CPU times might arise again.

HPCc: With the above question in mind, what problems are going to remain semi-permanent hurdles for CPU-heavy workloads in the cloud? Is there hope on the horizon that it will someday compete with native/physical hardware?

Kotthoff:  In my opinion, the major problem at the moment is the cost associated with running large computational experiments in the cloud. The largest type of virtual machine in the Amazon cloud is, in terms of variability of reported CPU times, as good as physical hardware. The cost of running it is relatively high though. For a research group that regularly runs large-scale computational experiments, it would almost always be cheaper to buy physical hardware themselves.

I should point out that we are not so much concerned with the actual performance we get as long as the variability is low. While the virtual machine instances we used did not seem to be significantly slower than physical hardware in general, we did not evaluate this aspect.

HPCc: In what ways (or cases) is it not a good idea to run acomputationally-intensive task on cloud-based resources?

Kotthoff: If the task cannot easily be parallelised and takes a long time to run. Cloud computing has disadvantages over physical hardware in this scenario; for example virtual machines might be shut down without warning and there is no way to access the disk of the terminated instance to recover results. Using cloud-based resources is also a question of the available funds; if there is only a small amount of money available, it might be better to buy and run the experiment on physical hardware.

HPCc: Taking this in another direction, what is the best type of case for running CPU-intensive tasks on cloud resources?

Kotthoff: The best case is probably running a lot of experiments or experiments that can be distributed across many virtual machines easily when there are not enough physical resources available, e.g. impending paper deadline. Provided that the experiment can be made to run in a virtual machine, it is easy to distribute it across virtual machines in the cloud.

You can find out more about Gent and Kotthoff’s experiments here.