People are notoriously bad at predicting the future. This very much includes experts. In the Forbes article “Why Most Predictions Are So Bad” Philip Tetlock discusses the largest and best-known test of the accuracy of expert predictions which show that any experts would do better if they make random guesses.
HPC users can be bad at predictions as well – specifically at predicting the memory consumption and job runtime required for their workloads. When we compare what users request with what is actually used, it can be significantly more than the actual amount of resources consumed by the job. This very human trait can drive down system utilization and throughput – which are detrimental to HPC environments.
More isn’t necessarily better
Humans are a curious bunch. Always looking for more; whether it’s asking for 2 scoops at the local ice cream parlor to our desire to win the big lottery jackpot, humans often strive to play the bigger, better game – it’s human nature.
Naturally, this desire for more manifests itself in the world of High-performance computing (HPC). Users of HPC environments often submit jobs requesting a specific amount of resources which are far more than the actual amount of resources consumed by the job. Without a good way to estimate the resource use of their applications, even well-seasoned HPC users tend to ask for more resources than actually required to avoid the risk of the application failing due to insufficient resources.
Are we there yet?
If you ask a system administrator what the number one question they get from users, the answer will be “why is my job not running?” The question they really should be asking is “when will my job complete?”. However, to predict the completion time of a given job you need to a) know it’s predicted runtime, and b) and when it will start. And estimating the start time is dependent upon knowing the runtime of every other job in the system.
A helping hand from prediction
For over 25 years, IBM Spectrum LSF has led the industry with innovative capabilities designed to optimize the utilization of valuable HPC assets, while maximizing the productivity of HPC users with powerful, yet easy to use interfaces for job submission and management.
Designed to maximize resource, the IBM Spectrum LSF VM Predictor technical preview is a self-contained VM image which provides IBM Spectrum LSF job resource usage prediction capabilities for job runtime and memory consumption, in addition to GUI-based analysis of prediction results. It uses historical data from the Spectrum LSF environment to model application patterns and the predictions give recommendations to adjust resource requirements specified by HPC users when submitting jobs. This helps takes the guesswork out of the hands of the HPC users, while helping administrators of HPC environments to reduce resource waste and overall degradation of performance due to overzealous requests for resources.
Improving time to insight
IBM Spectrum LSF includes a simulation function that can be used to estimate when a given job will start, and as previously mentioned, to do that with any degree of accuracy requires either the users to accurately specify job runtime or to have the system learn user/application behavior to predict the run time.
Figure 1 compares the user specified runtime (in most cases the user has just taken the system defaults) against LSF Predictor. For this dataset, the LSF Predictor gives a high degree of accuracy. While it isn’t perfect, it provides users with greater confidence of when their work will start, and more importantly, when they can expect results.
Overall system utilization and throughput is increased by better backfill scheduling, and fewer jobs get terminated due to reaching run limits.
Figure 1 Comparison of user specified and predicted job runtime limits
System utilization can further be improved by right sizing job requirements – such as how much memory a job reserves. Figure 2 shows percentage error of user specified memory requirement versus predicted values. What’s interesting to note in this dataset, is that for jobs where the actual memory utilization is small, users frequently specified significantly more memory than was actually used. For example, for jobs that only utilize a few Megabytes of memory, users ask for Gigabytes. Conversely, users seem to have a good handle on specifying required memory when jobs really do use large memory.
By right sizing the requirements, jobs are likely to wait for less time, thus further reducing time to insight.
Figure 2 Percentage error comparing user specified and predicated memory utilization for jobs.
Dollars and Sense?
By having a better understanding of what resources a job really needs, Administrators can optimize hardware procurement for what is actually needed versus what users may think they need.
When running workload on premise users rarely think about the cost of job, or the cost of inefficiencies in the job. But this is of paramount importance when running on the cloud. If your job only needs a few megabytes of memory and you are paying for large memory instances, the cost of that job is a lot more expensive. If you think your job is only going to run for minutes, and it runs for tens of hours, that is a lot more expensive.
Conversely, if you have a high degree of confidence in the resources your workload will require, you can make a much more informed decision on whether you want to wait and run on premise, or pay and run now on the cloud.
Setting up the Spectrum LSF VM Predictor technical preview is straightforward. First, one needs to import job accounting data from Spectrum LSF, then select jobs submitted during a user specified interval for training. The Spectrum LSF VM Predictor technical preview provides three machine learning algorithms to train models. With the model training complete, administrators can easily integrate the inference of resource prediction via the Spectrum LSF esub interface and compare with the resources requested during job submission and adjust as needed. This is all facilitated through RESTful APIs supporting queries and inference against the models built for Spectrum LSF jobs. Interested in trying the Spectrum LSF VM Predictor technical preview? Please contact us via email here.