LONDON, Nov. 23, 2021 — YellowDog, a UK cloud workload management company, has built a supercomputer in the cloud in just 33 minutes. The announcement marks a game changing step forward as, for the first time ever, thousands of servers across Europe and North America were able to work seamlessly together on a single problem. This releases big computing tasks from the constraints of a particular system or data center, dramatically reducing the cost of high performance computing and speeding up critical innovation, from months to hours.
This breakthrough means the cloud is now even more accessible and scalable for key areas of research, such as drug discovery and artificial intelligence. While high performance computing has always played an integral role in furthering important areas of exploration, supercomputers can take years to fund, with considerable investment required to both build and maintain the infrastructure. This prevents many organizations from realizing its benefits.
YellowDog has introduced a new reality, where growing computational need can be met instantly in the cloud. In sixty-five minutes, it ran a popular drug discovery application on 3.2 million virtual processor cores, as a single cluster. To provision one million pre-emptible instances took just seven minutes.
YellowDog also offers organizations the choice to access supercomputer performance with minimal environmental impact. With significant sustainability benefits, the technology not only provisions the exact resources needed, when they’re needed, but it also enables the selection of data centers based on renewable energy generation. This means high performance computing becomes much more efficient and computing tasks can be placed where clean energy is most available. This capability to move to where the sun is the brightest, or the wind the strongest, has never before been achieved.
“This number of concurrent cores, stretching across the Atlantic, represents a huge breakthrough. It reduces the time, cost and, so importantly, carbon impact of research into some of the most important societal issues,” said Simon Ponsford, CEO of YellowDog. “Our platform, through its ease of use, lets organizations focus on the problem itself, not on being an expert in exploiting the potential of the cloud. The realms of possibility are no longer bound by coordinating computing resources, but by the vision of the human mind.”
“In the past year alone, we have seen not only the continuation of the global pandemic, but also significant displacement caused by extreme weather events,” continued Ponsford. “The ability to solve these complex and multifaceted issues hinges on collecting, processing, and generating insight from massive data sets. This demonstrates the growing importance of high performance computing in the efforts to advance the fields of science, engineering, and technology.”
YellowDog, founded in 2015 in Bristol, UK, helps businesses scale across the cloud like no other. The company’s cloud workload management platform is used all over the world to accelerate compute-intensive applications.
The YellowDog Platform removes prohibitive barriers to entry, so organizations of all sizes can solve some of the most immediate threats faced by society including the global pandemic, the multiple and severe weather events, and the search for renewable energy. YellowDog works with cloud partners including Amazon Web Services, Microsoft Azure, Google Cloud Platform, Oracle Cloud Infrastructure, and Alibaba Cloud.
About the Run
To achieve this run, YellowDog launched 46,733 pre-emptible instances, utilizing 24 AWS EC2 Fleets and 8 instance types, provisioning a total of 3.2 million concurrent, virtual CPU cores to run a drug discovery application. The job consisted of over 200,000 discrete tasks, which were spread across Europe and North America.
Overall, the run was completed within 65 minutes. It took just 7 minutes to provision 1,000,000 cores. Not only was it fast, but the actual utilization of the instances kept track with provisioning. This meant that instances were made ready in step with those being used, maximizing utilization efficiency.