How Nvidia Could Use $700M Run.ai Acquisition for AI Consumption

By Agam Shah

May 6, 2024

Nvidia is touching $2 trillion in market cap purely on the brute force of its GPU sales, and there’s room for the company to grow with software. The company hopes to fill a big software gap with an agreement to acquire Run.ai for a whopping $700 million.

AI deployments are getting larger and more complex and spreading across more GPUs and accelerators. Run.ai provides the middleware to orchestrate and manage these deployments and ensure resources aren’t wasted.

The middleware includes tools to speed up workloads, manage resources, and ensure errors don’t bring down entire AI or high-performance computing operations. The middleware runs on a Kubernetes layer to virtualize AI workloads on GPUs.

Nvidia’s GPUs are hot-ticket items in the AI rush and are available to customers through all major cloud providers. 

The Run.ai acquisition will help Nvidia build its independent cloud service without building its data centers. 

(Source: run:al)

Nvidia wants to create its own network of GPUs and DGX systems across all major cloud providers. Run.ai’s middleware will provide an important hook for customers to reach more GPUs available online or on-premise. 

“Run:ai enables enterprise customers to manage and optimize their compute infrastructure, whether on-premise, in the cloud, or in hybrid environments,” Nvidia said in a blog entry.

At the top of Nvidia’s software stack is AI Enterprise, which includes the programming, deployment, and other tools. It has 300 libraries and 600 models.

The stack includes the proprietary CUDA parallel programming framework, compilers, AI large-language models, microservices, and other tools. The toolset also includes container toolkits, but Run.ai’s middleware supports open-source large language model deployments.

Nvidia GPUs are cloud-native, and Google, Amazon, and Oracle have strong Kubernetes stacks. Nvidia already has its own container runtime as a Kubernetes plugin for GPU devices, but Run.ai will bring more granular control to AI container management and orchestration. As a result, Nvidia will use more such tools rather than relying entirely on cloud provider configurations.

The Problem

Multiple GPU allocations for AI tasks are still not a straightforward deal. Nvidia’s GPUs are in its DGX server boxes deployed on premises of all major cloud providers.

Nvidia’s Triton Inference Server automatically distributes inferencing workloads across multiple GPUs in a configuration, but there are problems.

AI workloads also need Python code to point to cloud operators, only after which AI workloads will be executed on Nvidia GPUs within cloud services.

Nvidia is thinking ahead with its acquisition of Run.ai. The company wants to reduce its reliance on cloud operators — it’s one more step to lock down customers to its software stack. Customers can rent GPU time in the cloud and then go to Nvidia for all software needs.

At the same time, it fills a major need for Nvidia to provide a complete software stack.

Preparing for an AI Future

Currently, AI training and inferencing are mostly done on GPUs in data centers, but this will change in a few years.

Over time, AI — specifically inferencing — will be offloaded from data centers to the edge. AI PCs are already being used for inferencing.

The current state of AI processing, with power-hungry GPUs, is unsustainable. It is the same problem that crypto faced — tons of hungry GPUs running complex math at full speed and power to mine results quickly.

Nvidia has tried to reduce the power consumption of its chips with Blackwell. But the company is adding software to the equation, and Run.ai will help orchestrate workloads across GPUs and further down the network on AI PCs and edge devices.

AI processing will also be done on various waypoints, such as telecom chips, as it travels through wireless and wired networks. However, more demanding AI workloads will remain on servers with GPUs, while the less demanding workloads will offload to the edge.

Companies that include Rescale are already working with companies for high-priority tasks to remain on GPUs in the cloud, while low-priority tasks are delivered to low-end chips somewhere else. Run.ai’s orchestration can manage that via a strong combination of speed, power-efficiency, and utilization of resources.

The Run:ai Stack

One small error can bring down an entire AI operation. Run.ai’s stack has the three operations layers to prevent such mishaps, and deliver safe and efficient deployment.

The lowest layer is the AI Cluster Engine, which makes sure GPUs are well utilized and operate efficiently.

The engine provides granular insight into the entire AI stack, including the compute nodes, users, and workloads running on it. Companies can prioritize specific tasks, and make sure idle resources are utilized.

If a GPU seems busy, Run.ai will reallocate resources. It can also assign GPU quotas based on user or breakdown resources within GPUs to ensure proper allocation.

The second layer — called the Control Plane Engine — provides granular visibility on the resources used in the Cluster Engine, and cluster management tools to ensure metrics are being met. It also sets up policy on access control, resource management and workloads. It also has reporting tools.

The top layers include the API and development tools. The dev tools also support open-source models.

Falling in Line with Nvidia’s New GPUs

The big wild card is whether Run.ai will tap into some RAS (reliability, availability, and serviceability) features in Nvidia’s latest Blackwell GPUs. Blackwell GPUs were introduced in March and include more fine-grained features to ensure the chip runs predictably.

The GPUs have on-chip software to point out the healthy and unhealthy GPU nodes. “We’re looking at the trail of data from all those GPUs, monitoring thousands of data points every second to see how the job can get optimally done,” said Charlie Boyle, the vice president and general manager of DGX Systems unit at Nvidia, in a March interview. 

Run.ai’s efficiency might improve if it could tap into metrics or information from Blackwell. That kind of fine-grained reporting could go a big way in ensuring that AI tasks run smoothly.

Nvidia’s Acquisition History

Nvidia pulled in revenue of $22.1 billion in the most recent quarter, up by a whopping 265% from the same quarter a year ago.  Data center revenue was a whopping $18.4 billion.

The company is generating software revenue through the subscription model, and ultimately hopes it will become a multi-billion dollar market. The Run.ai acquisition will fit into that goal. 

Nvidia swung big with the failed acquisition attempt of ARM, which was way before the company became a $2 trillion behemoth. The ARM acquisition was blocked due to monopoly and regulatory concerns, but had the acquisition succeeded, the chip maker would have dominated the CPU and GPU market. ARM already dominates the mobile market and is making inroads in the server and PC markets.

The chip maker in 2011 paid $367 million for Icera, a software modem maker, which turned out to be a dud. Nvidia ultimately abandoned its chase of the mobile phone market, and the Icera product was dropped.

Subscribe to HPCwire's Weekly Update!

Be the most informed person in the room! Stay ahead of the tech trends with industry updates delivered to you every week!

ISC 2024 Takeaways: Love for Top500, Extending HPC Systems, and Media Bashing

May 23, 2024

The ISC High Performance show is typically about time-to-science, but breakout sessions also focused on Europe's tech sovereignty, server infrastructure, storage, throughput, and new computing technologies. This round Read more…

HPC Pioneer Gordon Bell Passed Away

May 22, 2024

Legendary computer scientist Gordon Bell passed away last Friday at his home in Coronado, CA. He was 89. The New York Times has a nice tribute piece. A long-time pioneer with Digital Equipment Corp, he pushed hard for de Read more…

ISC 2024 — A Few Quantum Gems and Slides from a Packed QC Agenda

May 22, 2024

If you were looking for quantum computing content, ISC 2024 was a good place to be last week — there were around 20 quantum computing related sessions. QC even earned a slide in Kathy Yelick’s opening keynote — Bey Read more…

Atos Outlines Plans to Get Acquired, and a Path Forward

May 21, 2024

Atos – via its subsidiary Eviden – is the second major supercomputer maker outside of HPE, while others have largely dropped out. The lack of integrators and Atos' financial turmoil have the HPC market worried. If Atos goes under, HPE will be the only major option for building large-scale systems. Read more…

Core42 Is Building Its 172 Million-core AI Supercomputer in Texas

May 20, 2024

UAE-based Core42 is building an AI supercomputer with 172 million cores which will become operational later this year. The system, Condor Galaxy 3, was announced earlier this year and will have 192 nodes with Cerebras Read more…

Google Announces Sixth-generation AI Chip, a TPU Called Trillium

May 17, 2024

On Tuesday May 14th, Google announced its sixth-generation TPU (tensor processing unit) called Trillium.  The chip, essentially a TPU v6, is the company's latest weapon in the AI battle with GPU maker Nvidia and clou Read more…

ISC 2024 Takeaways: Love for Top500, Extending HPC Systems, and Media Bashing

May 23, 2024

The ISC High Performance show is typically about time-to-science, but breakout sessions also focused on Europe's tech sovereignty, server infrastructure, storag Read more…

ISC 2024 — A Few Quantum Gems and Slides from a Packed QC Agenda

May 22, 2024

If you were looking for quantum computing content, ISC 2024 was a good place to be last week — there were around 20 quantum computing related sessions. QC eve Read more…

Atos Outlines Plans to Get Acquired, and a Path Forward

May 21, 2024

Atos – via its subsidiary Eviden – is the second major supercomputer maker outside of HPE, while others have largely dropped out. The lack of integrators and Atos' financial turmoil have the HPC market worried. If Atos goes under, HPE will be the only major option for building large-scale systems. Read more…

Google Announces Sixth-generation AI Chip, a TPU Called Trillium

May 17, 2024

On Tuesday May 14th, Google announced its sixth-generation TPU (tensor processing unit) called Trillium.  The chip, essentially a TPU v6, is the company's l Read more…

Europe’s Race towards Quantum-HPC Integration and Quantum Advantage

May 16, 2024

What an interesting panel, Quantum Advantage — Where are We and What is Needed? While the panelists looked slightly weary — their’s was, after all, one of Read more…

The Future of AI in Science

May 15, 2024

AI is one of the most transformative and valuable scientific tools ever developed. By harnessing vast amounts of data and computational power, AI systems can un Read more…

Some Reasons Why Aurora Didn’t Take First Place in the Top500 List

May 15, 2024

The makers of the Aurora supercomputer, which is housed at the Argonne National Laboratory, gave some reasons why the system didn't make the top spot on the Top Read more…

ISC 2024 Keynote: High-precision Computing Will Be a Foundation for AI Models

May 15, 2024

Some scientific computing applications cannot sacrifice accuracy and will always require high-precision computing. Therefore, conventional high-performance c Read more…

Synopsys Eats Ansys: Does HPC Get Indigestion?

February 8, 2024

Recently, it was announced that Synopsys is buying HPC tool developer Ansys. Started in Pittsburgh, Pa., in 1970 as Swanson Analysis Systems, Inc. (SASI) by John Swanson (and eventually renamed), Ansys serves the CAE (Computer Aided Engineering)/multiphysics engineering simulation market. Read more…

Nvidia H100: Are 550,000 GPUs Enough for This Year?

August 17, 2023

The GPU Squeeze continues to place a premium on Nvidia H100 GPUs. In a recent Financial Times article, Nvidia reports that it expects to ship 550,000 of its lat Read more…

Comparing NVIDIA A100 and NVIDIA L40S: Which GPU is Ideal for AI and Graphics-Intensive Workloads?

October 30, 2023

With long lead times for the NVIDIA H100 and A100 GPUs, many organizations are looking at the new NVIDIA L40S GPU, which it’s a new GPU optimized for AI and g Read more…

Atos Outlines Plans to Get Acquired, and a Path Forward

May 21, 2024

Atos – via its subsidiary Eviden – is the second major supercomputer maker outside of HPE, while others have largely dropped out. The lack of integrators and Atos' financial turmoil have the HPC market worried. If Atos goes under, HPE will be the only major option for building large-scale systems. Read more…

Choosing the Right GPU for LLM Inference and Training

December 11, 2023

Accelerating the training and inference processes of deep learning models is crucial for unleashing their true potential and NVIDIA GPUs have emerged as a game- Read more…

AMD MI3000A

How AMD May Get Across the CUDA Moat

October 5, 2023

When discussing GenAI, the term "GPU" almost always enters the conversation and the topic often moves toward performance and access. Interestingly, the word "GPU" is assumed to mean "Nvidia" products. (As an aside, the popular Nvidia hardware used in GenAI are not technically... Read more…

Nvidia’s New Blackwell GPU Can Train AI Models with Trillions of Parameters

March 18, 2024

Nvidia's latest and fastest GPU, codenamed Blackwell, is here and will underpin the company's AI plans this year. The chip offers performance improvements from Read more…

Some Reasons Why Aurora Didn’t Take First Place in the Top500 List

May 15, 2024

The makers of the Aurora supercomputer, which is housed at the Argonne National Laboratory, gave some reasons why the system didn't make the top spot on the Top Read more…

Leading Solution Providers

Contributors

Shutterstock 1606064203

Meta’s Zuckerberg Puts Its AI Future in the Hands of 600,000 GPUs

January 25, 2024

In under two minutes, Meta's CEO, Mark Zuckerberg, laid out the company's AI plans, which included a plan to build an artificial intelligence system with the eq Read more…

Eyes on the Quantum Prize – D-Wave Says its Time is Now

January 30, 2024

Early quantum computing pioneer D-Wave again asserted – that at least for D-Wave – the commercial quantum era has begun. Speaking at its first in-person Ana Read more…

The GenAI Datacenter Squeeze Is Here

February 1, 2024

The immediate effect of the GenAI GPU Squeeze was to reduce availability, either direct purchase or cloud access, increase cost, and push demand through the roof. A secondary issue has been developing over the last several years. Even though your organization secured several racks... Read more…

Shutterstock 1285747942

AMD’s Horsepower-packed MI300X GPU Beats Nvidia’s Upcoming H200

December 7, 2023

AMD and Nvidia are locked in an AI performance battle – much like the gaming GPU performance clash the companies have waged for decades. AMD has claimed it Read more…

The NASA Black Hole Plunge

May 7, 2024

We have all thought about it. No one has done it, but now, thanks to HPC, we see what it looks like. Hold on to your feet because NASA has released videos of wh Read more…

Intel Plans Falcon Shores 2 GPU Supercomputing Chip for 2026  

August 8, 2023

Intel is planning to onboard a new version of the Falcon Shores chip in 2026, which is code-named Falcon Shores 2. The new product was announced by CEO Pat Gel Read more…

GenAI Having Major Impact on Data Culture, Survey Says

February 21, 2024

While 2023 was the year of GenAI, the adoption rates for GenAI did not match expectations. Most organizations are continuing to invest in GenAI but are yet to Read more…

How the Chip Industry is Helping a Battery Company

May 8, 2024

Chip companies, once seen as engineering pure plays, are now at the center of geopolitical intrigue. Chip manufacturing firms, especially TSMC and Intel, have b Read more…

  • arrow
  • Click Here for More Headlines
  • arrow
HPCwire