Top Ten Ways AI Affects HPC in 2019

By James Reinders

March 26, 2019

AI workloads are becoming ubiquitous, including running on the world’s fastest computers — thereby changing what we call HPC forever. As every organization plans for the future, AI workloads are on our minds — how do they affect programming, software needs, hardware demands, and training needs? In the upcoming year, specialists and AI experts will continue to come together to create new and innovative solutions.

Here are the top ten ways that AI will most impact HPC in 2019.

10. Tensors: Lingua franca for AI computations

Vector algebra usage gave rise to computers designed for vector computing. Early supercomputers from Cray were vector-supercomputers, which in turn encouraged expressing applications as vector and matrix algebra problems, which in turn reinforced computers being designed to ensure vector computations run fast. This reinforcing cycle has strongly defined HPC over the years. Tensor algebra can be embraced as a generalized matrix algebra, so it is a natural evolution of supercomputer mathematical capabilities, not a revolution. Any machine supporting matrix operations can do tensors operations already. Today, users of CPUs= see support for vectors and tensors, with high performance, via support from general purpose compilers, accelerated Pythons, enhanced libraries, and optimized frameworks. All these allow software developers to use vectors and tensors from their favorite environments with high performance.

Tensors are leaving a deep mark, as vectors did before them, on HPC in hardware, software, and our thinking.

9. Languages: Higher level programming

Fortran programs dominate HPC today in terms of cycles consumed, with C and C++ programs using up almost all of the other cycles in HPC. Accelerator cycles are most often supported through C interfaces, extensions, and libraries. Attempts to disrupt this with new languages have failed because the incumbent languages have users, code, and support that fit the applications that make up HPC.

AI brings new users with new demands, which will expand what languages we associate with HPC. They will not change the activities of most physicists using Fortran code, but a data scientist using MATLAB and Python want solutions tailored to their needs.

Python, and a cast of other productivity languages and frameworks, will seemingly be the masters of more and more HPC cycles. Their secret will be that their actual number crunching routines will still be written in C/C++/Fortran, but the AI programmer will neither know that, nor care about it.

8. Freedom to think differently: Replace legacy code by using the opportunity (and peril) to rethink approaches

HPC is steeped in legacy, and AI is new and relatively legacy free. Obviously, as AI matures it will create important legacy of its own that will need supporting. For now, as the two interact – it will encourage conversations about reimplementing legacy code, which in some cases may have been overdue. The excuse might be “let’s add some AI features to this code” but the reality will be some beneficial efforts as well as some serious wastes of time. Remember the many ‘convert to Java’ efforts, in the early days of the Java craze?

Like those crazy early Java days, any rush to rewrite code into a new form will have winners and losers. ROI will be the key, but predicting the outcome of efforts to innovate are often highly flawed.

7. Portability and security: Virtualization and containers

Security and portability — “can I safely run on my machine?” and “does it work on my machine?” — are problems that virtualization and containers seek to solve. Of course, security comes from a well-constructed connection of hardware security features and software security features. For many, virtualization and containers seem to best establish that combination.

Containers have caught the attention of many developers over virtual machines, because they are viewed as more agile than virtual machines for deployment, patching, cloud versatility, and they may save on virtual machine licensing costs.

It is not surprising that talks about containers at any HPC or AI oriented conference always seem to be standing room only. The interest is there, and reality seems to be coming along to support the interest. Python and Julia, for instance, scale much better when carefully configured — something containers can help deploy.

Containers offer a natural way to give a well-tuned environment to users, and the HPC world will see more and more usage of containers in 2019, in part due to AI user interest. The HPC world here will undoubtedly stress performant instances – which require optimized eco-systems. There is plenty of fine work in this area going on in this area – the HPC community will help bring it to light for all, to satisfy this craving for containers.

6. Size Matters: Big Data

Where there is AI, there is Big Data. Much focus with the AI community is on squeezing meaning out of very large data sets using very large data models. Yes, there are enough HPC applications with big file needs, that many HPC centers already have much of the infrastructure for handling big data problems well.

All HPC centers will take big data into consideration as a major requirement for new systems, with AI workloads being a major motivation for big data requirements.

Because of the high cost of memory, we have seen the ratio of memory size to FLOP/s erode for many years. This is a trend against big data. New capabilities around persistent memory offer a hope to reverse this trend and support big data models in large machines, including HPC machines, that we obviously want and need. These new memory technologies offer expansion of main memory, as well as local storage (SSDs).

I’m writing today about how AI affects HPC, but I can’t resist pointing out that HPC’s love of visualization will have a role in HPC affecting AI. Putting data closer to the processor, which is best suited to do real data visualization, is one of the biggest ways that HPC will affect AI/ML. The concept of using and understanding big data, and visualizing the data and analysis, are very much intertwined.

5. Compute for the Masses: Cloud

AI developers may already embrace cloud computing more than HPC developers. While HPC “in the cloud” has already been emerging, high performance computing for AI applications will accelerate “HPC in the cloud.”

4. Hardware: Interactive capabilities, and focus on powering libraries and frameworks

The number of workloads for AI is not huge. This in turn means that a small number of library interfaces and frameworks dominate what any “AI accelerator” needs to tout as selling points.

Interactivity, a long-standing request that has generally stayed “on the back burner” for HPC systems, is squarely placed “front and center” by AI programmers. How quickly this changes “HPC” remains to be seen, but innovation in this area in 2019 will be notable even if scattered and somewhat hidden. Interactivity may be called “personalization” as well.

More hardware diversity, interactivity support, and additional library/framework abstractions optimized for performance, are in store for HPC to support AI workloads. The HPC community’s focus on performance, will help illuminate where additional convergence in infrastructure will benefit data center deployments as well. No one wants to give up performance if they do not have to do so, the HPC community’s expertise will help commoditize performance for AI/ML leading to even more convergence of hardware technologies between the communities.

3. Melding of people: User diversity and added excitement about HPC

AI will inject a lot of fresh talent with diverse backgrounds. AI will bring democratization to HPC at an unprecedented scale. In prior years, “democratization of HPC” is a phrase used to describe how HPC, previously accessible to only those in large organizations, has been made accessible to smaller groups of engineers and scientists. Mathematical and physics problems may have driven early supercomputing workloads, but more recently many more users have found HPC workloads to be indispensable in fields including medicine, weather forecasting, and risk management.

AI brings a much wider community of users than HPC has previously seen, bring a whole new dimension to democratization of HPC. Add AI to the list of reasons to do HPC, and we continue to add more excitement in the pursuit of the highest performance computing in the world — HPC specialists and AI experts are combining to generate excitement we can all enjoy.

2. New investments: Inferencing

Machine learning generally can be thought of as consisting of a learning phase called “training,” and a “doing” phase called inferencing. It appears that the world needs a lot more cycles doing inferencing than cycles doing training, especially as we see machine learning ubiquitously embedded into solutions all around us. Market analysts tend to estimate that the market for hardware to do inferencing is 5-10X the size of hardware to do training.

With such a large market opportunity it is no surprise that it feels like the whole world is aiming to get a bigger piece of the inferencing market. Inferencing has been implemented on processors, FPGAs, GPUs, DSPs and a plethora of custom ASICs. Power, latency, and overall cost are key factors that give us a field of options with different selling points. High performance CPUs, coupled with low latency, easily reprogrammable, and predictable latency FPGAs seem a logical choice to supplement the current CPU-dominated world of inferencing. Time will tell.

Follow the money, and you’ll see that inferencing workloads will substantially impact all of computing including HPC.

1. Melding of applications: Rather than replacing after “rethinking” – we “blend” with the best of both worlds – expanding workload diversity and seeing all manner of workload convergence

Those with vision have resoundingly proven that there are many opportunities when HPC and AI come together. Inspiring research ranges from having a neutral net learn to “act like a Monte Carlo simulation” with very good results, at a tiny fraction of the computational needs; to integrating systems to spot patterns that can predict extreme weather such, as hurricanes, into climate or weather forecasting systems. Ideas are popping up everywhere now. A generative adversarial network (GAN) is a class of machine learning systems that many hold in high regard, and GANs will no doubt help blend the HPC world and the AI/ML work.

While it is true that very few applications combine HPC algorithms and AI techniques today — based on early results in this area, it is easy for me to predict that this is the future of HPC applications, and will constitute the biggest change coming to HPC because of AI.

Making sense of these ten forces

The story of computing does not change in one sense: it’s all about what the complete system does for its users. While needs change, the fact that a complete system is made up of hardware and software does not change. It is easy to get distracted by a single technology (hardware or software); the best systems carefully apply new technology where it will help the most. I’m very partial to calling this “selective acceleration” – with an emphasis on using acceleration when it matters. I like Python acceleration (a software technology leaning on the CPU), when I use Python a lot. I like FPGA acceleration when I need lots of low latency inferencing. I don’t bother with either, when I only need a little. This is the art of building a balanced system. This top-ten list, doesn’t change the reality that balance gives the best overall result for multi-purpose machines.

Conclusion: AI will use HPC, and that will change HPC forever

It is clear that AI will use HPC, and that will change HPC forever. In fact, AI may be the biggest change agent for HPC in its history. HPC has continuously evolved as disciplines have arrived with their own workloads, and it will also evolve for AI. I do not think debating convergence vs. intersection gives enough credit to the concept that AI users will simply join the community of HPC and put their own mark on it. And they will use non-HPC systems too, just like other HPC users.

There will be custom high-performance machines designed and built primarily for AI workloads, and other machines have AI workloads run on more general high-performance facilities with non-AI workloads as well. Balanced machines will apply acceleration when it makes sense with a strong need for high-performance flexible machines. In all cases, the AI will contribute to the future definition of what makes a computer super, and therefore adjust the course of HPC forever.

James Reinders is an HPC enthusiast and author of eight books with more than 30 years of industry experience, including 27 years at Intel Corporation (retired June 2016).

Subscribe to HPCwire's Weekly Update!

Be the most informed person in the room! Stay ahead of the tech trends with industy updates delivered to you every week!

U.S. CTO Michael Kratsios Adds DoD Research & Engineering Title

July 13, 2020

Michael Kratsios, the U.S. Chief Technology Officer, has been appointed acting Undersecretary of Defense for research and engineering. He replaces Mike Griffin, who along with his deputy Lis Porter, stepped down last wee Read more…

By John Russell

Supercomputer Research Reveals Star Cluster Born Outside Our Galaxy

July 11, 2020

The Milky Way is our galactic home, containing our solar system and continuing into a giant band of densely packed stars that stretches across clear night skies around the world – but, it turns out, not all of those st Read more…

By Oliver Peckham

Max Planck Society Begins Installation of Liquid-Cooled Supercomputer from Lenovo

July 9, 2020

Lenovo announced today that it is supplying a new high performance computer to the Max Planck Society, one of Germany's premier research organizations. Comprised of Intel Xeon processors and Nvidia A100 GPUs, and featuri Read more…

By Tiffany Trader

Xilinx Announces First Adaptive Computing Challenge

July 9, 2020

A new contest is challenging the computing world. Xilinx has announced the first Xilinx Adaptive Computing Challenge, a competition that will task developers and startups with finding creative workload acceleration solutions. Xilinx is running the Adaptive Computing Challenge in partnership with, a developing community... Read more…

By Staff report

Reviving Moore’s Law? LBNL Researchers See Promise in Heterostructure Oxides

July 9, 2020

The reality of Moore’s law’s decline is no longer doubted for good empirical reasons. That said, never say never. Recent work by Lawrence Berkeley National Laboratory researchers suggests heterostructure oxides may b Read more…

By John Russell

AWS Solution Channel

Best Practices for Running Computational Fluid Dynamics (CFD) Workloads on AWS

The scalable nature and variable demand of CFD workloads makes them well-suited for a cloud computing environment. Many of the AWS instance types, such as the compute family instance types, are designed to include support for this type of workload.  Read more…

Intel® HPC + AI Pavilion

Supercomputing the Pandemic: Scientific Community Tackles COVID-19 from Multiple Perspectives

Since their inception, supercomputers have taken on the biggest, most complex, and most data-intensive computing challenges—from confirming Einstein’s theories about gravitational waves to predicting the impacts of climate change. Read more…

President’s Council Targets AI, Quantum, STEM; Recommends Spending Growth

July 9, 2020

Last week the President Council of Advisors on Science and Technology (PCAST) met (webinar) to review policy recommendations around three sub-committee reports: 1) Industries of the Future (IotF), chaired be Dario Gil (d Read more…

By John Russell

Max Planck Society Begins Installation of Liquid-Cooled Supercomputer from Lenovo

July 9, 2020

Lenovo announced today that it is supplying a new high performance computer to the Max Planck Society, one of Germany's premier research organizations. Comprise Read more…

By Tiffany Trader

President’s Council Targets AI, Quantum, STEM; Recommends Spending Growth

July 9, 2020

Last week the President Council of Advisors on Science and Technology (PCAST) met (webinar) to review policy recommendations around three sub-committee reports: Read more…

By John Russell

Google Cloud Debuts 16-GPU Ampere A100 Instances

July 7, 2020

On the heels of the Nvidia’s Ampere A100 GPU launch in May, Google Cloud is announcing alpha availability of the A100 “Accelerator Optimized” VM A2 instance family on Google Compute Engine. The instances are powered by the HGX A100 16-GPU platform, which combines two HGX A100 8-GPU baseboards using... Read more…

By Tiffany Trader

Q&A: HLRS’s Bastian Koller Tackles HPC and Industry in Germany and Europe

July 6, 2020

In this exclusive interview for HPCwire – sadly not face to face – Steve Conway, senior advisor for Hyperion Research, talks with Dr.-Ing Bastian Koller about the state of HPC and its collaboration with Industry in Europe. Koller is a familiar figure in HPC. He is the managing director at High Performance Computing Center Stuttgart (HLRS) and also serves... Read more…

By Steve Conway, Hyperion

OpenPOWER Reboot – New Director, New Silicon Partners, Leveraging Linux Foundation Connections

July 2, 2020

Earlier this week the OpenPOWER Foundation announced the contribution of IBM’s A21 Power processor core design to the open source community. Roughly this time Read more…

By John Russell

Hyperion Forecast – Headwinds in 2020 Won’t Stifle Cloud HPC Adoption or Arm’s Rise

June 30, 2020

The semiannual taking of HPC’s pulse by Hyperion Research – late fall at SC and early summer at ISC – is a much-watched indicator of things come. This yea Read more…

By John Russell

Racism and HPC: a Special Podcast

June 29, 2020

Promoting greater diversity in HPC is a much-discussed goal and ostensibly a long-sought goal in HPC. Yet it seems clear HPC is far from achieving this goal. Re Read more…

Top500 Trends: Movement on Top, but Record Low Turnover

June 25, 2020

The 55th installment of the Top500 list saw strong activity in the leadership segment with four new systems in the top ten and a crowning achievement from the f Read more…

By Tiffany Trader

Supercomputer Modeling Tests How COVID-19 Spreads in Grocery Stores

April 8, 2020

In the COVID-19 era, many people are treating simple activities like getting gas or groceries with caution as they try to heed social distancing mandates and protect their own health. Still, significant uncertainty surrounds the relative risk of different activities, and conflicting information is prevalent. A team of Finnish researchers set out to address some of these uncertainties by... Read more…

By Oliver Peckham

[email protected] Turns Its Massive Crowdsourced Computer Network Against COVID-19

March 16, 2020

For gamers, fighting against a global crisis is usually pure fantasy – but now, it’s looking more like a reality. As supercomputers around the world spin up Read more…

By Oliver Peckham

[email protected] Rallies a Legion of Computers Against the Coronavirus

March 24, 2020

Last week, we highlighted [email protected], a massive, crowdsourced computer network that has turned its resources against the coronavirus pandemic sweeping the globe – but [email protected] isn’t the only game in town. The internet is buzzing with crowdsourced computing... Read more…

By Oliver Peckham

Supercomputer Simulations Reveal the Fate of the Neanderthals

May 25, 2020

For hundreds of thousands of years, neanderthals roamed the planet, eventually (almost 50,000 years ago) giving way to homo sapiens, which quickly became the do Read more…

By Oliver Peckham

DoE Expands on Role of COVID-19 Supercomputing Consortium

March 25, 2020

After announcing the launch of the COVID-19 High Performance Computing Consortium on Sunday, the Department of Energy yesterday provided more details on its sco Read more…

By John Russell

Neocortex Will Be First-of-Its-Kind 800,000-Core AI Supercomputer

June 9, 2020

Pittsburgh Supercomputing Center (PSC - a joint research organization of Carnegie Mellon University and the University of Pittsburgh) has won a $5 million award Read more…

By Tiffany Trader

Honeywell’s Big Bet on Trapped Ion Quantum Computing

April 7, 2020

Honeywell doesn’t spring to mind when thinking of quantum computing pioneers, but a decade ago the high-tech conglomerate better known for its control systems waded deliberately into the then calmer quantum computing (QC) waters. Fast forward to March when Honeywell announced plans to introduce an ion trap-based quantum computer whose ‘performance’ would... Read more…

By John Russell

10nm, 7nm, 5nm…. Should the Chip Nanometer Metric Be Replaced?

June 1, 2020

The biggest cool factor in server chips is the nanometer. AMD beating Intel to a CPU built on a 7nm process node* – with 5nm and 3nm on the way – has been i Read more…

By Doug Black

Leading Solution Providers


Nvidia’s Ampere A100 GPU: Up to 2.5X the HPC, 20X the AI

May 14, 2020

Nvidia's first Ampere-based graphics card, the A100 GPU, packs a whopping 54 billion transistors on 826mm2 of silicon, making it the world's largest seven-nanom Read more…

By Tiffany Trader

‘Billion Molecules Against COVID-19’ Challenge to Launch with Massive Supercomputing Support

April 22, 2020

Around the world, supercomputing centers have spun up and opened their doors for COVID-19 research in what may be the most unified supercomputing effort in hist Read more…

By Oliver Peckham

Australian Researchers Break All-Time Internet Speed Record

May 26, 2020

If you’ve been stuck at home for the last few months, you’ve probably become more attuned to the quality (or lack thereof) of your internet connection. Even Read more…

By Oliver Peckham

15 Slides on Programming Aurora and Exascale Systems

May 7, 2020

Sometime in 2021, Aurora, the first planned U.S. exascale system, is scheduled to be fired up at Argonne National Laboratory. Cray (now HPE) and Intel are the k Read more…

By John Russell

Summit Supercomputer is Already Making its Mark on Science

September 20, 2018

Summit, now the fastest supercomputer in the world, is quickly making its mark in science – five of the six finalists just announced for the prestigious 2018 Read more…

By John Russell

TACC Supercomputers Run Simulations Illuminating COVID-19, DNA Replication

March 19, 2020

As supercomputers around the world spin up to combat the coronavirus, the Texas Advanced Computing Center (TACC) is announcing results that may help to illumina Read more…

By Staff report

$100B Plan Submitted for Massive Remake and Expansion of NSF

May 27, 2020

Legislation to reshape, expand - and rename - the National Science Foundation has been submitted in both the U.S. House and Senate. The proposal, which seems to Read more…

By John Russell

John Martinis Reportedly Leaves Google Quantum Effort

April 21, 2020

John Martinis, who led Google’s quantum computing effort since establishing its quantum hardware group in 2014, has left Google after being moved into an advi Read more…

By John Russell

  • arrow
  • Click Here for More Headlines
  • arrow
Do NOT follow this link or you will be banned from the site!
Share This