The care, feeding and sustained growth of the HPC industry increasingly is in the hands of the commercial market sector – in particular, it’s the hyperscale companies and their embrace of AI and deep learning – that will drive healthy and stable, if not spectacular, expansion of the market through 2021 and beyond.

Those are some of the top-line findings announced today by Addison Snell, CEO of HPC industry watcher Intersect 360 Research at the ISC conference in Frankfurt. Snell’s other key findings include:

• Total worldwide HPC market (servers, storage, software, etc.) reached $35.6 billion in 2016, up 3.5 percent from 2015.
• Servers were the largest component, reaching $11.5 billion.
• While servers grew by 3.3 percent over 2015, storage systems grew by a healthy 5.0 percent – Snell said this growth was spurred by commercial users.
• Total market is forecast to grow to $43.9 billion for 2021, for a 4.3 percent CAGR. The industrial sector is expected to deliver a CAGR of 6.5 percent, followed by government at 2 percent and a flat academia sector.
• Government represents 26 percent of the total market.

That 3.5 percent growth in 2016 is a significant come-down from 2015’s growth of 9 percent (mostly due to expansion in the storage sector) over 2014, according to Snell. His overall take on the coming years: long-term moderate, stable growth through 2021 and beyond.

Among HPC server vendors, HPE continued its leadership in HPC server sales, claiming more than 33 percent market share over second place Dell EMC’s 26.4 percent. HPE’s lead was boosted by “a couple of points” by the incorporation of sales by SGI, which HPE acquired last year. They were followed by Lenovo, IBM and Cray. Snell noted that it was only two years ago that IBM was the top seller of HPC servers until announcing the sale of its x86 server business to Lenovo (currently with 6.9 percent of the market), which he said has underperformed based on the market share it acquired from IBM.

On the storage side, Dell EMC continued its leadership with a 24.4 percent share, with NetApp slipping slightly to 15.8 percent, followed by HPE, IBM and HDS. DDN was reported to have a 3 percent share and Seagate at 2.9 percent.

Snell emphasized that AI and deep learning are the drivers for HPC in the commercial sector.

“Deep learning and AI drives a lot of HPC-like investments, they’re using a lot of GPUs, Infiniband, and so forth,” he said. “It’s being driven primarily by the hyperscale companies, like Baidu, Google, Amazon, Facebook, Microsoft. Traditional HPC companies might be looking at deep learning and AI but they’re not spending a lot differently than they would have spent otherwise. We’re not seeing big changes in budgets, even among major public sector supercomputer facilities.”

He said an academic institution may have spent money last year on a new supercomputer that it plans “to do some AI things with, it’s the same supercomputer they would have bought otherwise.”

Among non-hyperscale companies, he said HPC growth among commercial companies is led by the financial services sector, followed by manufacturing, then oil and gas.

Snell said the study’s most surprising finding: lack of market uptake for HPC in public clouds. Starting from a low base last year, Intersect 360 reported growth of 6.4 percent – not the 10 percent growth that had been expected. He said the market remains selective in the jobs it offloads to public cloud platforms, drawing an analogy to transportation decisions we make in everyday life: usually it makes the most sense to drive our own cars, but sometimes taking a taxi, the train or flying is the best choice. Similarly, he said, it usually makes the most sense for organizations to use own on-prem HPC capabilities.

With public clouds holding 2.5 percent of the HPC market, Snell said Amazon has the largest share, adding that he is “reasonably sure” Microsoft Azure overtook Google last year for second place among public cloud services providers, followed by IBM Softlayer.