OpenHPC Gathers Steam, Preps New Release for SC18

By John Russell

November 1, 2018

You may recall at SC15 an Intel-led group introduced OpenHPC – an effort to build a community project around an open source, plug-and-play HPC stack. One goal was to make standing up HPC systems easier and thereby accelerate HPC adoption beyond its traditional landscape, particularly in the enterprise. Another goal – given the diversity and ever-evolving nature of HPC stack options and practices – was to create a community-driven mechanism for testing and integrating new elements into the ‘standardized’ stack. Yet a third idea was to accommodate emerging vendor platforms and hardware (mostly processors) ecosystems.

So how’s it going?

The short answer is quite well, but with room for growth. HPCwire recently talked with Karl Schulz, project leader for OpenHPC’s technical steering committee, as he and the team were racing to finish release 1.3.6 in time for SC18 where for the first time OpenHPC will have a booth in addition to its popular Birds of a Feather (BOF) gathering. A fair amount has happened since OpenHPC’s start, which was greeted hopefully and perhaps a little warily because of Intel’s prominence in the group. Those fears seem to have been unwarranted and OpenHPC now also supports ARM. No sign of IBM/Power coming into the fold yet but there’s plenty on the OpenHPC roadmap.

HPCwire: Thanks for taking time to give an update. Maybe a quick organization summary would be helpful.

Karl Schulz, OpenHPC

Schulz: Sure. In 2015 when the project started it was a relatively small number of folks working together. It wasn’t until 2016 that we really became fully operational as a collaborative project under the Linux Foundation. That meant setting a governance structure which is two pronged. We have a board for budgetary items. All the technical leadership is done through a technical steering committee. Our steering committee members participate on a term for one year. Elections take place in the summer, so right now we have a mix of old and new folks. There are roughly 20 people on the steering committee. They represent interests that are similarly aligned to the overall interest in OpenHPC, so there are folks from supercomputing sites, the elite labs, academic sites; there are vendor folks, and developers. It’s a good mix of people and all of that information is available on our GitHub page.

HPCwire: What about growth in the OpenHPC community overall? 

Schulz: In terms of other adoption and interest in the community, when we started – and I’d have to go back for the exact numbers – but I’m guessing we had approximately 20-25 formal members. We have continued to grow year over year. As of right now we have about 38 members who are formally associated with the project. They have the same span of affiliations [as the technical steering committee]. Along the way, as you know, Arm joined, so we are doing builds and recipes that are for multiple architectures now. We have multiple OS vendors associated with the project in terms of Red Hat and SLES (SUSE Linux Enterprise Server) so we push out builds for both of those. We have seen continued growth in the membership.

HPCwire: As I understand it, the OpenHPC stack itself has grown considerably. 

Schulz: If we go back to the very first release and compare it to now, we almost double the number of software components that are in the curated OpenHPC stack. That has implications for all of our build infrastructure and our technical infrastructure, but we see it as important to be able to grow the stack over time. One of the first things we did on the technical steering committee side is we put together a mechanism by which the community could suggest or request packages to be added to the stack. You don’t have to be associated with the project. Anybody can come in to suggest and then the technical committee can review those requests on roughly a quarterly basis and make decisions. That’s how the majority of all this software has found its way into OpenHPC.

You know it’s things that are both administrative, like provisioners, and things of growing interest in the community like containerization. We’ve added Singularity and Charliecloud, a bunch of numerical packages in terms of finite elements and linear algebra. And in IO clients, we’ve added support for things like BeeGFS. We’re also adding ‘new’ software as it develops, so one example is an exascale runtime for job launches.

HPCwire: It’s good to have buy-in from contributors and vendors but what can you say about user traction?

Schulz: I get this question fairly frequently – who’s using OpenHPC? Of course we are an open source project and don’t require any registration so you don’t know necessarily who’s using it. But we do mine our logs just to see general growth trends. We’ve had roughly year-over-year in the last three years a doubling of the number of folks who are hitting our build repository. I think right now in 2018 we are averaging something like 3,000 visitors a month. People are accessing our binary builds to the tune of about a terabyte and a half of data downloaded every month. We have seen basically a 2x growth every year and we feel pretty good about that.

HPCwire: When OpenHPC was first announced at SC15 there was concern that Intel’s influence might make it “less open.” That said, the reality is clusters are still largely an x86 world. How’s progress on support for other ecosystems?  

Schulz: At the last SC is when we had the formal release supporting Arm. Before we had a test release preview and the reason we came out first with a test preview was we didn’t have any software which could actually provision bare metal Arm, at least not the software we were using. Once we got that sorted out, we formally had a real Arm release that coincided with when we started to see more and more Arm hardware work its way into the general public.

We have seen some uptake and can sort of delineate between x86 and Arm because we were starting from zero. We have seen that grow particularly this year. Use is still dominated by x86 as you might expect, but definitely there’s a steady growth in people interested in Arm. I think the community project is actually well suited to help in that area because there’s just not a lot of things you can use out-of-the-box on the Arm ecosystem. We’ve benefited a lot from having the Arm folks involved in the project.

Many of the component packages, particularly the development environment tools, may not have necessarily been built on Arm architecture upfront, so we had to work through a lot of mostly small issues. I’d say at least 70 percent of the packages that we had in OpenHPC needed a little bit of work to get up and running on Arm. We’re able to leverage our testing infrastructure for both x86 and Arm, where we can actually make sure not only are the builds working but we can actually run applications against these libraries in an HPC environment under a resource manager. Even people who aren’t using OpenHPC will benefit from this work because we got a lot of the packages up and running and it served upstream to the community so other people can build on Arm as well.

HPCwire: IBM’s Power-based systems are garnering attention with the success of the Summit and Sierra supercomputers. IBM told HPCwire back at SC16 it wasn’t opposed to joining OpenHPC but we’ve not heard of any movement in that direction. Do you have insight there?

Schulz: I don’t. We had some discussion about them possibly being interested in joining and they have had some discussion with the Linux Foundation. I don’t know what their thinking is. They are obviously welcome. We pointed them to the Linux Foundation folks several times. I believe there have been discussions [in the past] but until now they are not a member yet. It would not go through me anyway. I’m on the technical side of things. It wouldn’t surprise me if these discussions come up at SC.


HPCwire: Let’s shift gears for a second and talk about OpenHPC’s approach to creating these plug-and-play stacks and relate that to helping expand HPC use. 

Schulz: What we try to do is be very building block oriented. So for people who have a lot of expertise and have a strong opinion about how to do things, they can sort of pick and choose what they want. They can use their own provisioner and their own config management system that they probably spent years developing. We wanted to make sure that we as a community effort weren’t doing something that was so locked down in terms of having to do it one and only one way that it wouldn’t be of interest to those folks. So that’s one aspect.

So we have always kept everything very building block oriented, but to your leading point, we know that there’s a lot of folks who don’t have that kind of expertise and they are looking for guidance. That’s the other part of what we are trying to do is leverage the expertise and folks involved with the community who have been involved in standing up HPC systems for years and have codified some best practice, if you will, into a number of different recipes.

HPCwire: How’s that going?

Schulz: When we first launched OpenHPC, we had one recipe and when I say one recipe, for us that is a document that is validated. It’s something for people if they are starting from bare metal or they are doing something with virtual machines that they can take and follow end-to-end and have a working cluster at the end with all the software coming from the recipe they chose.

That initial recipe had a resource manager, one provisioner, one architecture, and one OS. Now I think we are up to 10 or 12 different recipes because we have support for multiple OSs, multiple architectures. We now have multiple provisioners so people can choose from Slurm or PBS professional, etc. We are trying to curate these recipes and give people options. That increases our testing time, but we think it is important because for some people to be able to make those kinds of decisions based on what they are familiar with.

HPCwire: Given the effort required to create multiple recipes, is it paying off in terms of expanding HPC use?

Schulz: I think we have seen that being helpful in areas that you might say are non-traditional HPC where people just want to get an HPC cluster up and running to run a piece of software. We have some examples of medical organizations doing this, sort of setting up their environment to do genomics and the like. We also have people using it in finance. So we do see evidence of people using OpenHPC in areas where we say ‘well that’s not traditional HPC but they have a need for doing parallel processing.’

HPCwire: Seems like adding AI processing capabilities should be a priority?

Schulz: Obviously AI is big now so people are using OpenHPC with AI packages. We’ve not added anything specific yet, but I am thinking that at some point it will bubble up. We did an internal survey recently asking that very question – do you have an interest in AI workloads, are you using this stuff? – and there was a strong interest. At some point we will get a community request for that, TensorFlow and Caffe and that kind of stuff, and that will make us review it. We might proactively do it, but today the components that are in OpenHPC are a little more generic-HPC.

HPCwire: When you do a new release, how much of the software is actually changing?

Schulz: You have to be aware that the open source software environment changes pretty fast and you can get out of date quickly. We see that as pretty important. When we are working on a new release we scan the upstream community, all the stuff that’s going on in HPC, and if there is a new stable piece of software, we start trying to use that right away and as long as we don’t find a problem with it. We always go with the new one. We have been doing this for a couple of years now and that allows us to go back and sort of look at the metrics for how fast the software is evolving.

Some software changes every time, like Slurm resource manager. There’s going to be a new release pretty often. Some stuff doesn’t change a lot. If you look at the aggregate of the software stack for OpenHPC – and we’ve been doing releases about every quarter – we’ve seen consistently about 40 percent of the components that are in OpenHPC have a new version. Every single release, we are updating and testing a new version. That requires a lot of work and upkeep, but we think that’s an important part of the delivery for the project because if you are starting from scratch, just now installing your system, you always want the latest version.

We’ve been trying to get a feel for how many people were doing upgrades. Of course, you also have some small cadre of folks that will never update. But it is a tiny amount. I think right now something like 90 percent of the folks accessing packages are on the latest release which certainly makes us happy.

HPCwire: What’s coming out at SC18?

Schulz: The release that is currently out is what we call 1.3.5 and the release that will be coming out at Supercomputing is 1.3.6. Some of the planned stuff is the usual number of software updates; so, the 40 percent of the software number I just quoted will have new versions and we are right now pretty close to that at 38 percent. We’ll be adding some new packages and those all came from this components submission request process. We have made some packaging changes based on some lessons along the way. We are introducing a new compiler variant. One of the things about OpenHPC is that we try to adopt a very hierarchical software environment. This is based on experience at large HPC sites where we have architected a system for which we know users will have different versions of different software. They may want to have major version of some components like compilers and other [older] components.

Like I said, we try to stay as current as we can. The 1.3.5 release out there today has the latest version of GCC which was at that time GCC7.  Now there’s a GCC8 introducing a whole family of builds that are supported with GCC8. The way that we do things is that people who have existing systems, they can opt in if they want to access a new compiler variant and all the builds that go with that; for someone building a system from scratch the defaults leverage that new version of GCC. It sounds like kind of a small thing, but because of the hierarchical nature of the software environment that we have, it is actually a lot of effort on our side because we are introducing every single development tool at the same time we roll out a new compiler variant.

HPCwire: As you’ve noted it’s a lot of work. In terms of hardware, my understanding is much of the infrastructure used by OpenHPC is housed at TACC. Is that correct?

Schulz: You may know I was at TACC for a number of years previously and UT/TACC is one of the formal academic members of the project thru the Linux Foundation. We have one member on the Technical Steering Committee from TACC presently (Cyrus Proctor), and the fine folks there have been kind enough to host continuous integration clusters in their data center along with some build servers. The CI infrastructure hosting at TACC has been there since the early days and is helped with hardware donations from Intel, Cavium, and Dell.  We also have additional build server support hosted at Packet through the Works on Arm Project and leverage additional cloud infrastructure through Amazon’s EC2.

: Looking ahead what’s on the roadmap?

Schulz: We won’t have it for this release but early in 2019 for the follow-on release 1.3.7 we’ll be introducing some additional support to leverage the Arm compiler. Today we do builds leveraging different compiler tool chains including GCC and on the x86 side we also to build on Intel compilers. Arm has their own HPC compiler toolchain and we have been working with them to get that build up and running against that toolchain, which is a little different because it is LLVM based. I would expect people should see in 2019 builds against the Arm compilers which will help for folks running on Arm because you will also get access to their math library and builds for all the open source software that can take advantage of any performance optimizations that they are putting in the math library.

I also think one of the efforts you’ll see also come out of the community project is recipes that are more automated that use one particular config management. Right now we’ve had people working on something that’s Ansible. I would expect that you would see us publish something that is Ansible that would allow people to install OpenHPC. Another possibility is we know there are people working with the OpenHPC stack in a cloud environment, but it’s up to them how to get that out. I see a possibility where we might publish some reference images for some of the more common cloud service providers to make it easier to spin up an HPC cloud on an OpenHPC cluster for example.

HPCwire: Anything more exotic in the wings? I know you’ve just returned from China where several processor efforts are ongoing?

Schulz:Everything that we are operating on requires processors that have been GA’d. We wouldn’t be in a position to push out a stack for something that’s not been GA’d. I’ll say that the same way we would like to stay current on the upstream software, as new hardware rolls out that is important for HPC, we also would like to stay current with that as well. I think you can imagine that’s also a motivation for some of the partner companies to be involved to make sure the community stack runs on their new technology. As a new processor comes out, the community will have to get access to it and we’ll be in a position to roll out builds against it.

HPCwire: Thanks again for your time Karl.

Bio: Karl W. Schulz, is Research Associate Professor – Institute for Computational Engineering and Sciences (ICES), Associate Professor – Women’s Health, Dell Medical School, The University of Texas at Austin

Slides source: OpenHPC Update, August 2018,

Subscribe to HPCwire's Weekly Update!

Be the most informed person in the room! Stay ahead of the tech trends with industy updates delivered to you every week!

TACC’s Upgraded Ranch Data Storage System Debuts New Features, Exabyte Potential

May 22, 2019

There's a joke attributed to comedian Steven Wright that goes, "You can't have everything. Where would you put it?" Users of advanced computing can likely relate to this. The exponential growth of data poses a steep c Read more…

By Jorge Salazar, TACC

Cray – and the Cray Brand – to Be Positioned at Tip of HPE’s HPC Spear

May 22, 2019

More so than with most acquisitions of this kind, HPE’s purchase of Cray for $1.3 billion, announced last week, seems to have elements of that overused, often abused term: transparency. Another surprise: HPE apparently Read more…

By Doug Black and Tiffany Trader

BlueField SmartNIC Backs Transformation to Bare Metal Kubernetes

May 21, 2019

Hardware vendors are betting the transition to 5G wireless networks supporting myriad connected consumer and industrial devices also will accelerate the shift to heavy-duty bare-metal servers as a way to provision cloud- Read more…

By George Leopold

HPE Extreme Performance Solutions

HPE and Intel® Omni-Path Architecture: How to Power a Cloud

Learn how HPE and Intel® Omni-Path Architecture provide critical infrastructure for leading Nordic HPC provider’s HPCFLOW cloud service.

For decades, HPE has been at the forefront of high-performance computing, and we’ve powered some of the fastest and most robust supercomputers in the world. Read more…

IBM Accelerated Insights

Smarter EDA: Leveraging New Technologies for Product Verification

There is perhaps no sector more competitive than the modern electronics industry. Macro-trends, including artificial intelligence, 5G, and the internet of things (IoT), continue to propel dramatic growth. Read more…

HPE to Acquire Cray for $1.3B

May 17, 2019

Venerable supercomputer pioneer Cray Inc. will be acquired by Hewlett Packard Enterprise for $1.3 billion under a definitive agreement announced this morning. The news follows HPE’s acquisition nearly three years ago o Read more…

By Doug Black & Tiffany Trader

Cray – and the Cray Brand – to Be Positioned at Tip of HPE’s HPC Spear

May 22, 2019

More so than with most acquisitions of this kind, HPE’s purchase of Cray for $1.3 billion, announced last week, seems to have elements of that overused, often Read more…

By Doug Black and Tiffany Trader

HPE to Acquire Cray for $1.3B

May 17, 2019

Venerable supercomputer pioneer Cray Inc. will be acquired by Hewlett Packard Enterprise for $1.3 billion under a definitive agreement announced this morning. T Read more…

By Doug Black & Tiffany Trader

Deep Learning Competitors Stalk Nvidia

May 14, 2019

There is no shortage of processing architectures emerging to accelerate deep learning workloads, with two more options emerging this week to challenge GPU leader Nvidia. First, Intel researchers claimed a new deep learning record for image classification on the ResNet-50 convolutional neural network. Separately, Israeli AI chip startup Read more…

By George Leopold

CCC Offers Draft 20-Year AI Roadmap; Seeks Comments

May 14, 2019

Artificial Intelligence in all its guises has captured much of the conversation in HPC and general computing today. The White House, DARPA, IARPA, and Departmen Read more…

By John Russell

Cascade Lake Shows Up to 84 Percent Gen-on-Gen Advantage on STAC Benchmarking

May 13, 2019

The Securities Technology Analysis Center (STAC) issued a report Friday comparing the performance of Intel's Cascade Lake processors with previous-gen Skylake u Read more…

By Tiffany Trader

Nvidia Claims 6000x Speed-Up for Stock Trading Backtest Benchmark

May 13, 2019

A stock trading backtesting algorithm used by hedge funds to simulate trading variants has received a massive, GPU-based performance boost, according to Nvidia, Read more…

By Doug Black

ASC19: NTHU Returns to Glory

May 11, 2019

As many of you Student Cluster Competition fanatics know by now, Taiwan’s National Tsing Hua University (NTHU) won the gold medal at the recently concluded AS Read more…

By Dan Olds

Intel 7nm GPU on Roadmap for 2021, OneAPI Coming This Year

May 8, 2019

At Intel's investor meeting today in Santa Clara, Calif., the company filled in details of its roadmap and product launch plans and sought to allay concerns about delays of its 10nm chips. In laying out its 10nm and 7nm timelines, Intel revealed that its first 7nm product would be... Read more…

By Tiffany Trader

Cray, AMD to Extend DOE’s Exascale Frontier

May 7, 2019

Cray and AMD are coming back to Oak Ridge National Laboratory to partner on the world’s largest and most expensive supercomputer. The Department of Energy’s Read more…

By Tiffany Trader

Graphene Surprises Again, This Time for Quantum Computing

May 8, 2019

Graphene is fascinating stuff with promise for use in a seeming endless number of applications. This month researchers from the University of Vienna and Institu Read more…

By John Russell

Why Nvidia Bought Mellanox: ‘Future Datacenters Will Be…Like High Performance Computers’

March 14, 2019

“Future datacenters of all kinds will be built like high performance computers,” said Nvidia CEO Jensen Huang during a phone briefing on Monday after Nvidia revealed scooping up the high performance networking company Mellanox for $6.9 billion. Read more…

By Tiffany Trader

ClusterVision in Bankruptcy, Fate Uncertain

February 13, 2019

ClusterVision, European HPC specialists that have built and installed over 20 Top500-ranked systems in their nearly 17-year history, appear to be in the midst o Read more…

By Tiffany Trader

It’s Official: Aurora on Track to Be First US Exascale Computer in 2021

March 18, 2019

The U.S. Department of Energy along with Intel and Cray confirmed today that an Intel/Cray supercomputer, "Aurora," capable of sustained performance of one exaf Read more…

By Tiffany Trader

Intel Reportedly in $6B Bid for Mellanox

January 30, 2019

The latest rumors and reports around an acquisition of Mellanox focus on Intel, which has reportedly offered a $6 billion bid for the high performance interconn Read more…

By Doug Black

Looking for Light Reading? NSF-backed ‘Comic Books’ Tackle Quantum Computing

January 28, 2019

Still baffled by quantum computing? How about turning to comic books (graphic novels for the well-read among you) for some clarity and a little humor on QC. The Read more…

By John Russell

The Case Against ‘The Case Against Quantum Computing’

January 9, 2019

It’s not easy to be a physicist. Richard Feynman (basically the Jimi Hendrix of physicists) once said: “The first principle is that you must not fool yourse Read more…

By Ben Criger

Leading Solution Providers

SC 18 Virtual Booth Video Tour

Advania @ SC18 AMD @ SC18
ASRock Rack @ SC18
DDN Storage @ SC18
HPE @ SC18
IBM @ SC18
Lenovo @ SC18 Mellanox Technologies @ SC18
One Stop Systems @ SC18
Oracle @ SC18 Panasas @ SC18
Supermicro @ SC18 SUSE @ SC18 TYAN @ SC18
Verne Global @ SC18

Deep Learning Competitors Stalk Nvidia

May 14, 2019

There is no shortage of processing architectures emerging to accelerate deep learning workloads, with two more options emerging this week to challenge GPU leader Nvidia. First, Intel researchers claimed a new deep learning record for image classification on the ResNet-50 convolutional neural network. Separately, Israeli AI chip startup Read more…

By George Leopold

Deep500: ETH Researchers Introduce New Deep Learning Benchmark for HPC

February 5, 2019

ETH researchers have developed a new deep learning benchmarking environment – Deep500 – they say is “the first distributed and reproducible benchmarking s Read more…

By John Russell

IBM Bets $2B Seeking 1000X AI Hardware Performance Boost

February 7, 2019

For now, AI systems are mostly machine learning-based and “narrow” – powerful as they are by today's standards, they're limited to performing a few, narro Read more…

By Doug Black

Arm Unveils Neoverse N1 Platform with up to 128-Cores

February 20, 2019

Following on its Neoverse roadmap announcement last October, Arm today revealed its next-gen Neoverse microarchitecture with compute and throughput-optimized si Read more…

By Tiffany Trader

Intel Launches Cascade Lake Xeons with Up to 56 Cores

April 2, 2019

At Intel's Data-Centric Innovation Day in San Francisco (April 2), the company unveiled its second-generation Xeon Scalable (Cascade Lake) family and debuted it Read more…

By Tiffany Trader

France to Deploy AI-Focused Supercomputer: Jean Zay

January 22, 2019

HPE announced today that it won the contract to build a supercomputer that will drive France’s AI and HPC efforts. The computer will be part of GENCI, the Fre Read more…

By Tiffany Trader

Announcing four new HPC capabilities in Google Cloud Platform

April 15, 2019

When you’re running compute-bound or memory-bound applications for high performance computing or large, data-dependent machine learning training workloads on Read more…

By Wyatt Gorman, HPC Specialist, Google Cloud; Brad Calder, VP of Engineering, Google Cloud; Bart Sano, VP of Platforms, Google Cloud

In Wake of Nvidia-Mellanox: Xilinx to Acquire Solarflare

April 25, 2019

With echoes of Nvidia’s recent acquisition of Mellanox, FPGA maker Xilinx has announced a definitive agreement to acquire Solarflare Communications, provider Read more…

By Doug Black

  • arrow
  • Click Here for More Headlines
  • arrow
Do NOT follow this link or you will be banned from the site!
Share This