OpenHPC Gathers Steam, Preps New Release for SC18

By John Russell

November 1, 2018

You may recall at SC15 an Intel-led group introduced OpenHPC – an effort to build a community project around an open source, plug-and-play HPC stack. One goal was to make standing up HPC systems easier and thereby accelerate HPC adoption beyond its traditional landscape, particularly in the enterprise. Another goal – given the diversity and ever-evolving nature of HPC stack options and practices – was to create a community-driven mechanism for testing and integrating new elements into the ‘standardized’ stack. Yet a third idea was to accommodate emerging vendor platforms and hardware (mostly processors) ecosystems.

So how’s it going?

The short answer is quite well, but with room for growth. HPCwire recently talked with Karl Schulz, project leader for OpenHPC’s technical steering committee, as he and the team were racing to finish release 1.3.6 in time for SC18 where for the first time OpenHPC will have a booth in addition to its popular Birds of a Feather (BOF) gathering. A fair amount has happened since OpenHPC’s start, which was greeted hopefully and perhaps a little warily because of Intel’s prominence in the group. Those fears seem to have been unwarranted and OpenHPC now also supports ARM. No sign of IBM/Power coming into the fold yet but there’s plenty on the OpenHPC roadmap.

HPCwire: Thanks for taking time to give an update. Maybe a quick organization summary would be helpful.

Karl Schulz, OpenHPC

Schulz: Sure. In 2015 when the project started it was a relatively small number of folks working together. It wasn’t until 2016 that we really became fully operational as a collaborative project under the Linux Foundation. That meant setting a governance structure which is two pronged. We have a board for budgetary items. All the technical leadership is done through a technical steering committee. Our steering committee members participate on a term for one year. Elections take place in the summer, so right now we have a mix of old and new folks. There are roughly 20 people on the steering committee. They represent interests that are similarly aligned to the overall interest in OpenHPC, so there are folks from supercomputing sites, the elite labs, academic sites; there are vendor folks, and developers. It’s a good mix of people and all of that information is available on our GitHub page.

HPCwire: What about growth in the OpenHPC community overall? 

Schulz: In terms of other adoption and interest in the community, when we started – and I’d have to go back for the exact numbers – but I’m guessing we had approximately 20-25 formal members. We have continued to grow year over year. As of right now we have about 38 members who are formally associated with the project. They have the same span of affiliations [as the technical steering committee]. Along the way, as you know, Arm joined, so we are doing builds and recipes that are for multiple architectures now. We have multiple OS vendors associated with the project in terms of Red Hat and SLES (SUSE Linux Enterprise Server) so we push out builds for both of those. We have seen continued growth in the membership.

HPCwire: As I understand it, the OpenHPC stack itself has grown considerably. 

Schulz: If we go back to the very first release and compare it to now, we almost double the number of software components that are in the curated OpenHPC stack. That has implications for all of our build infrastructure and our technical infrastructure, but we see it as important to be able to grow the stack over time. One of the first things we did on the technical steering committee side is we put together a mechanism by which the community could suggest or request packages to be added to the stack. You don’t have to be associated with the project. Anybody can come in to suggest and then the technical committee can review those requests on roughly a quarterly basis and make decisions. That’s how the majority of all this software has found its way into OpenHPC.

You know it’s things that are both administrative, like provisioners, and things of growing interest in the community like containerization. We’ve added Singularity and Charliecloud, a bunch of numerical packages in terms of finite elements and linear algebra. And in IO clients, we’ve added support for things like BeeGFS. We’re also adding ‘new’ software as it develops, so one example is an exascale runtime for job launches.

HPCwire: It’s good to have buy-in from contributors and vendors but what can you say about user traction?

Schulz: I get this question fairly frequently – who’s using OpenHPC? Of course we are an open source project and don’t require any registration so you don’t know necessarily who’s using it. But we do mine our logs just to see general growth trends. We’ve had roughly year-over-year in the last three years a doubling of the number of folks who are hitting our build repository. I think right now in 2018 we are averaging something like 3,000 visitors a month. People are accessing our binary builds to the tune of about a terabyte and a half of data downloaded every month. We have seen basically a 2x growth every year and we feel pretty good about that.

HPCwire: When OpenHPC was first announced at SC15 there was concern that Intel’s influence might make it “less open.” That said, the reality is clusters are still largely an x86 world. How’s progress on support for other ecosystems?  

Schulz: At the last SC is when we had the formal release supporting Arm. Before we had a test release preview and the reason we came out first with a test preview was we didn’t have any software which could actually provision bare metal Arm, at least not the software we were using. Once we got that sorted out, we formally had a real Arm release that coincided with when we started to see more and more Arm hardware work its way into the general public.

We have seen some uptake and can sort of delineate between x86 and Arm because we were starting from zero. We have seen that grow particularly this year. Use is still dominated by x86 as you might expect, but definitely there’s a steady growth in people interested in Arm. I think the community project is actually well suited to help in that area because there’s just not a lot of things you can use out-of-the-box on the Arm ecosystem. We’ve benefited a lot from having the Arm folks involved in the project.

Many of the component packages, particularly the development environment tools, may not have necessarily been built on Arm architecture upfront, so we had to work through a lot of mostly small issues. I’d say at least 70 percent of the packages that we had in OpenHPC needed a little bit of work to get up and running on Arm. We’re able to leverage our testing infrastructure for both x86 and Arm, where we can actually make sure not only are the builds working but we can actually run applications against these libraries in an HPC environment under a resource manager. Even people who aren’t using OpenHPC will benefit from this work because we got a lot of the packages up and running and it served upstream to the community so other people can build on Arm as well.

HPCwire: IBM’s Power-based systems are garnering attention with the success of the Summit and Sierra supercomputers. IBM told HPCwire back at SC16 it wasn’t opposed to joining OpenHPC but we’ve not heard of any movement in that direction. Do you have insight there?

Schulz: I don’t. We had some discussion about them possibly being interested in joining and they have had some discussion with the Linux Foundation. I don’t know what their thinking is. They are obviously welcome. We pointed them to the Linux Foundation folks several times. I believe there have been discussions [in the past] but until now they are not a member yet. It would not go through me anyway. I’m on the technical side of things. It wouldn’t surprise me if these discussions come up at SC.

 

HPCwire: Let’s shift gears for a second and talk about OpenHPC’s approach to creating these plug-and-play stacks and relate that to helping expand HPC use. 

Schulz: What we try to do is be very building block oriented. So for people who have a lot of expertise and have a strong opinion about how to do things, they can sort of pick and choose what they want. They can use their own provisioner and their own config management system that they probably spent years developing. We wanted to make sure that we as a community effort weren’t doing something that was so locked down in terms of having to do it one and only one way that it wouldn’t be of interest to those folks. So that’s one aspect.

So we have always kept everything very building block oriented, but to your leading point, we know that there’s a lot of folks who don’t have that kind of expertise and they are looking for guidance. That’s the other part of what we are trying to do is leverage the expertise and folks involved with the community who have been involved in standing up HPC systems for years and have codified some best practice, if you will, into a number of different recipes.

HPCwire: How’s that going?

Schulz: When we first launched OpenHPC, we had one recipe and when I say one recipe, for us that is a document that is validated. It’s something for people if they are starting from bare metal or they are doing something with virtual machines that they can take and follow end-to-end and have a working cluster at the end with all the software coming from the recipe they chose.

That initial recipe had a resource manager, one provisioner, one architecture, and one OS. Now I think we are up to 10 or 12 different recipes because we have support for multiple OSs, multiple architectures. We now have multiple provisioners so people can choose from Slurm or PBS professional, etc. We are trying to curate these recipes and give people options. That increases our testing time, but we think it is important because for some people to be able to make those kinds of decisions based on what they are familiar with.

HPCwire: Given the effort required to create multiple recipes, is it paying off in terms of expanding HPC use?

Schulz: I think we have seen that being helpful in areas that you might say are non-traditional HPC where people just want to get an HPC cluster up and running to run a piece of software. We have some examples of medical organizations doing this, sort of setting up their environment to do genomics and the like. We also have people using it in finance. So we do see evidence of people using OpenHPC in areas where we say ‘well that’s not traditional HPC but they have a need for doing parallel processing.’

HPCwire: Seems like adding AI processing capabilities should be a priority?

Schulz: Obviously AI is big now so people are using OpenHPC with AI packages. We’ve not added anything specific yet, but I am thinking that at some point it will bubble up. We did an internal survey recently asking that very question – do you have an interest in AI workloads, are you using this stuff? – and there was a strong interest. At some point we will get a community request for that, TensorFlow and Caffe and that kind of stuff, and that will make us review it. We might proactively do it, but today the components that are in OpenHPC are a little more generic-HPC.

HPCwire: When you do a new release, how much of the software is actually changing?

Schulz: You have to be aware that the open source software environment changes pretty fast and you can get out of date quickly. We see that as pretty important. When we are working on a new release we scan the upstream community, all the stuff that’s going on in HPC, and if there is a new stable piece of software, we start trying to use that right away and as long as we don’t find a problem with it. We always go with the new one. We have been doing this for a couple of years now and that allows us to go back and sort of look at the metrics for how fast the software is evolving.

Some software changes every time, like Slurm resource manager. There’s going to be a new release pretty often. Some stuff doesn’t change a lot. If you look at the aggregate of the software stack for OpenHPC – and we’ve been doing releases about every quarter – we’ve seen consistently about 40 percent of the components that are in OpenHPC have a new version. Every single release, we are updating and testing a new version. That requires a lot of work and upkeep, but we think that’s an important part of the delivery for the project because if you are starting from scratch, just now installing your system, you always want the latest version.

We’ve been trying to get a feel for how many people were doing upgrades. Of course, you also have some small cadre of folks that will never update. But it is a tiny amount. I think right now something like 90 percent of the folks accessing packages are on the latest release which certainly makes us happy.

HPCwire: What’s coming out at SC18?

Schulz: The release that is currently out is what we call 1.3.5 and the release that will be coming out at Supercomputing is 1.3.6. Some of the planned stuff is the usual number of software updates; so, the 40 percent of the software number I just quoted will have new versions and we are right now pretty close to that at 38 percent. We’ll be adding some new packages and those all came from this components submission request process. We have made some packaging changes based on some lessons along the way. We are introducing a new compiler variant. One of the things about OpenHPC is that we try to adopt a very hierarchical software environment. This is based on experience at large HPC sites where we have architected a system for which we know users will have different versions of different software. They may want to have major version of some components like compilers and other [older] components.

Like I said, we try to stay as current as we can. The 1.3.5 release out there today has the latest version of GCC which was at that time GCC7.  Now there’s a GCC8 introducing a whole family of builds that are supported with GCC8. The way that we do things is that people who have existing systems, they can opt in if they want to access a new compiler variant and all the builds that go with that; for someone building a system from scratch the defaults leverage that new version of GCC. It sounds like kind of a small thing, but because of the hierarchical nature of the software environment that we have, it is actually a lot of effort on our side because we are introducing every single development tool at the same time we roll out a new compiler variant.

HPCwire: As you’ve noted it’s a lot of work. In terms of hardware, my understanding is much of the infrastructure used by OpenHPC is housed at TACC. Is that correct?

Schulz: You may know I was at TACC for a number of years previously and UT/TACC is one of the formal academic members of the project thru the Linux Foundation. We have one member on the Technical Steering Committee from TACC presently (Cyrus Proctor), and the fine folks there have been kind enough to host continuous integration clusters in their data center along with some build servers. The CI infrastructure hosting at TACC has been there since the early days and is helped with hardware donations from Intel, Cavium, and Dell.  We also have additional build server support hosted at Packet through the Works on Arm Project and leverage additional cloud infrastructure through Amazon’s EC2.


HPCwire
: Looking ahead what’s on the roadmap?

Schulz: We won’t have it for this release but early in 2019 for the follow-on release 1.3.7 we’ll be introducing some additional support to leverage the Arm compiler. Today we do builds leveraging different compiler tool chains including GCC and on the x86 side we also to build on Intel compilers. Arm has their own HPC compiler toolchain and we have been working with them to get that build up and running against that toolchain, which is a little different because it is LLVM based. I would expect people should see in 2019 builds against the Arm compilers which will help for folks running on Arm because you will also get access to their math library and builds for all the open source software that can take advantage of any performance optimizations that they are putting in the math library.

I also think one of the efforts you’ll see also come out of the community project is recipes that are more automated that use one particular config management. Right now we’ve had people working on something that’s Ansible. I would expect that you would see us publish something that is Ansible that would allow people to install OpenHPC. Another possibility is we know there are people working with the OpenHPC stack in a cloud environment, but it’s up to them how to get that out. I see a possibility where we might publish some reference images for some of the more common cloud service providers to make it easier to spin up an HPC cloud on an OpenHPC cluster for example.

HPCwire: Anything more exotic in the wings? I know you’ve just returned from China where several processor efforts are ongoing?

Schulz:Everything that we are operating on requires processors that have been GA’d. We wouldn’t be in a position to push out a stack for something that’s not been GA’d. I’ll say that the same way we would like to stay current on the upstream software, as new hardware rolls out that is important for HPC, we also would like to stay current with that as well. I think you can imagine that’s also a motivation for some of the partner companies to be involved to make sure the community stack runs on their new technology. As a new processor comes out, the community will have to get access to it and we’ll be in a position to roll out builds against it.

HPCwire: Thanks again for your time Karl.

Bio: Karl W. Schulz, is Research Associate Professor – Institute for Computational Engineering and Sciences (ICES), Associate Professor – Women’s Health, Dell Medical School, The University of Texas at Austin

Slides source: OpenHPC Update, August 2018, https://github.com/openhpc/ohpc/wiki/Papers-and-Presentations

Subscribe to HPCwire's Weekly Update!

Be the most informed person in the room! Stay ahead of the tech trends with industy updates delivered to you every week!

What’s After Exascale? The Internet of Workflows Says HPE’s Nicolas Dubé

July 29, 2021

With the race to exascale computing in its final leg, it’s natural to wonder what the Post Exascale Era will look like. Nicolas Dubé, VP and chief technologist for HPE’s HPC business unit, agrees and shared his vision at Supercomputing Frontiers Europe 2021 held last week. The next big thing, he told the virtual audience at SFE21, is something that will connect HPC and (broadly) all of IT – into what Dubé calls The Internet of Workflows. Read more…

How UK Scientists Developed Transformative, HPC-Powered Coronavirus Sequencing System

July 29, 2021

In November 2020, the COVID-19 Genomics UK Consortium (COG-UK) won the HPCwire Readers’ Choice Award for Best HPC Collaboration for its CLIMB-COVID sequencing project. Launched in March 2020, CLIMB-COVID has now resulted in the sequencing of over 675,000 coronavirus genomes – an increasingly critical task as variants like Delta threaten the tenuous prospect of a return to normalcy in much of the world. Read more…

KAUST Leverages Mixed Precision for Geospatial Data

July 28, 2021

For many computationally intensive tasks, exacting precision is not necessary for every step of the entire task to obtain a suitably precise result. The alternative is mixed-precision computing: using high precision wher Read more…

Oak Ridge Supercomputer Enables Next-Gen Jet Turbine Research

July 27, 2021

Air travel is notoriously carbon-inefficient, with many airlines going as far as to offer purchasable carbon offsets to ease the guilt over large-footprint travel. But even over just the last decade, major aircraft model Read more…

IBM and University of Tokyo Roll Out Quantum System One in Japan

July 27, 2021

IBM and the University of Tokyo today unveiled an IBM Quantum System One as part of the IBM-Japan quantum program announced in 2019. The system is the second IBM Quantum System One assembled outside the U.S. and follows Read more…

AWS Solution Channel

Data compression with increased performance and lower costs

Many customers associate a performance cost with data compression, but that’s not the case with Amazon FSx for Lustre. With FSx for Lustre, data compression reduces storage costs and increases aggregate file system throughput. Read more…

Intel Unveils New Node Names; Sapphire Rapids Is Now an ‘Intel 7’ CPU

July 27, 2021

What's a preeminent chip company to do when its process node technology lags the competition by (roughly) one generation, but outmoded naming conventions make it seem like it's two nodes behind? For Intel, the response was to change how it refers to its nodes with the aim of better reflecting its positioning within the leadership semiconductor manufacturing space. Intel revealed its new node nomenclature, and... Read more…

What’s After Exascale? The Internet of Workflows Says HPE’s Nicolas Dubé

July 29, 2021

With the race to exascale computing in its final leg, it’s natural to wonder what the Post Exascale Era will look like. Nicolas Dubé, VP and chief technologist for HPE’s HPC business unit, agrees and shared his vision at Supercomputing Frontiers Europe 2021 held last week. The next big thing, he told the virtual audience at SFE21, is something that will connect HPC and (broadly) all of IT – into what Dubé calls The Internet of Workflows. Read more…

How UK Scientists Developed Transformative, HPC-Powered Coronavirus Sequencing System

July 29, 2021

In November 2020, the COVID-19 Genomics UK Consortium (COG-UK) won the HPCwire Readers’ Choice Award for Best HPC Collaboration for its CLIMB-COVID sequencing project. Launched in March 2020, CLIMB-COVID has now resulted in the sequencing of over 675,000 coronavirus genomes – an increasingly critical task as variants like Delta threaten the tenuous prospect of a return to normalcy in much of the world. Read more…

IBM and University of Tokyo Roll Out Quantum System One in Japan

July 27, 2021

IBM and the University of Tokyo today unveiled an IBM Quantum System One as part of the IBM-Japan quantum program announced in 2019. The system is the second IB Read more…

Intel Unveils New Node Names; Sapphire Rapids Is Now an ‘Intel 7’ CPU

July 27, 2021

What's a preeminent chip company to do when its process node technology lags the competition by (roughly) one generation, but outmoded naming conventions make it seem like it's two nodes behind? For Intel, the response was to change how it refers to its nodes with the aim of better reflecting its positioning within the leadership semiconductor manufacturing space. Intel revealed its new node nomenclature, and... Read more…

Will Approximation Drive Post-Moore’s Law HPC Gains?

July 26, 2021

“Hardware-based improvements are going to get more and more difficult,” said Neil Thompson, an innovation scholar at MIT’s Computer Science and Artificial Intelligence Lab (CSAIL). “I think that’s something that this crowd will probably, actually, be already familiar with.” Thompson, speaking... Read more…

With New Owner and New Roadmap, an Independent Omni-Path Is Staging a Comeback

July 23, 2021

Put on a shelf by Intel in 2019, Omni-Path faced a uncertain future, but under new custodian Cornelis Networks, OmniPath is looking to make a comeback as an independent high-performance interconnect solution. A "significant refresh" – called Omni-Path Express – is coming later this year according to the company. Cornelis Networks formed last September as a spinout of Intel's Omni-Path division. Read more…

Chameleon’s HPC Testbed Sharpens Its Edge, Presses ‘Replay’

July 22, 2021

“One way of saying what I do for a living is to say that I develop scientific instruments,” said Kate Keahey, a senior fellow at the University of Chicago a Read more…

Summer Reading: “High-Performance Computing Is at an Inflection Point”

July 21, 2021

At last month’s 11th International Symposium on Highly Efficient Accelerators and Reconfigurable Technologies (HEART), a group of researchers led by Martin Schulz of the Leibniz Supercomputing Center (Munich) presented a “position paper” in which they argue HPC architectural landscape... Read more…

AMD Chipmaker TSMC to Use AMD Chips for Chipmaking

May 8, 2021

TSMC has tapped AMD to support its major manufacturing and R&D workloads. AMD will provide its Epyc Rome 7702P CPUs – with 64 cores operating at a base cl Read more…

Intel Launches 10nm ‘Ice Lake’ Datacenter CPU with Up to 40 Cores

April 6, 2021

The wait is over. Today Intel officially launched its 10nm datacenter CPU, the third-generation Intel Xeon Scalable processor, codenamed Ice Lake. With up to 40 Read more…

Berkeley Lab Debuts Perlmutter, World’s Fastest AI Supercomputer

May 27, 2021

A ribbon-cutting ceremony held virtually at Berkeley Lab's National Energy Research Scientific Computing Center (NERSC) today marked the official launch of Perlmutter – aka NERSC-9 – the GPU-accelerated supercomputer built by HPE in partnership with Nvidia and AMD. Read more…

Ahead of ‘Dojo,’ Tesla Reveals Its Massive Precursor Supercomputer

June 22, 2021

In spring 2019, Tesla made cryptic reference to a project called Dojo, a “super-powerful training computer” for video data processing. Then, in summer 2020, Tesla CEO Elon Musk tweeted: “Tesla is developing a [neural network] training computer called Dojo to process truly vast amounts of video data. It’s a beast! … A truly useful exaflop at de facto FP32.” Read more…

Google Launches TPU v4 AI Chips

May 20, 2021

Google CEO Sundar Pichai spoke for only one minute and 42 seconds about the company’s latest TPU v4 Tensor Processing Units during his keynote at the Google I Read more…

CentOS Replacement Rocky Linux Is Now in GA and Under Independent Control

June 21, 2021

The Rocky Enterprise Software Foundation (RESF) is announcing the general availability of Rocky Linux, release 8.4, designed as a drop-in replacement for the soon-to-be discontinued CentOS. The GA release is launching six-and-a-half months after Red Hat deprecated its support for the widely popular, free CentOS server operating system. The Rocky Linux development effort... Read more…

CERN Is Betting Big on Exascale

April 1, 2021

The European Organization for Nuclear Research (CERN) involves 23 countries, 15,000 researchers, billions of dollars a year, and the biggest machine in the worl Read more…

Iran Gains HPC Capabilities with Launch of ‘Simorgh’ Supercomputer

May 18, 2021

Iran is said to be developing domestic supercomputing technology to advance the processing of scientific, economic, political and military data, and to strengthen the nation’s position in the age of AI and big data. On Sunday, Iran unveiled the Simorgh supercomputer, which will deliver.... Read more…

Leading Solution Providers

Contributors

HPE Launches Storage Line Loaded with IBM’s Spectrum Scale File System

April 6, 2021

HPE today launched a new family of storage solutions bundled with IBM’s Spectrum Scale Erasure Code Edition parallel file system (description below) and featu Read more…

Julia Update: Adoption Keeps Climbing; Is It a Python Challenger?

January 13, 2021

The rapid adoption of Julia, the open source, high level programing language with roots at MIT, shows no sign of slowing according to data from Julialang.org. I Read more…

10nm, 7nm, 5nm…. Should the Chip Nanometer Metric Be Replaced?

June 1, 2020

The biggest cool factor in server chips is the nanometer. AMD beating Intel to a CPU built on a 7nm process node* – with 5nm and 3nm on the way – has been i Read more…

GTC21: Nvidia Launches cuQuantum; Dips a Toe in Quantum Computing

April 13, 2021

Yesterday Nvidia officially dipped a toe into quantum computing with the launch of cuQuantum SDK, a development platform for simulating quantum circuits on GPU-accelerated systems. As Nvidia CEO Jensen Huang emphasized in his keynote, Nvidia doesn’t plan to build... Read more…

Microsoft to Provide World’s Most Powerful Weather & Climate Supercomputer for UK’s Met Office

April 22, 2021

More than 14 months ago, the UK government announced plans to invest £1.2 billion ($1.56 billion) into weather and climate supercomputing, including procuremen Read more…

Quantum Roundup: IBM, Rigetti, Phasecraft, Oxford QC, China, and More

July 13, 2021

IBM yesterday announced a proof for a quantum ML algorithm. A week ago, it unveiled a new topology for its quantum processors. Last Friday, the Technical Univer Read more…

Q&A with Jim Keller, CTO of Tenstorrent, and an HPCwire Person to Watch in 2021

April 22, 2021

As part of our HPCwire Person to Watch series, we are happy to present our interview with Jim Keller, president and chief technology officer of Tenstorrent. One of the top chip architects of our time, Keller has had an impactful career. Read more…

Senate Debate on Bill to Remake NSF – the Endless Frontier Act – Begins

May 18, 2021

The U.S. Senate today opened floor debate on the Endless Frontier Act which seeks to remake and expand the National Science Foundation by creating a technology Read more…

  • arrow
  • Click Here for More Headlines
  • arrow
HPCwire