OpenHPC Gathers Steam, Preps New Release for SC18

By John Russell

November 1, 2018

You may recall at SC15 an Intel-led group introduced OpenHPC – an effort to build a community project around an open source, plug-and-play HPC stack. One goal was to make standing up HPC systems easier and thereby accelerate HPC adoption beyond its traditional landscape, particularly in the enterprise. Another goal – given the diversity and ever-evolving nature of HPC stack options and practices – was to create a community-driven mechanism for testing and integrating new elements into the ‘standardized’ stack. Yet a third idea was to accommodate emerging vendor platforms and hardware (mostly processors) ecosystems.

So how’s it going?

The short answer is quite well, but with room for growth. HPCwire recently talked with Karl Schulz, project leader for OpenHPC’s technical steering committee, as he and the team were racing to finish release 1.3.6 in time for SC18 where for the first time OpenHPC will have a booth in addition to its popular Birds of a Feather (BOF) gathering. A fair amount has happened since OpenHPC’s start, which was greeted hopefully and perhaps a little warily because of Intel’s prominence in the group. Those fears seem to have been unwarranted and OpenHPC now also supports ARM. No sign of IBM/Power coming into the fold yet but there’s plenty on the OpenHPC roadmap.

HPCwire: Thanks for taking time to give an update. Maybe a quick organization summary would be helpful.

Karl Schulz, OpenHPC

Schulz: Sure. In 2015 when the project started it was a relatively small number of folks working together. It wasn’t until 2016 that we really became fully operational as a collaborative project under the Linux Foundation. That meant setting a governance structure which is two pronged. We have a board for budgetary items. All the technical leadership is done through a technical steering committee. Our steering committee members participate on a term for one year. Elections take place in the summer, so right now we have a mix of old and new folks. There are roughly 20 people on the steering committee. They represent interests that are similarly aligned to the overall interest in OpenHPC, so there are folks from supercomputing sites, the elite labs, academic sites; there are vendor folks, and developers. It’s a good mix of people and all of that information is available on our GitHub page.

HPCwire: What about growth in the OpenHPC community overall? 

Schulz: In terms of other adoption and interest in the community, when we started – and I’d have to go back for the exact numbers – but I’m guessing we had approximately 20-25 formal members. We have continued to grow year over year. As of right now we have about 38 members who are formally associated with the project. They have the same span of affiliations [as the technical steering committee]. Along the way, as you know, Arm joined, so we are doing builds and recipes that are for multiple architectures now. We have multiple OS vendors associated with the project in terms of Red Hat and SLES (SUSE Linux Enterprise Server) so we push out builds for both of those. We have seen continued growth in the membership.

HPCwire: As I understand it, the OpenHPC stack itself has grown considerably. 

Schulz: If we go back to the very first release and compare it to now, we almost double the number of software components that are in the curated OpenHPC stack. That has implications for all of our build infrastructure and our technical infrastructure, but we see it as important to be able to grow the stack over time. One of the first things we did on the technical steering committee side is we put together a mechanism by which the community could suggest or request packages to be added to the stack. You don’t have to be associated with the project. Anybody can come in to suggest and then the technical committee can review those requests on roughly a quarterly basis and make decisions. That’s how the majority of all this software has found its way into OpenHPC.

You know it’s things that are both administrative, like provisioners, and things of growing interest in the community like containerization. We’ve added Singularity and Charliecloud, a bunch of numerical packages in terms of finite elements and linear algebra. And in IO clients, we’ve added support for things like BeeGFS. We’re also adding ‘new’ software as it develops, so one example is an exascale runtime for job launches.

HPCwire: It’s good to have buy-in from contributors and vendors but what can you say about user traction?

Schulz: I get this question fairly frequently – who’s using OpenHPC? Of course we are an open source project and don’t require any registration so you don’t know necessarily who’s using it. But we do mine our logs just to see general growth trends. We’ve had roughly year-over-year in the last three years a doubling of the number of folks who are hitting our build repository. I think right now in 2018 we are averaging something like 3,000 visitors a month. People are accessing our binary builds to the tune of about a terabyte and a half of data downloaded every month. We have seen basically a 2x growth every year and we feel pretty good about that.

HPCwire: When OpenHPC was first announced at SC15 there was concern that Intel’s influence might make it “less open.” That said, the reality is clusters are still largely an x86 world. How’s progress on support for other ecosystems?  

Schulz: At the last SC is when we had the formal release supporting Arm. Before we had a test release preview and the reason we came out first with a test preview was we didn’t have any software which could actually provision bare metal Arm, at least not the software we were using. Once we got that sorted out, we formally had a real Arm release that coincided with when we started to see more and more Arm hardware work its way into the general public.

We have seen some uptake and can sort of delineate between x86 and Arm because we were starting from zero. We have seen that grow particularly this year. Use is still dominated by x86 as you might expect, but definitely there’s a steady growth in people interested in Arm. I think the community project is actually well suited to help in that area because there’s just not a lot of things you can use out-of-the-box on the Arm ecosystem. We’ve benefited a lot from having the Arm folks involved in the project.

Many of the component packages, particularly the development environment tools, may not have necessarily been built on Arm architecture upfront, so we had to work through a lot of mostly small issues. I’d say at least 70 percent of the packages that we had in OpenHPC needed a little bit of work to get up and running on Arm. We’re able to leverage our testing infrastructure for both x86 and Arm, where we can actually make sure not only are the builds working but we can actually run applications against these libraries in an HPC environment under a resource manager. Even people who aren’t using OpenHPC will benefit from this work because we got a lot of the packages up and running and it served upstream to the community so other people can build on Arm as well.

HPCwire: IBM’s Power-based systems are garnering attention with the success of the Summit and Sierra supercomputers. IBM told HPCwire back at SC16 it wasn’t opposed to joining OpenHPC but we’ve not heard of any movement in that direction. Do you have insight there?

Schulz: I don’t. We had some discussion about them possibly being interested in joining and they have had some discussion with the Linux Foundation. I don’t know what their thinking is. They are obviously welcome. We pointed them to the Linux Foundation folks several times. I believe there have been discussions [in the past] but until now they are not a member yet. It would not go through me anyway. I’m on the technical side of things. It wouldn’t surprise me if these discussions come up at SC.

 

HPCwire: Let’s shift gears for a second and talk about OpenHPC’s approach to creating these plug-and-play stacks and relate that to helping expand HPC use. 

Schulz: What we try to do is be very building block oriented. So for people who have a lot of expertise and have a strong opinion about how to do things, they can sort of pick and choose what they want. They can use their own provisioner and their own config management system that they probably spent years developing. We wanted to make sure that we as a community effort weren’t doing something that was so locked down in terms of having to do it one and only one way that it wouldn’t be of interest to those folks. So that’s one aspect.

So we have always kept everything very building block oriented, but to your leading point, we know that there’s a lot of folks who don’t have that kind of expertise and they are looking for guidance. That’s the other part of what we are trying to do is leverage the expertise and folks involved with the community who have been involved in standing up HPC systems for years and have codified some best practice, if you will, into a number of different recipes.

HPCwire: How’s that going?

Schulz: When we first launched OpenHPC, we had one recipe and when I say one recipe, for us that is a document that is validated. It’s something for people if they are starting from bare metal or they are doing something with virtual machines that they can take and follow end-to-end and have a working cluster at the end with all the software coming from the recipe they chose.

That initial recipe had a resource manager, one provisioner, one architecture, and one OS. Now I think we are up to 10 or 12 different recipes because we have support for multiple OSs, multiple architectures. We now have multiple provisioners so people can choose from Slurm or PBS professional, etc. We are trying to curate these recipes and give people options. That increases our testing time, but we think it is important because for some people to be able to make those kinds of decisions based on what they are familiar with.

HPCwire: Given the effort required to create multiple recipes, is it paying off in terms of expanding HPC use?

Schulz: I think we have seen that being helpful in areas that you might say are non-traditional HPC where people just want to get an HPC cluster up and running to run a piece of software. We have some examples of medical organizations doing this, sort of setting up their environment to do genomics and the like. We also have people using it in finance. So we do see evidence of people using OpenHPC in areas where we say ‘well that’s not traditional HPC but they have a need for doing parallel processing.’

HPCwire: Seems like adding AI processing capabilities should be a priority?

Schulz: Obviously AI is big now so people are using OpenHPC with AI packages. We’ve not added anything specific yet, but I am thinking that at some point it will bubble up. We did an internal survey recently asking that very question – do you have an interest in AI workloads, are you using this stuff? – and there was a strong interest. At some point we will get a community request for that, TensorFlow and Caffe and that kind of stuff, and that will make us review it. We might proactively do it, but today the components that are in OpenHPC are a little more generic-HPC.

HPCwire: When you do a new release, how much of the software is actually changing?

Schulz: You have to be aware that the open source software environment changes pretty fast and you can get out of date quickly. We see that as pretty important. When we are working on a new release we scan the upstream community, all the stuff that’s going on in HPC, and if there is a new stable piece of software, we start trying to use that right away and as long as we don’t find a problem with it. We always go with the new one. We have been doing this for a couple of years now and that allows us to go back and sort of look at the metrics for how fast the software is evolving.

Some software changes every time, like Slurm resource manager. There’s going to be a new release pretty often. Some stuff doesn’t change a lot. If you look at the aggregate of the software stack for OpenHPC – and we’ve been doing releases about every quarter – we’ve seen consistently about 40 percent of the components that are in OpenHPC have a new version. Every single release, we are updating and testing a new version. That requires a lot of work and upkeep, but we think that’s an important part of the delivery for the project because if you are starting from scratch, just now installing your system, you always want the latest version.

We’ve been trying to get a feel for how many people were doing upgrades. Of course, you also have some small cadre of folks that will never update. But it is a tiny amount. I think right now something like 90 percent of the folks accessing packages are on the latest release which certainly makes us happy.

HPCwire: What’s coming out at SC18?

Schulz: The release that is currently out is what we call 1.3.5 and the release that will be coming out at Supercomputing is 1.3.6. Some of the planned stuff is the usual number of software updates; so, the 40 percent of the software number I just quoted will have new versions and we are right now pretty close to that at 38 percent. We’ll be adding some new packages and those all came from this components submission request process. We have made some packaging changes based on some lessons along the way. We are introducing a new compiler variant. One of the things about OpenHPC is that we try to adopt a very hierarchical software environment. This is based on experience at large HPC sites where we have architected a system for which we know users will have different versions of different software. They may want to have major version of some components like compilers and other [older] components.

Like I said, we try to stay as current as we can. The 1.3.5 release out there today has the latest version of GCC which was at that time GCC7.  Now there’s a GCC8 introducing a whole family of builds that are supported with GCC8. The way that we do things is that people who have existing systems, they can opt in if they want to access a new compiler variant and all the builds that go with that; for someone building a system from scratch the defaults leverage that new version of GCC. It sounds like kind of a small thing, but because of the hierarchical nature of the software environment that we have, it is actually a lot of effort on our side because we are introducing every single development tool at the same time we roll out a new compiler variant.

HPCwire: As you’ve noted it’s a lot of work. In terms of hardware, my understanding is much of the infrastructure used by OpenHPC is housed at TACC. Is that correct?

Schulz: You may know I was at TACC for a number of years previously and UT/TACC is one of the formal academic members of the project thru the Linux Foundation. We have one member on the Technical Steering Committee from TACC presently (Cyrus Proctor), and the fine folks there have been kind enough to host continuous integration clusters in their data center along with some build servers. The CI infrastructure hosting at TACC has been there since the early days and is helped with hardware donations from Intel, Cavium, and Dell.  We also have additional build server support hosted at Packet through the Works on Arm Project and leverage additional cloud infrastructure through Amazon’s EC2.


HPCwire
: Looking ahead what’s on the roadmap?

Schulz: We won’t have it for this release but early in 2019 for the follow-on release 1.3.7 we’ll be introducing some additional support to leverage the Arm compiler. Today we do builds leveraging different compiler tool chains including GCC and on the x86 side we also to build on Intel compilers. Arm has their own HPC compiler toolchain and we have been working with them to get that build up and running against that toolchain, which is a little different because it is LLVM based. I would expect people should see in 2019 builds against the Arm compilers which will help for folks running on Arm because you will also get access to their math library and builds for all the open source software that can take advantage of any performance optimizations that they are putting in the math library.

I also think one of the efforts you’ll see also come out of the community project is recipes that are more automated that use one particular config management. Right now we’ve had people working on something that’s Ansible. I would expect that you would see us publish something that is Ansible that would allow people to install OpenHPC. Another possibility is we know there are people working with the OpenHPC stack in a cloud environment, but it’s up to them how to get that out. I see a possibility where we might publish some reference images for some of the more common cloud service providers to make it easier to spin up an HPC cloud on an OpenHPC cluster for example.

HPCwire: Anything more exotic in the wings? I know you’ve just returned from China where several processor efforts are ongoing?

Schulz:Everything that we are operating on requires processors that have been GA’d. We wouldn’t be in a position to push out a stack for something that’s not been GA’d. I’ll say that the same way we would like to stay current on the upstream software, as new hardware rolls out that is important for HPC, we also would like to stay current with that as well. I think you can imagine that’s also a motivation for some of the partner companies to be involved to make sure the community stack runs on their new technology. As a new processor comes out, the community will have to get access to it and we’ll be in a position to roll out builds against it.

HPCwire: Thanks again for your time Karl.

Bio: Karl W. Schulz, is Research Associate Professor – Institute for Computational Engineering and Sciences (ICES), Associate Professor – Women’s Health, Dell Medical School, The University of Texas at Austin

Slides source: OpenHPC Update, August 2018, https://github.com/openhpc/ohpc/wiki/Papers-and-Presentations

Subscribe to HPCwire's Weekly Update!

Be the most informed person in the room! Stay ahead of the tech trends with industy updates delivered to you every week!

Pfizer HPC Engineer Aims to Automate Software Stack Testing

January 17, 2019

Seeking to reign in the tediousness of manual software testing, Pfizer HPC Engineer Shahzeb Siddiqui is developing an open source software tool called buildtest, aimed at automating software stack testing by providing the community with a central repository of tests for common HPC apps and the ability to automate execution of testing. Read more…

By Tiffany Trader

Senegal Prepares to Take Delivery of Atos Supercomputer

January 16, 2019

Update (Jan. 21): HPCwire has received confirmation from Atos that the system will have a peak speed of 537.6 teraflops, not 320 teraflops as had previously been reported. We plan to report additional details as we recei Read more…

By Tiffany Trader

Google Cloud Platform Extends GPU Instance Options

January 16, 2019

If it's Nvidia GPUs you're after to power your AI/HPC/visualization workload, Google Cloud has them, now claiming "broadest GPU availability." Each of the three big public cloud vendors has by turn touted the latest and Read more…

By Tiffany Trader

HPE Extreme Performance Solutions

HPE Systems With Intel Omni-Path: Architected for Value and Accessible High-Performance Computing

Today’s high-performance computing (HPC) and artificial intelligence (AI) users value high performing clusters. And the higher the performance that their system can deliver, the better. Read more…

IBM Accelerated Insights

Resource Management in the Age of Artificial Intelligence

New challenges demand fresh approaches

Fueled by GPUs, big data, and rapid advances in software, the AI revolution is upon us. Read more…

STAC Floats ML Benchmark for Financial Services Workloads

January 16, 2019

STAC (Securities Technology Analysis Center) recently released an ‘exploratory’ benchmark for machine learning which it hopes will evolve into a firm benchmark or suite of benchmarking tools to compare the performanc Read more…

By John Russell

Google Cloud Platform Extends GPU Instance Options

January 16, 2019

If it's Nvidia GPUs you're after to power your AI/HPC/visualization workload, Google Cloud has them, now claiming "broadest GPU availability." Each of the three Read more…

By Tiffany Trader

STAC Floats ML Benchmark for Financial Services Workloads

January 16, 2019

STAC (Securities Technology Analysis Center) recently released an ‘exploratory’ benchmark for machine learning which it hopes will evolve into a firm benchm Read more…

By John Russell

A Big Data Journey While Seeking to Catalog our Universe

January 16, 2019

It turns out, astronomers have lots of photos of the sky but seek knowledge about what the photos mean. Sound familiar? Big data problems are often characterize Read more…

By James Reinders

Intel Bets Big on 2-Track Quantum Strategy

January 15, 2019

Quantum computing has lived so long in the future it’s taken on a futuristic life of its own, with a Gartner-style hype cycle that includes triggers of innovation, inflated expectations and – though a useful quantum system is still years away – anticipatory troughs of disillusionment. Read more…

By Doug Black

IBM Quantum Update: Q System One Launch, New Collaborators, and QC Center Plans

January 10, 2019

IBM made three significant quantum computing announcements at CES this week. One was introduction of IBM Q System One; it’s really the integration of IBM’s Read more…

By John Russell

IBM’s New Global Weather Forecasting System Runs on GPUs

January 9, 2019

Anyone who has checked a forecast to decide whether or not to pack an umbrella knows that weather prediction can be a mercurial endeavor. It is a Herculean task: the constant modeling of incredibly complex systems to a high degree of accuracy at a local level within very short spans of time. Read more…

By Oliver Peckham

The Case Against ‘The Case Against Quantum Computing’

January 9, 2019

It’s not easy to be a physicist. Richard Feynman (basically the Jimi Hendrix of physicists) once said: “The first principle is that you must not fool yourse Read more…

By Ben Criger

The Deep500 – Researchers Tackle an HPC Benchmark for Deep Learning

January 7, 2019

How do you know if an HPC system, particularly a larger-scale system, is well-suited for deep learning workloads? Today, that’s not an easy question to answer Read more…

By John Russell

Quantum Computing Will Never Work

November 27, 2018

Amid the gush of money and enthusiastic predictions being thrown at quantum computing comes a proposed cold shower in the form of an essay by physicist Mikhail Read more…

By John Russell

Cray Unveils Shasta, Lands NERSC-9 Contract

October 30, 2018

Cray revealed today the details of its next-gen supercomputing architecture, Shasta, selected to be the next flagship system at NERSC. We've known of the code-name "Shasta" since the Argonne slice of the CORAL project was announced in 2015 and although the details of that plan have changed considerably, Cray didn't slow down its timeline for Shasta. Read more…

By Tiffany Trader

AMD Sets Up for Epyc Epoch

November 16, 2018

It’s been a good two weeks, AMD’s Gary Silcott and Andy Parma told me on the last day of SC18 in Dallas at the restaurant where we met to discuss their show news and recent successes. Heck, it’s been a good year. Read more…

By Tiffany Trader

The Case Against ‘The Case Against Quantum Computing’

January 9, 2019

It’s not easy to be a physicist. Richard Feynman (basically the Jimi Hendrix of physicists) once said: “The first principle is that you must not fool yourse Read more…

By Ben Criger

US Leads Supercomputing with #1, #2 Systems & Petascale Arm

November 12, 2018

The 31st Supercomputing Conference (SC) - commemorating 30 years since the first Supercomputing in 1988 - kicked off in Dallas yesterday, taking over the Kay Ba Read more…

By Tiffany Trader

Contract Signed for New Finnish Supercomputer

December 13, 2018

After the official contract signing yesterday, configuration details were made public for the new BullSequana system that the Finnish IT Center for Science (CSC Read more…

By Tiffany Trader

Nvidia’s Jensen Huang Delivers Vision for the New HPC

November 14, 2018

For nearly two hours on Monday at SC18, Jensen Huang, CEO of Nvidia, presented his expansive view of the future of HPC (and computing in general) as only he can do. Animated. Backstopped by a stream of data charts, product photos, and even a beautiful image of supernovae... Read more…

By John Russell

HPE No. 1, IBM Surges, in ‘Bucking Bronco’ High Performance Server Market

September 27, 2018

Riding healthy U.S. and global economies, strong demand for AI-capable hardware and other tailwind trends, the high performance computing server market jumped 28 percent in the second quarter 2018 to $3.7 billion, up from $2.9 billion for the same period last year, according to industry analyst firm Hyperion Research. Read more…

By Doug Black

Leading Solution Providers

SC 18 Virtual Booth Video Tour

Advania @ SC18 AMD @ SC18
ASRock Rack @ SC18
DDN Storage @ SC18
HPE @ SC18
IBM @ SC18
Lenovo @ SC18 Mellanox Technologies @ SC18
NVIDIA @ SC18
One Stop Systems @ SC18
Oracle @ SC18 Panasas @ SC18
Supermicro @ SC18 SUSE @ SC18 TYAN @ SC18
Verne Global @ SC18

HPC Reflections and (Mostly Hopeful) Predictions

December 19, 2018

So much ‘spaghetti’ gets tossed on walls by the technology community (vendors and researchers) to see what sticks that it is often difficult to peer through Read more…

By John Russell

Intel Confirms 48-Core Cascade Lake-AP for 2019

November 4, 2018

As part of the run-up to SC18, taking place in Dallas next week (Nov. 11-16), Intel is doling out info on its next-gen Cascade Lake family of Xeon processors, specifically the “Advanced Processor” version (Cascade Lake-AP), architected for high-performance computing, artificial intelligence and infrastructure-as-a-service workloads. Read more…

By Tiffany Trader

Germany Celebrates Launch of Two Fastest Supercomputers

September 26, 2018

The new high-performance computer SuperMUC-NG at the Leibniz Supercomputing Center (LRZ) in Garching is the fastest computer in Germany and one of the fastest i Read more…

By Tiffany Trader

Microsoft to Buy Mellanox?

December 20, 2018

Networking equipment powerhouse Mellanox could be an acquisition target by Microsoft, according to a published report in an Israeli financial publication. Microsoft has reportedly gone so far as to engage Goldman Sachs to handle negotiations with Mellanox. Read more…

By Doug Black

Houston to Field Massive, ‘Geophysically Configured’ Cloud Supercomputer

October 11, 2018

Based on some news stories out today, one might get the impression that the next system to crack number one on the Top500 would be an industrial oil and gas mon Read more…

By Tiffany Trader

The Deep500 – Researchers Tackle an HPC Benchmark for Deep Learning

January 7, 2019

How do you know if an HPC system, particularly a larger-scale system, is well-suited for deep learning workloads? Today, that’s not an easy question to answer Read more…

By John Russell

Summit Supercomputer is Already Making its Mark on Science

September 20, 2018

Summit, now the fastest supercomputer in the world, is quickly making its mark in science – five of the six finalists just announced for the prestigious 2018 Read more…

By John Russell

IBM Quantum Update: Q System One Launch, New Collaborators, and QC Center Plans

January 10, 2019

IBM made three significant quantum computing announcements at CES this week. One was introduction of IBM Q System One; it’s really the integration of IBM’s Read more…

By John Russell

  • arrow
  • Click Here for More Headlines
  • arrow
Do NOT follow this link or you will be banned from the site!
Share This