OpenHPC Gathers Steam, Preps New Release for SC18

By John Russell

November 1, 2018

You may recall at SC15 an Intel-led group introduced OpenHPC – an effort to build a community project around an open source, plug-and-play HPC stack. One goal was to make standing up HPC systems easier and thereby accelerate HPC adoption beyond its traditional landscape, particularly in the enterprise. Another goal – given the diversity and ever-evolving nature of HPC stack options and practices – was to create a community-driven mechanism for testing and integrating new elements into the ‘standardized’ stack. Yet a third idea was to accommodate emerging vendor platforms and hardware (mostly processors) ecosystems.

So how’s it going?

The short answer is quite well, but with room for growth. HPCwire recently talked with Karl Schulz, project leader for OpenHPC’s technical steering committee, as he and the team were racing to finish release 1.3.6 in time for SC18 where for the first time OpenHPC will have a booth in addition to its popular Birds of a Feather (BOF) gathering. A fair amount has happened since OpenHPC’s start, which was greeted hopefully and perhaps a little warily because of Intel’s prominence in the group. Those fears seem to have been unwarranted and OpenHPC now also supports ARM. No sign of IBM/Power coming into the fold yet but there’s plenty on the OpenHPC roadmap.

HPCwire: Thanks for taking time to give an update. Maybe a quick organization summary would be helpful.

Karl Schulz, OpenHPC

Schulz: Sure. In 2015 when the project started it was a relatively small number of folks working together. It wasn’t until 2016 that we really became fully operational as a collaborative project under the Linux Foundation. That meant setting a governance structure which is two pronged. We have a board for budgetary items. All the technical leadership is done through a technical steering committee. Our steering committee members participate on a term for one year. Elections take place in the summer, so right now we have a mix of old and new folks. There are roughly 20 people on the steering committee. They represent interests that are similarly aligned to the overall interest in OpenHPC, so there are folks from supercomputing sites, the elite labs, academic sites; there are vendor folks, and developers. It’s a good mix of people and all of that information is available on our GitHub page.

HPCwire: What about growth in the OpenHPC community overall? 

Schulz: In terms of other adoption and interest in the community, when we started – and I’d have to go back for the exact numbers – but I’m guessing we had approximately 20-25 formal members. We have continued to grow year over year. As of right now we have about 38 members who are formally associated with the project. They have the same span of affiliations [as the technical steering committee]. Along the way, as you know, Arm joined, so we are doing builds and recipes that are for multiple architectures now. We have multiple OS vendors associated with the project in terms of Red Hat and SLES (SUSE Linux Enterprise Server) so we push out builds for both of those. We have seen continued growth in the membership.

HPCwire: As I understand it, the OpenHPC stack itself has grown considerably. 

Schulz: If we go back to the very first release and compare it to now, we almost double the number of software components that are in the curated OpenHPC stack. That has implications for all of our build infrastructure and our technical infrastructure, but we see it as important to be able to grow the stack over time. One of the first things we did on the technical steering committee side is we put together a mechanism by which the community could suggest or request packages to be added to the stack. You don’t have to be associated with the project. Anybody can come in to suggest and then the technical committee can review those requests on roughly a quarterly basis and make decisions. That’s how the majority of all this software has found its way into OpenHPC.

You know it’s things that are both administrative, like provisioners, and things of growing interest in the community like containerization. We’ve added Singularity and Charliecloud, a bunch of numerical packages in terms of finite elements and linear algebra. And in IO clients, we’ve added support for things like BeeGFS. We’re also adding ‘new’ software as it develops, so one example is an exascale runtime for job launches.

HPCwire: It’s good to have buy-in from contributors and vendors but what can you say about user traction?

Schulz: I get this question fairly frequently – who’s using OpenHPC? Of course we are an open source project and don’t require any registration so you don’t know necessarily who’s using it. But we do mine our logs just to see general growth trends. We’ve had roughly year-over-year in the last three years a doubling of the number of folks who are hitting our build repository. I think right now in 2018 we are averaging something like 3,000 visitors a month. People are accessing our binary builds to the tune of about a terabyte and a half of data downloaded every month. We have seen basically a 2x growth every year and we feel pretty good about that.

HPCwire: When OpenHPC was first announced at SC15 there was concern that Intel’s influence might make it “less open.” That said, the reality is clusters are still largely an x86 world. How’s progress on support for other ecosystems?  

Schulz: At the last SC is when we had the formal release supporting Arm. Before we had a test release preview and the reason we came out first with a test preview was we didn’t have any software which could actually provision bare metal Arm, at least not the software we were using. Once we got that sorted out, we formally had a real Arm release that coincided with when we started to see more and more Arm hardware work its way into the general public.

We have seen some uptake and can sort of delineate between x86 and Arm because we were starting from zero. We have seen that grow particularly this year. Use is still dominated by x86 as you might expect, but definitely there’s a steady growth in people interested in Arm. I think the community project is actually well suited to help in that area because there’s just not a lot of things you can use out-of-the-box on the Arm ecosystem. We’ve benefited a lot from having the Arm folks involved in the project.

Many of the component packages, particularly the development environment tools, may not have necessarily been built on Arm architecture upfront, so we had to work through a lot of mostly small issues. I’d say at least 70 percent of the packages that we had in OpenHPC needed a little bit of work to get up and running on Arm. We’re able to leverage our testing infrastructure for both x86 and Arm, where we can actually make sure not only are the builds working but we can actually run applications against these libraries in an HPC environment under a resource manager. Even people who aren’t using OpenHPC will benefit from this work because we got a lot of the packages up and running and it served upstream to the community so other people can build on Arm as well.

HPCwire: IBM’s Power-based systems are garnering attention with the success of the Summit and Sierra supercomputers. IBM told HPCwire back at SC16 it wasn’t opposed to joining OpenHPC but we’ve not heard of any movement in that direction. Do you have insight there?

Schulz: I don’t. We had some discussion about them possibly being interested in joining and they have had some discussion with the Linux Foundation. I don’t know what their thinking is. They are obviously welcome. We pointed them to the Linux Foundation folks several times. I believe there have been discussions [in the past] but until now they are not a member yet. It would not go through me anyway. I’m on the technical side of things. It wouldn’t surprise me if these discussions come up at SC.

 

HPCwire: Let’s shift gears for a second and talk about OpenHPC’s approach to creating these plug-and-play stacks and relate that to helping expand HPC use. 

Schulz: What we try to do is be very building block oriented. So for people who have a lot of expertise and have a strong opinion about how to do things, they can sort of pick and choose what they want. They can use their own provisioner and their own config management system that they probably spent years developing. We wanted to make sure that we as a community effort weren’t doing something that was so locked down in terms of having to do it one and only one way that it wouldn’t be of interest to those folks. So that’s one aspect.

So we have always kept everything very building block oriented, but to your leading point, we know that there’s a lot of folks who don’t have that kind of expertise and they are looking for guidance. That’s the other part of what we are trying to do is leverage the expertise and folks involved with the community who have been involved in standing up HPC systems for years and have codified some best practice, if you will, into a number of different recipes.

HPCwire: How’s that going?

Schulz: When we first launched OpenHPC, we had one recipe and when I say one recipe, for us that is a document that is validated. It’s something for people if they are starting from bare metal or they are doing something with virtual machines that they can take and follow end-to-end and have a working cluster at the end with all the software coming from the recipe they chose.

That initial recipe had a resource manager, one provisioner, one architecture, and one OS. Now I think we are up to 10 or 12 different recipes because we have support for multiple OSs, multiple architectures. We now have multiple provisioners so people can choose from Slurm or PBS professional, etc. We are trying to curate these recipes and give people options. That increases our testing time, but we think it is important because for some people to be able to make those kinds of decisions based on what they are familiar with.

HPCwire: Given the effort required to create multiple recipes, is it paying off in terms of expanding HPC use?

Schulz: I think we have seen that being helpful in areas that you might say are non-traditional HPC where people just want to get an HPC cluster up and running to run a piece of software. We have some examples of medical organizations doing this, sort of setting up their environment to do genomics and the like. We also have people using it in finance. So we do see evidence of people using OpenHPC in areas where we say ‘well that’s not traditional HPC but they have a need for doing parallel processing.’

HPCwire: Seems like adding AI processing capabilities should be a priority?

Schulz: Obviously AI is big now so people are using OpenHPC with AI packages. We’ve not added anything specific yet, but I am thinking that at some point it will bubble up. We did an internal survey recently asking that very question – do you have an interest in AI workloads, are you using this stuff? – and there was a strong interest. At some point we will get a community request for that, TensorFlow and Caffe and that kind of stuff, and that will make us review it. We might proactively do it, but today the components that are in OpenHPC are a little more generic-HPC.

HPCwire: When you do a new release, how much of the software is actually changing?

Schulz: You have to be aware that the open source software environment changes pretty fast and you can get out of date quickly. We see that as pretty important. When we are working on a new release we scan the upstream community, all the stuff that’s going on in HPC, and if there is a new stable piece of software, we start trying to use that right away and as long as we don’t find a problem with it. We always go with the new one. We have been doing this for a couple of years now and that allows us to go back and sort of look at the metrics for how fast the software is evolving.

Some software changes every time, like Slurm resource manager. There’s going to be a new release pretty often. Some stuff doesn’t change a lot. If you look at the aggregate of the software stack for OpenHPC – and we’ve been doing releases about every quarter – we’ve seen consistently about 40 percent of the components that are in OpenHPC have a new version. Every single release, we are updating and testing a new version. That requires a lot of work and upkeep, but we think that’s an important part of the delivery for the project because if you are starting from scratch, just now installing your system, you always want the latest version.

We’ve been trying to get a feel for how many people were doing upgrades. Of course, you also have some small cadre of folks that will never update. But it is a tiny amount. I think right now something like 90 percent of the folks accessing packages are on the latest release which certainly makes us happy.

HPCwire: What’s coming out at SC18?

Schulz: The release that is currently out is what we call 1.3.5 and the release that will be coming out at Supercomputing is 1.3.6. Some of the planned stuff is the usual number of software updates; so, the 40 percent of the software number I just quoted will have new versions and we are right now pretty close to that at 38 percent. We’ll be adding some new packages and those all came from this components submission request process. We have made some packaging changes based on some lessons along the way. We are introducing a new compiler variant. One of the things about OpenHPC is that we try to adopt a very hierarchical software environment. This is based on experience at large HPC sites where we have architected a system for which we know users will have different versions of different software. They may want to have major version of some components like compilers and other [older] components.

Like I said, we try to stay as current as we can. The 1.3.5 release out there today has the latest version of GCC which was at that time GCC7.  Now there’s a GCC8 introducing a whole family of builds that are supported with GCC8. The way that we do things is that people who have existing systems, they can opt in if they want to access a new compiler variant and all the builds that go with that; for someone building a system from scratch the defaults leverage that new version of GCC. It sounds like kind of a small thing, but because of the hierarchical nature of the software environment that we have, it is actually a lot of effort on our side because we are introducing every single development tool at the same time we roll out a new compiler variant.

HPCwire: As you’ve noted it’s a lot of work. In terms of hardware, my understanding is much of the infrastructure used by OpenHPC is housed at TACC. Is that correct?

Schulz: You may know I was at TACC for a number of years previously and UT/TACC is one of the formal academic members of the project thru the Linux Foundation. We have one member on the Technical Steering Committee from TACC presently (Cyrus Proctor), and the fine folks there have been kind enough to host continuous integration clusters in their data center along with some build servers. The CI infrastructure hosting at TACC has been there since the early days and is helped with hardware donations from Intel, Cavium, and Dell.  We also have additional build server support hosted at Packet through the Works on Arm Project and leverage additional cloud infrastructure through Amazon’s EC2.


HPCwire
: Looking ahead what’s on the roadmap?

Schulz: We won’t have it for this release but early in 2019 for the follow-on release 1.3.7 we’ll be introducing some additional support to leverage the Arm compiler. Today we do builds leveraging different compiler tool chains including GCC and on the x86 side we also to build on Intel compilers. Arm has their own HPC compiler toolchain and we have been working with them to get that build up and running against that toolchain, which is a little different because it is LLVM based. I would expect people should see in 2019 builds against the Arm compilers which will help for folks running on Arm because you will also get access to their math library and builds for all the open source software that can take advantage of any performance optimizations that they are putting in the math library.

I also think one of the efforts you’ll see also come out of the community project is recipes that are more automated that use one particular config management. Right now we’ve had people working on something that’s Ansible. I would expect that you would see us publish something that is Ansible that would allow people to install OpenHPC. Another possibility is we know there are people working with the OpenHPC stack in a cloud environment, but it’s up to them how to get that out. I see a possibility where we might publish some reference images for some of the more common cloud service providers to make it easier to spin up an HPC cloud on an OpenHPC cluster for example.

HPCwire: Anything more exotic in the wings? I know you’ve just returned from China where several processor efforts are ongoing?

Schulz:Everything that we are operating on requires processors that have been GA’d. We wouldn’t be in a position to push out a stack for something that’s not been GA’d. I’ll say that the same way we would like to stay current on the upstream software, as new hardware rolls out that is important for HPC, we also would like to stay current with that as well. I think you can imagine that’s also a motivation for some of the partner companies to be involved to make sure the community stack runs on their new technology. As a new processor comes out, the community will have to get access to it and we’ll be in a position to roll out builds against it.

HPCwire: Thanks again for your time Karl.

Bio: Karl W. Schulz, is Research Associate Professor – Institute for Computational Engineering and Sciences (ICES), Associate Professor – Women’s Health, Dell Medical School, The University of Texas at Austin

Slides source: OpenHPC Update, August 2018, https://github.com/openhpc/ohpc/wiki/Papers-and-Presentations

Subscribe to HPCwire's Weekly Update!

Be the most informed person in the room! Stay ahead of the tech trends with industy updates delivered to you every week!

Talk to Me: Nvidia Claims NLP Inference, Training Records

August 15, 2019

Nvidia says it’s achieved significant advances in conversation natural language processing (NLP) training and inference, enabling more complex, immediate-response interchanges between customers and chatbots. And the co Read more…

By Doug Black

Trump Administration and NIST Issue AI Standards Development Plan

August 14, 2019

Efforts to develop AI are gathering steam fast. On Monday, the White House issued a federal plan to help develop technical standards for AI following up on a mandate contained in the Administration’s AI Executive Order Read more…

By John Russell

Scientists to Tap Exascale Computing to Unlock the Mystery of our Accelerating Universe

August 14, 2019

The universe and everything in it roared to life with the Big Bang approximately 13.8 billion years ago. It has continued expanding ever since. While we have a good understanding of the early universe, its fate billions Read more…

By Rob Johnson

AWS Solution Channel

Efficiency and Cost-Optimization for HPC Workloads – AWS Batch and Amazon EC2 Spot Instances

High Performance Computing on AWS leverages the power of cloud computing and the extreme scale it offers to achieve optimal HPC price/performance. With AWS you can right size your services to meet exactly the capacity requirements you need without having to overprovision or compromise capacity. Read more…

HPE Extreme Performance Solutions

Bring the combined power of HPC and AI to your business transformation

FPGA (Field Programmable Gate Array) acceleration cards are not new, as they’ve been commercially available since 1984. Typically, the emphasis around FPGAs has centered on the fact that they’re programmable accelerators, and that they can truly offer workload specific hardware acceleration solutions without requiring custom silicon. Read more…

IBM Accelerated Insights

Cloudy with a Chance of Mainframes

[Connect with HPC users and learn new skills in the IBM Spectrum LSF User Community.]

Rapid rates of change sometimes result in unexpected bedfellows. Read more…

Argonne Supercomputer Accelerates Cancer Prediction Research

August 13, 2019

In the fight against cancer, early prediction, which drastically improves prognoses, is critical. Now, new research by a team from Northwestern University – and accelerated by supercomputing resources at Argonne Nation Read more…

By Oliver Peckham

Scientists to Tap Exascale Computing to Unlock the Mystery of our Accelerating Universe

August 14, 2019

The universe and everything in it roared to life with the Big Bang approximately 13.8 billion years ago. It has continued expanding ever since. While we have a Read more…

By Rob Johnson

AI is the Next Exascale – Rick Stevens on What that Means and Why It’s Important

August 13, 2019

Twelve years ago the Department of Energy (DOE) was just beginning to explore what an exascale computing program might look like and what it might accomplish. Today, DOE is repeating that process for AI, once again starting with science community town halls to gather input and stimulate conversation. The town hall program... Read more…

By Tiffany Trader and John Russell

Cray Wins NNSA-Livermore ‘El Capitan’ Exascale Contract

August 13, 2019

Cray has won the bid to build the first exascale supercomputer for the National Nuclear Security Administration (NNSA) and Lawrence Livermore National Laborator Read more…

By Tiffany Trader

AMD Launches Epyc Rome, First 7nm CPU

August 8, 2019

From a gala event at the Palace of Fine Arts in San Francisco yesterday (Aug. 7), AMD launched its second-generation Epyc Rome x86 chips, based on its 7nm proce Read more…

By Tiffany Trader

Lenovo Drives Single-Socket Servers with AMD Epyc Rome CPUs

August 7, 2019

No summer doldrums here. As part of the AMD Epyc Rome launch event in San Francisco today, Lenovo announced two new single-socket servers, the ThinkSystem SR635 Read more…

By Doug Black

Building Diversity and Broader Engagement in the HPC Community

August 7, 2019

Increasing diversity and inclusion in HPC is a community-building effort. Representation of both issues and individuals matters - the more people see HPC in a w Read more…

By AJ Lauer

Xilinx vs. Intel: FPGA Market Leaders Launch Server Accelerator Cards

August 6, 2019

The two FPGA market leaders, Intel and Xilinx, both announced new accelerator cards this week designed to handle specialized, compute-intensive workloads and un Read more…

By Doug Black

Upcoming NSF Cyberinfrastructure Projects to Support ‘Long-Tail’ Users, AI and Big Data

August 5, 2019

The National Science Foundation is well positioned to support national priorities, as new NSF-funded HPC systems to come online in the upcoming year promise to Read more…

By Ken Chiacchia, Pittsburgh Supercomputing Center/XSEDE

High Performance (Potato) Chips

May 5, 2006

In this article, we focus on how Procter & Gamble is using high performance computing to create some common, everyday supermarket products. Tom Lange, a 27-year veteran of the company, tells us how P&G models products, processes and production systems for the betterment of consumer package goods. Read more…

By Michael Feldman

Supercomputer-Powered AI Tackles a Key Fusion Energy Challenge

August 7, 2019

Fusion energy is the Holy Grail of the energy world: low-radioactivity, low-waste, zero-carbon, high-output nuclear power that can run on hydrogen or lithium. T Read more…

By Oliver Peckham

Cray, AMD to Extend DOE’s Exascale Frontier

May 7, 2019

Cray and AMD are coming back to Oak Ridge National Laboratory to partner on the world’s largest and most expensive supercomputer. The Department of Energy’s Read more…

By Tiffany Trader

Graphene Surprises Again, This Time for Quantum Computing

May 8, 2019

Graphene is fascinating stuff with promise for use in a seeming endless number of applications. This month researchers from the University of Vienna and Institu Read more…

By John Russell

AMD Verifies Its Largest 7nm Chip Design in Ten Hours

June 5, 2019

AMD announced last week that its engineers had successfully executed the first physical verification of its largest 7nm chip design – in just ten hours. The AMD Radeon Instinct Vega20 – which boasts 13.2 billion transistors – was tested using a TSMC-certified Calibre nmDRC software platform from Mentor. Read more…

By Oliver Peckham

TSMC and Samsung Moving to 5nm; Whither Moore’s Law?

June 12, 2019

With reports that Taiwan Semiconductor Manufacturing Co. (TMSC) and Samsung are moving quickly to 5nm manufacturing, it’s a good time to again ponder whither goes the venerable Moore’s law. Shrinking feature size has of course been the primary hallmark of achieving Moore’s law... Read more…

By John Russell

Deep Learning Competitors Stalk Nvidia

May 14, 2019

There is no shortage of processing architectures emerging to accelerate deep learning workloads, with two more options emerging this week to challenge GPU leader Nvidia. First, Intel researchers claimed a new deep learning record for image classification on the ResNet-50 convolutional neural network. Separately, Israeli AI chip startup Hailo.ai... Read more…

By George Leopold

Nvidia Embraces Arm, Declares Intent to Accelerate All CPU Architectures

June 17, 2019

As the Top500 list was being announced at ISC in Frankfurt today with an upgraded petascale Arm supercomputer in the top third of the list, Nvidia announced its Read more…

By Tiffany Trader

Leading Solution Providers

ISC 2019 Virtual Booth Video Tour

CRAY
CRAY
DDN
DDN
DELL EMC
DELL EMC
GOOGLE
GOOGLE
ONE STOP SYSTEMS
ONE STOP SYSTEMS
PANASAS
PANASAS
VERNE GLOBAL
VERNE GLOBAL

Cray Wins NNSA-Livermore ‘El Capitan’ Exascale Contract

August 13, 2019

Cray has won the bid to build the first exascale supercomputer for the National Nuclear Security Administration (NNSA) and Lawrence Livermore National Laborator Read more…

By Tiffany Trader

Top500 Purely Petaflops; US Maintains Performance Lead

June 17, 2019

With the kick-off of the International Supercomputing Conference (ISC) in Frankfurt this morning, the 53rd Top500 list made its debut, and this one's for petafl Read more…

By Tiffany Trader

A Behind-the-Scenes Look at the Hardware That Powered the Black Hole Image

June 24, 2019

Two months ago, the first-ever image of a black hole took the internet by storm. A team of scientists took years to produce and verify the striking image – an Read more…

By Oliver Peckham

Cray – and the Cray Brand – to Be Positioned at Tip of HPE’s HPC Spear

May 22, 2019

More so than with most acquisitions of this kind, HPE’s purchase of Cray for $1.3 billion, announced last week, seems to have elements of that overused, often Read more…

By Doug Black and Tiffany Trader

AMD Launches Epyc Rome, First 7nm CPU

August 8, 2019

From a gala event at the Palace of Fine Arts in San Francisco yesterday (Aug. 7), AMD launched its second-generation Epyc Rome x86 chips, based on its 7nm proce Read more…

By Tiffany Trader

Chinese Company Sugon Placed on US ‘Entity List’ After Strong Showing at International Supercomputing Conference

June 26, 2019

After more than a decade of advancing its supercomputing prowess, operating the world’s most powerful supercomputer from June 2013 to June 2018, China is keep Read more…

By Tiffany Trader

In Wake of Nvidia-Mellanox: Xilinx to Acquire Solarflare

April 25, 2019

With echoes of Nvidia’s recent acquisition of Mellanox, FPGA maker Xilinx has announced a definitive agreement to acquire Solarflare Communications, provider Read more…

By Doug Black

Qualcomm Invests in RISC-V Startup SiFive

June 7, 2019

Investors are zeroing in on the open standard RISC-V instruction set architecture and the processor intellectual property being developed by a batch of high-flying chip startups. Last fall, Esperanto Technologies announced a $58 million funding round. Read more…

By George Leopold

  • arrow
  • Click Here for More Headlines
  • arrow
Do NOT follow this link or you will be banned from the site!
Share This