OpenHPC Gathers Steam, Preps New Release for SC18

By John Russell

November 1, 2018

You may recall at SC15 an Intel-led group introduced OpenHPC – an effort to build a community project around an open source, plug-and-play HPC stack. One goal was to make standing up HPC systems easier and thereby accelerate HPC adoption beyond its traditional landscape, particularly in the enterprise. Another goal – given the diversity and ever-evolving nature of HPC stack options and practices – was to create a community-driven mechanism for testing and integrating new elements into the ‘standardized’ stack. Yet a third idea was to accommodate emerging vendor platforms and hardware (mostly processors) ecosystems.

So how’s it going?

The short answer is quite well, but with room for growth. HPCwire recently talked with Karl Schulz, project leader for OpenHPC’s technical steering committee, as he and the team were racing to finish release 1.3.6 in time for SC18 where for the first time OpenHPC will have a booth in addition to its popular Birds of a Feather (BOF) gathering. A fair amount has happened since OpenHPC’s start, which was greeted hopefully and perhaps a little warily because of Intel’s prominence in the group. Those fears seem to have been unwarranted and OpenHPC now also supports ARM. No sign of IBM/Power coming into the fold yet but there’s plenty on the OpenHPC roadmap.

HPCwire: Thanks for taking time to give an update. Maybe a quick organization summary would be helpful.

Karl Schulz, OpenHPC

Schulz: Sure. In 2015 when the project started it was a relatively small number of folks working together. It wasn’t until 2016 that we really became fully operational as a collaborative project under the Linux Foundation. That meant setting a governance structure which is two pronged. We have a board for budgetary items. All the technical leadership is done through a technical steering committee. Our steering committee members participate on a term for one year. Elections take place in the summer, so right now we have a mix of old and new folks. There are roughly 20 people on the steering committee. They represent interests that are similarly aligned to the overall interest in OpenHPC, so there are folks from supercomputing sites, the elite labs, academic sites; there are vendor folks, and developers. It’s a good mix of people and all of that information is available on our GitHub page.

HPCwire: What about growth in the OpenHPC community overall? 

Schulz: In terms of other adoption and interest in the community, when we started – and I’d have to go back for the exact numbers – but I’m guessing we had approximately 20-25 formal members. We have continued to grow year over year. As of right now we have about 38 members who are formally associated with the project. They have the same span of affiliations [as the technical steering committee]. Along the way, as you know, Arm joined, so we are doing builds and recipes that are for multiple architectures now. We have multiple OS vendors associated with the project in terms of Red Hat and SLES (SUSE Linux Enterprise Server) so we push out builds for both of those. We have seen continued growth in the membership.

HPCwire: As I understand it, the OpenHPC stack itself has grown considerably. 

Schulz: If we go back to the very first release and compare it to now, we almost double the number of software components that are in the curated OpenHPC stack. That has implications for all of our build infrastructure and our technical infrastructure, but we see it as important to be able to grow the stack over time. One of the first things we did on the technical steering committee side is we put together a mechanism by which the community could suggest or request packages to be added to the stack. You don’t have to be associated with the project. Anybody can come in to suggest and then the technical committee can review those requests on roughly a quarterly basis and make decisions. That’s how the majority of all this software has found its way into OpenHPC.

You know it’s things that are both administrative, like provisioners, and things of growing interest in the community like containerization. We’ve added Singularity and Charliecloud, a bunch of numerical packages in terms of finite elements and linear algebra. And in IO clients, we’ve added support for things like BeeGFS. We’re also adding ‘new’ software as it develops, so one example is an exascale runtime for job launches.

HPCwire: It’s good to have buy-in from contributors and vendors but what can you say about user traction?

Schulz: I get this question fairly frequently – who’s using OpenHPC? Of course we are an open source project and don’t require any registration so you don’t know necessarily who’s using it. But we do mine our logs just to see general growth trends. We’ve had roughly year-over-year in the last three years a doubling of the number of folks who are hitting our build repository. I think right now in 2018 we are averaging something like 3,000 visitors a month. People are accessing our binary builds to the tune of about a terabyte and a half of data downloaded every month. We have seen basically a 2x growth every year and we feel pretty good about that.

HPCwire: When OpenHPC was first announced at SC15 there was concern that Intel’s influence might make it “less open.” That said, the reality is clusters are still largely an x86 world. How’s progress on support for other ecosystems?  

Schulz: At the last SC is when we had the formal release supporting Arm. Before we had a test release preview and the reason we came out first with a test preview was we didn’t have any software which could actually provision bare metal Arm, at least not the software we were using. Once we got that sorted out, we formally had a real Arm release that coincided with when we started to see more and more Arm hardware work its way into the general public.

We have seen some uptake and can sort of delineate between x86 and Arm because we were starting from zero. We have seen that grow particularly this year. Use is still dominated by x86 as you might expect, but definitely there’s a steady growth in people interested in Arm. I think the community project is actually well suited to help in that area because there’s just not a lot of things you can use out-of-the-box on the Arm ecosystem. We’ve benefited a lot from having the Arm folks involved in the project.

Many of the component packages, particularly the development environment tools, may not have necessarily been built on Arm architecture upfront, so we had to work through a lot of mostly small issues. I’d say at least 70 percent of the packages that we had in OpenHPC needed a little bit of work to get up and running on Arm. We’re able to leverage our testing infrastructure for both x86 and Arm, where we can actually make sure not only are the builds working but we can actually run applications against these libraries in an HPC environment under a resource manager. Even people who aren’t using OpenHPC will benefit from this work because we got a lot of the packages up and running and it served upstream to the community so other people can build on Arm as well.

HPCwire: IBM’s Power-based systems are garnering attention with the success of the Summit and Sierra supercomputers. IBM told HPCwire back at SC16 it wasn’t opposed to joining OpenHPC but we’ve not heard of any movement in that direction. Do you have insight there?

Schulz: I don’t. We had some discussion about them possibly being interested in joining and they have had some discussion with the Linux Foundation. I don’t know what their thinking is. They are obviously welcome. We pointed them to the Linux Foundation folks several times. I believe there have been discussions [in the past] but until now they are not a member yet. It would not go through me anyway. I’m on the technical side of things. It wouldn’t surprise me if these discussions come up at SC.

 

HPCwire: Let’s shift gears for a second and talk about OpenHPC’s approach to creating these plug-and-play stacks and relate that to helping expand HPC use. 

Schulz: What we try to do is be very building block oriented. So for people who have a lot of expertise and have a strong opinion about how to do things, they can sort of pick and choose what they want. They can use their own provisioner and their own config management system that they probably spent years developing. We wanted to make sure that we as a community effort weren’t doing something that was so locked down in terms of having to do it one and only one way that it wouldn’t be of interest to those folks. So that’s one aspect.

So we have always kept everything very building block oriented, but to your leading point, we know that there’s a lot of folks who don’t have that kind of expertise and they are looking for guidance. That’s the other part of what we are trying to do is leverage the expertise and folks involved with the community who have been involved in standing up HPC systems for years and have codified some best practice, if you will, into a number of different recipes.

HPCwire: How’s that going?

Schulz: When we first launched OpenHPC, we had one recipe and when I say one recipe, for us that is a document that is validated. It’s something for people if they are starting from bare metal or they are doing something with virtual machines that they can take and follow end-to-end and have a working cluster at the end with all the software coming from the recipe they chose.

That initial recipe had a resource manager, one provisioner, one architecture, and one OS. Now I think we are up to 10 or 12 different recipes because we have support for multiple OSs, multiple architectures. We now have multiple provisioners so people can choose from Slurm or PBS professional, etc. We are trying to curate these recipes and give people options. That increases our testing time, but we think it is important because for some people to be able to make those kinds of decisions based on what they are familiar with.

HPCwire: Given the effort required to create multiple recipes, is it paying off in terms of expanding HPC use?

Schulz: I think we have seen that being helpful in areas that you might say are non-traditional HPC where people just want to get an HPC cluster up and running to run a piece of software. We have some examples of medical organizations doing this, sort of setting up their environment to do genomics and the like. We also have people using it in finance. So we do see evidence of people using OpenHPC in areas where we say ‘well that’s not traditional HPC but they have a need for doing parallel processing.’

HPCwire: Seems like adding AI processing capabilities should be a priority?

Schulz: Obviously AI is big now so people are using OpenHPC with AI packages. We’ve not added anything specific yet, but I am thinking that at some point it will bubble up. We did an internal survey recently asking that very question – do you have an interest in AI workloads, are you using this stuff? – and there was a strong interest. At some point we will get a community request for that, TensorFlow and Caffe and that kind of stuff, and that will make us review it. We might proactively do it, but today the components that are in OpenHPC are a little more generic-HPC.

HPCwire: When you do a new release, how much of the software is actually changing?

Schulz: You have to be aware that the open source software environment changes pretty fast and you can get out of date quickly. We see that as pretty important. When we are working on a new release we scan the upstream community, all the stuff that’s going on in HPC, and if there is a new stable piece of software, we start trying to use that right away and as long as we don’t find a problem with it. We always go with the new one. We have been doing this for a couple of years now and that allows us to go back and sort of look at the metrics for how fast the software is evolving.

Some software changes every time, like Slurm resource manager. There’s going to be a new release pretty often. Some stuff doesn’t change a lot. If you look at the aggregate of the software stack for OpenHPC – and we’ve been doing releases about every quarter – we’ve seen consistently about 40 percent of the components that are in OpenHPC have a new version. Every single release, we are updating and testing a new version. That requires a lot of work and upkeep, but we think that’s an important part of the delivery for the project because if you are starting from scratch, just now installing your system, you always want the latest version.

We’ve been trying to get a feel for how many people were doing upgrades. Of course, you also have some small cadre of folks that will never update. But it is a tiny amount. I think right now something like 90 percent of the folks accessing packages are on the latest release which certainly makes us happy.

HPCwire: What’s coming out at SC18?

Schulz: The release that is currently out is what we call 1.3.5 and the release that will be coming out at Supercomputing is 1.3.6. Some of the planned stuff is the usual number of software updates; so, the 40 percent of the software number I just quoted will have new versions and we are right now pretty close to that at 38 percent. We’ll be adding some new packages and those all came from this components submission request process. We have made some packaging changes based on some lessons along the way. We are introducing a new compiler variant. One of the things about OpenHPC is that we try to adopt a very hierarchical software environment. This is based on experience at large HPC sites where we have architected a system for which we know users will have different versions of different software. They may want to have major version of some components like compilers and other [older] components.

Like I said, we try to stay as current as we can. The 1.3.5 release out there today has the latest version of GCC which was at that time GCC7.  Now there’s a GCC8 introducing a whole family of builds that are supported with GCC8. The way that we do things is that people who have existing systems, they can opt in if they want to access a new compiler variant and all the builds that go with that; for someone building a system from scratch the defaults leverage that new version of GCC. It sounds like kind of a small thing, but because of the hierarchical nature of the software environment that we have, it is actually a lot of effort on our side because we are introducing every single development tool at the same time we roll out a new compiler variant.

HPCwire: As you’ve noted it’s a lot of work. In terms of hardware, my understanding is much of the infrastructure used by OpenHPC is housed at TACC. Is that correct?

Schulz: You may know I was at TACC for a number of years previously and UT/TACC is one of the formal academic members of the project thru the Linux Foundation. We have one member on the Technical Steering Committee from TACC presently (Cyrus Proctor), and the fine folks there have been kind enough to host continuous integration clusters in their data center along with some build servers. The CI infrastructure hosting at TACC has been there since the early days and is helped with hardware donations from Intel, Cavium, and Dell.  We also have additional build server support hosted at Packet through the Works on Arm Project and leverage additional cloud infrastructure through Amazon’s EC2.


HPCwire
: Looking ahead what’s on the roadmap?

Schulz: We won’t have it for this release but early in 2019 for the follow-on release 1.3.7 we’ll be introducing some additional support to leverage the Arm compiler. Today we do builds leveraging different compiler tool chains including GCC and on the x86 side we also to build on Intel compilers. Arm has their own HPC compiler toolchain and we have been working with them to get that build up and running against that toolchain, which is a little different because it is LLVM based. I would expect people should see in 2019 builds against the Arm compilers which will help for folks running on Arm because you will also get access to their math library and builds for all the open source software that can take advantage of any performance optimizations that they are putting in the math library.

I also think one of the efforts you’ll see also come out of the community project is recipes that are more automated that use one particular config management. Right now we’ve had people working on something that’s Ansible. I would expect that you would see us publish something that is Ansible that would allow people to install OpenHPC. Another possibility is we know there are people working with the OpenHPC stack in a cloud environment, but it’s up to them how to get that out. I see a possibility where we might publish some reference images for some of the more common cloud service providers to make it easier to spin up an HPC cloud on an OpenHPC cluster for example.

HPCwire: Anything more exotic in the wings? I know you’ve just returned from China where several processor efforts are ongoing?

Schulz:Everything that we are operating on requires processors that have been GA’d. We wouldn’t be in a position to push out a stack for something that’s not been GA’d. I’ll say that the same way we would like to stay current on the upstream software, as new hardware rolls out that is important for HPC, we also would like to stay current with that as well. I think you can imagine that’s also a motivation for some of the partner companies to be involved to make sure the community stack runs on their new technology. As a new processor comes out, the community will have to get access to it and we’ll be in a position to roll out builds against it.

HPCwire: Thanks again for your time Karl.

Bio: Karl W. Schulz, is Research Associate Professor – Institute for Computational Engineering and Sciences (ICES), Associate Professor – Women’s Health, Dell Medical School, The University of Texas at Austin

Slides source: OpenHPC Update, August 2018, https://github.com/openhpc/ohpc/wiki/Papers-and-Presentations

Subscribe to HPCwire's Weekly Update!

Be the most informed person in the room! Stay ahead of the tech trends with industy updates delivered to you every week!

What’s New in HPC Research: Rabies, Smog, Robots & More

October 14, 2019

In this bimonthly feature, HPCwire highlights newly published research in the high-performance computing community and related domains. From parallel programming to exascale to quantum computing, the details are here. Read more…

By Oliver Peckham

Crystal Ball Gazing: IBM’s Vision for the Future of Computing

October 14, 2019

Dario Gil, IBM’s relatively new director of research, painted a intriguing portrait of the future of computing along with a rough idea of how IBM thinks we’ll get there at last month’s MIT-IBM Watson AI Lab’s AI Read more…

By John Russell

Summit Simulates Braking – on Mars

October 14, 2019

NASA is planning to send humans to Mars by the 2030s – and landing on the surface will be considerably trickier than landing a rover like Curiosity. To solve the problem, NASA researchers are using the world’s fastes Read more…

By Staff report

Chaminade University’s Immersion Program Builds Capacity for Data Science in Hawaii, Pacific Region

October 10, 2019

Kuleana is a uniquely Hawaiian value and practice which embodies responsibility to self, community, and the ‘aina' (land). At Chaminade University, a federally designated Native Hawaiian serving university in Hawai‘i Read more…

By Faith Singer-Villalobos

Trovares Drives Memory-Driven, Property Graph Analytics Strategy with HPE

October 10, 2019

Trovares, a high performance property graph analytics company, has partnered with HPE and its Superdome Flex memory-driven servers on a cybersecurity capability the companies say “routinely” runs near-time workloads on 24TB-capacity systems... Read more…

By Doug Black

AWS Solution Channel

Making High Performance Computing Affordable and Accessible for Small and Medium Businesses with HPC on AWS

High performance computing (HPC) brings a powerful set of tools to a broad range of industries, helping to drive innovation and boost revenue in finance, genomics, oil and gas extraction, and other fields. Read more…

HPE Extreme Performance Solutions

Intel FPGAs: More Than Just an Accelerator Card

FPGA (Field Programmable Gate Array) acceleration cards are not new, as they’ve been commercially available since 1984. Typically, the emphasis around FPGAs has centered on the fact that they’re programmable accelerators, and that they can truly offer workload specific hardware acceleration solutions without requiring custom silicon. Read more…

IBM Accelerated Insights

HPC in the Cloud: Avoid These Common Pitfalls

[Connect with LSF users and learn new skills in the IBM Spectrum LSF User Community.]

It seems that everyone is experimenting about cloud computing. Read more…

Intel, Lenovo Join Forces on HPC Cluster for Flatiron

October 9, 2019

An HPC cluster with deep learning techniques will be used to process petabytes of scientific data as part of workload-intensive projects spanning astrophysics to genomics. AI partners Intel and Lenovo said they are providing... Read more…

By George Leopold

Crystal Ball Gazing: IBM’s Vision for the Future of Computing

October 14, 2019

Dario Gil, IBM’s relatively new director of research, painted a intriguing portrait of the future of computing along with a rough idea of how IBM thinks we’ Read more…

By John Russell

Summit Simulates Braking – on Mars

October 14, 2019

NASA is planning to send humans to Mars by the 2030s – and landing on the surface will be considerably trickier than landing a rover like Curiosity. To solve Read more…

By Staff report

Trovares Drives Memory-Driven, Property Graph Analytics Strategy with HPE

October 10, 2019

Trovares, a high performance property graph analytics company, has partnered with HPE and its Superdome Flex memory-driven servers on a cybersecurity capability the companies say “routinely” runs near-time workloads on 24TB-capacity systems... Read more…

By Doug Black

Intel, Lenovo Join Forces on HPC Cluster for Flatiron

October 9, 2019

An HPC cluster with deep learning techniques will be used to process petabytes of scientific data as part of workload-intensive projects spanning astrophysics to genomics. AI partners Intel and Lenovo said they are providing... Read more…

By George Leopold

Optimizing Offshore Wind Farms with Supercomputer Simulations

October 9, 2019

Offshore wind farms offer a number of benefits; many of the areas with the strongest winds are located offshore, and siting wind farms offshore ameliorates many of the land use concerns associated with onshore wind farms. Some estimates say that, if leveraged, offshore wind power... Read more…

By Oliver Peckham

Harvard Deploys Cannon, New Lenovo Water-Cooled HPC Cluster

October 9, 2019

Harvard's Faculty of Arts & Sciences Research Computing (FASRC) center announced a refresh of their primary HPC resource. The new cluster, called Cannon after the pioneering American astronomer Annie Jump Cannon, is supplied by Lenovo... Read more…

By Tiffany Trader

NSF Announces New AI Program; Plans $120M in Funding Next Year

October 8, 2019

As the saying goes, when you’re hot, you’re hot. Right now, AI is scalding. Today the National Science Foundation announced a new AI initiative – The National Artificial Intelligence Research Institutes program – with plans to invest about “$120 million in grants next year... Read more…

By Staff report

DOE Sets Sights on Accelerating AI (and other) Technology Transfer

October 3, 2019

For the past two days DOE leaders along with ~350 members from academia and industry gathered in Chicago to discuss AI development and the ways in which industr Read more…

By John Russell

Supercomputer-Powered AI Tackles a Key Fusion Energy Challenge

August 7, 2019

Fusion energy is the Holy Grail of the energy world: low-radioactivity, low-waste, zero-carbon, high-output nuclear power that can run on hydrogen or lithium. T Read more…

By Oliver Peckham

DARPA Looks to Propel Parallelism

September 4, 2019

As Moore’s law runs out of steam, new programming approaches are being pursued with the goal of greater hardware performance with less coding. The Defense Advanced Projects Research Agency is launching a new programming effort aimed at leveraging the benefits of massive distributed parallelism with less sweat. Read more…

By George Leopold

Cray Wins NNSA-Livermore ‘El Capitan’ Exascale Contract

August 13, 2019

Cray has won the bid to build the first exascale supercomputer for the National Nuclear Security Administration (NNSA) and Lawrence Livermore National Laborator Read more…

By Tiffany Trader

AMD Launches Epyc Rome, First 7nm CPU

August 8, 2019

From a gala event at the Palace of Fine Arts in San Francisco yesterday (Aug. 7), AMD launched its second-generation Epyc Rome x86 chips, based on its 7nm proce Read more…

By Tiffany Trader

Ayar Labs to Demo Photonics Chiplet in FPGA Package at Hot Chips

August 19, 2019

Silicon startup Ayar Labs continues to gain momentum with its DARPA-backed optical chiplet technology that puts advanced electronics and optics on the same chip Read more…

By Tiffany Trader

Chinese Company Sugon Placed on US ‘Entity List’ After Strong Showing at International Supercomputing Conference

June 26, 2019

After more than a decade of advancing its supercomputing prowess, operating the world’s most powerful supercomputer from June 2013 to June 2018, China is keep Read more…

By Tiffany Trader

D-Wave’s Path to 5000 Qubits; Google’s Quantum Supremacy Claim

September 24, 2019

On the heels of IBM’s quantum news last week come two more quantum items. D-Wave Systems today announced the name of its forthcoming 5000-qubit system, Advantage (yes the name choice isn’t serendipity), at its user conference being held this week in Newport, RI. Read more…

By John Russell

A Behind-the-Scenes Look at the Hardware That Powered the Black Hole Image

June 24, 2019

Two months ago, the first-ever image of a black hole took the internet by storm. A team of scientists took years to produce and verify the striking image – an Read more…

By Oliver Peckham

Leading Solution Providers

ISC 2019 Virtual Booth Video Tour

CRAY
CRAY
DDN
DDN
DELL EMC
DELL EMC
GOOGLE
GOOGLE
ONE STOP SYSTEMS
ONE STOP SYSTEMS
PANASAS
PANASAS
VERNE GLOBAL
VERNE GLOBAL

Intel Confirms Retreat on Omni-Path

August 1, 2019

Intel Corp.’s plans to make a big splash in the network fabric market for linking HPC and other workloads has apparently belly-flopped. The chipmaker confirmed to us the outlines of an earlier report by the website CRN that it has jettisoned plans for a second-generation version of its Omni-Path interconnect... Read more…

By Staff report

Kubernetes, Containers and HPC

September 19, 2019

Software containers and Kubernetes are important tools for building, deploying, running and managing modern enterprise applications at scale and delivering enterprise software faster and more reliably to the end user — while using resources more efficiently and reducing costs. Read more…

By Daniel Gruber, Burak Yenier and Wolfgang Gentzsch, UberCloud

Intel Debuts Pohoiki Beach, Its 8M Neuron Neuromorphic Development System

July 17, 2019

Neuromorphic computing has received less fanfare of late than quantum computing whose mystery has captured public attention and which seems to have generated mo Read more…

By John Russell

Rise of NIH’s Biowulf Mirrors the Rise of Computational Biology

July 29, 2019

The story of NIH’s supercomputer Biowulf is fascinating, important, and in many ways representative of the transformation of life sciences and biomedical res Read more…

By John Russell

Quantum Bits: Neven’s Law (Who Asked for That), D-Wave’s Steady Push, IBM’s Li-O2- Simulation

July 3, 2019

Quantum computing’s (QC) many-faceted R&D train keeps slogging ahead and recently Japan is taking a leading role. Yesterday D-Wave Systems announced it ha Read more…

By John Russell

With the Help of HPC, Astronomers Prepare to Deflect a Real Asteroid

September 26, 2019

For years, NASA has been running simulations of asteroid impacts to understand the risks (and likelihoods) of asteroids colliding with Earth. Now, NASA and the European Space Agency (ESA) are preparing for the next, crucial step in planetary defense against asteroid impacts: physically deflecting a real asteroid. Read more…

By Oliver Peckham

ISC Keynote: Thomas Sterling’s Take on Whither HPC

June 20, 2019

Entertaining, insightful, and unafraid to launch the occasional verbal ICBM, HPC pioneer Thomas Sterling delivered his 16th annual closing keynote at ISC yesterday. He explored, among other things: exascale machinations; quantum’s bubbling money pot; Arm’s new HPC viability; Europe’s... Read more…

By John Russell

Argonne Team Makes Record Globus File Transfer

July 10, 2019

A team of scientists at Argonne National Laboratory has broken a data transfer record by moving a staggering 2.9 petabytes of data for a research project.  The data – from three large cosmological simulations – was generated and stored on the Summit supercomputer at the Oak Ridge Leadership Computing Facility (OLCF)... Read more…

By Oliver Peckham

  • arrow
  • Click Here for More Headlines
  • arrow
Do NOT follow this link or you will be banned from the site!
Share This