NERSC’s ‘Shifter’ Makes Container-based HPC a Breeze

By Kathy Kincade, Communications Specialist, Berkeley Lab Computing Sciences

August 7, 2015

The explosive growth in data coming out of experiments in cosmology, particle physics, bioinformatics and nuclear physics is pushing computational scientists to design novel software tools that will help users better access, manage and analyze that data on current and next-generation HPC architectures.

One increasingly popular approach is container-based computing, designed to support flexible, scalable computing. Linux containers, which are just now beginning to find their way into the HPC environment, allow an application to be packaged with its entire software stack, including portions of the base operating system files, user environment variables and application “entry points.”

With the growing adoption of container-based computing, new tools to create and manage containers have emerged — notably Docker, an open-source, automated container deployment service that allows users to package an application with all of its dependencies into a standardized unit for software development. Docker containers wrap up a piece of software in a complete filesystem that houses everything it needs to run, including code, runtime, system tools and system libraries. This guarantees that it will always operate the same, regardless of the environment in which it is running.

While Docker has taken the general computing world by storm in recent years, like container-based computing it has yet to be fully recognized in HPC. To facilitate the use of both of these tools in HPC, NERSC is enabling Docker-like container technology on its systems through a new, customized software package known as Shifter. Shifter, designed as a scalable method for deploying containers and user-defined images in an HPC environment, was developed at NERSC to improve flexibility and usability of its HPC systems for increasingly data-intensive workloads. It is initially being tested on NERSC’s Edison system — a Cray XC30 supercomputer — by users in particle physics and nuclear physics and will eventually be made available as an open source tool for the general HPC community.

Here, NERSC’s Doug Jacobsen, Shane Canon, Lisa Gerhardt and Deborah Bard — who have been instrumental in developing, deploying and testing Shifter — discuss how Shifter works and how it will help the scientific community better utilize resources at NERSC and other supercomputing facilities and increase their scientific productivity in the process.

Q. How is container-based computing changing the way applications are developed and deployed in HPC?

Canon: That remains to be seen, but my expectation is that, similar to the way it is having an impact in the enterprise space, it will carry over into scientific computing and HPC as well. One reason I think it will be powerful is because it is a productivity enhancer. It makes it easier for users to develop something locally on their laptop and push to another place. Another key factor is scientific reproducibility; being able to take that image and know that you can reliably instantiate it over and over or share it with others is really powerful. I think the enterprise world wants the same things but for different reasons — for them it is about compliance and stability, while for scientists it’s about reproducibility and verifiability. But it wouldn’t surprise me if, five years from now, the way people build and deliver software is through something like containers, that it becomes the predominant way people share their software, both in general and in HPC.

Q: With the growing adoption of container-based computing, Docker is gaining popularity in the HPC community as well. What motivated you to develop Shifter rather than modify Docker for use on HPC systems?

Jacobsen: Shifter is strongly focused on the needs and requirements of the HPC workload, which means it can deliver the functionality we are looking for while still meeting the overall performance requirements and constraints that a large-scale HPC platform imposes. Shifter allows the user to supply us with a Docker image that we can then convert to a form that is easier and more efficient for us to distribute to the compute nodes. We are leveraging the user interface that Docker makes available to people to create their software images and leveraging that ecosystem, but not directly using their software internally. Shifter it is not a replacement for Docker functionality; it is specifically focused on the HPC use cases.

Canon: What Docker has done is develop a framework that makes it easy for people to create images and then publish those to something like Dockerhub, which makes it easy for them to share. So we are leveraging that and trying to preserve the things we think are most useful for scientists. We felt it was important for the implementation to make it easy for users to create a software environment and then instantiate that on our systems and also leverage what is happening in the Docker ecosystem so they can use some of the existing images out there or publish to Docker and share with other scientists but also easily run on NERSC systems. So we are preserving the best parts of Docker.

Q. How does Shifter work?

Jacobsen: Shifter works by converting user- or staff-generated images in Docker, virtual machines or CHOS (another method for delivering flexible environments) to a common format that provides a tunable point to allow images to be distributed on the Cray supercomputers at NERSC. Through the user interface, a user can select an image from their Dockerhub account or private Docker registry. An image manager at NERSC then automatically converts the image to the common format based on an automated analysis of the image. The image is then copied to the Lustre scratch filesystem and the user can begin submitting jobs — all of which run entirely within the container — specifying which image to use.

Shifter workflow graphic 1000x

Q. In addition to enabling user-defined images and automating the image conversion process, what other advantages does Shifter bring to NERSC users?

Jacobsen: What makes this software a big deal is that it is enabling science on our systems that has been inaccessible in the past. For example, for data-intensive users such as researchers with experimental apparatus who want to analyze data versus just-run simulations on our systems — their codes typically tend to be very different from the way most calculations on Edison run. They tend to have very large, complex software stacks with many different dependencies.

With Shifter, users can prepare a Docker container on their own system, bring it onto the Edison system through already constructed pipelines and it just works. Applications go from not working on Edison to immediately working. And as a critical side benefit, Shifter provides a lot of performance benefits to data-intensive codes that rely on many different dependencies because of the way the software shifts to the compute nodes. With Shifter, it is very performant to start them up and run them. Previously, we relied a lot on the centralized resources at NERSC to make that happen.

Bard: I’ve been working with Shifter since I came to NERSC, evaluating it for running simulations for the LSST (Large Synoptic Survey Telescope). One of the things I have learned is that you have root access when you use Docker, which is not good because you can accidentally screw things up. This is a barrier to running Docker in a lot of places (in HPC) that Shifter fixes, which is huge. People don’t want to deploy Docker because of security issues, but Shifter controls the external connections in a way that means it works at NERSC. And they are going to make it open source, which is brilliant. When you’re thinking about software for a large collaboration, for example, you want to be able to develop a software environment that people can run anywhere, and with Shifter you can run it safely anywhere.

I’ve also been learning about how to incorporate Shifter into workflow tools, and it is very easy, which is nice. I am particularly interested in how we are going to be supporting it from the users’ perspective, not just within large collaborations but for all users of Cori (NERSC’s next-generation Cray X40). With Shifter, users will be able to get running very quickly on Cori.

Q. Are there certain applications or science domains in which Shifter will have greater impact, or is it designed to improve data management across the board?

Canon: I think it can work for a large range of applications. Initially it’s more important for some of the nontraditional and data-intensive areas because they are the ones that often come in with the most challenging software requirements, and trying to take those requirements and deploy their software on an HPC system can sometimes be very challenging. It’s the thing they stumble on right out of the gate. They don’t even get to the point where they are effectively running their applications because they can’t get all the prerequisite software requirements satisfied first.

What’s happened is that as datasets have gotten larger these communities’ computing demands have grown, so that they now have problems that are similar in scale to traditional HPC users. In the past maybe they could have gotten by just running something on a workstation or very small cluster, but now they have problems that have gotten big enough that they can take advantage of facilities like NERSC. But then when they tried to make that leap they would struggle. We may be able to work around the problems, but a lot of the time it is very time consuming and tedious. So now we believe we have a solution — Shifter — that will allow them to get past those problems more quickly, and the early results are very encouraging.

Can you share some of those results and/or successes?

Jacobsen: There are two ways in which Shifter has already been successful. First, we have two major groups beginning to use it right now: the LCLS experimental facility at SLAC, and the high energy physics community at CERN. The LCLS, for example, has its own software environment, and it is rather challenging to take it out of its context and put it into our environment. LCLS spent a lot of time trying to adapt to Edison, which they eventually did, but it took them months to make a Docker image. Using Shifter, however, we were able to create a Docker image in one day and demonstrated that the staff effort in migrating applications is greatly reduced, which was the original purpose of Shifter: to make our system more adaptable to external software.

The other benefit is that, because of the way we present images to the Edison system in Shifter, it turns out that the software can load much faster than before. So in the case of the LCLS, before Shifter once an image had been ported to NERSC it could sometimes take up to half an hour just for the software to start. With Shifter, everything starts in a matter of seconds — somewhere between 5-20 seconds. So this results in much better utilization of our resources.

Gerhardt: With scientists from the Large Hadron Collider’s (LHC) ALICE, ATLAS and CMS experiments at CERN, we are testing Shifter in conjunction with their CVMFS (CERN Virtual Machine File System) software package. I’ve been working to bring the CVMFS software onto Edison, but it’s a huge software repository. For example, with ATLAS (one of two general-purpose detectors at the LHC), if we do a straight-up rsync over scratch, we’re working with more than 3.5 TB of data and 20 million inodes. We found that Shifter is a good tool for handling this because you can build a filesystem image on the local node and deliver startup times around half a minute on a single node. So with Shifter the jobs run as efficiently, if not more so, than they do in their current configurations and without the user having to jump through any special hoops. It just works.

Canon: From a big-picture perspective, Shifter is really about trying to enable and simplify the process of science. Scientists really struggle with the fact that they create some sort of code or simulation and it can be really difficult for another user to replicate the computing environment that was used. It’s just as challenging as it used to be for scientists to replicate experimental conditions. Shifter is potentially one way to address that challenge.

Q: One last question; are you going to make this available to other centers? How can people learn more about it?

Canon: Doug has already gone through the steps to open source it and release it through a BSD (Berkeley Software Distribution) license. The intent is that others can download it and use it at their centers. While that might take away from it being a unique capability for NERSC, we think in the end it is important for scientists because the more available this capability is for users, the more they will adopt it and make it a standard for how they operate.

Also, as we’ve been developing Shifter we’ve been discussing it with Cray, and hopefully this is something that will become a mainstream capability for Cray systems. The fact that they are working with us on this means they recognize the potential for it.

We also plan to offer demonstrations of Shifter in the Department of Energy booth at SC15 in November.

Subscribe to HPCwire's Weekly Update!

Be the most informed person in the room! Stay ahead of the tech trends with industy updates delivered to you every week!

UCSD, AIST Forge Tighter Alliance with AI-Focused MOU

January 18, 2018

The rich history of collaboration between UC San Diego and AIST in Japan is getting richer. The organizations entered into a five-year memorandum of understanding on January 10. The MOU represents the continuation of a 1 Read more…

By Tiffany Trader

New Blueprint for Converging HPC, Big Data

January 18, 2018

After five annual workshops on Big Data and Extreme-Scale Computing (BDEC), a group of international HPC heavyweights including Jack Dongarra (University of Tennessee), Satoshi Matsuoka (Tokyo Institute of Technology), Read more…

By John Russell

Researchers Measure Impact of ‘Meltdown’ and ‘Spectre’ Patches on HPC Workloads

January 17, 2018

Computer scientists from the Center for Computational Research, State University of New York (SUNY), University at Buffalo have examined the effect of Meltdown and Spectre security updates on the performance of popular H Read more…

By Tiffany Trader

HPE Extreme Performance Solutions

HPE and NREL Take Steps to Create a Sustainable, Energy-Efficient Data Center with an H2 Fuel Cell

As enterprises attempt to manage rising volumes of data, unplanned data center outages are becoming more common and more expensive. As the cost of downtime rises, enterprises lose out on productivity and valuable competitive advantage without access to their critical data. Read more…

Fostering Lustre Advancement Through Development and Contributions

January 17, 2018

Six months after organizational changes at Intel's High Performance Data (HPDD) division, most in the Lustre community have shed any initial apprehension around the potential changes that could affect or disrupt Lustre Read more…

By Carlos Aoki Thomaz

UCSD, AIST Forge Tighter Alliance with AI-Focused MOU

January 18, 2018

The rich history of collaboration between UC San Diego and AIST in Japan is getting richer. The organizations entered into a five-year memorandum of understandi Read more…

By Tiffany Trader

New Blueprint for Converging HPC, Big Data

January 18, 2018

After five annual workshops on Big Data and Extreme-Scale Computing (BDEC), a group of international HPC heavyweights including Jack Dongarra (University of Te Read more…

By John Russell

Researchers Measure Impact of ‘Meltdown’ and ‘Spectre’ Patches on HPC Workloads

January 17, 2018

Computer scientists from the Center for Computational Research, State University of New York (SUNY), University at Buffalo have examined the effect of Meltdown Read more…

By Tiffany Trader

Fostering Lustre Advancement Through Development and Contributions

January 17, 2018

Six months after organizational changes at Intel's High Performance Data (HPDD) division, most in the Lustre community have shed any initial apprehension aroun Read more…

By Carlos Aoki Thomaz

When the Chips Are Down

January 11, 2018

In the last article, "The High Stakes Semiconductor Game that Drives HPC Diversity," I alluded to the challenges facing the semiconductor industry and how that may impact the evolution of HPC systems over the next few years. I thought I’d lift the covers a little and look at some of the commercial challenges that impact the component technology we use in HPC. Read more…

By Dairsie Latimer

How Meltdown and Spectre Patches Will Affect HPC Workloads

January 10, 2018

There have been claims that the fixes for the Meltdown and Spectre security vulnerabilities, named the KPTI (aka KAISER) patches, are going to affect applicatio Read more…

By Rosemary Francis

Momentum Builds for US Exascale

January 9, 2018

2018 looks to be a great year for the U.S. exascale program. The last several months of 2017 revealed a number of important developments that help put the U.S. Read more…

By Alex R. Larzelere

ANL’s Rick Stevens on CANDLE, ARM, Quantum, and More

January 8, 2018

Late last year HPCwire caught up with Rick Stevens, associate laboratory director for computing, environment and life Sciences at Argonne National Laboratory, f Read more…

By John Russell

Inventor Claims to Have Solved Floating Point Error Problem

January 17, 2018

"The decades-old floating point error problem has been solved," proclaims a press release from inventor Alan Jorgensen. The computer scientist has filed for and Read more…

By Tiffany Trader

US Coalesces Plans for First Exascale Supercomputer: Aurora in 2021

September 27, 2017

At the Advanced Scientific Computing Advisory Committee (ASCAC) meeting, in Arlington, Va., yesterday (Sept. 26), it was revealed that the "Aurora" supercompute Read more…

By Tiffany Trader

Japan Unveils Quantum Neural Network

November 22, 2017

The U.S. and China are leading the race toward productive quantum computing, but it's early enough that ultimate leadership is still something of an open questi Read more…

By Tiffany Trader

AMD Showcases Growing Portfolio of EPYC and Radeon-based Systems at SC17

November 13, 2017

AMD’s charge back into HPC and the datacenter is on full display at SC17. Having launched the EPYC processor line in June along with its MI25 GPU the focus he Read more…

By John Russell

Nvidia Responds to Google TPU Benchmarking

April 10, 2017

Nvidia highlights strengths of its newest GPU silicon in response to Google's report on the performance and energy advantages of its custom tensor processor. Read more…

By Tiffany Trader

IBM Begins Power9 Rollout with Backing from DOE, Google

December 6, 2017

After over a year of buildup, IBM is unveiling its first Power9 system based on the same architecture as the Department of Energy CORAL supercomputers, Summit a Read more…

By Tiffany Trader

Fast Forward: Five HPC Predictions for 2018

December 21, 2017

What’s on your list of high (and low) lights for 2017? Volta 100’s arrival on the heels of the P100? Appearance, albeit late in the year, of IBM’s Power9? Read more…

By John Russell

Chip Flaws ‘Meltdown’ and ‘Spectre’ Loom Large

January 4, 2018

The HPC and wider tech community have been abuzz this week over the discovery of critical design flaws that impact virtually all contemporary microprocessors. T Read more…

By Tiffany Trader

Leading Solution Providers

Perspective: What Really Happened at SC17?

November 22, 2017

SC is over. Now comes the myriad of follow-ups. Inboxes are filled with templated emails from vendors and other exhibitors hoping to win a place in the post-SC thinking of booth visitors. Attendees of tutorials, workshops and other technical sessions will be inundated with requests for feedback. Read more…

By Andrew Jones

Tensors Come of Age: Why the AI Revolution Will Help HPC

November 13, 2017

Thirty years ago, parallel computing was coming of age. A bitter battle began between stalwart vector computing supporters and advocates of various approaches to parallel computing. IBM skeptic Alan Karp, reacting to announcements of nCUBE’s 1024-microprocessor system and Thinking Machines’ 65,536-element array, made a public $100 wager that no one could get a parallel speedup of over 200 on real HPC workloads. Read more…

By John Gustafson & Lenore Mullin

Researchers Measure Impact of ‘Meltdown’ and ‘Spectre’ Patches on HPC Workloads

January 17, 2018

Computer scientists from the Center for Computational Research, State University of New York (SUNY), University at Buffalo have examined the effect of Meltdown Read more…

By Tiffany Trader

Delays, Smoke, Records & Markets – A Candid Conversation with Cray CEO Peter Ungaro

October 5, 2017

Earlier this month, Tom Tabor, publisher of HPCwire and I had a very personal conversation with Cray CEO Peter Ungaro. Cray has been on something of a Cinderell Read more…

By Tiffany Trader & Tom Tabor

Flipping the Flops and Reading the Top500 Tea Leaves

November 13, 2017

The 50th edition of the Top500 list, the biannual publication of the world’s fastest supercomputers based on public Linpack benchmarking results, was released Read more…

By Tiffany Trader

How Meltdown and Spectre Patches Will Affect HPC Workloads

January 10, 2018

There have been claims that the fixes for the Meltdown and Spectre security vulnerabilities, named the KPTI (aka KAISER) patches, are going to affect applicatio Read more…

By Rosemary Francis

GlobalFoundries, Ayar Labs Team Up to Commercialize Optical I/O

December 4, 2017

GlobalFoundries (GF) and Ayar Labs, a startup focused on using light, instead of electricity, to transfer data between chips, today announced they've entered in Read more…

By Tiffany Trader

HPC Chips – A Veritable Smorgasbord?

October 10, 2017

For the first time since AMD's ill-fated launch of Bulldozer the answer to the question, 'Which CPU will be in my next HPC system?' doesn't have to be 'Whichever variety of Intel Xeon E5 they are selling when we procure'. Read more…

By Dairsie Latimer

  • arrow
  • Click Here for More Headlines
  • arrow
Share This