NERSC’s ‘Shifter’ Makes Container-based HPC a Breeze

By Kathy Kincade, Communications Specialist, Berkeley Lab Computing Sciences

August 7, 2015

The explosive growth in data coming out of experiments in cosmology, particle physics, bioinformatics and nuclear physics is pushing computational scientists to design novel software tools that will help users better access, manage and analyze that data on current and next-generation HPC architectures.

One increasingly popular approach is container-based computing, designed to support flexible, scalable computing. Linux containers, which are just now beginning to find their way into the HPC environment, allow an application to be packaged with its entire software stack, including portions of the base operating system files, user environment variables and application “entry points.”

With the growing adoption of container-based computing, new tools to create and manage containers have emerged — notably Docker, an open-source, automated container deployment service that allows users to package an application with all of its dependencies into a standardized unit for software development. Docker containers wrap up a piece of software in a complete filesystem that houses everything it needs to run, including code, runtime, system tools and system libraries. This guarantees that it will always operate the same, regardless of the environment in which it is running.

While Docker has taken the general computing world by storm in recent years, like container-based computing it has yet to be fully recognized in HPC. To facilitate the use of both of these tools in HPC, NERSC is enabling Docker-like container technology on its systems through a new, customized software package known as Shifter. Shifter, designed as a scalable method for deploying containers and user-defined images in an HPC environment, was developed at NERSC to improve flexibility and usability of its HPC systems for increasingly data-intensive workloads. It is initially being tested on NERSC’s Edison system — a Cray XC30 supercomputer — by users in particle physics and nuclear physics and will eventually be made available as an open source tool for the general HPC community.

Here, NERSC’s Doug Jacobsen, Shane Canon, Lisa Gerhardt and Deborah Bard — who have been instrumental in developing, deploying and testing Shifter — discuss how Shifter works and how it will help the scientific community better utilize resources at NERSC and other supercomputing facilities and increase their scientific productivity in the process.

Q. How is container-based computing changing the way applications are developed and deployed in HPC?

Canon: That remains to be seen, but my expectation is that, similar to the way it is having an impact in the enterprise space, it will carry over into scientific computing and HPC as well. One reason I think it will be powerful is because it is a productivity enhancer. It makes it easier for users to develop something locally on their laptop and push to another place. Another key factor is scientific reproducibility; being able to take that image and know that you can reliably instantiate it over and over or share it with others is really powerful. I think the enterprise world wants the same things but for different reasons — for them it is about compliance and stability, while for scientists it’s about reproducibility and verifiability. But it wouldn’t surprise me if, five years from now, the way people build and deliver software is through something like containers, that it becomes the predominant way people share their software, both in general and in HPC.

Q: With the growing adoption of container-based computing, Docker is gaining popularity in the HPC community as well. What motivated you to develop Shifter rather than modify Docker for use on HPC systems?

Jacobsen: Shifter is strongly focused on the needs and requirements of the HPC workload, which means it can deliver the functionality we are looking for while still meeting the overall performance requirements and constraints that a large-scale HPC platform imposes. Shifter allows the user to supply us with a Docker image that we can then convert to a form that is easier and more efficient for us to distribute to the compute nodes. We are leveraging the user interface that Docker makes available to people to create their software images and leveraging that ecosystem, but not directly using their software internally. Shifter it is not a replacement for Docker functionality; it is specifically focused on the HPC use cases.

Canon: What Docker has done is develop a framework that makes it easy for people to create images and then publish those to something like Dockerhub, which makes it easy for them to share. So we are leveraging that and trying to preserve the things we think are most useful for scientists. We felt it was important for the implementation to make it easy for users to create a software environment and then instantiate that on our systems and also leverage what is happening in the Docker ecosystem so they can use some of the existing images out there or publish to Docker and share with other scientists but also easily run on NERSC systems. So we are preserving the best parts of Docker.

Q. How does Shifter work?

Jacobsen: Shifter works by converting user- or staff-generated images in Docker, virtual machines or CHOS (another method for delivering flexible environments) to a common format that provides a tunable point to allow images to be distributed on the Cray supercomputers at NERSC. Through the user interface, a user can select an image from their Dockerhub account or private Docker registry. An image manager at NERSC then automatically converts the image to the common format based on an automated analysis of the image. The image is then copied to the Lustre scratch filesystem and the user can begin submitting jobs — all of which run entirely within the container — specifying which image to use.

Shifter workflow graphic 1000x

Q. In addition to enabling user-defined images and automating the image conversion process, what other advantages does Shifter bring to NERSC users?

Jacobsen: What makes this software a big deal is that it is enabling science on our systems that has been inaccessible in the past. For example, for data-intensive users such as researchers with experimental apparatus who want to analyze data versus just-run simulations on our systems — their codes typically tend to be very different from the way most calculations on Edison run. They tend to have very large, complex software stacks with many different dependencies.

With Shifter, users can prepare a Docker container on their own system, bring it onto the Edison system through already constructed pipelines and it just works. Applications go from not working on Edison to immediately working. And as a critical side benefit, Shifter provides a lot of performance benefits to data-intensive codes that rely on many different dependencies because of the way the software shifts to the compute nodes. With Shifter, it is very performant to start them up and run them. Previously, we relied a lot on the centralized resources at NERSC to make that happen.

Bard: I’ve been working with Shifter since I came to NERSC, evaluating it for running simulations for the LSST (Large Synoptic Survey Telescope). One of the things I have learned is that you have root access when you use Docker, which is not good because you can accidentally screw things up. This is a barrier to running Docker in a lot of places (in HPC) that Shifter fixes, which is huge. People don’t want to deploy Docker because of security issues, but Shifter controls the external connections in a way that means it works at NERSC. And they are going to make it open source, which is brilliant. When you’re thinking about software for a large collaboration, for example, you want to be able to develop a software environment that people can run anywhere, and with Shifter you can run it safely anywhere.

I’ve also been learning about how to incorporate Shifter into workflow tools, and it is very easy, which is nice. I am particularly interested in how we are going to be supporting it from the users’ perspective, not just within large collaborations but for all users of Cori (NERSC’s next-generation Cray X40). With Shifter, users will be able to get running very quickly on Cori.

Q. Are there certain applications or science domains in which Shifter will have greater impact, or is it designed to improve data management across the board?

Canon: I think it can work for a large range of applications. Initially it’s more important for some of the nontraditional and data-intensive areas because they are the ones that often come in with the most challenging software requirements, and trying to take those requirements and deploy their software on an HPC system can sometimes be very challenging. It’s the thing they stumble on right out of the gate. They don’t even get to the point where they are effectively running their applications because they can’t get all the prerequisite software requirements satisfied first.

What’s happened is that as datasets have gotten larger these communities’ computing demands have grown, so that they now have problems that are similar in scale to traditional HPC users. In the past maybe they could have gotten by just running something on a workstation or very small cluster, but now they have problems that have gotten big enough that they can take advantage of facilities like NERSC. But then when they tried to make that leap they would struggle. We may be able to work around the problems, but a lot of the time it is very time consuming and tedious. So now we believe we have a solution — Shifter — that will allow them to get past those problems more quickly, and the early results are very encouraging.

Can you share some of those results and/or successes?

Jacobsen: There are two ways in which Shifter has already been successful. First, we have two major groups beginning to use it right now: the LCLS experimental facility at SLAC, and the high energy physics community at CERN. The LCLS, for example, has its own software environment, and it is rather challenging to take it out of its context and put it into our environment. LCLS spent a lot of time trying to adapt to Edison, which they eventually did, but it took them months to make a Docker image. Using Shifter, however, we were able to create a Docker image in one day and demonstrated that the staff effort in migrating applications is greatly reduced, which was the original purpose of Shifter: to make our system more adaptable to external software.

The other benefit is that, because of the way we present images to the Edison system in Shifter, it turns out that the software can load much faster than before. So in the case of the LCLS, before Shifter once an image had been ported to NERSC it could sometimes take up to half an hour just for the software to start. With Shifter, everything starts in a matter of seconds — somewhere between 5-20 seconds. So this results in much better utilization of our resources.

Gerhardt: With scientists from the Large Hadron Collider’s (LHC) ALICE, ATLAS and CMS experiments at CERN, we are testing Shifter in conjunction with their CVMFS (CERN Virtual Machine File System) software package. I’ve been working to bring the CVMFS software onto Edison, but it’s a huge software repository. For example, with ATLAS (one of two general-purpose detectors at the LHC), if we do a straight-up rsync over scratch, we’re working with more than 3.5 TB of data and 20 million inodes. We found that Shifter is a good tool for handling this because you can build a filesystem image on the local node and deliver startup times around half a minute on a single node. So with Shifter the jobs run as efficiently, if not more so, than they do in their current configurations and without the user having to jump through any special hoops. It just works.

Canon: From a big-picture perspective, Shifter is really about trying to enable and simplify the process of science. Scientists really struggle with the fact that they create some sort of code or simulation and it can be really difficult for another user to replicate the computing environment that was used. It’s just as challenging as it used to be for scientists to replicate experimental conditions. Shifter is potentially one way to address that challenge.

Q: One last question; are you going to make this available to other centers? How can people learn more about it?

Canon: Doug has already gone through the steps to open source it and release it through a BSD (Berkeley Software Distribution) license. The intent is that others can download it and use it at their centers. While that might take away from it being a unique capability for NERSC, we think in the end it is important for scientists because the more available this capability is for users, the more they will adopt it and make it a standard for how they operate.

Also, as we’ve been developing Shifter we’ve been discussing it with Cray, and hopefully this is something that will become a mainstream capability for Cray systems. The fact that they are working with us on this means they recognize the potential for it.

We also plan to offer demonstrations of Shifter in the Department of Energy booth at SC15 in November.

Subscribe to HPCwire's Weekly Update!

Be the most informed person in the room! Stay ahead of the tech trends with industy updates delivered to you every week!

Talk to Me: Nvidia Claims NLP Inference, Training Records

August 15, 2019

Nvidia says it’s achieved significant advances in conversation natural language processing (NLP) training and inference, enabling more complex, immediate-response interchanges between customers and chatbots. And the co Read more…

By Doug Black

Trump Administration and NIST Issue AI Standards Development Plan

August 14, 2019

Efforts to develop AI are gathering steam fast. On Monday, the White House issued a federal plan to help develop technical standards for AI following up on a mandate contained in the Administration’s AI Executive Order Read more…

By John Russell

Scientists to Tap Exascale Computing to Unlock the Mystery of our Accelerating Universe

August 14, 2019

The universe and everything in it roared to life with the Big Bang approximately 13.8 billion years ago. It has continued expanding ever since. While we have a good understanding of the early universe, its fate billions Read more…

By Rob Johnson

AWS Solution Channel

Efficiency and Cost-Optimization for HPC Workloads – AWS Batch and Amazon EC2 Spot Instances

High Performance Computing on AWS leverages the power of cloud computing and the extreme scale it offers to achieve optimal HPC price/performance. With AWS you can right size your services to meet exactly the capacity requirements you need without having to overprovision or compromise capacity. Read more…

HPE Extreme Performance Solutions

Bring the combined power of HPC and AI to your business transformation

FPGA (Field Programmable Gate Array) acceleration cards are not new, as they’ve been commercially available since 1984. Typically, the emphasis around FPGAs has centered on the fact that they’re programmable accelerators, and that they can truly offer workload specific hardware acceleration solutions without requiring custom silicon. Read more…

IBM Accelerated Insights

Cloudy with a Chance of Mainframes

[Connect with HPC users and learn new skills in the IBM Spectrum LSF User Community.]

Rapid rates of change sometimes result in unexpected bedfellows. Read more…

Argonne Supercomputer Accelerates Cancer Prediction Research

August 13, 2019

In the fight against cancer, early prediction, which drastically improves prognoses, is critical. Now, new research by a team from Northwestern University – and accelerated by supercomputing resources at Argonne Nation Read more…

By Oliver Peckham

Scientists to Tap Exascale Computing to Unlock the Mystery of our Accelerating Universe

August 14, 2019

The universe and everything in it roared to life with the Big Bang approximately 13.8 billion years ago. It has continued expanding ever since. While we have a Read more…

By Rob Johnson

AI is the Next Exascale – Rick Stevens on What that Means and Why It’s Important

August 13, 2019

Twelve years ago the Department of Energy (DOE) was just beginning to explore what an exascale computing program might look like and what it might accomplish. Today, DOE is repeating that process for AI, once again starting with science community town halls to gather input and stimulate conversation. The town hall program... Read more…

By Tiffany Trader and John Russell

Cray Wins NNSA-Livermore ‘El Capitan’ Exascale Contract

August 13, 2019

Cray has won the bid to build the first exascale supercomputer for the National Nuclear Security Administration (NNSA) and Lawrence Livermore National Laborator Read more…

By Tiffany Trader

AMD Launches Epyc Rome, First 7nm CPU

August 8, 2019

From a gala event at the Palace of Fine Arts in San Francisco yesterday (Aug. 7), AMD launched its second-generation Epyc Rome x86 chips, based on its 7nm proce Read more…

By Tiffany Trader

Lenovo Drives Single-Socket Servers with AMD Epyc Rome CPUs

August 7, 2019

No summer doldrums here. As part of the AMD Epyc Rome launch event in San Francisco today, Lenovo announced two new single-socket servers, the ThinkSystem SR635 Read more…

By Doug Black

Building Diversity and Broader Engagement in the HPC Community

August 7, 2019

Increasing diversity and inclusion in HPC is a community-building effort. Representation of both issues and individuals matters - the more people see HPC in a w Read more…

By AJ Lauer

Xilinx vs. Intel: FPGA Market Leaders Launch Server Accelerator Cards

August 6, 2019

The two FPGA market leaders, Intel and Xilinx, both announced new accelerator cards this week designed to handle specialized, compute-intensive workloads and un Read more…

By Doug Black

Upcoming NSF Cyberinfrastructure Projects to Support ‘Long-Tail’ Users, AI and Big Data

August 5, 2019

The National Science Foundation is well positioned to support national priorities, as new NSF-funded HPC systems to come online in the upcoming year promise to Read more…

By Ken Chiacchia, Pittsburgh Supercomputing Center/XSEDE

High Performance (Potato) Chips

May 5, 2006

In this article, we focus on how Procter & Gamble is using high performance computing to create some common, everyday supermarket products. Tom Lange, a 27-year veteran of the company, tells us how P&G models products, processes and production systems for the betterment of consumer package goods. Read more…

By Michael Feldman

Supercomputer-Powered AI Tackles a Key Fusion Energy Challenge

August 7, 2019

Fusion energy is the Holy Grail of the energy world: low-radioactivity, low-waste, zero-carbon, high-output nuclear power that can run on hydrogen or lithium. T Read more…

By Oliver Peckham

Cray, AMD to Extend DOE’s Exascale Frontier

May 7, 2019

Cray and AMD are coming back to Oak Ridge National Laboratory to partner on the world’s largest and most expensive supercomputer. The Department of Energy’s Read more…

By Tiffany Trader

Graphene Surprises Again, This Time for Quantum Computing

May 8, 2019

Graphene is fascinating stuff with promise for use in a seeming endless number of applications. This month researchers from the University of Vienna and Institu Read more…

By John Russell

AMD Verifies Its Largest 7nm Chip Design in Ten Hours

June 5, 2019

AMD announced last week that its engineers had successfully executed the first physical verification of its largest 7nm chip design – in just ten hours. The AMD Radeon Instinct Vega20 – which boasts 13.2 billion transistors – was tested using a TSMC-certified Calibre nmDRC software platform from Mentor. Read more…

By Oliver Peckham

TSMC and Samsung Moving to 5nm; Whither Moore’s Law?

June 12, 2019

With reports that Taiwan Semiconductor Manufacturing Co. (TMSC) and Samsung are moving quickly to 5nm manufacturing, it’s a good time to again ponder whither goes the venerable Moore’s law. Shrinking feature size has of course been the primary hallmark of achieving Moore’s law... Read more…

By John Russell

Deep Learning Competitors Stalk Nvidia

May 14, 2019

There is no shortage of processing architectures emerging to accelerate deep learning workloads, with two more options emerging this week to challenge GPU leader Nvidia. First, Intel researchers claimed a new deep learning record for image classification on the ResNet-50 convolutional neural network. Separately, Israeli AI chip startup Hailo.ai... Read more…

By George Leopold

Nvidia Embraces Arm, Declares Intent to Accelerate All CPU Architectures

June 17, 2019

As the Top500 list was being announced at ISC in Frankfurt today with an upgraded petascale Arm supercomputer in the top third of the list, Nvidia announced its Read more…

By Tiffany Trader

Leading Solution Providers

ISC 2019 Virtual Booth Video Tour

CRAY
CRAY
DDN
DDN
DELL EMC
DELL EMC
GOOGLE
GOOGLE
ONE STOP SYSTEMS
ONE STOP SYSTEMS
PANASAS
PANASAS
VERNE GLOBAL
VERNE GLOBAL

Cray Wins NNSA-Livermore ‘El Capitan’ Exascale Contract

August 13, 2019

Cray has won the bid to build the first exascale supercomputer for the National Nuclear Security Administration (NNSA) and Lawrence Livermore National Laborator Read more…

By Tiffany Trader

Top500 Purely Petaflops; US Maintains Performance Lead

June 17, 2019

With the kick-off of the International Supercomputing Conference (ISC) in Frankfurt this morning, the 53rd Top500 list made its debut, and this one's for petafl Read more…

By Tiffany Trader

A Behind-the-Scenes Look at the Hardware That Powered the Black Hole Image

June 24, 2019

Two months ago, the first-ever image of a black hole took the internet by storm. A team of scientists took years to produce and verify the striking image – an Read more…

By Oliver Peckham

Cray – and the Cray Brand – to Be Positioned at Tip of HPE’s HPC Spear

May 22, 2019

More so than with most acquisitions of this kind, HPE’s purchase of Cray for $1.3 billion, announced last week, seems to have elements of that overused, often Read more…

By Doug Black and Tiffany Trader

AMD Launches Epyc Rome, First 7nm CPU

August 8, 2019

From a gala event at the Palace of Fine Arts in San Francisco yesterday (Aug. 7), AMD launched its second-generation Epyc Rome x86 chips, based on its 7nm proce Read more…

By Tiffany Trader

Chinese Company Sugon Placed on US ‘Entity List’ After Strong Showing at International Supercomputing Conference

June 26, 2019

After more than a decade of advancing its supercomputing prowess, operating the world’s most powerful supercomputer from June 2013 to June 2018, China is keep Read more…

By Tiffany Trader

In Wake of Nvidia-Mellanox: Xilinx to Acquire Solarflare

April 25, 2019

With echoes of Nvidia’s recent acquisition of Mellanox, FPGA maker Xilinx has announced a definitive agreement to acquire Solarflare Communications, provider Read more…

By Doug Black

Qualcomm Invests in RISC-V Startup SiFive

June 7, 2019

Investors are zeroing in on the open standard RISC-V instruction set architecture and the processor intellectual property being developed by a batch of high-flying chip startups. Last fall, Esperanto Technologies announced a $58 million funding round. Read more…

By George Leopold

  • arrow
  • Click Here for More Headlines
  • arrow
Do NOT follow this link or you will be banned from the site!
Share This