SC17: Singularity Preps Version 3.0, Nears 1M Containers Served Daily

By John Russell

November 1, 2017

Just a few months ago about half a million jobs were being run daily using Singularity containers, the LBNL-founded container platform intended for HPC. That was already a big number for the young open source project. Today, the number is closer to one million per day says Gregory Kurtzer, former HPC Systems Architect for Lawrence Berkeley National, the original driving force behind Singularity, and now the CEO of SingularityWare LCC.

It’s increasingly clear that the appetite for containers in scientific computing is every bit as strong as it is in enterprise computing where several technologies are jostling but Docker remains firmly the king. Containers, of course, aren’t new. (I’ve probably written exactly those words before) Like virtual machines (VM) before them, containers embody the idea of encapsulating compute environments to enhance application and workflow portability (and reproducibility) across varying compute infrastructures. Gather what you need into a single ‘box’, wrap it with enough standardized hooks to play nicely on other machines running Singularity, and game on. Build it once. Use (and share) it many times.

This separation of the software environment (and all of its dependencies) from the systems it runs on is an important paradigm change, says Kurtzer, especially in HPC where tight coupling of software to the specific underlying hardware has long been the guiding principle for squeezing out maximum performance. It still is. But for many jobs squeezing out optimum performance is less important than achieving portability and consistency of adequate performance.

HPCwire recently talked with Kurtzer about Singularity’s rapid growth, its technology roadmap, its staffing challenge, and Kurtzer’s plans to create a little excitement for Singularity at SC17. He is chary of spending on an SC booth given that SC newcomers often land in far corners with limited traffic. Instead Kurtzer is sponsoring a scavenger hunt involving many booths of Singularity stakeholders (preliminary list of participating booths at the end of the article). We’ll see how that works out.

In the meantime there’s lots to talk about. Singularity 2.4 was released last month with 3.0 expected early in 2018. (The Singularity FAQ page provides a good overview of features.) Importantly support from the big and small government and academic computing centers has been significant. Many of the familiar supercomputing centers, for example OCLF, TACC, NASA Ames, SDSC, Sandia, NIH, and LBNL, are using Singularity in production. Though not comprehensive the Singularity registry (voluntary) provides snapshot of Singularity user base.

It’s also noteworthy that now that Singularity has made substantial inroads into the traditional HPC community, Kurtzer is planning expansion into the enterprise computing arena. That seems in keeping with many traditional HPC technology suppliers who have pivoted to the enterprise in search of growth. What follows is a portion of our conversation plus a few slides from a recent presentation by Kurtzer to a national lab.

HPCwire: Seems like the use of Singularity is growing quickly, even in the relatively short time since we talked last (see, Singularity HPC Container Technology Moves Out of the Lab). Besides adoption what been happening with Singularity itself.

Gregory Kurtzer, SingularityWare CEO

Greg Kurtzer: We’ve been focusing a lot on the 2.4 release which I am hoping will actually be out maybe even tonight (it’s now out). There are a couple important new features in 2.4. The biggest is something we are calling instance support. Basically it’s the ability to run persistent namespaces such that you can come back and join it after the fact. It’s almost like a virtual machine; you can start up a Singularity container, run your jobs and everything you want in it, and when you want to leave it and exit, you can leave and exit and then you can come back to it later and it is still running.

The other big change is we have moved to compressed, immutable images. By default, previously everything was embedded in a read write format that emulates a file system. Now what we are doing is something that will be a little bit different because it is much more optimal in terms of space consumption on your hard drive because it is always compressed and actually even runs compressed. You can execute and use a compressed container without ever having to un-compress it.

Also, we now have ability to do things like persistent overlays so you can capture all the deltas in a container and then use that as a data container, a new concept we are playing with.

HPCwire: I understand that 3.0 is actually not far off. Can you give a preview? What’s on the technology roadmap.

Kurtzer: I can give you a little bit of a heads ups on what is coming in future versions. We’re coining a new term, or at least I think we are as I have not heard it used before us, called data containers which is really along the line of encapsulation of data. Currently we have encapsulations of environments, operating environments, etc. but there are also really good reasons to encapsulate your data. We are going to have additional abilities with security to further limit security exposure to host systems. That’s a big one. A lot of supercomputing centers won’t even have to make Singularity “setuid” root; they’ll basically just use additional Linux capabilities for increasing security privileges in order to run containers.

The big change we are going for [in 3.0] is introducing something called the Singularity image format, SIF for short. We are changing the major version name from 2.x to 3.x because we are changing the image format. Every time we have changed the image format we have changed the major number. SIF is basically a single file image because the whole context of Singularity is a single image; that’s really the main point of what a singularity container is, differentiating itself from other container systems or at least that is one of the main ones. We are spending a lot of effort defining what that single image looks like.

This SIF file will have the ability to have multiple data regions within the file. In a manner of speaking you can have multiple containers or multiple container layers, within a single file. Let’s say it’s running CentOS with MySQL and some sort of genomics application that queries the SQL database and then can do other things. You can have that in the base image and that base is immutable, it cannot change. Then when we have another data region which is writeable where we capture all of the modifications to that image in the writable layer of the SIF file. This is important because we can checksum and sign the multiple data regions independently while still allowing the data within the container to change or evolve.

Also we are bringing forth the idea of cryptographically signing of containers which is actually huge because the other container systems that are out there, even in enterprise, don’t have the ability to cryptographicly sign a container in its runtime form. Thanks to the SIF format we can have sections of that file which are signed and other sections which are using overlays. To give a use case, let’s stick with the genomics example; let’s say you have a database server running and you are querying and updating that genomics database and as time goes on that database is going to grow. If you think about this in a traditional way, that would break the signing, it would break the validation of your container. You wouldn’t be able to verify it anymore because you just changed your image. But because it is based on an immutable base and the data in that base layer can be signed independently within the image, we have the ability to have containers that evolve with time without breaking signatures.

HPCwire: What else is on the docket for 3.0 or beyond?

Kurtzer: The SIF image format includes image signing and image verification/validation, and we are also additional support for network namespaces. Right now we have network namespace isolation, but this new set of features will give us the ability for a container to come up as a virtual IP address. That obviously will require a privileged user; that can’t be done with a regular user because it will be changing network configurations. We’re also looking into CGroup support (control groups, Linux), and virtual booting of instances. The idea of virtual booting of those instances would basically allow you to run a Singularity container just like a VM. We have already done prototypes of this and it only takes it a fraction of a second to boot so it is very fast.

Container performance monitoring is also on the roadmap but I am not sure if we are going to get that into 3.0. It’s basically the ability to do performance profiling through a container. Our goal, really, is to make Singularity a feature-full, science enabling platform.

HPCwire: Maybe this is a good spot to review the Singularity user base. How is support from the major science computing centers? What does a typical user look like? Is it mostly the “long tail of science” type of users and organizations?

Kurtzer: Well, Titan (OCLF supercomputer center at Oak Ridge National Lab) among several other top 10 supercomputers on the Top500 are running Singularity. We have had very strong support from a wide range of major computing centers including older and new machines. As for who the user is, we have people that are not only on the long tail of science but also people on the cutting edge of their computational domain. This is because Singularity changes the software distribution and archival paradigm as the container can fit within a researcher’s existing data management solutions.

Singularity is already running on most of the large centers; what I want to focus on now is enabling the independent scientists as much as we can and to focus on this notion of enterprise HPC. I’ve been contacted by multiple vendors, tier 1 and tier 2, on something that they are calling “Enterprise HPC.” It’s basically enterprise sites who have little or no expertise in HPC but who are starting to run HPC-like workloads (machine learning and compute driven analytics). Singularity is being targeted by many organizations as the vector for basically dealing with these applications and these workloads because these enterprises don’t have the HPC expertise to be building or maintaining all of these workflows. The vendors are talking about building workflows – e.g., machine learning or analytics workflows – and saying ‘we want to distribute those as singularity containers.’ So singularity is going to be really big, I think, in this kind of hybrid mix between enterprise and HPC.

Many of these enterprise HPC jobs are not the tightly-coupled highly parallelized jobs we’ve come to see as commonplace on HPC. When someone says HPC it means something really specific to traditional HPC folks; it’s tightly coupled, we’ve got some sort of low latency interconnect, parallel file systems, designed to run high performance, highly scalable custom applications. But today, this has changed. HPC has come to mean pretty much any form of scientific computing and as a result, its breadth has grown in terms of what kind of applications we need to support. The traditional HPC architecture is not as applicable to a general wide use case scenario.

Singularity has lowered the barrier of entry to HPC considerably. People can create their own workflows, can leverage Docker, can and leverage other people’s containers via singularity hub to recapitulate people’s workflows and then further expand on this basis.

Even people who are doing more traditional HPC type jobs that are tightly coupled and whatnot are looking into containers to escape some of the dependency issues and some of the difficulties in creating those workflows and archive and/or distribute their software stacks.

HPCwire: Surely not all HPC applications are easily containerized.

Kurtzer: True. Fluent (CFD simulation, ANSYS) for example is extremely difficult to containerize. This is because to do a multi-node parallel processing job Fluent wants to run the MPI but the host resource manager should be controlling the MPI. We end up in this chicken and egg problem as the MPI within the container actually wants to be the MPI outside the container. Having vendor buy-in and support in how we properly containerize these applications is critical (hint, hint ANSYS, let’s talk.).

HPCwire: Let’s change gears and talk about the Singularity organization. How’s it going?

Kurtzer: It is fantastic. I’ve created a few companies previously, and several open source projects, including CentOS Linux, and the growth and commercial interest in Singularity has surpassed all of them! At this point, I am building my core team. I am looking for experienced developers, great minds, and people who want to change the face of computing. It is an extremely surreal experience, and I am looking for the most fantastic of people to join in this project. Funding is available so it just comes down to finding the right people.

To that point, if there are any readers out there interested in being part of this endeavor reach out to me, and let’s talk.

Preliminary List of Organizations Supporting Singularity at Their Booths

  1. Bright Computing
  2. Globus
  3. HPCwire
  4. MVAPICH2/Ohio State University
  5. Penguin Computing
  6. RedBarn Computing
  7. Rutgers
  8. SSERCA (Sunshine State Educational and Research Computing Alliance)
  9. Texas Tech University Booth
  10. University of Michigan / Michigan State

Slide Source: Kurtzer

Subscribe to HPCwire's Weekly Update!

Be the most informed person in the room! Stay ahead of the tech trends with industy updates delivered to you every week!

Automated Optimization Boosts ResNet50 Performance by 1.77x on Intel CPUs

October 23, 2018

From supercomputers to cell phones, every system and software device in our digital panoply has a growing number of settings that, if not optimized, constrain performance, wasting precious cycles and watts. In the f Read more…

By Tiffany Trader

South Africa CHPC: Home Grown Dynasty

October 22, 2018

Before the build up to the final event in the 2018 Student Cluster Competition season (the SC18 competition in Dallas), I want to take a moment to write about one of the great inspirational stories of these competitions. Read more…

By Dan Olds

NSF Launches Quantum Computing Faculty Fellows Program

October 22, 2018

Efforts to expand quantum computing research capacity continue to accelerate. The National Science Foundation today announced a Quantum Computing & Information Science Faculty Fellows (QCIS-FF) program aimed at devel Read more…

By John Russell

HPE Extreme Performance Solutions

One Small Step Toward Mars: One Giant Leap for Supercomputing

Since the days of the Space Race between the U.S. and the former Soviet Union, we have continually sought ways to perform experiments in space. Read more…

IBM Accelerated Insights

Join IBM at SC18 and Learn to Harness the Next Generation of AI-focused Supercomputing

Blurring the lines between HPC and AI

Today’s high performance computers are helping clients gain insights at an unprecedented pace. The intersection of artificial intelligence (AI) and HPC can transform industries while solving some of the world’s toughest challenges. Read more…

Democratization of HPC Part 3: Ninth Graders Tap HPC in the Cloud to Design Flying Boats

October 18, 2018

This is the third in a series of articles demonstrating the growing acceptance of high-performance computing (HPC) in new user communities and application areas. In this article we present UberCloud use case #208 on how Read more…

By Wolfgang Gentzsch and Håkon Bull Hove

Automated Optimization Boosts ResNet50 Performance by 1.77x on Intel CPUs

October 23, 2018

From supercomputers to cell phones, every system and software device in our digital panoply has a growing number of settings that, if not optimized, constrain  Read more…

By Tiffany Trader

South Africa CHPC: Home Grown Dynasty

October 22, 2018

Before the build up to the final event in the 2018 Student Cluster Competition season (the SC18 competition in Dallas), I want to take a moment to write about o Read more…

By Dan Olds

Penguin Computing Launches Consultancy for Piecing AI Strategies Together

October 18, 2018

AI stands before the HPC industry as a beacon of great expectations, yet market research repeatedly shows that AI adoption is commonly stuck in the talking phas Read more…

By Tiffany Trader

When Water Quality—Not Quantity—Hinders HPC Cooling

October 18, 2018

Attention has been paid to the sheer quantity of water consumed by supercomputers’ cooling towers – and rightly so, as they can require thousands of gallons per minute to cool. But in the background, another factor can emerge, bottlenecking efficiency and raising costs: water quality. Read more…

By Oliver Peckham

Paper Offers ‘Proof’ of Quantum Advantage on Some Problems

October 18, 2018

Is quantum computing worth all the effort being poured into it or should we just wait for classical computing to catch up? An IBM blog today posed those questio Read more…

By John Russell

Dell EMC to Supply U Michigan’s Great Lakes Cluster

October 16, 2018

The University of Michigan (U-M) today announced Dell EMC is the lead vendor for U-M’s $4.8 million Great Lakes HPC cluster scheduled for deployment in first Read more…

By John Russell

Houston to Field Massive, ‘Geophysically Configured’ Cloud Supercomputer

October 11, 2018

Based on some news stories out today, one might get the impression that the next system to crack number one on the Top500 would be an industrial oil and gas mon Read more…

By Tiffany Trader

Nvidia Platform Pushes GPUs into Machine Learning, High Performance Data Analytics

October 10, 2018

GPU leader Nvidia, generally associated with deep learning, autonomous vehicles and other higher-end enterprise and scientific workloads (and gaming, of course) Read more…

By Doug Black

TACC Wins Next NSF-funded Major Supercomputer

July 30, 2018

The Texas Advanced Computing Center (TACC) has won the next NSF-funded big supercomputer beating out rivals including the National Center for Supercomputing Ap Read more…

By John Russell

IBM at Hot Chips: What’s Next for Power

August 23, 2018

With processor, memory and networking technologies all racing to fill in for an ailing Moore’s law, the era of the heterogeneous datacenter is well underway, Read more…

By Tiffany Trader

Requiem for a Phi: Knights Landing Discontinued

July 25, 2018

On Monday, Intel made public its end of life strategy for the Knights Landing "KNL" Phi product set. The announcement makes official what has already been wide Read more…

By Tiffany Trader

CERN Project Sees Orders-of-Magnitude Speedup with AI Approach

August 14, 2018

An award-winning effort at CERN has demonstrated potential to significantly change how the physics based modeling and simulation communities view machine learni Read more…

By Rob Farber

House Passes $1.275B National Quantum Initiative

September 17, 2018

Last Thursday the U.S. House of Representatives passed the National Quantum Initiative Act (NQIA) intended to accelerate quantum computing research and developm Read more…

By John Russell

Summit Supercomputer is Already Making its Mark on Science

September 20, 2018

Summit, now the fastest supercomputer in the world, is quickly making its mark in science – five of the six finalists just announced for the prestigious 2018 Read more…

By John Russell

New Deep Learning Algorithm Solves Rubik’s Cube

July 25, 2018

Solving (and attempting to solve) Rubik’s Cube has delighted millions of puzzle lovers since 1974 when the cube was invented by Hungarian sculptor and archite Read more…

By John Russell

D-Wave Breaks New Ground in Quantum Simulation

July 16, 2018

Last Friday D-Wave scientists and colleagues published work in Science which they say represents the first fulfillment of Richard Feynman’s 1982 notion that Read more…

By John Russell

Leading Solution Providers

HPC on Wall Street 2018 Booth Video Tours Playlist

Arista

Dell EMC

IBM

Intel

RStor

VMWare

TACC’s ‘Frontera’ Supercomputer Expands Horizon for Extreme-Scale Science

August 29, 2018

The National Science Foundation and the Texas Advanced Computing Center announced today that a new system, called Frontera, will overtake Stampede 2 as the fast Read more…

By Tiffany Trader

HPE No. 1, IBM Surges, in ‘Bucking Bronco’ High Performance Server Market

September 27, 2018

Riding healthy U.S. and global economies, strong demand for AI-capable hardware and other tailwind trends, the high performance computing server market jumped 28 percent in the second quarter 2018 to $3.7 billion, up from $2.9 billion for the same period last year, according to industry analyst firm Hyperion Research. Read more…

By Doug Black

Intel Announces Cooper Lake, Advances AI Strategy

August 9, 2018

Intel's chief datacenter exec Navin Shenoy kicked off the company's Data-Centric Innovation Summit Wednesday, the day-long program devoted to Intel's datacenter Read more…

By Tiffany Trader

Germany Celebrates Launch of Two Fastest Supercomputers

September 26, 2018

The new high-performance computer SuperMUC-NG at the Leibniz Supercomputing Center (LRZ) in Garching is the fastest computer in Germany and one of the fastest i Read more…

By Tiffany Trader

MLPerf – Will New Machine Learning Benchmark Help Propel AI Forward?

May 2, 2018

Let the AI benchmarking wars begin. Today, a diverse group from academia and industry – Google, Baidu, Intel, AMD, Harvard, and Stanford among them – releas Read more…

By John Russell

Houston to Field Massive, ‘Geophysically Configured’ Cloud Supercomputer

October 11, 2018

Based on some news stories out today, one might get the impression that the next system to crack number one on the Top500 would be an industrial oil and gas mon Read more…

By Tiffany Trader

Aerodynamic Simulation Reveals Best Position in a Peloton of Cyclists

July 5, 2018

Eindhoven University of Technology (TU/e) and KU Leuven research group conducts the largest numerical simulation ever done in the sport industry and cycling discipline. The goal was to understand the aerodynamic interactions in the peloton, i.e., the main pack of cyclists in a race. Read more…

No Go for GloFo at 7nm; and the Fujitsu A64FX post-K CPU

September 5, 2018

It’s been a news worthy couple of weeks in the semiconductor and HPC industry. There were several HPC relevant disclosures at Hot Chips 2018 to whet appetites Read more…

By Dairsie Latimer

  • arrow
  • Click Here for More Headlines
  • arrow
Do NOT follow this link or you will be banned from the site!
Share This