SC17: Singularity Preps Version 3.0, Nears 1M Containers Served Daily

By John Russell

November 1, 2017

Just a few months ago about half a million jobs were being run daily using Singularity containers, the LBNL-founded container platform intended for HPC. That was already a big number for the young open source project. Today, the number is closer to one million per day says Gregory Kurtzer, former HPC Systems Architect for Lawrence Berkeley National, the original driving force behind Singularity, and now the CEO of SingularityWare LCC.

It’s increasingly clear that the appetite for containers in scientific computing is every bit as strong as it is in enterprise computing where several technologies are jostling but Docker remains firmly the king. Containers, of course, aren’t new. (I’ve probably written exactly those words before) Like virtual machines (VM) before them, containers embody the idea of encapsulating compute environments to enhance application and workflow portability (and reproducibility) across varying compute infrastructures. Gather what you need into a single ‘box’, wrap it with enough standardized hooks to play nicely on other machines running Singularity, and game on. Build it once. Use (and share) it many times.

This separation of the software environment (and all of its dependencies) from the systems it runs on is an important paradigm change, says Kurtzer, especially in HPC where tight coupling of software to the specific underlying hardware has long been the guiding principle for squeezing out maximum performance. It still is. But for many jobs squeezing out optimum performance is less important than achieving portability and consistency of adequate performance.

HPCwire recently talked with Kurtzer about Singularity’s rapid growth, its technology roadmap, its staffing challenge, and Kurtzer’s plans to create a little excitement for Singularity at SC17. He is chary of spending on an SC booth given that SC newcomers often land in far corners with limited traffic. Instead Kurtzer is sponsoring a scavenger hunt involving many booths of Singularity stakeholders (preliminary list of participating booths at the end of the article). We’ll see how that works out.

In the meantime there’s lots to talk about. Singularity 2.4 was released last month with 3.0 expected early in 2018. (The Singularity FAQ page provides a good overview of features.) Importantly support from the big and small government and academic computing centers has been significant. Many of the familiar supercomputing centers, for example OCLF, TACC, NASA Ames, SDSC, Sandia, NIH, and LBNL, are using Singularity in production. Though not comprehensive the Singularity registry (voluntary) provides snapshot of Singularity user base.

It’s also noteworthy that now that Singularity has made substantial inroads into the traditional HPC community, Kurtzer is planning expansion into the enterprise computing arena. That seems in keeping with many traditional HPC technology suppliers who have pivoted to the enterprise in search of growth. What follows is a portion of our conversation plus a few slides from a recent presentation by Kurtzer to a national lab.

HPCwire: Seems like the use of Singularity is growing quickly, even in the relatively short time since we talked last (see, Singularity HPC Container Technology Moves Out of the Lab). Besides adoption what been happening with Singularity itself.

Gregory Kurtzer, SingularityWare CEO

Greg Kurtzer: We’ve been focusing a lot on the 2.4 release which I am hoping will actually be out maybe even tonight (it’s now out). There are a couple important new features in 2.4. The biggest is something we are calling instance support. Basically it’s the ability to run persistent namespaces such that you can come back and join it after the fact. It’s almost like a virtual machine; you can start up a Singularity container, run your jobs and everything you want in it, and when you want to leave it and exit, you can leave and exit and then you can come back to it later and it is still running.

The other big change is we have moved to compressed, immutable images. By default, previously everything was embedded in a read write format that emulates a file system. Now what we are doing is something that will be a little bit different because it is much more optimal in terms of space consumption on your hard drive because it is always compressed and actually even runs compressed. You can execute and use a compressed container without ever having to un-compress it.

Also, we now have ability to do things like persistent overlays so you can capture all the deltas in a container and then use that as a data container, a new concept we are playing with.

HPCwire: I understand that 3.0 is actually not far off. Can you give a preview? What’s on the technology roadmap.

Kurtzer: I can give you a little bit of a heads ups on what is coming in future versions. We’re coining a new term, or at least I think we are as I have not heard it used before us, called data containers which is really along the line of encapsulation of data. Currently we have encapsulations of environments, operating environments, etc. but there are also really good reasons to encapsulate your data. We are going to have additional abilities with security to further limit security exposure to host systems. That’s a big one. A lot of supercomputing centers won’t even have to make Singularity “setuid” root; they’ll basically just use additional Linux capabilities for increasing security privileges in order to run containers.

The big change we are going for [in 3.0] is introducing something called the Singularity image format, SIF for short. We are changing the major version name from 2.x to 3.x because we are changing the image format. Every time we have changed the image format we have changed the major number. SIF is basically a single file image because the whole context of Singularity is a single image; that’s really the main point of what a singularity container is, differentiating itself from other container systems or at least that is one of the main ones. We are spending a lot of effort defining what that single image looks like.

This SIF file will have the ability to have multiple data regions within the file. In a manner of speaking you can have multiple containers or multiple container layers, within a single file. Let’s say it’s running CentOS with MySQL and some sort of genomics application that queries the SQL database and then can do other things. You can have that in the base image and that base is immutable, it cannot change. Then when we have another data region which is writeable where we capture all of the modifications to that image in the writable layer of the SIF file. This is important because we can checksum and sign the multiple data regions independently while still allowing the data within the container to change or evolve.

Also we are bringing forth the idea of cryptographically signing of containers which is actually huge because the other container systems that are out there, even in enterprise, don’t have the ability to cryptographicly sign a container in its runtime form. Thanks to the SIF format we can have sections of that file which are signed and other sections which are using overlays. To give a use case, let’s stick with the genomics example; let’s say you have a database server running and you are querying and updating that genomics database and as time goes on that database is going to grow. If you think about this in a traditional way, that would break the signing, it would break the validation of your container. You wouldn’t be able to verify it anymore because you just changed your image. But because it is based on an immutable base and the data in that base layer can be signed independently within the image, we have the ability to have containers that evolve with time without breaking signatures.

HPCwire: What else is on the docket for 3.0 or beyond?

Kurtzer: The SIF image format includes image signing and image verification/validation, and we are also additional support for network namespaces. Right now we have network namespace isolation, but this new set of features will give us the ability for a container to come up as a virtual IP address. That obviously will require a privileged user; that can’t be done with a regular user because it will be changing network configurations. We’re also looking into CGroup support (control groups, Linux), and virtual booting of instances. The idea of virtual booting of those instances would basically allow you to run a Singularity container just like a VM. We have already done prototypes of this and it only takes it a fraction of a second to boot so it is very fast.

Container performance monitoring is also on the roadmap but I am not sure if we are going to get that into 3.0. It’s basically the ability to do performance profiling through a container. Our goal, really, is to make Singularity a feature-full, science enabling platform.

HPCwire: Maybe this is a good spot to review the Singularity user base. How is support from the major science computing centers? What does a typical user look like? Is it mostly the “long tail of science” type of users and organizations?

Kurtzer: Well, Titan (OCLF supercomputer center at Oak Ridge National Lab) among several other top 10 supercomputers on the Top500 are running Singularity. We have had very strong support from a wide range of major computing centers including older and new machines. As for who the user is, we have people that are not only on the long tail of science but also people on the cutting edge of their computational domain. This is because Singularity changes the software distribution and archival paradigm as the container can fit within a researcher’s existing data management solutions.

Singularity is already running on most of the large centers; what I want to focus on now is enabling the independent scientists as much as we can and to focus on this notion of enterprise HPC. I’ve been contacted by multiple vendors, tier 1 and tier 2, on something that they are calling “Enterprise HPC.” It’s basically enterprise sites who have little or no expertise in HPC but who are starting to run HPC-like workloads (machine learning and compute driven analytics). Singularity is being targeted by many organizations as the vector for basically dealing with these applications and these workloads because these enterprises don’t have the HPC expertise to be building or maintaining all of these workflows. The vendors are talking about building workflows – e.g., machine learning or analytics workflows – and saying ‘we want to distribute those as singularity containers.’ So singularity is going to be really big, I think, in this kind of hybrid mix between enterprise and HPC.

Many of these enterprise HPC jobs are not the tightly-coupled highly parallelized jobs we’ve come to see as commonplace on HPC. When someone says HPC it means something really specific to traditional HPC folks; it’s tightly coupled, we’ve got some sort of low latency interconnect, parallel file systems, designed to run high performance, highly scalable custom applications. But today, this has changed. HPC has come to mean pretty much any form of scientific computing and as a result, its breadth has grown in terms of what kind of applications we need to support. The traditional HPC architecture is not as applicable to a general wide use case scenario.

Singularity has lowered the barrier of entry to HPC considerably. People can create their own workflows, can leverage Docker, can and leverage other people’s containers via singularity hub to recapitulate people’s workflows and then further expand on this basis.

Even people who are doing more traditional HPC type jobs that are tightly coupled and whatnot are looking into containers to escape some of the dependency issues and some of the difficulties in creating those workflows and archive and/or distribute their software stacks.

HPCwire: Surely not all HPC applications are easily containerized.

Kurtzer: True. Fluent (CFD simulation, ANSYS) for example is extremely difficult to containerize. This is because to do a multi-node parallel processing job Fluent wants to run the MPI but the host resource manager should be controlling the MPI. We end up in this chicken and egg problem as the MPI within the container actually wants to be the MPI outside the container. Having vendor buy-in and support in how we properly containerize these applications is critical (hint, hint ANSYS, let’s talk.).

HPCwire: Let’s change gears and talk about the Singularity organization. How’s it going?

Kurtzer: It is fantastic. I’ve created a few companies previously, and several open source projects, including CentOS Linux, and the growth and commercial interest in Singularity has surpassed all of them! At this point, I am building my core team. I am looking for experienced developers, great minds, and people who want to change the face of computing. It is an extremely surreal experience, and I am looking for the most fantastic of people to join in this project. Funding is available so it just comes down to finding the right people.

To that point, if there are any readers out there interested in being part of this endeavor reach out to me, and let’s talk.

Preliminary List of Organizations Supporting Singularity at Their Booths

  1. Bright Computing
  2. Globus
  3. HPCwire
  4. MVAPICH2/Ohio State University
  5. Penguin Computing
  6. RedBarn Computing
  7. Rutgers
  8. SSERCA (Sunshine State Educational and Research Computing Alliance)
  9. Texas Tech University Booth
  10. University of Michigan / Michigan State

Slide Source: Kurtzer

Subscribe to HPCwire's Weekly Update!

Be the most informed person in the room! Stay ahead of the tech trends with industy updates delivered to you every week!

African Supercomputing Center Inaugurates ‘Toubkal,’ Most Powerful Supercomputer on the Continent

February 25, 2021

Historically, Africa hasn’t exactly been synonymous with supercomputing. There are only a handful of supercomputers on the continent, with few ranking on the global stage. Now, the Mohammed VI Polytechnic University (U Read more…

By Oliver Peckham

Supercomputer-Powered Machine Learning Supports Fusion Energy Reactor Design

February 25, 2021

Energy researchers have been reaching for the stars for decades in their attempt to artificially recreate a stable fusion energy reactor. If successful, such a reactor would revolutionize the world’s energy supply over Read more…

By Oliver Peckham

Japan to Debut Integrated Fujitsu HPC/AI Supercomputer This Spring

February 25, 2021

The integrated Fujitsu HPC/AI Supercomputer, Wisteria, is coming to Japan this spring. The University of Tokyo is preparing to deploy a heterogeneous computing system, called "Wisteria/BDEC-01," that will tackle simulati Read more…

By Tiffany Trader

President Biden Signs Executive Order to Review Chip, Other Supply Chains

February 24, 2021

U.S. President Biden signed an executive order late today calling for a 100-day review of key supply chains including semiconductors, large capacity batteries, pharmaceuticals, and rare-earth elements. The scarcity of ch Read more…

By John Russell

Xilinx Launches Alveo SN1000 SmartNIC

February 24, 2021

FPGA vendor Xilinx has debuted its latest SmartNIC model, the Alveo SN1000, with integrated “composability” features that allow enterprise users to add their own custom networking functions to supplement its built-in networking. By providing deep flexibility... Read more…

By Todd R. Weiss

AWS Solution Channel

Introducing AWS HPC Tech Shorts

Amazon Web Services (AWS) is excited to announce a new videos series focused on running HPC workloads on AWS. This new video series will cover HPC workloads from genomics, computational chemistry, to computational fluid dynamics (CFD) and more. Read more…

ASF Keynotes Showcase How HPC and Big Data Have Pervaded the Pandemic

February 24, 2021

Last Thursday, a range of experts joined the Advanced Scale Forum (ASF) in a rapid-fire roundtable to discuss how advanced technologies have transformed the way humanity responded to the COVID-19 pandemic in indelible ways. The roundtable, held near the one-year mark of the first... Read more…

By Oliver Peckham

Japan to Debut Integrated Fujitsu HPC/AI Supercomputer This Spring

February 25, 2021

The integrated Fujitsu HPC/AI Supercomputer, Wisteria, is coming to Japan this spring. The University of Tokyo is preparing to deploy a heterogeneous computing Read more…

By Tiffany Trader

Xilinx Launches Alveo SN1000 SmartNIC

February 24, 2021

FPGA vendor Xilinx has debuted its latest SmartNIC model, the Alveo SN1000, with integrated “composability” features that allow enterprise users to add their own custom networking functions to supplement its built-in networking. By providing deep flexibility... Read more…

By Todd R. Weiss

ASF Keynotes Showcase How HPC and Big Data Have Pervaded the Pandemic

February 24, 2021

Last Thursday, a range of experts joined the Advanced Scale Forum (ASF) in a rapid-fire roundtable to discuss how advanced technologies have transformed the way humanity responded to the COVID-19 pandemic in indelible ways. The roundtable, held near the one-year mark of the first... Read more…

By Oliver Peckham

IBM’s Prototype Low-Power 7nm AI Chip Offers ‘Precision Scaling’

February 23, 2021

IBM has released details of a prototype AI chip geared toward low-precision training and inference across different AI model types while retaining model quality within AI applications. In a paper delivered during this year’s International Solid-State Circuits Virtual Conference, IBM... Read more…

By George Leopold

IBM Continues Mainstreaming Power Systems and Integrating Red Hat in Pivot to Cloud

February 23, 2021

As IBM continues its massive pivot to the cloud, its Power-microprocessor-based products are being mainstreamed and realigned with the corporate-wide strategy. Read more…

By John Russell

Livermore’s El Capitan Supercomputer to Debut HPE ‘Rabbit’ Near Node Local Storage

February 18, 2021

A near node local storage innovation called Rabbit factored heavily into Lawrence Livermore National Laboratory’s decision to select Cray’s proposal for its CORAL-2 machine, the lab’s first exascale-class supercomputer, El Capitan. Details of this new storage technology were revealed... Read more…

By Tiffany Trader

ENIAC at 75: Celebrating the World’s First Supercomputer

February 15, 2021

With little fanfare, today’s computer revolution was arguably born and announced through a small, innocuous, two-column story at the bottom of the front page of The New York Times on Feb. 15, 1946. In that story and others, the previously classified project, ENIAC... Read more…

By Todd R. Weiss

Microsoft, HPE Bringing AI, Edge, Cloud to Earth Orbit in Preparation for Mars Missions

February 12, 2021

The International Space Station will soon get a delivery of powerful AI, edge and cloud computing tools from HPE and Microsoft Azure to expand technology experi Read more…

By Todd R. Weiss

Julia Update: Adoption Keeps Climbing; Is It a Python Challenger?

January 13, 2021

The rapid adoption of Julia, the open source, high level programing language with roots at MIT, shows no sign of slowing according to data from Julialang.org. I Read more…

By John Russell

Esperanto Unveils ML Chip with Nearly 1,100 RISC-V Cores

December 8, 2020

At the RISC-V Summit today, Art Swift, CEO of Esperanto Technologies, announced a new, RISC-V based chip aimed at machine learning and containing nearly 1,100 low-power cores based on the open-source RISC-V architecture. Esperanto Technologies, headquartered in... Read more…

By Oliver Peckham

Azure Scaled to Record 86,400 Cores for Molecular Dynamics

November 20, 2020

A new record for HPC scaling on the public cloud has been achieved on Microsoft Azure. Led by Dr. Jer-Ming Chia, the cloud provider partnered with the Beckman I Read more…

By Oliver Peckham

NICS Unleashes ‘Kraken’ Supercomputer

April 4, 2008

A Cray XT4 supercomputer, dubbed Kraken, is scheduled to come online in mid-summer at the National Institute for Computational Sciences (NICS). The soon-to-be petascale system, and the resulting NICS organization, are the result of an NSF Track II award of $65 million to the University of Tennessee and its partners to provide next-generation supercomputing for the nation's science community. Read more…

Programming the Soon-to-Be World’s Fastest Supercomputer, Frontier

January 5, 2021

What’s it like designing an app for the world’s fastest supercomputer, set to come online in the United States in 2021? The University of Delaware’s Sunita Chandrasekaran is leading an elite international team in just that task. Chandrasekaran, assistant professor of computer and information sciences, recently was named... Read more…

By Tracey Bryant

10nm, 7nm, 5nm…. Should the Chip Nanometer Metric Be Replaced?

June 1, 2020

The biggest cool factor in server chips is the nanometer. AMD beating Intel to a CPU built on a 7nm process node* – with 5nm and 3nm on the way – has been i Read more…

By Doug Black

Top500: Fugaku Keeps Crown, Nvidia’s Selene Climbs to #5

November 16, 2020

With the publication of the 56th Top500 list today from SC20's virtual proceedings, Japan's Fugaku supercomputer – now fully deployed – notches another win, Read more…

By Tiffany Trader

Gordon Bell Special Prize Goes to Massive SARS-CoV-2 Simulations

November 19, 2020

2020 has proven a harrowing year – but it has produced remarkable heroes. To that end, this year, the Association for Computing Machinery (ACM) introduced the Read more…

By Oliver Peckham

Leading Solution Providers

Contributors

Texas A&M Announces Flagship ‘Grace’ Supercomputer

November 9, 2020

Texas A&M University has announced its next flagship system: Grace. The new supercomputer, named for legendary programming pioneer Grace Hopper, is replacing the Ada system (itself named for mathematician Ada Lovelace) as the primary workhorse for Texas A&M’s High Performance Research Computing (HPRC). Read more…

By Oliver Peckham

Saudi Aramco Unveils Dammam 7, Its New Top Ten Supercomputer

January 21, 2021

By revenue, oil and gas giant Saudi Aramco is one of the largest companies in the world, and it has historically employed commensurate amounts of supercomputing Read more…

By Oliver Peckham

Intel Xe-HP GPU Deployed for Aurora Exascale Development

November 17, 2020

At SC20, Intel announced that it is making its Xe-HP high performance discrete GPUs available to early access developers. Notably, the new chips have been deplo Read more…

By Tiffany Trader

Intel Teases Ice Lake-SP, Shows Competitive Benchmarking

November 17, 2020

At SC20 this week, Intel teased its forthcoming third-generation Xeon "Ice Lake-SP" server processor, claiming competitive benchmarking results against AMD's second-generation Epyc "Rome" processor. Ice Lake-SP, Intel's first server processor with 10nm technology... Read more…

By Tiffany Trader

New Deep Learning Algorithm Solves Rubik’s Cube

July 25, 2018

Solving (and attempting to solve) Rubik’s Cube has delighted millions of puzzle lovers since 1974 when the cube was invented by Hungarian sculptor and archite Read more…

By John Russell

At Oak Ridge, ‘End of Life’ Sometimes Isn’t

October 31, 2020

Sometimes, the old dog actually does go live on a farm. HPC systems are often cursed with short lifespans, as they are continually supplanted by the latest and Read more…

By Oliver Peckham

It’s Fugaku vs. COVID-19: How the World’s Top Supercomputer Is Shaping Our New Normal

November 9, 2020

Fugaku is currently the most powerful publicly ranked supercomputer in the world – but we weren’t supposed to have it yet. The supercomputer, situated at Japan’s Riken scientific research institute, was scheduled to come online in 2021. When the pandemic struck... Read more…

By Oliver Peckham

MIT Makes a Big Breakthrough in Nonsilicon Transistors

December 10, 2020

What if Silicon Valley moved beyond silicon? In the 80’s, Seymour Cray was asking the same question, delivering at Supercomputing 1988 a talk titled “What’s All This About Gallium Arsenide?” The supercomputing legend intended to make gallium arsenide (GaA) the material of the future... Read more…

By Oliver Peckham

  • arrow
  • Click Here for More Headlines
  • arrow
HPCwire