Fostering Lustre Advancement Through Development and Contributions

By Carlos Aoki Thomaz

January 17, 2018

In this contributed feature, Carlos Aoki Thomaz, DDN senior product manager, provides perspective on the path of Lustre since Intel ended its commercially supported version last April and open sourced its Lustre activities. In a technically-detailed accounting, Thomaz spells out a number of strategic investments DDN is making in Lustre’s development.

Six months after organizational changes at Intel’s High Performance Data (HPDD) division, most in the Lustre community have shed any initial apprehension around the potential changes that could affect or disrupt Lustre development. Customers who have adopted the technology as their main parallel file system now have a clearer picture of what the future holds for the world’s most utilized parallel file system. Lustre remains strong and will continue to dominate the persistent parallel file system arena, at least for the foreseeable future.

Carlos Aoki Thomaz, Senior Product Manager at DDN

The new Lustre development and adoption strategy has turned out to be surprisingly simple, and more clear and consistent than anticipated. Like the old Whamcloud days, Lustre development has returned to a single code stream, thereby avoiding confusion and lack of discernment regarding different distributions, features, capabilities, and source code differentiation. Quietly released in July 2017, 2.10 is the LTS (Long Term Support) release of Lustre that should be the mainstream version through mid to early 2019.

As a major contributor to the Lustre community, DataDirect Networks (DDN) announced in 2016 that all of its Lustre features will be merged over time into the Lustre master branch. This convergence gives the entire community transparent access to the code, reducing the overhead of code development management and better aligning with the new features released in Lustre 2.10.

A very sophisticated set of features has been announced on Lustre 2.10 such as Progressive File Layouts (PFL), Project Quotas, IB Multi-rail and NRS Delay policy. Progressive file layouts allow system administrators and users to adjust file layouts, and how a file is stripped – the number of stripes and stripe block size now may vary according to the file size. There are several use cases that would take huge advantage leveraging PFL while simplifying the storage administration in the process. The storage administrator could define standard default layouts for different types of files, minimizing the need of users to manipulate file layouts by themselves (although the user is still able to define their own layouts). With the increasing utilization of flash technologies in a hybrid parallel file system (SSDs and NVMe devices mixed with standard rotational drives) it is now possible to create sophisticated mechanisms to optimize data location using PFL and OST pools.

Another feature, possibly the most latent need among the current Lustre users, is the Project Quotas. Project Quotas allows quota definition per “Project” which could be, for example, associated with a specific directory. Previously, Lustre only allowed standard POSIX User and group quotas. With Project Quotas we move one step ahead on the realm of managing spaces among users, groups and projects and planning for capacity and growth. Project Quota adds space accounting and enforcements of capacity utilization based on OSTs, sub-directories and file-sets, providing the granularity needed to manage several different use cases.

Some have asked about the impact of performance related to Project Quotas. Results of various tests have been impressive and encouraging, showing no degradation compared to the standard POSIX quota. Project Quotas is a feature available for Lustre running with a LDISKFS backend.

Although the feature has been only landed on Lustre 2.10, as the developer responsible for this feature, DDN has backported it into its Exascaler 3.2 (based on Lustre 2.7). Historically speaking, the latest and greatest version of Lustre usually brings the most advanced technologies with a price to pay, which is the un-tested and unproven chunk of codes that usually require a few cycles to stabilize. Since Project Quotas is a need for a huge range of customers that are not ready to move to Lustre 2.10 currently, Lustre 2.7 users can get the ability to run Project Quotas and get full support for it. In the case of customers running Project Quotas on Lustre 2.7, once they decide to upgrade to Lustre 2.10, data will be totally preserved (note that any users going from Lustre versions prior to 2.7, to Lustre 2.10 and activating Project Quotas require a reformat of the file system).

LNET IB Multi Rail allows users to take advantage of multiple infiniBand adapters, aggregating the bandwidth for Lustre LNET. This technique is widely used by Ethernet users through Ethernet Bonding. InfiniBand users were previously unable to “bond” interfaces and they were somehow limited to the performance of a single IB card. There was a need for increased bandwidth, especially on the client side. New architectures, such as HPE UV, have multiple sockets and a huge amount of memory capable to run multiple and much larger compute jobs. Those scenarios bring an unbalanced CPU/MEMORY to IO ratio, where even an IB EDR running 100Gbps may turn into a bottleneck. IB Multi Rail leverages Lustre on larger SMP like nodes, aggregating network bandwidth performance and proving a balanced CPU/Memory to IO ratio. On the server side, the biggest advantage is on the high availability capabilities. Having more than one IB link provides redundancy, avoiding scenarios that trigger server failover due an IB failure. Now in network failure scenarios the failures and their recovery are handled transparently, without compromising on performance.

NRS delay policy, which simulates high server load as way of validating the resilience of Lustre under load, is another feature introduced in Lustre 2.10. This is one valid way to perform fault injection and load simulation, usually very important during stabilization phases, performance characterization and overall debugging techniques.

Along with these recently announced features, a new approach has been proposed for Lustre’s policy engine (LiPE), designed to reduce installation and deployment complexity while delivering significantly faster results when executing and managing storage policies. LiPE relies on a set of components that allows the engine to:

  • Scan Lustre metadata targets (MDTs) quickly,
  • Create an in-memory map of the file system’s objects, and
  • Implement data management policies based on that mapped information.

This approach would allow users to define policies that trigger data automation via Lustre HSM hooks or external data management (copy tools, for example) mechanisms.

In the next stage of development, LiPE may be integrated with a File Heat Map mechanism for more automated and transparent data management, resulting in a better utilization of parallel storage infrastructure.

In regard to Lustre performance, a new initiative within the community is investigating the implementation of high-level tools, possibly at the user level, that would improve utilization and configuration of Lustre Quality of Service (QoS). In support of those efforts, a new QoS approach has been developed that is based on the Token Bucket Filter algorithm on the OST level. It allows system administrators to define the maximum number of RPCs to be issued by a user/group or job ID to a given OST. Throttling performance provides I/O control and bandwidth reservation that can guarantee that higher priority jobs run in a more predictable time, avoiding performance variations due to I/O delays.

In keeping with new HPC trends, a tremendous amount of work has also been invested in the integration of Lustre with Linux container-based workloads, providing native Lustre file system capabilities within containers, support for new kernel and specialized Artificial Intelligence and Machine Learning appliances.

2017 was a productive year for Lustre that showcased a very active and growing Lustre community and that positioned Lustre as the “go to” choice for many high-performance computing organizations and data centers. Moving into 2018, look for Lustre roadmaps to solidify this position with enhanced security, performance, Remote Access Service (RAS), and data management capabilities, as well as the addition of more enterprise-class features.

 

Subscribe to HPCwire's Weekly Update!

Be the most informed person in the room! Stay ahead of the tech trends with industy updates delivered to you every week!

HOKUSAI’s BigWaterfall Cluster Extends RIKEN’s Supercomputing Performance

February 21, 2018

RIKEN, Japan’s largest comprehensive research institution, recently expanded the capacity and capabilities of its HOKUSAI supercomputer, a key resource managed by the institution’s Advanced Center for Computing and C Read more…

By Ken Strandberg

Neural Networking Shows Promise in Earthquake Monitoring

February 21, 2018

A team of Harvard University and MIT researchers report their new neural networking method for monitoring earthquakes is more accurate and orders of magnitude faster than traditional approaches. Read more…

By John Russell

HPE Wins $57 Million DoD Supercomputing Contract

February 20, 2018

Hewlett Packard Enterprise (HPE) today revealed details of its massive $57 million HPC contract with the U.S. Department of Defense (DoD). The deal calls for HPE to provide the DoD High Performance Computing Modernizatio Read more…

By Tiffany Trader

HPE Extreme Performance Solutions

Experience Memory & Storage Solutions that will Transform Your Data Performance

High performance computing (HPC) has revolutionized the way we harness insight, leading to a dramatic increase in both the size and complexity of HPC systems. Read more…

Topological Quantum Superconductor Progress Reported

February 20, 2018

Overcoming sensitivity to decoherence is a persistent stumbling block in efforts to build effective quantum computers. Now, a group of researchers from Chalmers University of Technology (Sweden) report progress in devisi Read more…

By John Russell

HOKUSAI’s BigWaterfall Cluster Extends RIKEN’s Supercomputing Performance

February 21, 2018

RIKEN, Japan’s largest comprehensive research institution, recently expanded the capacity and capabilities of its HOKUSAI supercomputer, a key resource manage Read more…

By Ken Strandberg

Neural Networking Shows Promise in Earthquake Monitoring

February 21, 2018

A team of Harvard University and MIT researchers report their new neural networking method for monitoring earthquakes is more accurate and orders of magnitude faster than traditional approaches. Read more…

By John Russell

Fluid HPC: How Extreme-Scale Computing Should Respond to Meltdown and Spectre

February 15, 2018

The Meltdown and Spectre vulnerabilities are proving difficult to fix, and initial experiments suggest security patches will cause significant performance penal Read more…

By Pete Beckman

Brookhaven Ramps Up Computing for National Security Effort

February 14, 2018

Last week, Dan Coats, the director of Director of National Intelligence for the U.S., warned the Senate Intelligence Committee that Russia was likely to meddle in the 2018 mid-term U.S. elections, much as it stands accused of doing in the 2016 Presidential election. Read more…

By John Russell

AI Cloud Competition Heats Up: Google’s TPUs, Amazon Building AI Chip

February 12, 2018

Competition in the white hot AI (and public cloud) market pits Google against Amazon this week, with Google offering AI hardware on its cloud platform intended Read more…

By Doug Black

Russian Nuclear Engineers Caught Cryptomining on Lab Supercomputer

February 12, 2018

Nuclear scientists working at the All-Russian Research Institute of Experimental Physics (RFNC-VNIIEF) have been arrested for using lab supercomputing resources to mine crypto-currency, according to a report in Russia’s Interfax News Agency. Read more…

By Tiffany Trader

The Food Industry’s Next Journey — from Mars to Exascale

February 12, 2018

Global food producer and one of the world's leading chocolate companies Mars Inc. has a unique perspective on the impact that exascale computing will have on the food industry. Read more…

By Scott Gibson, Oak Ridge National Laboratory

Singularity HPC Container Start-Up – Sylabs – Emerges from Stealth

February 8, 2018

The driving force behind Singularity, the popular HPC container technology, is bringing the open source platform to the enterprise with the launch of a new vent Read more…

By George Leopold

Inventor Claims to Have Solved Floating Point Error Problem

January 17, 2018

"The decades-old floating point error problem has been solved," proclaims a press release from inventor Alan Jorgensen. The computer scientist has filed for and Read more…

By Tiffany Trader

Japan Unveils Quantum Neural Network

November 22, 2017

The U.S. and China are leading the race toward productive quantum computing, but it's early enough that ultimate leadership is still something of an open questi Read more…

By Tiffany Trader

AMD Showcases Growing Portfolio of EPYC and Radeon-based Systems at SC17

November 13, 2017

AMD’s charge back into HPC and the datacenter is on full display at SC17. Having launched the EPYC processor line in June along with its MI25 GPU the focus he Read more…

By John Russell

Researchers Measure Impact of ‘Meltdown’ and ‘Spectre’ Patches on HPC Workloads

January 17, 2018

Computer scientists from the Center for Computational Research, State University of New York (SUNY), University at Buffalo have examined the effect of Meltdown Read more…

By Tiffany Trader

IBM Begins Power9 Rollout with Backing from DOE, Google

December 6, 2017

After over a year of buildup, IBM is unveiling its first Power9 system based on the same architecture as the Department of Energy CORAL supercomputers, Summit a Read more…

By Tiffany Trader

Nvidia Responds to Google TPU Benchmarking

April 10, 2017

Nvidia highlights strengths of its newest GPU silicon in response to Google's report on the performance and energy advantages of its custom tensor processor. Read more…

By Tiffany Trader

Fast Forward: Five HPC Predictions for 2018

December 21, 2017

What’s on your list of high (and low) lights for 2017? Volta 100’s arrival on the heels of the P100? Appearance, albeit late in the year, of IBM’s Power9? Read more…

By John Russell

Russian Nuclear Engineers Caught Cryptomining on Lab Supercomputer

February 12, 2018

Nuclear scientists working at the All-Russian Research Institute of Experimental Physics (RFNC-VNIIEF) have been arrested for using lab supercomputing resources to mine crypto-currency, according to a report in Russia’s Interfax News Agency. Read more…

By Tiffany Trader

Leading Solution Providers

Chip Flaws ‘Meltdown’ and ‘Spectre’ Loom Large

January 4, 2018

The HPC and wider tech community have been abuzz this week over the discovery of critical design flaws that impact virtually all contemporary microprocessors. T Read more…

By Tiffany Trader

Perspective: What Really Happened at SC17?

November 22, 2017

SC is over. Now comes the myriad of follow-ups. Inboxes are filled with templated emails from vendors and other exhibitors hoping to win a place in the post-SC thinking of booth visitors. Attendees of tutorials, workshops and other technical sessions will be inundated with requests for feedback. Read more…

By Andrew Jones

How Meltdown and Spectre Patches Will Affect HPC Workloads

January 10, 2018

There have been claims that the fixes for the Meltdown and Spectre security vulnerabilities, named the KPTI (aka KAISER) patches, are going to affect applicatio Read more…

By Rosemary Francis

GlobalFoundries, Ayar Labs Team Up to Commercialize Optical I/O

December 4, 2017

GlobalFoundries (GF) and Ayar Labs, a startup focused on using light, instead of electricity, to transfer data between chips, today announced they've entered in Read more…

By Tiffany Trader

Tensors Come of Age: Why the AI Revolution Will Help HPC

November 13, 2017

Thirty years ago, parallel computing was coming of age. A bitter battle began between stalwart vector computing supporters and advocates of various approaches to parallel computing. IBM skeptic Alan Karp, reacting to announcements of nCUBE’s 1024-microprocessor system and Thinking Machines’ 65,536-element array, made a public $100 wager that no one could get a parallel speedup of over 200 on real HPC workloads. Read more…

By John Gustafson & Lenore Mullin

Flipping the Flops and Reading the Top500 Tea Leaves

November 13, 2017

The 50th edition of the Top500 list, the biannual publication of the world’s fastest supercomputers based on public Linpack benchmarking results, was released Read more…

By Tiffany Trader

V100 Good but not Great on Select Deep Learning Aps, Says Xcelerit

November 27, 2017

Wringing optimum performance from hardware to accelerate deep learning applications is a challenge that often depends on the specific application in use. A benc Read more…

By John Russell

SC17: Singularity Preps Version 3.0, Nears 1M Containers Served Daily

November 1, 2017

Just a few months ago about half a million jobs were being run daily using Singularity containers, the LBNL-founded container platform intended for HPC. That wa Read more…

By John Russell

  • arrow
  • Click Here for More Headlines
  • arrow
Share This