Lustre Gets Backing of Non-Profit Corporation

By Michael Feldman

October 21, 2010

Some of the most prominent organizations in the HPC community have joined together to bootstrap a non-profit corporation devoted to scalable file system technologies. On Tuesday, Cray, Data Direct Networks, Lawrence Livermore National Laboratory (LLNL) and Oak Ridge National Laboratory (ORNL) announced the incorporation of Open Scalable File Systems, Inc. (OpenSFS). The newly-hatched group has cast itself as the focal point for development of Lustre and other open source file system technologies aimed at high performance computing.

According to OpenSFS CEO Norman Morse, the organization’s mission is to bring together the stakeholders for high-end scalable file systems and provide a formal structure for moving the associated software forward. Today that effort will focus on Lustre, the open source parallel file system that grew up in HPC. The Lustre source repository is currently in the hands of Oracle, who inherited the technology when it acquired Sun Microsystems, (who itself had acquired Lustre a year before it got swallowed up). Since Oracle will focus post-Lustre 2.0 development on OpenSolaris and its own database products, Linux-based Lustre for HPC has been left to a disparate group of vendors, research labs, and academic institutions who have a common need to see the technology move forward.

OpenSFS’ role will be to gather requirements from HPC stakeholders, prioritize them, and then fund the efforts to implement them. “We’ll develop feature sets that are important for the entire community, within the context of OpenSFS, and then over time those feature sets will make their way back into the canonical Lustre release,” explains Galen Shipman, group leader of technology integration at Oak Ridge National Laboratory and OpenSFS board member.

That model is pretty much the same as before, prior to Oracle’s control of the Lustre code. The rationale is to fold all software fixes and enhancements back into official Lustre source repository, in order to avoid the prospect of multiple (and incompatible) implementations roaming around the ecosystem. “We absolutely refuse to fork the system,” declares Morse. “We intend for Oracle to be the canonical definition of Lustre.”

The initial focus for OpenSFS will be to support and stabilize the current Linux-based Lustre storage systems in production at HPC installations around the world. This is especially critical for the array of US Department of Energy labs, who have very large Lustre storage systems deployed, and even larger ones on the drawing board. The longer term goal for OpenSFS is to morph Lustre and related parallel file technologies into something that supports the transition to exascale systems several years down the road.

Requirements for new features will come out of technical working groups organized by OpenSFS, and those enhancements deemed most important will be brought forward as RFPs to the community. As a non-profit entity, OpenSFS won’t be doing the development itself, but vendors who have aggregated Lustre expertise — Whamcloud, Terascala, Xyratex, SGI, Cray, DataDirect Networks, and others — would be likely to bid on these contracts.

Funding for this work will be derived from OpenSFS membership dues, which depending on your organization’s commitment to this effort can be quite expensive. There are three different levels: The promoter level costs $500K per year, which buys you a seat on the OpenSFS board; the contributor/adopter level runs $50K, and lets you manage a working group; finally, for $5K per year you can become a support member, which allows you to participate in the working group. As you might imagine, the further you go up the membership food chain, the more influence you have over which work gets funded.

Since Lustre development and testing requires large-scale computing and storage, support for this OpenSFS-initiated development will be provided by national labs, such as Lawrence Livermore and Oak Ridge, which already have resources in place for this type of work. At LLNL, the Hyperion system is available on the lab’s unclassified network as a test bed for scaling different types of Linux cluster technologies. For the past year, Sun Microsystems (and then Oracle) used the machine for its Lustre 2.0 development. Likewise, Oak Ridge has its own test bed of storage systems from various vendors for developer access. Much of the SMP scalability work for Lustre was developed and tested at ORNL. Other research labs, both in the US and elsewhere, may end up donating their own HPC resources for Lustre development, especially if they’re looking to drive specific file system development for their own programs.

Morse says members are already lining up to join the alliance. According to him, more than 20 organizations — vendors, universities, and government labs — are ready to sign on (although he wouldn’t say at what membership levels). As soon as certain legalities of OpenSFS incorporation are finalized, they’ll begin bringing them aboard. Morse expects to attract in the neighborhood of 50 to 60 organizations.

To help that process along, next month OpenSFS is going to host an introductory meeting about the organization in conjunction with Supercomputing Conference (SC10) in New Orleans. Although they were too late to reserve a session at SC10 proper, the meeting will take place in parallel with the conference festivities. The meeting is tentatively scheduled for Tuesday, September 16 at the Ritz Carlton. Registration information will soon be available on the OpenSFS website.

Subscribe to HPCwire's Weekly Update!

Be the most informed person in the room! Stay ahead of the tech trends with industry updates delivered to you every week!

MLPerf Inference 4.0 Results Showcase GenAI; Nvidia Still Dominates

March 28, 2024

There were no startling surprises in the latest MLPerf Inference benchmark (4.0) results released yesterday. Two new workloads — Llama 2 and Stable Diffusion XL — were added to the benchmark suite as MLPerf continues Read more…

Q&A with Nvidia’s Chief of DGX Systems on the DGX-GB200 Rack-scale System

March 27, 2024

Pictures of Nvidia's new flagship mega-server, the DGX GB200, on the GTC show floor got favorable reactions on social media for the sheer amount of computing power it brings to artificial intelligence.  Nvidia's DGX Read more…

Call for Participation in Workshop on Potential NSF CISE Quantum Initiative

March 26, 2024

Editor’s Note: Next month there will be a workshop to discuss what a quantum initiative led by NSF’s Computer, Information Science and Engineering (CISE) directorate could entail. The details are posted below in a Ca Read more…

Waseda U. Researchers Reports New Quantum Algorithm for Speeding Optimization

March 25, 2024

Optimization problems cover a wide range of applications and are often cited as good candidates for quantum computing. However, the execution time for constrained combinatorial optimization applications on quantum device Read more…

NVLink: Faster Interconnects and Switches to Help Relieve Data Bottlenecks

March 25, 2024

Nvidia’s new Blackwell architecture may have stolen the show this week at the GPU Technology Conference in San Jose, California. But an emerging bottleneck at the network layer threatens to make bigger and brawnier pro Read more…

Who is David Blackwell?

March 22, 2024

During GTC24, co-founder and president of NVIDIA Jensen Huang unveiled the Blackwell GPU. This GPU itself is heavily optimized for AI work, boasting 192GB of HBM3E memory as well as the the ability to train 1 trillion pa Read more…

MLPerf Inference 4.0 Results Showcase GenAI; Nvidia Still Dominates

March 28, 2024

There were no startling surprises in the latest MLPerf Inference benchmark (4.0) results released yesterday. Two new workloads — Llama 2 and Stable Diffusion Read more…

Q&A with Nvidia’s Chief of DGX Systems on the DGX-GB200 Rack-scale System

March 27, 2024

Pictures of Nvidia's new flagship mega-server, the DGX GB200, on the GTC show floor got favorable reactions on social media for the sheer amount of computing po Read more…

NVLink: Faster Interconnects and Switches to Help Relieve Data Bottlenecks

March 25, 2024

Nvidia’s new Blackwell architecture may have stolen the show this week at the GPU Technology Conference in San Jose, California. But an emerging bottleneck at Read more…

Who is David Blackwell?

March 22, 2024

During GTC24, co-founder and president of NVIDIA Jensen Huang unveiled the Blackwell GPU. This GPU itself is heavily optimized for AI work, boasting 192GB of HB Read more…

Nvidia Looks to Accelerate GenAI Adoption with NIM

March 19, 2024

Today at the GPU Technology Conference, Nvidia launched a new offering aimed at helping customers quickly deploy their generative AI applications in a secure, s Read more…

The Generative AI Future Is Now, Nvidia’s Huang Says

March 19, 2024

We are in the early days of a transformative shift in how business gets done thanks to the advent of generative AI, according to Nvidia CEO and cofounder Jensen Read more…

Nvidia’s New Blackwell GPU Can Train AI Models with Trillions of Parameters

March 18, 2024

Nvidia's latest and fastest GPU, codenamed Blackwell, is here and will underpin the company's AI plans this year. The chip offers performance improvements from Read more…

Nvidia Showcases Quantum Cloud, Expanding Quantum Portfolio at GTC24

March 18, 2024

Nvidia’s barrage of quantum news at GTC24 this week includes new products, signature collaborations, and a new Nvidia Quantum Cloud for quantum developers. Wh Read more…

Alibaba Shuts Down its Quantum Computing Effort

November 30, 2023

In case you missed it, China’s e-commerce giant Alibaba has shut down its quantum computing research effort. It’s not entirely clear what drove the change. Read more…

Nvidia H100: Are 550,000 GPUs Enough for This Year?

August 17, 2023

The GPU Squeeze continues to place a premium on Nvidia H100 GPUs. In a recent Financial Times article, Nvidia reports that it expects to ship 550,000 of its lat Read more…

Shutterstock 1285747942

AMD’s Horsepower-packed MI300X GPU Beats Nvidia’s Upcoming H200

December 7, 2023

AMD and Nvidia are locked in an AI performance battle – much like the gaming GPU performance clash the companies have waged for decades. AMD has claimed it Read more…

DoD Takes a Long View of Quantum Computing

December 19, 2023

Given the large sums tied to expensive weapon systems – think $100-million-plus per F-35 fighter – it’s easy to forget the U.S. Department of Defense is a Read more…

Synopsys Eats Ansys: Does HPC Get Indigestion?

February 8, 2024

Recently, it was announced that Synopsys is buying HPC tool developer Ansys. Started in Pittsburgh, Pa., in 1970 as Swanson Analysis Systems, Inc. (SASI) by John Swanson (and eventually renamed), Ansys serves the CAE (Computer Aided Engineering)/multiphysics engineering simulation market. Read more…

Choosing the Right GPU for LLM Inference and Training

December 11, 2023

Accelerating the training and inference processes of deep learning models is crucial for unleashing their true potential and NVIDIA GPUs have emerged as a game- Read more…

Intel’s Server and PC Chip Development Will Blur After 2025

January 15, 2024

Intel's dealing with much more than chip rivals breathing down its neck; it is simultaneously integrating a bevy of new technologies such as chiplets, artificia Read more…

Baidu Exits Quantum, Closely Following Alibaba’s Earlier Move

January 5, 2024

Reuters reported this week that Baidu, China’s giant e-commerce and services provider, is exiting the quantum computing development arena. Reuters reported � Read more…

Leading Solution Providers

Contributors

Comparing NVIDIA A100 and NVIDIA L40S: Which GPU is Ideal for AI and Graphics-Intensive Workloads?

October 30, 2023

With long lead times for the NVIDIA H100 and A100 GPUs, many organizations are looking at the new NVIDIA L40S GPU, which it’s a new GPU optimized for AI and g Read more…

Shutterstock 1179408610

Google Addresses the Mysteries of Its Hypercomputer 

December 28, 2023

When Google launched its Hypercomputer earlier this month (December 2023), the first reaction was, "Say what?" It turns out that the Hypercomputer is Google's t Read more…

AMD MI3000A

How AMD May Get Across the CUDA Moat

October 5, 2023

When discussing GenAI, the term "GPU" almost always enters the conversation and the topic often moves toward performance and access. Interestingly, the word "GPU" is assumed to mean "Nvidia" products. (As an aside, the popular Nvidia hardware used in GenAI are not technically... Read more…

Shutterstock 1606064203

Meta’s Zuckerberg Puts Its AI Future in the Hands of 600,000 GPUs

January 25, 2024

In under two minutes, Meta's CEO, Mark Zuckerberg, laid out the company's AI plans, which included a plan to build an artificial intelligence system with the eq Read more…

Google Introduces ‘Hypercomputer’ to Its AI Infrastructure

December 11, 2023

Google ran out of monikers to describe its new AI system released on December 7. Supercomputer perhaps wasn't an apt description, so it settled on Hypercomputer Read more…

China Is All In on a RISC-V Future

January 8, 2024

The state of RISC-V in China was discussed in a recent report released by the Jamestown Foundation, a Washington, D.C.-based think tank. The report, entitled "E Read more…

Intel Won’t Have a Xeon Max Chip with New Emerald Rapids CPU

December 14, 2023

As expected, Intel officially announced its 5th generation Xeon server chips codenamed Emerald Rapids at an event in New York City, where the focus was really o Read more…

IBM Quantum Summit: Two New QPUs, Upgraded Qiskit, 10-year Roadmap and More

December 4, 2023

IBM kicks off its annual Quantum Summit today and will announce a broad range of advances including its much-anticipated 1121-qubit Condor QPU, a smaller 133-qu Read more…

  • arrow
  • Click Here for More Headlines
  • arrow
HPCwire