Lustre Gets Backing of Non-Profit Corporation

By Michael Feldman

October 21, 2010

Some of the most prominent organizations in the HPC community have joined together to bootstrap a non-profit corporation devoted to scalable file system technologies. On Tuesday, Cray, Data Direct Networks, Lawrence Livermore National Laboratory (LLNL) and Oak Ridge National Laboratory (ORNL) announced the incorporation of Open Scalable File Systems, Inc. (OpenSFS). The newly-hatched group has cast itself as the focal point for development of Lustre and other open source file system technologies aimed at high performance computing.

According to OpenSFS CEO Norman Morse, the organization’s mission is to bring together the stakeholders for high-end scalable file systems and provide a formal structure for moving the associated software forward. Today that effort will focus on Lustre, the open source parallel file system that grew up in HPC. The Lustre source repository is currently in the hands of Oracle, who inherited the technology when it acquired Sun Microsystems, (who itself had acquired Lustre a year before it got swallowed up). Since Oracle will focus post-Lustre 2.0 development on OpenSolaris and its own database products, Linux-based Lustre for HPC has been left to a disparate group of vendors, research labs, and academic institutions who have a common need to see the technology move forward.

OpenSFS’ role will be to gather requirements from HPC stakeholders, prioritize them, and then fund the efforts to implement them. “We’ll develop feature sets that are important for the entire community, within the context of OpenSFS, and then over time those feature sets will make their way back into the canonical Lustre release,” explains Galen Shipman, group leader of technology integration at Oak Ridge National Laboratory and OpenSFS board member.

That model is pretty much the same as before, prior to Oracle’s control of the Lustre code. The rationale is to fold all software fixes and enhancements back into official Lustre source repository, in order to avoid the prospect of multiple (and incompatible) implementations roaming around the ecosystem. “We absolutely refuse to fork the system,” declares Morse. “We intend for Oracle to be the canonical definition of Lustre.”

The initial focus for OpenSFS will be to support and stabilize the current Linux-based Lustre storage systems in production at HPC installations around the world. This is especially critical for the array of US Department of Energy labs, who have very large Lustre storage systems deployed, and even larger ones on the drawing board. The longer term goal for OpenSFS is to morph Lustre and related parallel file technologies into something that supports the transition to exascale systems several years down the road.

Requirements for new features will come out of technical working groups organized by OpenSFS, and those enhancements deemed most important will be brought forward as RFPs to the community. As a non-profit entity, OpenSFS won’t be doing the development itself, but vendors who have aggregated Lustre expertise — Whamcloud, Terascala, Xyratex, SGI, Cray, DataDirect Networks, and others — would be likely to bid on these contracts.

Funding for this work will be derived from OpenSFS membership dues, which depending on your organization’s commitment to this effort can be quite expensive. There are three different levels: The promoter level costs $500K per year, which buys you a seat on the OpenSFS board; the contributor/adopter level runs $50K, and lets you manage a working group; finally, for $5K per year you can become a support member, which allows you to participate in the working group. As you might imagine, the further you go up the membership food chain, the more influence you have over which work gets funded.

Since Lustre development and testing requires large-scale computing and storage, support for this OpenSFS-initiated development will be provided by national labs, such as Lawrence Livermore and Oak Ridge, which already have resources in place for this type of work. At LLNL, the Hyperion system is available on the lab’s unclassified network as a test bed for scaling different types of Linux cluster technologies. For the past year, Sun Microsystems (and then Oracle) used the machine for its Lustre 2.0 development. Likewise, Oak Ridge has its own test bed of storage systems from various vendors for developer access. Much of the SMP scalability work for Lustre was developed and tested at ORNL. Other research labs, both in the US and elsewhere, may end up donating their own HPC resources for Lustre development, especially if they’re looking to drive specific file system development for their own programs.

Morse says members are already lining up to join the alliance. According to him, more than 20 organizations — vendors, universities, and government labs — are ready to sign on (although he wouldn’t say at what membership levels). As soon as certain legalities of OpenSFS incorporation are finalized, they’ll begin bringing them aboard. Morse expects to attract in the neighborhood of 50 to 60 organizations.

To help that process along, next month OpenSFS is going to host an introductory meeting about the organization in conjunction with Supercomputing Conference (SC10) in New Orleans. Although they were too late to reserve a session at SC10 proper, the meeting will take place in parallel with the conference festivities. The meeting is tentatively scheduled for Tuesday, September 16 at the Ritz Carlton. Registration information will soon be available on the OpenSFS website.

Subscribe to HPCwire's Weekly Update!

Be the most informed person in the room! Stay ahead of the tech trends with industy updates delivered to you every week!

Machine Learning at HPC User Forum: Drilling into Specific Use Cases

September 22, 2017

The 66th HPC User Forum held September 5-7, in Milwaukee, Wisconsin, at the elegant and historic Pfister Hotel, highlighting the 1893 Victorian décor and art of “The Grand Hotel Of The West,” contrasted nicely with Read more…

By Arno Kolster

Google Cloud Makes Good on Promise to Add Nvidia P100 GPUs

September 21, 2017

Google has taken down the notice on its cloud platform website that says Nvidia Tesla P100s are “coming soon.” That's because the search giant has announced the beta launch of the high-end P100 Nvidia Tesla GPUs on t Read more…

By George Leopold

Cray Wins $48M Supercomputer Contract from KISTI

September 21, 2017

It was a good day for Cray which won a $48 million contract from the Korea Institute of Science and Technology Information (KISTI) for a 128-rack CS500 cluster supercomputer. The new system, equipped with Intel Xeon Scal Read more…

By John Russell

HPE Extreme Performance Solutions

HPE Prepares Customers for Success with the HPC Software Portfolio

High performance computing (HPC) software is key to harnessing the full power of HPC environments. Development and management tools enable IT departments to streamline installation and maintenance of their systems as well as create, optimize, and run their HPC applications. Read more…

Adolfy Hoisie to Lead Brookhaven’s Computing for National Security Effort

September 21, 2017

Brookhaven National Laboratory announced today that Adolfy Hoisie will chair its newly formed Computing for National Security department, which is part of Brookhaven’s new Computational Science Initiative (CSI). Read more…

By John Russell

Machine Learning at HPC User Forum: Drilling into Specific Use Cases

September 22, 2017

The 66th HPC User Forum held September 5-7, in Milwaukee, Wisconsin, at the elegant and historic Pfister Hotel, highlighting the 1893 Victorian décor and art o Read more…

By Arno Kolster

Stanford University and UberCloud Achieve Breakthrough in Living Heart Simulations

September 21, 2017

Cardiac arrhythmia can be an undesirable and potentially lethal side effect of drugs. During this condition, the electrical activity of the heart turns chaotic, Read more…

By Wolfgang Gentzsch, UberCloud, and Francisco Sahli, Stanford University

PNNL’s Center for Advanced Tech Evaluation Seeks Wider HPC Community Ties

September 21, 2017

Two years ago the Department of Energy established the Center for Advanced Technology Evaluation (CENATE) at Pacific Northwest National Laboratory (PNNL). CENAT Read more…

By John Russell

Exascale Computing Project Names Doug Kothe as Director

September 20, 2017

The Department of Energy’s Exascale Computing Project (ECP) has named Doug Kothe as its new director effective October 1. He replaces Paul Messina, who is stepping down after two years to return to Argonne National Laboratory. Kothe is a 32-year veteran of DOE’s National Laboratory System. Read more…

Takeaways from the Milwaukee HPC User Forum

September 19, 2017

Milwaukee’s elegant Pfister Hotel hosted approximately 100 attendees for the 66th HPC User Forum (September 5-7, 2017). In the original home city of Pabst Blu Read more…

By Merle Giles

Kathy Yelick Charts the Promise and Progress of Exascale Science

September 15, 2017

On Friday, Sept. 8, Kathy Yelick of Lawrence Berkeley National Laboratory and the University of California, Berkeley, delivered the keynote address on “Breakthrough Science at the Exascale” at the ACM Europe Conference in Barcelona. In conjunction with her presentation, Yelick agreed to a short Q&A discussion with HPCwire. Read more…

By Tiffany Trader

DARPA Pledges Another $300 Million for Post-Moore’s Readiness

September 14, 2017

The Defense Advanced Research Projects Agency (DARPA) launched a giant funding effort to ensure the United States can sustain the pace of electronic innovation vital to both a flourishing economy and a secure military. Under the banner of the Electronics Resurgence Initiative (ERI), some $500-$800 million will be invested in post-Moore’s Law technologies. Read more…

By Tiffany Trader

IBM Breaks Ground for Complex Quantum Chemistry

September 14, 2017

IBM has reported the use of a novel algorithm to simulate BeH2 (beryllium-hydride) on a quantum computer. This is the largest molecule so far simulated on a quantum computer. The technique, which used six qubits of a seven-qubit system, is an important step forward and may suggest an approach to simulating ever larger molecules. Read more…

By John Russell

How ‘Knights Mill’ Gets Its Deep Learning Flops

June 22, 2017

Intel, the subject of much speculation regarding the delayed, rewritten or potentially canceled “Aurora” contract (the Argonne Lab part of the CORAL “ Read more…

By Tiffany Trader

Reinders: “AVX-512 May Be a Hidden Gem” in Intel Xeon Scalable Processors

June 29, 2017

Imagine if we could use vector processing on something other than just floating point problems.  Today, GPUs and CPUs work tirelessly to accelerate algorithms Read more…

By James Reinders

NERSC Scales Scientific Deep Learning to 15 Petaflops

August 28, 2017

A collaborative effort between Intel, NERSC and Stanford has delivered the first 15-petaflops deep learning software running on HPC platforms and is, according Read more…

By Rob Farber

Oracle Layoffs Reportedly Hit SPARC and Solaris Hard

September 7, 2017

Oracle’s latest layoffs have many wondering if this is the end of the line for the SPARC processor and Solaris OS development. As reported by multiple sources Read more…

By John Russell

Six Exascale PathForward Vendors Selected; DoE Providing $258M

June 15, 2017

The much-anticipated PathForward awards for hardware R&D in support of the Exascale Computing Project were announced today with six vendors selected – AMD Read more…

By John Russell

Top500 Results: Latest List Trends and What’s in Store

June 19, 2017

Greetings from Frankfurt and the 2017 International Supercomputing Conference where the latest Top500 list has just been revealed. Although there were no major Read more…

By Tiffany Trader

IBM Clears Path to 5nm with Silicon Nanosheets

June 5, 2017

Two years since announcing the industry’s first 7nm node test chip, IBM and its research alliance partners GlobalFoundries and Samsung have developed a proces Read more…

By Tiffany Trader

Nvidia Responds to Google TPU Benchmarking

April 10, 2017

Nvidia highlights strengths of its newest GPU silicon in response to Google's report on the performance and energy advantages of its custom tensor processor. Read more…

By Tiffany Trader

Leading Solution Providers

Graphcore Readies Launch of 16nm Colossus-IPU Chip

July 20, 2017

A second $30 million funding round for U.K. AI chip developer Graphcore sets up the company to go to market with its “intelligent processing unit” (IPU) in Read more…

By Tiffany Trader

Google Releases Deeplearn.js to Further Democratize Machine Learning

August 17, 2017

Spreading the use of machine learning tools is one of the goals of Google’s PAIR (People + AI Research) initiative, which was introduced in early July. Last w Read more…

By John Russell

Russian Researchers Claim First Quantum-Safe Blockchain

May 25, 2017

The Russian Quantum Center today announced it has overcome the threat of quantum cryptography by creating the first quantum-safe blockchain, securing cryptocurrencies like Bitcoin, along with classified government communications and other sensitive digital transfers. Read more…

By Doug Black

Google Debuts TPU v2 and will Add to Google Cloud

May 25, 2017

Not long after stirring attention in the deep learning/AI community by revealing the details of its Tensor Processing Unit (TPU), Google last week announced the Read more…

By John Russell

EU Funds 20 Million Euro ARM+FPGA Exascale Project

September 7, 2017

At the Barcelona Supercomputer Centre on Wednesday (Sept. 6), 16 partners gathered to launch the EuroEXA project, which invests €20 million over three-and-a-half years into exascale-focused research and development. Led by the Horizon 2020 program, EuroEXA picks up the banner of a triad of partner projects — ExaNeSt, EcoScale and ExaNoDe — building on their work... Read more…

By Tiffany Trader

Amazon Debuts New AMD-based GPU Instances for Graphics Acceleration

September 12, 2017

Last week Amazon Web Services (AWS) streaming service, AppStream 2.0, introduced a new GPU instance called Graphics Design intended to accelerate graphics. The Read more…

By John Russell

Cray Moves to Acquire the Seagate ClusterStor Line

July 28, 2017

This week Cray announced that it is picking up Seagate's ClusterStor HPC storage array business for an undisclosed sum. "In short we're effectively transitioning the bulk of the ClusterStor product line to Cray," said CEO Peter Ungaro. Read more…

By Tiffany Trader

GlobalFoundries: 7nm Chips Coming in 2018, EUV in 2019

June 13, 2017

GlobalFoundries has formally announced that its 7nm technology is ready for customer engagement with product tape outs expected for the first half of 2018. The Read more…

By Tiffany Trader

  • arrow
  • Click Here for More Headlines
  • arrow
Share This