Visit additional Tabor Communication Publications
February 21, 2013
With the announcement this week that storage maker Xyratex has acquired Oracle's Lustre assets, the popular open source parallel file system is once again completely under the control of HPC stakeholders. The deal gives Xyratex ownership of the Lustre trademark, logo, and lustre.org website, as well as related intellectual property still under the Oracle banner.
Xyratex will also take on Oracle's remaining Lustre support contracts and bring over an undetermined number of employees from the database giant that were involved in that effort. The deal was announced on Tuesday, with neither company disclosing the terms of the transaction.
Although UK-based Xyratex has been providing more generic storage hardware to OEMs for some time, the company took a deep dive into the HPC storage space in 2010 when it acquired ClusterStor, a Colorado-startup that built HPC cluster storage systems of the same name. Shortly thereafter Xyratex brought Peter Braam and Peter Bojanic on board to lead the company's Lustre development. Braam and Bojanic were two of the original Lustre architects and have been key to keeping the technology on course as it's made a circuitous route around the industry, from Carnegie-Mellon University (1999) to Cluster File Systems (2001) to Sun Microsystems (2007) to Oracle (2010), and now to Xyratex (2013).
Thanks to its open source pedigree, the file system was able to survive the 14-year journey intact. In fact, as a result of its growing developer and customer base, the Lustre software has become more reliable and robust as it's aged. Its moment of crisis came under Oracle, a company that has expressed little interest in the high performance computing market. The database maker developed a version for its own commercial offerings, but had no obvious motivation to move the technology forward as an open source HPC solution.
When Lustre was treading water at Oracle, Whamcloud emerged as a key player in coalescing the community. The company helped stabilize the Lustre source tree and put the technology back on an HPC-centric roadmap. When Intel bought Whamcloud in 2012, it added even more credibility to Lustre's long-term prospects.
Building on Whamcloud's momentum was the Open Scalable File Systems (OpenSFS) consortium, which brought together all the major Lustre developers: Cray, DataDirect Networks, Intel/Whamcloud, Lawrence Livermore National Lab, Oak Ridge National Lab, and Xyratex. OpenSFS is committed to the continuing development of Lustre for HPC vendors and customers and using the technology as a basis for future exascale file systems.
Xyratex is also engaged with the European Open File System (EOFS) organization, another consortium committed to the adoption of open source parallel file systems. Like OpenSFS, EOFS is interested in a supporting and adding new features to Lustre. The organization brings with it some notable European HPC vendors and end users, including Bull, Eurotech, ParTec, CEA, the Max Planck Society, Forschungszentrum Jülich, and Leibniz-Rechenzentrum, to name a few.
With some of the biggest users behind it, Lustre has come to dominate the top tier of HPC systems, inhabiting six of the fastest 10 supercomputers and more than 60 of the speediest 100. Although less frequently encountered in smaller and medium-sized HPC clusters, Lustre is still the most common parallel file system across high performance computing. In our 2012 survey, Intersect360 Research found that the most frequently mentioned file management software (disregarding in-house or vendor-supplied tools) was Lustre, at 23 percent.
The technology continues to push the envelope in performance and scalability as systems enter the double-digit petabyte realm. For example, Lustre is currently slinging files for Blue Waters, the newly minted multi-petaflop Cray supercomputer housed at the National Center for Supercomputing Applications (NCSA). That machine's 25 petabytes of Cray Sonexion storage, which is based on Xyratex ClusterStor gear and Lustre file management, delivers a terabyte per second of I/O to more than 25 thousand compute nodes.
Besides Cray, Xyratex has also hooked up with Dell and HP, who will resell ClusterStor systems for HPC and "big data" installations. Now that it has cut itself a bigger slice of the Lustre pie, Xyratex may be able to entice other OEMs into reseller partnerships.
For the rest of the Lustre community, it should be business as usual. The code repository will remain open source and be available to vendors, government labs, and academics alike, whether they do business with Xyratex or not. OpenSFS' mission to shepherd the technology for Lustre stakeholders remains unchanged. In fact with one of its top tier members now in control of the trademark and website, it should be even easier for the organization to promote Lustre adoption and keep the technology evolving into the exascale realm.
May 23, 2013 |
he study of climate change is one of those scientific problems where it is almost essential to model the entire Earth to attain accurate results and make worthwhile predictions. In an attempt to make climate science more accessible to smaller research facilities, NASA introduced what they call ‘Climate in a Box,’ a system they note acts as a desktop supercomputer.
May 22, 2013 |
At some point in the not-too-distant future, building powerful, miniature computing systems will be considered a hobby for high schoolers, just as robotics or even Lego-building are today. That could be made possible through recent advancements made with the Raspberry Pi computers.
May 16, 2013 |
When it comes to cloud, long distances mean unacceptably high latencies. Researchers from the University of Bonn in Germany examined those latency issues of doing CFD modeling in the cloud by utilizing a common CFD and its utilization in HPC instance types including both CPU and GPU cores of Amazon EC2.
May 15, 2013 |
Supercomputers at the Department of Energy’s National Energy Research Scientific Computing Center (NERSC) have worked on important computational problems such as collapse of the atomic state, the optimization of chemical catalysts, and now modeling popping bubbles.
May 10, 2013 |
Program provides cash awards up to $10,000 for the best open-source end-user applications deployed on 100G network.
05/10/2013 | Cleversafe, Cray, DDN, NetApp, & Panasas | From Wall Street to Hollywood, drug discovery to homeland security, companies and organizations of all sizes and stripes are coming face to face with the challenges – and opportunities – afforded by Big Data. Before anyone can utilize these extraordinary data repositories, however, they must first harness and manage their data stores, and do so utilizing technologies that underscore affordability, security, and scalability.
04/15/2013 | Bull | “50% of HPC users say their largest jobs scale to 120 cores or less.” How about yours? Are your codes ready to take advantage of today’s and tomorrow’s ultra-parallel HPC systems? Download this White Paper by Analysts Intersect360 Research to see what Bull and Intel’s Center for Excellence in Parallel Programming can do for your codes.
In this demonstration of SGI DMF ZeroWatt disk solution, Dr. Eng Lim Goh, SGI CTO, discusses a function of SGI DMF software to reduce costs and power consumption in an exascale (Big Data) storage datacenter.
The Cray CS300-AC cluster supercomputer offers energy efficient, air-cooled design based on modular, industry-standard platforms featuring the latest processor and network technologies and a wide range of datacenter cooling requirements.