CUPERTINO, Calif., Nov. 18 — Seagate Technology plc, a world leader in storage solutions, today announced that as part of its continued commitment to open source communities, the contribution of an Apache Hadoop on Lustre Connector. The Hadoop on Lustre Connector improves workflow efficiency by eliminating the need to copy data to the Hadoop Distributed File System (HDFS) prior to running Apache Hadoop jobs. The Hadoop on Lustre Connector also provides an alternative to Hadoop’s reliance on the HDFS file system and enables Hadoop ecosystem tools such as Mahout, Hive and Pig to take advantage of the Lustre file system. In addition, Seagate is also releasing source code for a patch to Hadoop that allows Map and Reduce processes to share files and enables the use of “diskless” Hadoop compute clusters, allowing Hadoop to function with HPC architectures that use Lustre for storage. HPC customers in the Life Science and Energy fields are increasingly using Hadoop and Lustre together as part of their data analysis workflows. The Hadoop on Lustre Connector helps HPC customers streamline their Hadoop workflows and accelerate time to results.
Also announced today is an agreement to transfer assets relating to Lustre.org to Open Scalable File Systems, Inc. (OpenSFS) and European Open Filesystem SCE (EOFS). OpenSFS and EOFS are trusted stewards of the Lustre distributed file software community and will jointly manage Lustre.org. Seagate continues to demonstrate a commitment to Lustre through financial contributions to OpenSFS at the highest ‘Promoter’ level and as an active board member. Seagate has deep involvement with OpenSFS and EOFS on all levels and is one of the largest code contributors to the Lustre code tree.
“Seagate believes that direct involvement enabling core capabilities as well as fostering the addition of new application environments is critical to open source community vitality, especially for Lustre which is a foundation for much of the success of high performance computing among science, government and business community leaders. Our work with OpenStack Swift, the Open Compute Project (OCP), OpenSFS, EOFS and now Hadoop is just the beginning,” said Ken Claffey, Vice President of ClusterStor, Seagate Cloud Systems and Solutions. “We are committed to driving open source innovation and partnering with open source communities as they develop cutting-edge enabling technologies that are foundational for the entire industry.”
This news follows Seagate’s recent announcement to make its Ethernet Drive interface specification and T-Card development adapter available to the Open Compute Project in January of this year.
Seagate is exhibiting at SC14, November 16-21 in New Orleans during which it will demonstrate the Hadoop on Lustre Connector at booth 3239. Those interested in a meeting should contact a member of the Seagate team or stop by the booth during the event.
Seagate is a world leader in storage solutions. Learn more at www.seagate.com.