PALO ALTO, Calif., Nov. 14, 2021 — Zettar today announced it will showcase a fully optimized, at-scale, enterprise AI data migration solution, developed in collaboration with DDN and NVIDIA, at Supercomputing 2021 (Nov. 14-19, 2021).
NVIDIA technology-driven research is gaining ever increasing prominence and momentum in industries and academia. NVIDIA DGX SuperPOD, the first cloud-native, multi-tenant AI supercomputer and a turnkey data center, integrated with DDN’s AI400X AI storage appliances using NVIDIA Quantum InfiniBand networking, is the de-facto standard of large-scale AI supercomputing in data-intensive enterprises.
With the fast-growing and ever larger datasets for AI training and processing, feeding GPUs in a timely and adequate manner has become challenging. This is particularly true when data needs to be migrated for cases from the edge, repatriated from cloud, ingested from another on-prem storage, or balanced between existing and newly added storage. Under such circumstances, using a simple, scalable, and efficient software data mover is essential for boosting the productivity of large-scale AI research.
Zettar’s AI data migration solution, developed in collaboration with DDN and NVIDIA, is production ready, using commercially available hardware and software, and attains full storage and network performance. It is compact, consisting of only 11 rack units: 8 RU for the 4 AI400x, 1 RU for the NVIDIA Quantum 8700 InfiniBand switch, and 2 RU for servers. It is also linearly scalable and highly available, with a distributed and modular architecture that offers intelligent task management, is easy to use and secure, and provides extensive reporting. Furthermore, it covers all common use cases: single-site, multi-site, and multi-cloud.
The companies began a solution testbed on October 15. Ten days later, the testbed was moving 2.9 PB/day (33GB/s). With further tuning, it is anticipated to achieve a data migration rate of 4PB/day (50GB/s) by NVIDIA GTC next spring, a 34 percent boost using the same hardware and software. A recorded lightning talk is available at the NVIDIA SC21 Virtual Theater. There will be a follow-up whitepaper that describes the solution, testing procedures and results, as well as open-source tools by Zettar, including the elbencho storage sweep tools.
“When data needs to be migrated from any other storage services — such as NAS systems, a parallel file system, or an object storage — across any distance, the basic architecture of the solution is still applicable. Once data is in the DDN storage, excellent NVIDIA DGX SuperPOD AI processing efficiency is expected,” according to Chin Fang, founder and CEO, Zettar Inc.
“Zettar’s software uses NVIDIA Quantum InfiniBand technology to deliver at-scale, high-performance data migration, even over long distances when using the NVIDIA MetroX-2 for edge-to-datacenter use cases,” said Gilad Shainer, SVP, Marketing, NVIDIA.
“The Zettar solution is an ideal complement to our A3I storage appliances,” said Kurt Kuckein, SVP of marketing at DDN. “Data movement is a frequent challenge for our customers because they operate at a scale typical data management tools don’t address. Delivering high-speed data transfer into a centralized repository gives customers quicker access to more of their data, which in turn generates faster and more accurate results.”
Link to presentation: here.
Zettar’s data mover software has earned a global, peerless reputation in the high-end HPC space for being the simplest, most efficient, and versatile data manager for moving data at scale, at speed of light. For more information, contact [email protected], [email protected], or [email protected].