SambaNova Systems is announcing – and shipping – its second-generation DataScale system, the DataScale SN30. Powered by the eponymous Cardinal SN30 RDU (Reconfigurable Data Unit), SambaNova claims that the DataScale SN30 is capable of a 6x speedup (on certain AI workloads) relative to a comparable system powered by Nvidia A100 GPUs. The new hardware is shipping today. Ahead of the announcement, HPCwire spoke to Marshall Choy, SambaNova’s senior vice president for product, about the launch.

The Cardinal SN30 RDU powering the new DataScale system is manufactured on TSMC’s 7nm process, contains 86 billion transistors and is capable of 688 teraflops at bfloat16 precision. Choy said they weren’t going into “Hot Chips-level detail” on the chip because they don’t actually sell the chip itself – they only use them for SambaNova-sold systems and service offerings. Each SN30 system is equipped with eight SN30 RDUs, along with other improvements. “Beyond the chip, we’ve also thought a lot about memory,” Choy said. “Massive models require massive amounts of memory.” To that end, the SN30 reportedly offers “12.8x more memory capacity than [Nvidia’s] DGX A100” – “a full terabyte versus 80 gigabytes,” Choy said. “We basically doubled all the compute and memory on the chip from the previous generation.”

“Put that all together with our SambaFlow software stack, with enhancements around enterprise integration, with things like native Kubernetes support for the orchestration of containerized and virtualized models and applications,” Choy continued, “and you get our new DataScale SN30 hardware platform.”

Choy said that the new DataScale system is comparable to a DGX system, but completely rack-integrated for ease-of-use (“roll them in, plug it in and it works”). Customers can access the SN30 by purchasing systems or via SambaNova’s “Dataflow-as-a-Service” model, which can be run either through a cloud service provider or on-premises through a private cloud (which Choy said that some customers prefer due to firewalls and privacy). Choy said that while the full list of supported public cloud providers had yet to be finalized, some providers – like Aicadium, Cirrascale and ORock – were already on board.

SambaNova bills its offering as “a fully integrated AI platform innovating in every level of the stack,” and the company is positioning this offering against Nvidia’s suite in its comparisons. According to SambaNova, when training a 13-billion parameter GPT-3 model, the new DataScale SN30 system outperformed an eight-socket DGX A100 system sixfold. Compared to SambaNova’s previous system, the DataScale SN10, Choy said that “we’re saying generally 2-6x” better performance.

Customers

So far, SambaNova is disclosing just a few of the initial customers for the DataScale SN30 system, including Argonne National Laboratory and Lawrence Livermore National Laboratory (LLNL) – both of which have reputations as early adopters of new HPC technologies, particularly with respect to AI-focused systems and novel accelerators.

“We look forward to deploying a larger, multirack system of the next generation of SambaNova’s DataScale systems,” said Bronis de Supinski, CTO for Livermore Computing at LLNL. “Integration of this solution with traditional clusters throughout our center will enable the technology to have a deeper programmatic impact. We anticipate a 2 to 6x performance increase, as the new DataScale system promises to significantly improve overall speed, performance, and productivity.”

More customers, Choy said, would be announced by the end of the year.

“What we’re also seeing is as end users run into these deep learning type models, foundation models, they’re now getting a little bit more experimental in their approach,” commented Chirag Dekate, vice president, analyst at Gartner. “They’re actively exploring the technology landscape, so to speak.” Dekate added that by “dramatically improving the accelerator performance” and “exposing the dataflow ecosystem as a service,” SambaNova was creating a “unique technology offering for end users exploring large, more complex models.”

“For end users that want to experiment and explore diverse innovation options, they’ve just lowered the threshold for entry dramatically,” Dekate said.

A new foundation

SambaNova views its offerings as well-suited to a perceived boom in foundation models – large AI models that are able to handle wide ranges of specialized tasks. “We’re certainly seeing an evolution in artificial intelligence,” Choy said. “We’ve come from the world of predictive analytics, the machine learning, the deep learning – and the next frontier is all about foundation models. We think that foundation models are going to be as necessary for enterprises and organizations from now and into the future as a high-speed internet connection and a mobile application are to business today.”

By way of explanation, Choy cited a customer case – a large bank that used to have thousands of different deployments of the BERT language model, all fine-tuned for specific tasks. With foundation models like GPT-3, Choy said, that has changed.

“The old thinking was you needed a smaller model that was specifically tuned to a specific task with a large dataset to get the highest levels of accuracy,” he said. “We’ve proven that you can achieve the same or better levels of accuracy with a GPT-based foundation model.”

“The big challenge, of course, is there’s not a lot of people out in the world who can actually build and deliver a foundation model deployment. … You look at GPT-3, for example – your choices are basically OpenAI, AWS, Meta and SambaNova.”

To that end, of course, SambaNova’s Dataflow-as-a-Service offering has a GPT model – a model that Choy said would soon see further enhancements, and which would soon be complemented by a vision-oriented foundation model, as well.

“Gartner agrees that a lot of the focus going forward will include a very heavy emphasis on foundational models,” Dekate commented. “What is starting to happen in end-user ecosystems is they are working very aggressively to not just productionalize a lot of the deep learning-based innovations in their ecosystems – some homegrown, others borrowed from elsewhere – but in many cases they’re also turning towards leading-edge technologies like foundation models or multimodal AI where you can combine different types of AI techniques together.”

SambaNova has raised $1.1 billion at a $5 billion valuation and employs more than 500 people. The company will have a presence at SC22 this November in Dallas, and its speakers are featured several times at this year’s AI Hardware Summit, which began yesterday and continues through Thursday.

Header image: LLNL’s existing first-gen SambaNova deployment.