In under two minutes, Meta’s CEO, Mark Zuckerberg, laid out the company’s AI plans, which included a plan to build an artificial intelligence system with the equivalent of 600,000 Nvidia GPUs.
“I’m bringing Meta’s AI research efforts closer together to support our long-term goals of building general intelligence, open sourcing it responsibly, and making it available and useful to everyone in all of our daily lives,” Zuckerberg said in a video posted on Twitter.
Zuckerberg’s announcement was an updated roadmap of Meta’s AI plans, which is built around the upcoming Llama3, which is currently being trained. It will succeed last year’s Llama2 model weights and tokenizers, which were major successes with just under 2 million combined downloads on Huggingface. Open-source developers have also released thousands of Llama2 forks.
Llama3 will compete with Google’s recently released Gemini model and OpenAI’s GPT-4 and upcoming GPT-5 models. OpenAI CEO Sam Altman has not talked about GPT-5 yet but has hinted that it would be much easier to handle text, speech, and images by supporting more data sources.
“We are building an absolutely massive amount of infrastructure to support this by the end of this year. We will have around 350,000 Nvidia H100 or around 600,000 H100 equivalents of compute if you include other GPUs,” Zuckerberg said.
That means Meta’s total number of GPUs will be more than the population of around 70 countries. Meta will also be using AMD’s MI300X GPUs. In fact, Meta deployed servers with the MI300X GPU in record time.
“[MI300X] leverages the OCP module, standard, and platform, which has helped us adopt it in record time. In fact, MI300X is one of the fastest deployment solutions in Meta’s history,” said Ajit Mathews, senior director of engineering at Meta, in an on-stage appearance at an AMD event last year.
Nvidia will start shipping the H200 GPUs next year to supplant its H100 GPUs. There are signs that Nvidia will scale up H100 GPU shipments, which have been in big demand, as it ramps up H200 shipments.
Companies are waiting many quarters to receive Nvidia GPUs, and Meta’s 350,000 GPUs are a giant order. But Nvidia is slowly fulfilling shipments, prioritizing larger customers, but also meeting smaller orders placed by bitcoin-turned-AI data center companies.
Meta is also reconstructing its data centers to focus on GPU computing. Meta is building mega clusters with thousands of accelerators. The network of cores is organized as a mesh, with a bandwidth of 1 terabyte per second among accelerators. The company has 21 data centers globally, and the most recent one, opened in DeKalb, Illinois, runs on renewable energy.
But even more GPUs will be needed to meet Zuckerberg’s ultimate goal: to build artificial general intelligence, which is more a digital version of the human brain.
“It’s become clear that the next generation of services requires building full general intelligence, building best AI assistants, AIs for creators as businesses, and more that needs advances in every area of AI from reasoning to planning to coding to memory and other cognitive abilities,” Zuckerberg said.
Zuckerberg explained how the metaverse strategy is tied to its AI plans. Sensory input from humans — images, sounds, speech — will all be fed into multimodal AIs, and the output will be on metaverse devices like glasses and headsets. Right now, AI models are separated in the form of chatbots, text-to-image generators, and more, but ultimately, those will all merge to present information in a way that matches how humans process information.
“People are also going to need new devices for AI, and this brings together AI and metaverse because, over time, I think a lot of us are going to talk to AI as frequently throughout the day,” Zuckerberg said.
One example is Rayban’s Meta smart glasses, which are built on Qualcomm’s chips. With the product, Meta “now carries the flag for smart glasses in terms of day-to-day functionality,” said Moor Insights and Strategy analyst Anshel Sag in a product review posted last month.
“These glasses are the ideal form factor for letting an AI see what you see and hear what you hear. So it’s always available to help out,” Zuckerberg said.
Zuckerberg hinted that Llama3 will likely be Meta’s first multimodal model that supports multimedia and speech input. Llama 2 was exclusively a chatbot that provided a prompt where users could ask questions and compile stories.
Meta has a distribution advantage for Llama 3. The open-source nature means that anyone can host the model, and Llama 2 was offered cloud services by Google, AWS, and Microsoft. Meta never released its own hosted service for a Llama2 chatbot, and users could use it through services like HuggingChat or by downloading it to their PC.
“This technology is so important, and the opportunities are so great that we should open source and make it as widely available as we responsibly can so that everyone can benefit,” Zuckerberg said.
Zuckerberg is also emerging as the face of open-source AI, and he indirectly slammed OpenAI and Google for their closed-sourced approach to AI. OpenAI and Google exploited public research and open-source developers to build their AI strategy but went closed source to cash in on AI.
Transformer models from OpenAI and Google are only available through their websites (except Microsoft, which has built its AI strategy on OpenAI’s GPT models).
Meta “has common software platforms like Pytorch that everybody uses. It makes the entire field accelerate because you have a lot of people working on it,” said Yann LeCun, chief AI scientist, during a discussion at the World Economic Forum in Davos last week.
Meta prioritizes safety and responsibility in its AI technologies, which are already featured in Llama2. The company has already dealt with many political controversies and may want to avoid any public or political backlash. Llama2 has supervised learning techniques to weigh results, which is a change from the switch to unsupervised models in newer transformer models.
Zuckerberg did not comment on when the company would release Llama 3. But the company shipped Llama 1 and 2 last year, with the releases ten months apart.