Meta is training Llama 3 — and hoarding 340,000 H100s

For the past year, tech giants have been obsessing over artificial general intelligence, but Mark Zuckerberg had largely avoided leaning into the term. Now, he's jumping into the ring with both feet — and bringing an absurd amount of silicon with him.

Zuckerberg announced that Meta has officially begun training Llama 3. The goal isn't just another incremental update; Meta is openly aiming to build AGI, ramping up advanced reasoning, planning, and code-generation capabilities.

Flexing a 340,000-chip computing fortress

Training frontier models requires serious compute, and Zuckerberg isn't shy about showing off Meta's arsenal. He confirmed that by the end of 2024, Meta will own more than 340,000 Nvidia H100 GPUs.

When counting its existing Nvidia A100s and other AI chips, Meta's compute stockpile will hit roughly 600,000 H100 equivalents. Research shows Meta received around 150,000 H100s in 2023 alone, matching Microsoft and outpacing every other competitor at least three to one. Zuckerberg believes Meta's capacity is now larger than almost any individual company on Earth.

Behind the chips is an equally ruthless war for talent. With top researchers commanding packages north of $1 million a year, Zuckerberg noted he is routinely pulled into recruitment calls to woo engineers or stop key staff from defecting.

To move faster, Meta also overhauled its org chart. The Fundamental AI Research team (FAIR), led by Yann LeCun and Joelle Pineau, is moving out of Andrew Bosworth's Reality Labs hardware division and into Chris Cox's Family of Apps team. The intent is simple: bridge foundational research directly with the apps used by billions of people.

Learning from Llama 2 to target the frontier

Zuckerberg admitted that Meta learned a major lesson from Llama 2. Originally, the team questioned the need for heavy coding chops, assuming people wouldn't ask WhatsApp for programming advice. But they quickly realized that code generation is structural glue — it teaches models rigorous logic and hierarchical knowledge structure.

With Llama 3, coding, reasoning, and planning are central. While Llama 2 was the best open-source model of its time, Zuckerberg acknowledged it wasn't the overall industry leader. With Llama 3 and beyond, Meta's goal is to hit the absolute state of the art.

Zuckerberg also took a jab at closed-door rivals like OpenAI. He pointed out how companies that once championed open science shifted toward secrecy once real money was on the table, often hiding behind safety rhetoric. Still, Zuckerberg left himself an escape hatch: while Meta leans toward open source, he stopped short of promising that a future AGI would definitely be released freely.

Meanwhile, he insisted this is not a pivot away from the metaverse. Meta still burns over $15 billion annually on Reality Labs. Between AI-powered Ray-Ban smart glasses and an upcoming platform for users to spawn custom AI personas across Instagram and Threads, the lines between social feeds and AI are blurring fast.

Why it matters

Zuckerberg's announcement cements Meta's role as the great disrupter of the AI ecosystem.

For developers, open-weights access to a genuine frontier model would be massive. If Llama 3 can close the gap with GPT-4, it gives builders an enterprise-grade base model without recurring API tolls or platform lock-in. Fine-tuning and private self-hosting will become accessible to a much broader cohort of startups.

For the industry, Meta's open playbook is a calculated pricing attack. Unlike OpenAI or Anthropic, Meta doesn't rely on selling model access for survival; its money comes from social ads. By commoditizing the underlying model, Meta forces closed rivals to defend their profit margins while rallying the global developer community to build tools for Meta's ecosystem.

The catch? Computing power is now more centralized than ever. When one company wields 600,000 GPUs, independent open-source efforts stand little chance without Big Tech patronage. Furthermore, Zuckerberg holds ultimate voting control at Meta. His open-source strategy is a brilliant business lever today, but if true AGI ever materializes, whether the world gets access still comes down to one man's decision.