Meta just dropped LLaMA, a 65B open research model

For the past few months, the AI world has been obsessed with massive parameter counts that only tech giants could afford to run.

Meta just flipped that script. On February 24, Mark Zuckerberg unveiled LLaMA (Large Language Model Meta AI), a suite of foundational models designed to bring frontier-level performance to researchers without requiring massive supercomputers.

Meta released four model sizes: 7B, 13B, 33B, and 65B parameters. And despite their relatively compact footprint, the benchmark numbers are turning heads.

Punching way above its weight class

According to Meta's research team, the 13B parameter LLaMA model already outperforms OpenAI's 175B parameter GPT-3 on most standard language benchmarks. Even better, Meta notes that this 13B model can run on a single Nvidia Tesla V100 GPU.

Meanwhile, the flagship 65B model goes toe-to-toe with Google's massive 540B PaLM and matches DeepMind’s Chinchilla.

How did Meta pull this off? By training smaller architectures for much longer on far more data. While GPT-3 was trained on roughly 300 billion tokens, Meta trained the 7B and 13B versions on 1 trillion tokens, and the 33B and 65B variants on 1.4 trillion tokens.

Meta researcher Guillaume Lample pointed out on Twitter that even after a full trillion tokens, the smallest 7B model was still actively improving.

Open data and gated access

Unlike proprietary models trained on opaque web scrapes, Meta built LLaMA entirely on publicly available datasets. Roughly 67% comes from a cleaned version of Common Crawl, alongside public GitHub code and Wikipedia across 20 languages.

Meta isn't making this a completely wild free-for-all just yet. To guard against toxicity and misuse, the models are distributed under a non-commercial research license.

Access is granted on a case-by-case basis to academic researchers, civil society organizations, government affiliates, and industry labs.

Why it matters

This is a massive shift in how the industry thinks about model scaling.

Until now, standard practice was to blow up parameter counts to reach better reasoning. But as the researchers noted, that completely ignores the inference budget. A smaller model trained longer might take more upfront compute, but running it at scale is exponentially cheaper.

For independent researchers and universities, running a GPT-3-class model on a single graphics card changes everything. It democratizes alignment, fine-tuning, and red-teaming research that previously required deep corporate pockets.

We think Meta made a brilliant strategic play here against closed-source rivals like OpenAI and Google. The obvious lingering question is enforcement: distributing model weights under a research agreement rarely stays contained for long, and keeping tabs on misuse once the weights are out in the wild will be an uphill battle.