Stability AI drops Stable Beluga, takes top spot on LLM leaderboard

Stable Beluga 1 and Stable Beluga 2 are two open-access instruction-tuned language models developed by Stability AI and its CarperAI lab. Built specifically to tackle intricate reasoning, catch linguistic subtleties, and answer complex questions in domains like law and mathematics, both models bring frontier-level reasoning to researchers.

Internally, the project went by the codename FreeWilly, a nod to Microsoft's influential Orca paper. The team eventually renamed the models to Beluga—a gentler whale compared to the fierce killer whale—reflecting their emphasis on politeness and internal red-teaming for harmlessness.

10% of the data, top-tier performance

The most impressive feat behind Stable Beluga is its data efficiency. Stability AI followed the general training blueprint from Microsoft's Orca research, which teaches models by progressive distillation from GPT-4 explanation traces, but switched out the underlying data entirely.

Instead of burning massive compute on millions of examples, the team synthetically generated an instruction dataset of roughly 600,000 data points using high-quality instructions created by Enrico Shippole. That is just about 10% of the dataset size Microsoft used in the original Orca paper.

To ensure results were clean, researchers carefully filtered the dataset to remove any overlap with benchmark evaluations like the ARC-Challenge. Despite using a fraction of the sample size, the streamlined training significantly slashed both compute costs and carbon footprints without hurting accuracy.

Benchmarks and model weights

The small-data approach paid off immediately on standard benchmarks. Stability AI evaluated the models using EleutherAI's lm-eval-harness alongside AGIEval and zero-shot GPT4ALL tests, showing remarkable performance on complex logic and math tasks.

Hugging Face independently verified the numbers. On its Open LLM Leaderboard, Stable Beluga 2 jumped straight to the number one position overall, while Stable Beluga 1 claimed the fourth spot.

In terms of specs, Stable Beluga 2 is fine-tuned on top of Meta's LLaMA 2 70B, and its weights are available as-is on Hugging Face. Running it locally will demand serious hardware—think multiple enterprise-grade GPUs or aggressive quantization.

Stable Beluga 1 is built on the original LLaMA 65B using standard Alpaca-style Supervised Fine-Tuning. Due to the original Meta licensing terms, its weights are provided only as delta files that must be applied over the base LLaMA weights.

Both models are released under the non-commercial Stable Beluga Research License, meaning they are strictly meant for academic study and open research rather than commercial deployment.

Why it matters

Stable Beluga is a massive proof-of-concept for synthetic data. It shows that open-source teams do not need massive web scraping operations or millions of uncurated examples to reach state-of-the-art results. A curated set of 600,000 high-grade synthetic instructions can squeeze world-class reasoning out of a 70B base model.

That said, the non-commercial license and 70B hardware footprint keep it out of reach for instant production apps. If you are an AI researcher or developer experimenting with local reasoning models, grabbing the weights from Hugging Face is well worth your time, but builders looking to ship commercial software will likely treat it as an architectural blueprint rather than a plug-and-play engine.