Meet Smaug-72B, the open-source model beating GPT-3.5

Smaug-72B is a 72-billion parameter open-source language model released by AI startup Abacus AI. If you need a heavy-duty, self-hosted base model that can handle complex reasoning without calling proprietary APIs, this is currently the highest-scoring model on public leaderboards.

Technically, it is not a ground-up pretraining project. Smaug-72B is a refined fine-tune of Alibaba's Qwen-72B. On Hugging Face's Open LLM leaderboard, it just scored an average above 80 points, making it the first open-source model in history to cross that milestone.

What it does and benchmark results

Smaug-72B outperforms proprietary mainstays like OpenAI's GPT-3.5 and Mistral's Mistral Medium across several standard industry benchmarks. It also beats its base model, Qwen-72B, by a noticeable margin across multiple evaluation categories.

Its biggest edge lies in multi-step reasoning and mathematics. Abacus AI CEO Bindu Reddy noted that the team applied targeted fine-tuning techniques designed specifically to shore up common model weaknesses in logic and math, resulting in exceptionally high marks on evaluations like GSM8K.

While an 80-point average still sits below the 90-to-100 zone typically associated with human-level benchmarks, it marks a rare moment where an open-weights model officially outscores commercial models in standardized testing.

How it works and availability

Abacus AI credits novel fine-tuning methods for the performance jump. The team plans to detail the exact recipe in an upcoming research paper and intends to apply the same post-training pipeline to other open models, including miqu (a 70B fine-tune of LLaMA-2).

The model weights are live on Hugging Face right now for anyone to download, inspect, and modify. Abacus AI also announced plans to submit Smaug-72B to the crowdsourced LMSYS human evaluation arena for real-world blind testing.

In terms of deployment, running an unquantized 72B model requires serious enterprise hardware. However, with modern 4-bit or 8-bit quantization techniques, developers can spin it up on dual-GPU workstations or cloud instances for internal testing.

Why it matters

For enterprise teams and developers bound by strict data privacy mandates, Smaug-72B offers a legitimate alternative to proprietary APIs like GPT-3.5. It shows that smart post-training can squeeze flagship-level performance out of existing open weights without spending millions on raw pretraining compute.

That said, benchmark superiority does not always guarantee flawless conversational quality or instruction following in messy, real-world edge cases. We recommend testing your own domain-specific prompts against Smaug-72B before ripping out any existing production pipelines.