LMArena open-sources its leaderboard engine Arena-Rank
LMArena just open-sourced Arena-Rank, the standalone Python package behind its widely watched AI model leaderboards. Rewritten with a JAX backend, the toolkit calculates rankings over 30x faster than before and introduces reweighting to fairly evaluate models with fewer match records. Beyond LLM evaluations, it can also rank head-to-head match data in sports and gaming.
Arena-Rank is the standalone Python package behind LMArena's widely watched LLM leaderboards. It extracts the ranking machinery previously buried inside the unmaintained FastChat repo, completely overhauls the math, and ships under an Apache 2.0 license.
At its core, it takes pairwise head-to-head competition data and fits a Bradley-Terry model to calculate ratings and confidence intervals. While tuned for AI evaluation, it is fundamentally a general-purpose ranking engine that can score sports leagues or esports tournaments just as easily.
How it hits a 30x speedup
The biggest technical leap is raw performance. Arena-Rank runs more than 30 times faster than the old NumPy and SciPy scripts from FastChat.
That gain comes down to two major shifts. First, the team swapped the compute backend to JAX. That means just-in-time compilation and automatic differentiation out of the box, with native support ready for scaling on GPUs and TPUs down the line.
Second, it ditches slow bootstrap resampling. Instead, it computes confidence intervals using closed-form calculations, removing massive computational overhead without sacrificing statistical rigor.
The update also bakes in recent methodological fixes. There is a sample reweighting feature to handle models with fewer total battles more fairly, and an extended Bradley-Terry variant that controls for contextual features like verbosity, matching the style-controlled leaderboards on the live site.
Architecturally, data preparation is now cleanly separated from model fitting. You process your raw match history once into a PairDataset class, then fit multiple ranking variants or sweep hyperparameters across it without touching the data pipeline again.
How to run it
The package is live on PyPI and installs cleanly through pip or uv. It works out of the box in standard Python environments without exotic hardware.
Pairing it with LMArena's public 140k human preference dataset takes less than ten lines of code. You load the matchups, instantiate the Bradley-Terry model, compute ratings with 95% confidence intervals, and print a formatted leaderboard.
Beyond basic LLM ratings, the repo provides working notebooks for PRISM alignment voter patterns, as well as real-world examples scoring professional basketball seasons and Super Smash Bros. brackets.
Why it matters
If you run custom model evaluations or analyze pairwise test results, you can now replicate LMArena's exact math locally instead of treating the leaderboard as an opaque score.
We think this is a smart move for LMArena. Their real defensibility has always been the stream of genuine human blind tests, not the math behind the rankings. Open-sourcing the engine builds institutional trust, directly answers community questions about small-sample bias, and positions Arena-Rank as the default ranking standard across AI and beyond.
The catch is that this library only solves the downstream math. If you do not already have high-volume, clean pairwise preference data, the package alone will not give you a credible evaluation pipeline.
Sources (1)
- news.lmarena.ai(opens in a new tab)Originalnews.lmarena.ai