Inclusion AI pits LLMs against each other in real-world apps

Every time a frontier AI lab drops a new model, we get the same shiny benchmark charts showing near-perfect scores on static test sets. But anyone who has plugged an LLM into an actual product knows lab performance rarely translates cleanly to production.

Inclusion AI, a research team affiliated with Ant Group, is trying to fix this disconnect with a new live leaderboard called Inclusion Arena.

Secretly testing models in live apps

Instead of asking benchmarkers to solve standardized exam questions, Inclusion Arena embeds model evaluations directly into production consumer software.

The system is currently hooked into two live applications: a character roleplay chat platform called Joyland, and an educational communication tool called T-Box. When regular people chat inside these apps, their prompts are sent to multiple anonymous LLMs behind the scenes.

Users simply pick the response they like best during their normal multi-turn conversations, without knowing which model generated what. It effectively turns everyday user activity into continuous, blind A/B testing.

By capping its initial dataset in July 2025, the team analyzed 501,003 pairwise comparisons across more than 46,611 active users.

The early results show Anthropic and DeepSeek leading the pack. Anthropic's Claude 3.7 Sonnet took the top spot, followed closely by DeepSeek v3-0324, Claude 3.5 Sonnet, DeepSeek v3, and Alibaba's Qwen Max-0125.

Ditching traditional Elo for efficiency

Most competitive leaderboards, including Chatbot Arena, lean heavily on the chess-style Elo rating system. Inclusion Arena instead relies on the Bradley-Terry probabilistic framework, which researchers argue delivers far more stable rankings over time.

Running exhaustive matchups across dozens of models gets prohibitively expensive fast. To solve this, the researchers introduced two clever matchmaking shortcuts.

First is a placement match mechanism that quickly guesses an incoming model's baseline rank. Second is proximity sampling, which pairs models only against rivals within the same general performance tier to maximize information gain per battle.

The researchers openly acknowledge that two apps are not enough to represent the entire internet, noting plans to build an open alliance to onboard more third-party products.

Why it matters

For engineering teams choosing an API, static benchmarks like MMLU have become largely useless due to contamination and benchmark gaming. Seeing how models perform when judged by ordinary users in messy, real-world dialogues offers much more practical signal.

Still, we think you should take these initial rankings with a grain of salt. The current data comes strictly from roleplay and education apps. A model that excels at emotional roleplay or patient tutoring might completely fall apart at structured JSON generation or enterprise workflows.

For the AI industry, this represents a crucial shift away from prompt-testing enthusiasts toward genuine end-user evaluation. Chatbot Arena often reflects what tech-savvy users look for when trying to break a model, whereas Inclusion Arena captures what normal consumers actually prefer.

The strategy is smart because it uses passive, ambient human feedback. The real challenge will be ecosystem expansion: unless Inclusion AI can recruit business tools, search interfaces, and workflow apps into its alliance, the leaderboard will remain a specialized gauge rather than a universal standard.