Stop benchmarking in the lab: Inclusion Arena shows how LLMs perform in production

Benchmark testing models have become essential for enterprises, allowing them to choose the type of performance that resonates with their needs. But not all benchmarks are built the same and many test models are based on static datasets or testing environments. 

Researchers from Inclusion AI, which is affiliated with Alibaba’s Ant Group, proposed a new model leaderboard and benchmark that focuses more on a model’s performance in real-life scenarios. They argue that LLMs need a leaderboard that takes into account how people use them and how much people prefer their answers compared to the static knowledge capabilities models have. 

In a paper, the researchers laid out the foundation for Inclusion Arena, which ranks models based on user preferences.  

“To address these gaps, we propose Inclusion Arena, a live leaderboard that bridges real-world AI-powered applications with state-of-the-art LLMs and MLLMs. Unlike crowdsourced platforms, our system randomly triggers model battles during multi-turn human-AI dialogues in real-world apps,” the paper said. 

Inclusion Arena stands out among other model leaderboards, such as MMLU and OpenLLM, due to its real-life aspect and its unique method of ranking models. It employs the Bradley-Terry modeling method, similar to the one used by Chatbot Arena. 

Inclusion Arena works by integrating the benchmark into AI applications to gather datasets and conduct human evaluations. The researchers admit that “the number of initially integrated AI-powered applications is limited, but we aim to build an open alliance to expand the ecosystem.”

By now, most people are familiar with the leaderboards and benchmarks touting the performance of each new LLM released by companies like OpenAI, Google or Anthropic. VentureBeat is no stranger to these leaderboards since some models, like xAI’s Grok 3, show their might by topping the Chatbot Arena leaderboard. The Inclusion AI researchers argue that their new leaderboard “ensures evaluations reflect practical usage scenarios,” so enterprises have better information around models they plan to choose. 

Using the Bradley-Terry method 

Inclusion Arena draws inspiration from Chatbot Arena, utilizing the Bradley-Terry method, while Chatbot Arena also employs the Elo ranking method concurrently. 

Most leaderboards rely on the Elo method to set rankings and performance. Elo refers to the Elo rating in chess, which determines the relative skill of players. Both Elo and Bradley-Terry are probabilistic frameworks, but the researchers said Bradley-Terry produces more stable ratings. 

“The Bradley-Terry model provides a robust framework for inferring latent abilities from pairwise comparison outcomes,” the paper said. “However, in practical scenarios, particularly with a large and growing number of models, the prospect of exhaustive pairwise comparisons becomes computationally prohibitive and resource-intensive. This highlights a critical need for intelligent battle strategies that maximize information gain within a limited budget.” 

To make ranking more efficient in the face of a large number of LLMs, Inclusion Arena has two other components: the placement match mechanism and proximity sampling. The placement match mechanism estimates an initial ranking for new models registered for the leaderboard. Proximity sampling then limits those comparisons to models within the same trust region. 

How it works

So how does it work? 

Inclusion Arena’s framework integrates into AI-powered applications. Currently, there are two apps available on Inclusion Arena: the character chat app Joyland and the education communication app T-Box. When people use the apps, the prompts are sent to multiple LLMs behind the scenes for responses. The users then choose which answer they like best, though they don’t know which model generated the response. 

The framework considers user preferences to generate pairs of models for comparison. The Bradley-Terry algorithm is then used to calculate a score for each model, which then leads to the final leaderboard. 

Inclusion AI capped its experiment at data up to July 2025, comprising 501,003 pairwise comparisons. 

According to the initial experiments with Inclusion Arena, the most performant model is Anthropic’s Claude 3.7 Sonnet, DeepSeek v3-0324, Claude 3.5 Sonnet, DeepSeek v3 and Qwen Max-0125. 

Of course, this was data from two apps with more than 46,611 active users, according to the paper. The researchers said they can create a more robust and precise leaderboard with more data. 

More leaderboards, more choices

The increasing number of models being released makes it more challenging for enterprises to select which LLMs to begin evaluating. Leaderboards and benchmarks guide technical decision makers to models that could provide the best performance for their needs. Of course, organizations should then conduct internal evaluations to ensure the LLMs are effective for their applications. 

It also provides an idea of the broader LLM landscape, highlighting which models are becoming competitive compared to their peers. Recent benchmarks such as RewardBench 2 from the Allen Institute for AI attempt to align models with real-life use cases for enterprises. 

Daily insights on business use cases with VB Daily

If you want to impress your boss, VB Daily has you covered. We give you the inside scoop on what companies are doing with generative AI, from regulatory shifts to practical deployments, so you can share insights for maximum ROI.

Read our Privacy Policy

Thanks for subscribing. Check out more VB newsletters here.

An error occured.