Alibaba’s AI division is facing scrutiny after a new academic study alleged its Qwen2.5 model achieved top scores on a key math benchmark, alledgedly by “cheating.” Researchers from Fudan University and UC Davis published a paper on July 18, 2025, providing evidence that the model was trained on the test questions.
This allowed it to memorize answers rather than demonstrate genuine reasoning. The findings challenge Alibaba’s previous claims of superior AI performance. They also raise broader questions about the reliability of industry-standard benchmarks like MATH-500, which are crucial for evaluating the world’s most powerful AI systems.
The controversy strikes at the heart of how AI progress is measured. Data contamination, where test data inadvertently leaks into training sets, can inflate performance metrics, which can creat a false impression of a model’s true capabilities and misleads the industry.
From Benchmark Champion to Cheating Scandal
The allegations in the research paper stand in stark contrast to Alibaba’s confident announcements earlier this year. In January 2025, the company launched its Qwen2.5 series, positioning it as a direct competitor to models from OpenAI and DeepSeek. The Qwen team claimed, “these models are not just about recognition; they actively engage with tools, making them capable of performing complex tasks across devices.”
Shortly after, Alibaba unveiled Qwen 2.5-Max, claiming it outperformed rivals on multiple leaderboards. At the time, the company stated, “our base models have demonstrated significant advantages across most benchmarks, and we are optimistic that advancements in post-training techniques will elevate the next version of Qwen 2.5-Max to new heights.” Those claims are now under a cloud of suspicion.
The Evidence: Reasoning or Rote Memorization?
The study from Fudan University provides compelling evidence of memorization. Researchers tested the model using a “partial prompt” method. They fed Qwen2.5 only the first 60% of problems from the MATH-500 benchmark to see if it could complete them.
The results were telling. Qwen2.5 was able to complete the problems verbatim with 54.6% accuracy and still solve them correctly. By contrast, Meta’s Llama3.1-8B model, which was not trained on the test data, managed a completion rate of just 3.8%.
This discrepancy strongly suggests that Qwen2.5 was not reasoning from first principles. Instead, it appears to have been retrieving information it had already seen during its training, effectively “regurgitating” answers it had memorized from sources like GitHub where such datasets are common.
A Test on Clean Ground Reveals the Truth
To confirm their hypothesis, the researchers created a completely new, uncontaminated dataset called “RandomCalculation”. This benchmark consists of fully synthetic arithmetic problems, guaranteeing that no existing AI model could have seen them before.
On this clean dataset, Qwen2.5’s performance changed dramatically. Its accuracy declined as the problems became more complex, which is the expected behavior for a model that is genuinely attempting to reason. This stands in sharp contrast to its flawless recall on the contaminated MATH-500 test.
Furthermore, the study debunked recent theories that random or incorrect reward signals could enhance reasoning. On the clean dataset, Qwen2.5’s performance only improved when it was trained with correct rewards. Random or inverted rewards caused its performance to become unstable or collapse entirely.
A Wider Crisis of Trust for AI Benchmarks
This incident highlights a growing crisis of trust within the AI industry. As companies race to top competitive leaderboards, the integrity of the benchmarks themselves is coming into question. The pressure to perform can lead to “teaching to the test,” where models are trained specifically to ace evaluations.
This is a systemic issue that extends beyond any single company. The reliability of benchmarks is fundamental for tracking real progress in AI. When these evaluation tools are compromised, it becomes difficult to distinguish between genuine innovation and clever optimization.
In response to the study, Alibaba’s Qwen team has issued a public statement addressing the data contamination claims and reaffirming their commitment to benchmark integrity.