Reasoning has emerged as the next major frontier for language models (LMs),
with rapid advances from both academic and industrial labs. However, this
progress often outpaces methodological rigor, with many evaluations relying on
benchmarking practices that lack transparency, robustness, or statistical
grounding. In this work, we conduct a comprehensive empirical study and find
that current mathematical reasoning benchmarks are highly sensitive to subtle
implementation choices – including decoding parameters, random seeds, prompt
formatting, and even hardware and software-framework configurations.
Performance gains reported in recent studies frequently hinge on unclear
comparisons or unreported sources of variance. To address these issues, we
propose a standardized evaluation framework with clearly defined best practices
and reporting standards. Using this framework, we reassess recent methods and
find that reinforcement learning (RL) approaches yield only modest improvements
– far below prior claims – and are prone to overfitting, especially on
small-scale benchmarks like AIME24. In contrast, supervised finetuning (SFT)
methods show consistently stronger generalization. To foster reproducibility,
we release all code, prompts, and model outputs, for reasoning benchmarks,
establishing more rigorous foundations for future work.
