Reinforcement learning with verifiable rewards (RLVR) has achieved remarkable
success in enhancing the reasoning capabilities of large language models
(LLMs). However, existing RLVR methods often suffer from exploration
inefficiency due to mismatches between the training data’s difficulty and the
model’s capability. LLMs fail to discover viable reasoning paths when problems
are overly difficult, while learning little new capability when problems are
too simple. In this work, we formalize the impact of problem difficulty by
quantifying the relationship between loss descent speed and rollout accuracy.
Building on this analysis, we propose SEELE, a novel supervision-aided RLVR
framework that dynamically adjusts problem difficulty to stay within the
high-efficiency region. SEELE augments each training sample by appending a hint
(part of a full solution) after the original problem. Unlike previous
hint-based approaches, SEELE deliberately and adaptively adjusts the hint
length for each problem to achieve an optimal difficulty. To determine the
optimal hint length, SEELE employs a multi-round rollout sampling strategy. In
each round, it fits an item response theory model to the accuracy-hint pairs
collected in preceding rounds to predict the required hint length for the next
round. This instance-level, real-time difficulty adjustment aligns problem
difficulty with the evolving model capability, thereby improving exploration
efficiency. Experimental results show that SEELE outperforms Group Relative
Policy Optimization (GRPO) and Supervised Fine-tuning (SFT) by +11.8 and +10.5
points, respectively, and surpasses the best previous supervision-aided
approach by +3.6 points on average across six math reasoning benchmarks.