Reinforcement Learning with Verifiable Rewards (RLVR) has emerged as a key
ingredient for unlocking complex reasoning capabilities in large language
models. Recent work ProRL has shown promise in scaling RL by increasing the
number of training steps. However, performance plateaus after thousands of
steps, with clear diminishing returns from allocating more computation to
additional training. In this work, we investigate a complementary paradigm for
scaling RL, BroR-Lincreasing the number of rollouts per example to hundreds to
exhaustively Broaden exploration, which yields continuous performance gains
beyond the saturation point observed in ProRL when scaling the number of
training steps. Our approach is motivated by a mass balance equation analysis
allowing us to characterize the rate of change in probability mass for correct
and incorrect tokens during the reinforcement process. We show that under a
one-step RL assumption, sampled rollout tokens always contribute to
correct-mass expansion, while unsampled tokens outside rollouts may lead to
gains or losses depending on their distribution and the net reward balance.
Importantly, as the number of rollouts per example N increases, the effect of
unsampled terms diminishes, ensuring overall correct-mass expansion. To
validate our theoretical analysis, we conduct simulations under more relaxed
conditions and find that a sufficiently large rollout size N-corresponding to
ample exploration-guarantees an increase in the probability mass of all correct
tokens. Empirically, BroRL revives models saturated after 3K ProRL training
steps and demonstrates robust, continuous improvement, achieving
state-of-the-art results for the 1.5B model across diverse benchmarks.