A novel reward modeling framework DuaShepherd integrates correctness and potential signals into a unified multi-head architecture to enhance LLMs’ mathematical reasoning capabilities and achieve state-of-the-art performance.
In this paper, we propose DuaShepherd, a novel reward modeling framework that
integrates two complementary reward signals, correctness and potential, to
enhance the mathematical reasoning capabilities of Large Language Models
(LLMs). While correctness-based signals emphasize identification of stepwise
errors, potential-based signals focus on the likelihood of reaching the correct
final answer. We developed an automated pipeline for constructing large-scale
reward modeling dataset with both signals. A unified, multi-head architecture
was explored to train the two reward models in a multi-task setup,
demonstrating benefits from learning both correctness and potential in
parallel. By combining these two signals into a compound probability, our model
achieves consistent performance improvements across multiple benchmarks.
Empirical evaluations on MATH500 and ProcessBench confirm that this combined
reward significantly outperforms models trained on either reward type alone,
achieving state-of-the-art performance under comparable resource constraints.