We introduce S1-Bench, a novel benchmark designed to evaluate Large Reasoning
Models’ (LRMs) performance on simple tasks that favor intuitive system 1
thinking rather than deliberative system 2 reasoning. While LRMs have achieved
significant breakthroughs in complex reasoning tasks through explicit chains of
thought, their reliance on deep analytical thinking may limit their system 1
thinking capabilities. Moreover, a lack of benchmark currently exists to
evaluate LRMs’ performance in tasks that require such capabilities. To fill
this gap, S1-Bench presents a set of simple, diverse, and naturally clear
questions across multiple domains and languages, specifically designed to
assess LRMs’ performance in such tasks. Our comprehensive evaluation of 22 LRMs
reveals significant lower efficiency tendencies, with outputs averaging 15.5
times longer than those of traditional small LLMs. Additionally, LRMs often
identify correct answers early but continue unnecessary deliberation, with some
models even producing numerous errors. These findings highlight the rigid
reasoning patterns of current LRMs and underscore the substantial development
needed to achieve balanced dual-system thinking capabilities that can adapt
appropriately to task complexity.
