TON, a two-stage training strategy combining supervised fine-tuning with thought dropout and Group Relative Policy Optimization, reduces unnecessary reasoning steps in vision-language models without sacrificing performance.
Reinforcement Learning (RL) has proven to be an effective post-training
strategy for enhancing reasoning in vision-language models (VLMs). Group
Relative Policy Optimization (GRPO) is a recent prominent method that
encourages models to generate complete reasoning traces before answering,
leading to increased token usage and computational cost. Inspired by the
human-like thinking process-where people skip reasoning for easy questions but
think carefully when needed-we explore how to enable VLMs to first decide when
reasoning is necessary. To realize this, we propose TON, a two-stage training
strategy: (i) a supervised fine-tuning (SFT) stage with a simple yet effective
‘thought dropout’ operation, where reasoning traces are randomly replaced with
empty thoughts. This introduces a think-or-not format that serves as a cold
start for selective reasoning; (ii) a GRPO stage that enables the model to
freely explore when to think or not, while maximizing task-aware outcome
rewards. Experimental results show that TON can reduce the completion length by
up to 90% compared to vanilla GRPO, without sacrificing performance or even
improving it. Further evaluations across diverse vision-language tasks-covering
a range of reasoning difficulties under both 3B and 7B models-consistently
reveal that the model progressively learns to bypass unnecessary reasoning
steps as training advances. These findings shed light on the path toward
human-like reasoning patterns in reinforcement learning approaches. Our code is
available at https://github.com/kokolerk/TON.
