The use of continuous instead of discrete tokens during the Chain-of-Thought
(CoT) phase of reasoning LLMs has garnered attention recently, based on the
intuition that a continuous mixture of discrete tokens could simulate a
superposition of several reasoning paths simultaneously. Theoretical results
have formally proven that continuous tokens have much greater expressivity and
can solve specific problems more efficiently. However, practical use of
continuous tokens has been limited by strong training difficulties: previous
works either just use continuous tokens at inference time on a pre-trained
discrete-token model, or must distill the continuous CoT from ground-truth
discrete CoTs and face computational costs that limit the CoT to very few
tokens.
This is the first work introducing a scalable method to learn continuous CoTs
via reinforcement learning (RL), without distilling from reference discrete
CoTs. We use “soft” tokens: mixtures of tokens together with noise on the input
embedding to provide RL exploration. Computational overhead is minimal,
enabling us to learn continuous CoTs with hundreds of tokens. On math reasoning
benchmarks with Llama and Qwen models up to 8B, training with continuous CoTs
match discrete-token CoTs for pass@1 and surpass them for pass@32, showing
greater CoT diversity. In systematic comparisons, the best-performing scenario
is to train with continuous CoT tokens then use discrete tokens for inference,
meaning the “soft” models can be deployed in a standard way. Finally, we show
continuous CoT RL training better preserves the predictions of the base model
on out-of-domain tasks, thus providing a softer touch to the base model.