Implicit Chain-of-Thought (CoT) methods present a promising, token-efficient
alternative to explicit CoT reasoning in Large Language Models (LLMs), but a
persistent performance gap has limited the application of implicit CoT. We
identify a core latent instability issue by scaling the computational budget of
implicit CoT approaches: as we increase the number of implicit reasoning tokens
to enhance performance, the training process often becomes unstable and
collapses. Our analysis reveals that this instability arises from the latent
representations becoming homogeneous and losing their semantic diversity, a
failure caused by insufficient step-level supervision in existing implicit CoT
approaches. To address this issue, we propose SIM-CoT, a plug-and-play training
module that introduces step-level supervision to stabilize and enrich the
latent reasoning space. Specifically, SIM-CoT employs an auxiliary decoder
during training to align each implicit token with its corresponding explicit
reasoning step, ensuring that latent states capture distinct and meaningful
information. The proposed auxiliary decoder is removed during inference,
preserving the computational efficiency of implicit CoT methods with no added
overhead. In addition, the auxiliary decoder affords interpretability of
implicit reasoning by projecting each latent token onto an explicit reasoning
vocabulary, enabling per-step visualization of semantic roles and diagnosis.
SIM-CoT significantly enhances both the in-domain accuracy and out-of-domain
stability of various implicit CoT methods, boosting baselines like Coconut by
+8.2% on GPT-2 and CODI by +3.0% on LLaMA-3.1 8B. Demonstrating strong
scalability, SIM-CoT also surpasses the explicit CoT baseline on GPT-2 by 2.1%
with 2.3\times greater token efficiency, while substantially closing the
performance gap on larger models like LLaMA-3.1 8B.