Research investigates reasoning failures in language models for multi-hop question answering, introducing a framework to categorize errors and improve model fidelity.
The emergence of reasoning models and their integration into practical AI
chat bots has led to breakthroughs in solving advanced math, deep search, and
extractive question answering problems that requires a complex and multi-step
thought process. Yet, a complete understanding of why these models hallucinate
more than general purpose language models is missing. In this investigative
study, we systematicallyexplore reasoning failures of contemporary language
models on multi-hop question answering tasks. We introduce a novel, nuanced
error categorization framework that examines failures across three critical
dimensions: the diversity and uniqueness of source documents involved (“hops”),
completeness in capturing relevant information (“coverage”), and cognitive
inefficiency (“overthinking”). Through rigorous hu-man annotation, supported by
complementary automated metrics, our exploration uncovers intricate error
patterns often hidden by accuracy-centric evaluations. This investigative
approach provides deeper insights into the cognitive limitations of current
models and offers actionable guidance toward enhancing reasoning fidelity,
transparency, and robustness in future language modeling efforts.