We introduce Drivelology, a unique linguistic phenomenon characterised as
“nonsense with depth”, utterances that are syntactically coherent yet
pragmatically paradoxical, emotionally loaded, or rhetorically subversive.
While such expressions may resemble surface-level nonsense, they encode
implicit meaning requiring contextual inference, moral reasoning, or emotional
interpretation. We find that current large language models (LLMs), despite
excelling at many natural language processing (NLP) tasks, consistently fail to
grasp the layered semantics of Drivelological text. To investigate this, we
construct a small but diverse benchmark dataset of over 1,200 meticulously
curated examples, with select instances in English, Mandarin, Spanish, French,
Japanese, and Korean. Annotation was especially challenging: each of the
examples required careful expert review to verify that it truly reflected
Drivelological characteristics. The process involved multiple rounds of
discussion and adjudication to address disagreements, highlighting the subtle
and subjective nature of the Drivelology. We evaluate a range of LLMs on
classification, generation, and reasoning tasks. Our results reveal clear
limitations of LLMs: models often confuse Drivelology with shallow nonsense,
produce incoherent justifications, or miss the implied rhetorical function
altogether. These findings highlight a deeper representational gap in LLMs’
pragmatic understanding and challenge the assumption that statistical fluency
implies cognitive comprehension. We release our dataset and code to facilitate
further research in modelling linguistic depth beyond surface-level coherence.