Large Language Models (LLMs) effectiveness is usually evaluated by means of
benchmarks such as MMLU, ARC-C, or HellaSwag, where questions are presented in
their original wording, thus in a fixed, standardized format. However,
real-world applications involve linguistic variability, requiring models to
maintain their effectiveness across diverse rewordings of the same question or
query. In this study, we systematically assess the robustness of LLMs to
paraphrased benchmark questions and investigate whether benchmark-based
evaluations provide a reliable measure of model capabilities. We systematically
generate various paraphrases of all the questions across six different common
benchmarks, and measure the resulting variations in effectiveness of 34
state-of-the-art LLMs, of different size and effectiveness. Our findings reveal
that while LLM rankings remain relatively stable across paraphrased inputs,
absolute effectiveness scores change, and decline significantly. This suggests
that LLMs struggle with linguistic variability, raising concerns about their
generalization abilities and evaluation methodologies. Furthermore, the
observed performance drop challenges the reliability of benchmark-based
evaluations, indicating that high benchmark scores may not fully capture a
model’s robustness to real-world input variations. We discuss the implications
of these findings for LLM evaluation methodologies, emphasizing the need for
robustness-aware benchmarks that better reflect practical deployment scenarios.