The study evaluates various LLMs on diverse text tasks using a new dataset, revealing distinct personality traits and improving model interpretability.
Large Language Models (LLMs) are increasingly integrated into everyday
applications. As their influence grows, understanding their decision making and
underlying personality becomes essential. In this work, we interpret model
personality using our proposed Supernova Event Dataset, a novel dataset with
diverse articles spanning biographies, historical events, news, and scientific
discoveries. We use this dataset to benchmark LLMs on extracting and ranking
key events from text, a subjective and complex challenge that requires
reasoning over long-range context and modeling causal chains. We evaluate small
models like Phi-4, Orca 2, and Qwen 2.5, and large, stronger models such as
Claude 3.7, Gemini 2.5, and OpenAI o3, and propose a framework where another
LLM acts as a judge to infer each model’s personality based on its selection
and classification of events. Our analysis shows distinct personality traits:
for instance, Orca 2 demonstrates emotional reasoning focusing on interpersonal
dynamics, while Qwen 2.5 displays a more strategic, analytical style. When
analyzing scientific discovery events, Claude Sonnet 3.7 emphasizes conceptual
framing, Gemini 2.5 Pro prioritizes empirical validation, and o3 favors
step-by-step causal reasoning. This analysis improves model interpretability,
making them user-friendly for a wide range of diverse applications.