In recent years, multi-agent frameworks powered by large language models
(LLMs) have advanced rapidly. Despite this progress, there is still a notable
absence of benchmark datasets specifically tailored to evaluate their
performance. To bridge this gap, we introduce Auto-SLURP, a benchmark dataset
aimed at evaluating LLM-based multi-agent frameworks in the context of
intelligent personal assistants. Auto-SLURP extends the original SLURP dataset
— initially developed for natural language understanding tasks — by
relabeling the data and integrating simulated servers and external services.
This enhancement enables a comprehensive end-to-end evaluation pipeline,
covering language understanding, task execution, and response generation. Our
experiments demonstrate that Auto-SLURP presents a significant challenge for
current state-of-the-art frameworks, highlighting that truly reliable and
intelligent multi-agent personal assistants remain a work in progress. The
dataset and related code are available at
https://github.com/lorashen/Auto-SLURP/.
