To enhance the efficiency of GUI agents on various platforms like smartphones
and computers, a hybrid paradigm that combines flexible GUI operations with
efficient shortcuts (e.g., API, deep links) is emerging as a promising
direction. However, a framework for systematically benchmarking these hybrid
agents is still underexplored. To take the first step in bridging this gap, we
introduce MAS-Bench, a benchmark that pioneers the evaluation of GUI-shortcut
hybrid agents with a specific focus on the mobile domain. Beyond merely using
predefined shortcuts, MAS-Bench assesses an agent’s capability to autonomously
generate shortcuts by discovering and creating reusable, low-cost workflows. It
features 139 complex tasks across 11 real-world applications, a knowledge base
of 88 predefined shortcuts (APIs, deep-links, RPA scripts), and 7 evaluation
metrics. The tasks are designed to be solvable via GUI-only operations, but can
be significantly accelerated by intelligently embedding shortcuts. Experiments
show that hybrid agents achieve significantly higher success rates and
efficiency than their GUI-only counterparts. This result also demonstrates the
effectiveness of our method for evaluating an agent’s shortcut generation
capabilities. MAS-Bench fills a critical evaluation gap, providing a
foundational platform for future advancements in creating more efficient and
robust intelligent agents.