GUI agents powered by LLMs show promise in interacting with diverse digital
environments. Among these, video games offer a valuable testbed due to their
varied interfaces, with adventure games posing additional challenges through
complex, narrative-driven interactions. Existing game benchmarks, however, lack
diversity and rarely evaluate agents on completing entire storylines. To
address this, we introduce FlashAdventure, a benchmark of 34 Flash-based
adventure games designed to test full story arc completion and tackle the
observation-behavior gap: the challenge of remembering and acting on earlier
gameplay information. We also propose CUA-as-a-Judge, an automated gameplay
evaluator, and COAST, an agentic framework leveraging long-term clue memory to
better plan and solve sequential tasks. Experiments show current GUI agents
struggle with full story arcs, while COAST improves milestone completion by
bridging the observation-behavior gap. Nonetheless, a marked discrepancy
between humans and best-performing agents warrants continued research efforts
to narrow this divide.