Web agents enable users to perform tasks on web browsers through natural
language interaction. Evaluating web agents trajectories is an important
problem, since it helps us determine whether the agent successfully completed
the tasks. Rule-based methods are widely used for this purpose, but they are
challenging to extend to new tasks and may not always recognize successful
trajectories. We may achieve higher accuracy through human evaluation, but the
process would be substantially slower and more expensive. Automatic evaluations
with LLMs may avoid the challenges of designing new rules and manually
annotating trajectories, enabling faster and cost-effective evaluation.
However, it is unclear how effective they are at evaluating web agents. To this
end, we propose AgentRewardBench, the first benchmark to assess the
effectiveness of LLM judges for evaluating web agents. AgentRewardBench
contains 1302 trajectories across 5 benchmarks and 4 LLMs. Each trajectory in
AgentRewardBench is reviewed by an expert, who answers questions pertaining to
the success, side effects, and repetitiveness of the agent. Using our
benchmark, we evaluate 12 LLM judges and find that no single LLM excels across
all benchmarks. We also find that the rule-based evaluation used by common
benchmarks tends to underreport the success rate of web agents, highlighting a
key weakness of rule-based evaluation and the need to develop more flexible
automatic evaluations. We release the benchmark at:
https://agent-reward-bench.github.io
