EOC-Bench introduces a benchmark to evaluate dynamic object-centric cognition in egocentric vision applications, focusing on temporal and interactive aspects not covered by existing benchmarks.
The emergence of multimodal large language models (MLLMs) has driven
breakthroughs in egocentric vision applications. These applications necessitate
persistent, context-aware understanding of objects, as users interact with
tools in dynamic and cluttered environments. However, existing embodied
benchmarks primarily focus on static scene exploration, emphasizing object’s
appearance and spatial attributes while neglecting the assessment of dynamic
changes arising from users’ interactions. To address this gap, we introduce
EOC-Bench, an innovative benchmark designed to systematically evaluate
object-centric embodied cognition in dynamic egocentric scenarios. Specially,
EOC-Bench features 3,277 meticulously annotated QA pairs categorized into three
temporal categories: Past, Present, and Future, covering 11 fine-grained
evaluation dimensions and 3 visual object referencing types. To ensure thorough
assessment, we develop a mixed-format human-in-the-loop annotation framework
with four types of questions and design a novel multi-scale temporal accuracy
metric for open-ended temporal evaluation. Based on EOC-Bench, we conduct
comprehensive evaluations of various proprietary, open-source, and object-level
MLLMs. EOC-Bench serves as a crucial tool for advancing the embodied object
cognitive capabilities of MLLMs, establishing a robust foundation for
developing reliable core models for embodied systems.