This paper introduces MovieCORE, a novel video question answering (VQA)
dataset designed to probe deeper cognitive understanding of movie content.
Unlike existing datasets that focus on surface-level comprehension, MovieCORE
emphasizes questions that engage System-2 thinking while remaining specific to
the video material. We present an innovative agentic brainstorming approach,
utilizing multiple large language models (LLMs) as thought agents to generate
and refine high-quality question-answer pairs. To evaluate dataset quality, we
develop a set of cognitive tests assessing depth, thought-provocation
potential, and syntactic complexity. We also propose a comprehensive evaluation
scheme for assessing VQA model performance on deeper cognitive tasks. To
address the limitations of existing video-language models (VLMs), we introduce
an agentic enhancement module, Agentic Choice Enhancement (ACE), which improves
model reasoning capabilities post-training by up to 25%. Our work contributes
to advancing movie understanding in AI systems and provides valuable insights
into the capabilities and limitations of current VQA models when faced with
more challenging, nuanced questions about cinematic content. Our project page,
dataset and code can be found at
https://joslefaure.github.io/assets/html/moviecore.html.