Robo2VLM, a framework for generating Visual Question Answering datasets using robot trajectory data, enhances and evaluates Vision-Language Models by leveraging sensory modalities and 3D property understanding.
Vision-Language Models (VLMs) acquire real-world knowledge and general
reasoning ability through Internet-scale image-text corpora. They can augment
robotic systems with scene understanding and task planning, and assist
visuomotor policies that are trained on robot trajectory data. We explore the
reverse paradigm – using rich, real, multi-modal robot trajectory data to
enhance and evaluate VLMs. In this paper, we present Robo2VLM, a Visual
Question Answering (VQA) dataset generation framework for VLMs. Given a human
tele-operated robot trajectory, Robo2VLM derives ground-truth from non-visual
and non-descriptive sensory modalities, such as end-effector pose, gripper
aperture, and force sensing. Based on these modalities, it segments the robot
trajectory into a sequence of manipulation phases. At each phase, Robo2VLM uses
scene and interaction understanding to identify 3D properties of the robot,
task goal, and the target object. The properties are used to generate
representative VQA queries – images with textural multiple-choice questions –
based on spatial, goal-conditioned, and interaction reasoning question
templates. We curate Robo2VLM-1, a large-scale in-the-wild dataset with 684,710
questions covering 463 distinct scenes and 3,396 robotic manipulation tasks
from 176k real robot trajectories. Results suggest that Robo2VLM-1 can
benchmark and improve VLM capabilities in spatial and interaction reasoning.