Reinforcement Learning enhances generalizable spatial reasoning and interaction in 3D environments through cross-view goal specification and automated task synthesis, achieving zero-shot generalization and improved interaction success rates.
While Reinforcement Learning (RL) has achieved remarkable success in language
modeling, its triumph hasn’t yet fully translated to visuomotor agents. A
primary challenge in RL models is their tendency to overfit specific tasks or
environments, thereby hindering the acquisition of generalizable behaviors
across diverse settings. This paper provides a preliminary answer to this
challenge by demonstrating that RL-finetuned visuomotor agents in Minecraft can
achieve zero-shot generalization to unseen worlds. Specifically, we explore
RL’s potential to enhance generalizable spatial reasoning and interaction
capabilities in 3D worlds. To address challenges in multi-task RL
representation, we analyze and establish cross-view goal specification as a
unified multi-task goal space for visuomotor policies. Furthermore, to overcome
the significant bottleneck of manual task design, we propose automated task
synthesis within the highly customizable Minecraft environment for large-scale
multi-task RL training, and we construct an efficient distributed RL framework
to support this. Experimental results show RL significantly boosts interaction
success rates by 4times and enables zero-shot generalization of spatial
reasoning across diverse environments, including real-world settings. Our
findings underscore the immense potential of RL training in 3D simulated
environments, especially those amenable to large-scale task generation, for
significantly advancing visuomotor agents’ spatial reasoning.