The rapid advancement of diffusion models holds the promise of
revolutionizing the application of VR and AR technologies, which typically
require scene-level 4D assets for user experience. Nonetheless, existing
diffusion models predominantly concentrate on modeling static 3D scenes or
object-level dynamics, constraining their capacity to provide truly immersive
experiences. To address this issue, we propose HoloTime, a framework that
integrates video diffusion models to generate panoramic videos from a single
prompt or reference image, along with a 360-degree 4D scene reconstruction
method that seamlessly transforms the generated panoramic video into 4D assets,
enabling a fully immersive 4D experience for users. Specifically, to tame video
diffusion models for generating high-fidelity panoramic videos, we introduce
the 360World dataset, the first comprehensive collection of panoramic videos
suitable for downstream 4D scene reconstruction tasks. With this curated
dataset, we propose Panoramic Animator, a two-stage image-to-video diffusion
model that can convert panoramic images into high-quality panoramic videos.
Following this, we present Panoramic Space-Time Reconstruction, which leverages
a space-time depth estimation method to transform the generated panoramic
videos into 4D point clouds, enabling the optimization of a holistic 4D
Gaussian Splatting representation to reconstruct spatially and temporally
consistent 4D scenes. To validate the efficacy of our method, we conducted a
comparative analysis with existing approaches, revealing its superiority in
both panoramic video generation and 4D scene reconstruction. This demonstrates
our method’s capability to create more engaging and realistic immersive
environments, thereby enhancing user experiences in VR and AR applications.