Digitizing the physical world into accurate simulation-ready virtual
environments offers significant opportunities in a variety of fields such as
augmented and virtual reality, gaming, and robotics. However, current 3D
reconstruction and scene-understanding methods commonly fall short in one or
more critical aspects, such as geometry completeness, object interactivity,
physical plausibility, photorealistic rendering, or realistic physical
properties for reliable dynamic simulation. To address these limitations, we
introduce HoloScene, a novel interactive 3D reconstruction framework that
simultaneously achieves these requirements. HoloScene leverages a comprehensive
interactive scene-graph representation, encoding object geometry, appearance,
and physical properties alongside hierarchical and inter-object relationships.
Reconstruction is formulated as an energy-based optimization problem,
integrating observational data, physical constraints, and generative priors
into a unified, coherent objective. Optimization is efficiently performed via a
hybrid approach combining sampling-based exploration with gradient-based
refinement. The resulting digital twins exhibit complete and precise geometry,
physical stability, and realistic rendering from novel viewpoints. Evaluations
conducted on multiple benchmark datasets demonstrate superior performance,
while practical use-cases in interactive gaming and real-time digital-twin
manipulation illustrate HoloScene’s broad applicability and effectiveness.
Project page: https://xiahongchi.github.io/HoloScene.