Inferring the physical properties of 3D scenes from visual information is a
critical yet challenging task for creating interactive and realistic virtual
worlds. While humans intuitively grasp material characteristics such as
elasticity or stiffness, existing methods often rely on slow, per-scene
optimization, limiting their generalizability and application. To address this
problem, we introduce PIXIE, a novel method that trains a generalizable neural
network to predict physical properties across multiple scenes from 3D visual
features purely using supervised losses. Once trained, our feed-forward network
can perform fast inference of plausible material fields, which coupled with a
learned static scene representation like Gaussian Splatting enables realistic
physics simulation under external forces. To facilitate this research, we also
collected PIXIEVERSE, one of the largest known datasets of paired 3D assets and
physic material annotations. Extensive evaluations demonstrate that PIXIE is
about 1.46-4.39x better and orders of magnitude faster than test-time
optimization methods. By leveraging pretrained visual features like CLIP, our
method can also zero-shot generalize to real-world scenes despite only ever
been trained on synthetic data. https://pixie-3d.github.io/