3D content inherently encompasses multi-modal characteristics and can be
projected into different modalities (e.g., RGB images, RGBD, and point clouds).
Each modality exhibits distinct advantages in 3D asset modeling: RGB images
contain vivid 3D textures, whereas point clouds define fine-grained 3D
geometries. However, most existing 3D-native generative architectures either
operate predominantly within single-modality paradigms-thus overlooking the
complementary benefits of multi-modality data-or restrict themselves to 3D
structures, thereby limiting the scope of available training datasets. To
holistically harness multi-modalities for 3D modeling, we present TriMM, the
first feed-forward 3D-native generative model that learns from basic
multi-modalities (e.g., RGB, RGBD, and point cloud). Specifically, 1) TriMM
first introduces collaborative multi-modal coding, which integrates
modality-specific features while preserving their unique representational
strengths. 2) Furthermore, auxiliary 2D and 3D supervision are introduced to
raise the robustness and performance of multi-modal coding. 3) Based on the
embedded multi-modal code, TriMM employs a triplane latent diffusion model to
generate 3D assets of superior quality, enhancing both the texture and the
geometric detail. Extensive experiments on multiple well-known datasets
demonstrate that TriMM, by effectively leveraging multi-modality, achieves
competitive performance with models trained on large-scale datasets, despite
utilizing a small amount of training data. Furthermore, we conduct additional
experiments on recent RGB-D datasets, verifying the feasibility of
incorporating other multi-modal datasets into 3D generation.