EPiC is a framework for efficient 3D camera control in video diffusion models that constructs high-quality anchor videos through first-frame visibility masking, integrates them using a lightweight ControlNet module, and achieves state-of-the-art performance on I2V tasks with minimal resources.
Recent approaches on 3D camera control in video diffusion models (VDMs) often
create anchor videos to guide diffusion models as a structured prior by
rendering from estimated point clouds following annotated camera trajectories.
However, errors inherent in point cloud estimation often lead to inaccurate
anchor videos. Moreover, the requirement for extensive camera trajectory
annotations further increases resource demands. To address these limitations,
we introduce EPiC, an efficient and precise camera control learning framework
that automatically constructs high-quality anchor videos without expensive
camera trajectory annotations. Concretely, we create highly precise anchor
videos for training by masking source videos based on first-frame visibility.
This approach ensures high alignment, eliminates the need for camera trajectory
annotations, and thus can be readily applied to any in-the-wild video to
generate image-to-video (I2V) training pairs. Furthermore, we introduce
Anchor-ControlNet, a lightweight conditioning module that integrates anchor
video guidance in visible regions to pretrained VDMs, with less than 1% of
backbone model parameters. By combining the proposed anchor video data and
ControlNet module, EPiC achieves efficient training with substantially fewer
parameters, training steps, and less data, without requiring modifications to
the diffusion model backbone typically needed to mitigate rendering
misalignments. Although being trained on masking-based anchor videos, our
method generalizes robustly to anchor videos made with point clouds during
inference, enabling precise 3D-informed camera control. EPiC achieves SOTA
performance on RealEstate10K and MiraData for I2V camera control task,
demonstrating precise and robust camera control ability both quantitatively and
qualitatively. Notably, EPiC also exhibits strong zero-shot generalization to
video-to-video scenarios.