HOComp uses MLLMs and attention mechanisms to achieve seamless human-object interactions with consistent appearances in image compositing.
While existing image-guided composition methods may help insert a foreground
object onto a user-specified region of a background image, achieving natural
blending inside the region with the rest of the image unchanged, we observe
that these existing methods often struggle in synthesizing seamless
interaction-aware compositions when the task involves human-object
interactions. In this paper, we first propose HOComp, a novel approach for
compositing a foreground object onto a human-centric background image, while
ensuring harmonious interactions between the foreground object and the
background person and their consistent appearances. Our approach includes two
key designs: (1) MLLMs-driven Region-based Pose Guidance (MRPG), which utilizes
MLLMs to identify the interaction region as well as the interaction type (e.g.,
holding and lefting) to provide coarse-to-fine constraints to the generated
pose for the interaction while incorporating human pose landmarks to track
action variations and enforcing fine-grained pose constraints; and (2)
Detail-Consistent Appearance Preservation (DCAP), which unifies a shape-aware
attention modulation mechanism, a multi-view appearance loss, and a background
consistency loss to ensure consistent shapes/textures of the foreground and
faithful reproduction of the background human. We then propose the first
dataset, named Interaction-aware Human-Object Composition (IHOC), for the task.
Experimental results on our dataset show that HOComp effectively generates
harmonious human-object interactions with consistent appearances, and
outperforms relevant methods qualitatively and quantitatively.