Lip synchronization, known as the task of aligning lip movements in an
existing video with new input audio, is typically framed as a simpler variant
of audio-driven facial animation. However, as well as suffering from the usual
issues in talking head generation (e.g., temporal consistency), lip
synchronization presents significant new challenges such as expression leakage
from the input video and facial occlusions, which can severely impact
real-world applications like automated dubbing, but are often neglected in
existing works. To address these shortcomings, we present KeySync, a two-stage
framework that succeeds in solving the issue of temporal consistency, while
also incorporating solutions for leakage and occlusions using a carefully
designed masking strategy. We show that KeySync achieves state-of-the-art
results in lip reconstruction and cross-synchronization, improving visual
quality and reducing expression leakage according to LipLeak, our novel leakage
metric. Furthermore, we demonstrate the effectiveness of our new masking
approach in handling occlusions and validate our architectural choices through
several ablation studies. Code and model weights can be found at
https://antonibigata.github.io/KeySync.
