We introduce Perception Encoder (PE), a state-of-the-art encoder for image
and video understanding trained via simple vision-language learning.
Traditionally, vision encoders have relied on a variety of pretraining
objectives, each tailored to specific downstream tasks such as classification,
captioning, or localization. Surprisingly, after scaling our carefully tuned
image pretraining recipe and refining with our robust video data engine, we
find that contrastive vision-language training alone can produce strong,
general embeddings for all of these downstream tasks. There is only one caveat:
these embeddings are hidden within the intermediate layers of the network. To
draw them out, we introduce two alignment methods, language alignment for
multimodal language modeling, and spatial alignment for dense prediction.
Together with the core contrastive checkpoint, our PE family of models achieves
state-of-the-art performance on a wide variety of tasks, including zero-shot
image and video classification and retrieval; document, image, and video Q&A;
and spatial tasks such as detection, depth estimation, and tracking. To foster
further research, we are releasing our models, code, and a novel dataset of
synthetically and human-annotated videos.
