Perception Encoder: The best visual embeddings are not at the output of
  the network
We introduce Perception Encoder (PE), a state-of-the-art encoder for image
and video understanding trained via simple vision-language learning.
Traditionally, vision encoders have relied on a variety of pretraining
objectives, each tailored to specific downstream tasks such as classification,
captioning, or localization. Surprisingly, after scaling our carefully tuned
image pretraining recipe and refining with our robust video data engine, we
find that contrastive vision-language training alone can produce strong,
general embeddings for all of these downstream tasks. There is only one caveat:
these embeddings are hidden within the intermediate layers of the network. To
draw them out, we introduce two alignment methods, language alignment for
multimodal language modeling, and spatial alignment for dense prediction.
Together with the core contrastive checkpoint, our PE family of models achieves
state-of-the-art performance on a wide variety of tasks, including zero-shot
image and video classification and retrieval; document, image, and video Q&A;
and spatial tasks such as detection, depth estimation, and tracking. To foster
further research, we are releasing our models, code, and a novel dataset of
synthetically and human-annotated videos.
680237af2724430ec1defd6f
Daniel Bolya
680237af2724430ec1defd70
Po-Yao Huang
680237af2724430ec1defd71
Peize Sun
claimed_verified
2025-04-20T15:02:49.858Z
640dc9bf8512ec51d7f0ac1a
https://cdn-avatars.huggingface.co/v1/production/uploads/640dc9bf8512ec51d7f0ac1a/sT4rdEoQbzfW6D3xDVdqt.jpeg
peizesun
user
peizesun
680237af2724430ec1defd72
Jang Hyun Cho
claimed_verified
2025-04-20T15:02:53.798Z
647a4ae6c6077fb66fbd0036
/avatars/ee08791366311da83bd5395012f3b1c2.svg
Jang Hyun (Vincent) Cho
user
janghyuncho7
680237af2724430ec1defd73
Andrea Madotto
680237af2724430ec1defd74
Chen Wei
680237af2724430ec1defd75
Tengyu Ma
680237af2724430ec1defd76
Jiale Zhi
claimed_verified
2025-04-20T15:02:57.074Z
64c9e07ef7f4ccb5ea6a5f8d
/avatars/57e990d4dddd749c6cb50c3f0150a332.svg
Jiale Zhi
user
jz2023
680237af2724430ec1defd77
Jathushan Rajasegaran
680237af2724430ec1defd78
Hanoona Rasheed
680237af2724430ec1defd79
Junke Wang
680237af2724430ec1defd7a
Marco Monteiro
680237af2724430ec1defd7b
Hu Xu
680237af2724430ec1defd7c
Shiyu Dong
680237af2724430ec1defd7d
Nikhila Ravi
680237af2724430ec1defd7e
Daniel Li
680237af2724430ec1defd7f
Piotr Dollár
680237af2724430ec1defd80
Christoph Feichtenhofer
Perception Encoder: The best visual embeddings are not at the output of
  the network
We introduce Perception Encoder (PE), a state-of-the-art encoder for image
and video understanding trained via simple vision-language learning.
Traditionally, vision encoders have relied on a variety of pretraining
objectives, each tailored to specific downstream tasks such as classification,
captioning, or localization. Surprisingly, after scaling our carefully tuned
image pretraining recipe and refining with our robust video data engine, we
find that contrastive vision-language training alone can produce strong,
general embeddings for all of these downstream tasks. There is only one caveat:
these embeddings are hidden within the intermediate layers of the network. To
draw them out, we introduce two alignment methods, language alignment for
multimodal language modeling, and spatial alignment for dense prediction.
Together with the core contrastive checkpoint, our PE family of models achieves
state-of-the-art performance on a wide variety of tasks, including zero-shot
image and video classification and retrieval; document, image, and video Q&A;
and spatial tasks such as detection, depth estimation, and tracking. To foster
further research, we are releasing our models, code, and a novel dataset of
synthetically and human-annotated videos.
Perception Encoder, trained via contrastive vision-language learning, achieves state-of-the-art performance across various image and video tasks using intermediate embeddings extracted through alignment methods.
Perception Encoder contrastive vision-language training language alignment spatial alignment zero-shot classification zero-shot retrieval document Q&A image Q&A video Q&A detection depth estimation tracking