Gaze-LLE: Gaze Target Estimation via Large-Scale Learned Encoders
We address the problem of gaze target estimation, which aims to predict where
a person is looking in a scene. Predicting a person's gaze target requires
reasoning both about the person's appearance and the contents of the scene.
Prior works have developed increasingly complex, hand-crafted pipelines for
gaze target estimation that carefully fuse features from separate scene
encoders, head encoders, and auxiliary models for signals like depth and pose.
Motivated by the success of general-purpose feature extractors on a variety of
visual tasks, we propose Gaze-LLE, a novel transformer framework that
streamlines gaze target estimation by leveraging features from a frozen DINOv2
encoder. We extract a single feature representation for the scene, and apply a
person-specific positional prompt to decode gaze with a lightweight module. We
demonstrate state-of-the-art performance across several gaze benchmarks and
provide extensive analysis to validate our design choices. Our code is
available at: http://github.com/fkryan/gazelle .
675ba24cc7ceaec7e65066d0
Fiona Ryan
claimed_verified
2024-12-13T15:22:25.678Z
656a4b89e8bf55919a313920
/avatars/c903ec6e1c7952c5a200edd9f8cfcee7.svg
Fiona Ryan
user
fkryan
675ba24cc7ceaec7e65066d1
Ajay Bati
admin_assigned
2024-12-16T10:58:25.614Z
638ed41149de7ae552de38cf
/avatars/eb0af76e6557b7ec26893902b223254d.svg
Ajay Bati
user
ajbot
675ba24cc7ceaec7e65066d2
Sangmin Lee
675ba24cc7ceaec7e65066d3
Daniel Bolya
admin_assigned
2024-12-16T10:58:54.549Z
62ff05ab9bfda579504d81b2
/avatars/94590ed960b6df3fc1ababa31cd13f17.svg
Daniel Bolya
user
dbolya
675ba24cc7ceaec7e65066d4
Judy Hoffman
admin_assigned
2024-12-16T10:59:00.557Z
6477790c4ae93470ffc297bb
/avatars/f8d5e08719297a335731a117416d6336.svg
Judy Hoffman
user
judyhoffman
675ba24cc7ceaec7e65066d5
James M. Rehg
Gaze-LLE: Gaze Target Estimation via Large-Scale Learned Encoders
We address the problem of gaze target estimation, which aims to predict where
a person is looking in a scene. Predicting a person's gaze target requires
reasoning both about the person's appearance and the contents of the scene.
Prior works have developed increasingly complex, hand-crafted pipelines for
gaze target estimation that carefully fuse features from separate scene
encoders, head encoders, and auxiliary models for signals like depth and pose.
Motivated by the success of general-purpose feature extractors on a variety of
visual tasks, we propose Gaze-LLE, a novel transformer framework that
streamlines gaze target estimation by leveraging features from a frozen DINOv2
encoder. We extract a single feature representation for the scene, and apply a
person-specific positional prompt to decode gaze with a lightweight module. We
demonstrate state-of-the-art performance across several gaze benchmarks and
provide extensive analysis to validate our design choices. Our code is
available at: http://github.com/fkryan/gazelle .
Gaze-LLE, a transformer framework using features from a DINOv2 encoder, achieves state-of-the-art performance in gaze target estimation by integrating scene and person-specific information.
gaze target estimation transformer framework DINOv2 encoder feature representation positional prompt scene encoders head encoders auxiliary models gaze benchmarks