Object Recognition as Next Token Prediction
We present an approach to pose object recognition as next token prediction.
The idea is to apply a language decoder that auto-regressively predicts the
text tokens from image embeddings to form labels. To ground this prediction
process in auto-regression, we customize a non-causal attention mask for the
decoder, incorporating two key features: modeling tokens from different labels
to be independent, and treating image tokens as a prefix. This masking
mechanism inspires an efficient method - one-shot sampling - to simultaneously
sample tokens of multiple labels in parallel and rank generated labels by their
probabilities during inference. To further enhance the efficiency, we propose a
simple strategy to construct a compact decoder by simply discarding the
intermediate blocks of a pretrained language model. This approach yields a
decoder that matches the full model's performance while being notably more
efficient. The code is available at https://github.com/kaiyuyue/nxtp
656e9a231f8ce3366d9eb166
Kaiyu Yue
admin_assigned
2023-12-05T11:15:12.208Z
640d0dbc8036cc2142273a83
https://cdn-avatars.huggingface.co/v1/production/uploads/640d0dbc8036cc2142273a83/cicTWJVqqvQv_DgDucWgY.jpeg
Kaiyu Yue
user
kaiyuyue
656e9a231f8ce3366d9eb167
Bor-Chun Chen
656e9a231f8ce3366d9eb168
Jonas Geiping
admin_assigned
2023-12-05T11:15:27.829Z
63d86dbf3130cadcaf8bdd11
/avatars/29d79a0c6dcec01111ef192fecd0fa7a.svg
Jonas Geiping
user
JonasGeiping
656e9a231f8ce3366d9eb169
Hengduo Li
656e9a231f8ce3366d9eb16a
Tom Goldstein
admin_assigned
2023-12-05T11:15:48.070Z
6381ca7d65dc156aba0b933d
/avatars/84dfdca8e1cd6fbf50d6fb2a6f1b488d.svg
Tom Goldstein
user
tomgoldstein
656e9a231f8ce3366d9eb16b
Ser-Nam Lim
claimed_verified
2023-12-26T10:38:01.652Z
65884498ce38d143c435d279
/avatars/96a917c40680a79a37dcb8fa75014c21.svg
Ser Nam Lim
user
sernam
Object Recognition as Next Token Prediction
We present an approach to pose object recognition as next token prediction.
The idea is to apply a language decoder that auto-regressively predicts the
text tokens from image embeddings to form labels. To ground this prediction
process in auto-regression, we customize a non-causal attention mask for the
decoder, incorporating two key features: modeling tokens from different labels
to be independent, and treating image tokens as a prefix. This masking
mechanism inspires an efficient method - one-shot sampling - to simultaneously
sample tokens of multiple labels in parallel and rank generated labels by their
probabilities during inference. To further enhance the efficiency, we propose a
simple strategy to construct a compact decoder by simply discarding the
intermediate blocks of a pretrained language model. This approach yields a
decoder that matches the full model's performance while being notably more
efficient. The code is available at https://github.com/kaiyuyue/nxtp
Pose object recognition is approached using next token prediction with customized non-causal attention masks and a compact language decoder, enhancing efficiency.
pose object recognition next token prediction language decoder auto-regressive prediction non-causal attention mask image embeddings one-shot sampling compact decoder pretrained language model