Subobject-level Image Tokenization
Transformer-based vision models typically tokenize images into fixed-size
square patches as input units, which lacks the adaptability to image content
and overlooks the inherent pixel grouping structure. Inspired by the subword
tokenization widely adopted in language models, we propose an image tokenizer
at a subobject level, where the subobjects are represented by semantically
meaningful image segments obtained by segmentation models (e.g., segment
anything models). To implement a learning system based on subobject
tokenization, we first introduced a Sequence-to-sequence AutoEncoder (SeqAE) to
compress subobject segments of varying sizes and shapes into compact embedding
vectors, then fed the subobject embeddings into a large language model for
vision language learning. Empirical results demonstrated that our
subobject-level tokenization significantly facilitates efficient learning of
translating images into object and attribute descriptions compared to the
traditional patch-level tokenization. Codes and models will be open-sourced at
https://github.com/ChenDelong1999/subobjects.
65d81106c790f52e937f94eb
Delong Chen
admin_assigned
2024-02-23T12:56:34.973Z
630491107424d937fa3258be
/avatars/b8bd81bc8544674ee26b78702afdb87c.svg
Delong Chen
user
chendelong
65d81106c790f52e937f94ec
Samuel Cahyawijaya
admin_assigned
2024-02-23T12:56:40.946Z
5f5c4d49e56d546cd623309b
https://cdn-avatars.huggingface.co/v1/production/uploads/1599884443706-noauth.jpeg
Samuel Cahyawijaya
user
samuelcahyawijaya
65d81106c790f52e937f94ed
Jianfeng Liu
admin_assigned
2024-02-23T12:56:47.609Z
62f1db70a39cf6f5c63fc5e5
/avatars/ff4a0f4f4ddab4cc87cf3b3abea0f232.svg
Jianfeng Liu
user
docacola
65d81106c790f52e937f94ee
Baoyuan Wang
admin_assigned
2024-02-23T12:56:55.119Z
64b80f042dde7191bf229d85
/avatars/34ea357b9c42ac8c40133936992e0005.svg
Baoyuan Wang
user
baoyuanw
65d81106c790f52e937f94ef
Pascale Fung
Subobject-level Image Tokenization
Transformer-based vision models typically tokenize images into fixed-size
square patches as input units, which lacks the adaptability to image content
and overlooks the inherent pixel grouping structure. Inspired by the subword
tokenization widely adopted in language models, we propose an image tokenizer
at a subobject level, where the subobjects are represented by semantically
meaningful image segments obtained by segmentation models (e.g., segment
anything models). To implement a learning system based on subobject
tokenization, we first introduced a Sequence-to-sequence AutoEncoder (SeqAE) to
compress subobject segments of varying sizes and shapes into compact embedding
vectors, then fed the subobject embeddings into a large language model for
vision language learning. Empirical results demonstrated that our
subobject-level tokenization significantly facilitates efficient learning of
translating images into object and attribute descriptions compared to the
traditional patch-level tokenization. Codes and models will be open-sourced at
https://github.com/ChenDelong1999/subobjects.
Subobject-level tokenization in vision models improves image description learning by using semantically meaningful segments instead of fixed-size patches.
Transformer-based vision models image tokenizer subobject tokenization segment anything models Sequence-to-sequence AutoEncoder (SeqAE) compact embedding vectors vision language learning patch-level tokenization