End-to-End Vision Tokenizer Tuning
Existing vision tokenization isolates the optimization of vision tokenizers
from downstream training, implicitly assuming the visual tokens can generalize
well across various tasks, e.g., image generation and visual question
answering. The vision tokenizer optimized for low-level reconstruction is
agnostic to downstream tasks requiring varied representations and semantics.
This decoupled paradigm introduces a critical misalignment: The loss of the
vision tokenization can be the representation bottleneck for target tasks. For
example, errors in tokenizing text in a given image lead to poor results when
recognizing or generating them. To address this, we propose ETT, an end-to-end
vision tokenizer tuning approach that enables joint optimization between vision
tokenization and target autoregressive tasks. Unlike prior autoregressive
models that use only discrete indices from a frozen vision tokenizer, ETT
leverages the visual embeddings of the tokenizer codebook, and optimizes the
vision tokenizers end-to-end with both reconstruction and caption objectives.
ETT can be seamlessly integrated into existing training pipelines with minimal
architecture modifications. Our ETT is simple to implement and integrate,
without the need to adjust the original codebooks or architectures of the
employed large language models. Extensive experiments demonstrate that our
proposed end-to-end vision tokenizer tuning unlocks significant performance
gains, i.e., 2-6% for multimodal understanding and visual generation tasks
compared to frozen tokenizer baselines, while preserving the original
reconstruction capability. We hope this very simple and strong method can
empower multimodal foundation models besides image generation and
understanding.
6826a5d8154611642ada50da
Wenxuan Wang
claimed_verified
2025-05-16T13:24:09.061Z
656428b5462e5ebcbf537d4e
/avatars/cbedecc9c6f2afee2ca6b72efb593561.svg
Wenxuan Wang
user
Rookielion
6826a5d8154611642ada50db
Fan Zhang
claimed_verified
2025-05-16T07:12:04.101Z
640ed40dc025ddf618950af7
/avatars/a99abd4346fa649ad4144f284ebcc972.svg
Fan Zhang
user
ryanzhangfan
6826a5d8154611642ada50dc
Yufeng Cui
admin_assigned
2025-05-16T08:16:58.537Z
648683de623b5f050213f2be
/avatars/83ecbcf4a21f68d2893de79f0444d6e3.svg
Yufeng Cui
user
YufengCui
6826a5d8154611642ada50dd
Haiwen Diao
admin_assigned
2025-05-16T08:17:04.879Z
64b4a717aa03b6520839e9b8
https://cdn-avatars.huggingface.co/v1/production/uploads/64b4a717aa03b6520839e9b8/Rt3ERG-6BVEA4hAwOz0_I.jpeg
Haiwen Diao
user
Paranioar
6826a5d8154611642ada50de
Zhuoyan Luo
admin_assigned
2025-05-16T08:17:11.336Z
657187de7644d1128571495e
/avatars/89412c94fd6136f6680055551de3ddc4.svg
Zhuoyan Luo
user
RobertLuo1
6826a5d8154611642ada50df
Huchuan Lu
6826a5d8154611642ada50e0
Jing Liu
6826a5d8154611642ada50e1
Xinlong Wang
admin_assigned
2025-05-16T08:17:32.558Z
63ca558304c979828311c5a5
/avatars/2a439d79fba2f987cabe780d10c94d25.svg
Xinlong Wang
user
xinlongwang
End-to-End Vision Tokenizer Tuning
Existing vision tokenization isolates the optimization of vision tokenizers
from downstream training, implicitly assuming the visual tokens can generalize
well across various tasks, e.g., image generation and visual question
answering. The vision tokenizer optimized for low-level reconstruction is
agnostic to downstream tasks requiring varied representations and semantics.
This decoupled paradigm introduces a critical misalignment: The loss of the
vision tokenization can be the representation bottleneck for target tasks. For
example, errors in tokenizing text in a given image lead to poor results when
recognizing or generating them. To address this, we propose ETT, an end-to-end
vision tokenizer tuning approach that enables joint optimization between vision
tokenization and target autoregressive tasks. Unlike prior autoregressive
models that use only discrete indices from a frozen vision tokenizer, ETT
leverages the visual embeddings of the tokenizer codebook, and optimizes the
vision tokenizers end-to-end with both reconstruction and caption objectives.
ETT can be seamlessly integrated into existing training pipelines with minimal
architecture modifications. Our ETT is simple to implement and integrate,
without the need to adjust the original codebooks or architectures of the
employed large language models. Extensive experiments demonstrate that our
proposed end-to-end vision tokenizer tuning unlocks significant performance
gains, i.e., 2-6% for multimodal understanding and visual generation tasks
compared to frozen tokenizer baselines, while preserving the original
reconstruction capability. We hope this very simple and strong method can
empower multimodal foundation models besides image generation and
understanding.
ETT is an end-to-end vision tokenizer tuning method that integrates visual tokenizer training with autoregressive tasks, significantly improving performance in multimodal understanding and visual generation.
vision tokenization end-to-end vision tokenizer tuning autoregressive models visual embeddings codebook multimodal understanding visual generation