Democratizing Text-to-Image Masked Generative Models with Compact
  Text-Aware One-Dimensional Tokens
Image tokenizers form the foundation of modern text-to-image generative
models but are notoriously difficult to train. Furthermore, most existing
text-to-image models rely on large-scale, high-quality private datasets, making
them challenging to replicate. In this work, we introduce Text-Aware
Transformer-based 1-Dimensional Tokenizer (TA-TiTok), an efficient and powerful
image tokenizer that can utilize either discrete or continuous 1-dimensional
tokens. TA-TiTok uniquely integrates textual information during the tokenizer
decoding stage (i.e., de-tokenization), accelerating convergence and enhancing
performance. TA-TiTok also benefits from a simplified, yet effective, one-stage
training process, eliminating the need for the complex two-stage distillation
used in previous 1-dimensional tokenizers. This design allows for seamless
scalability to large datasets. Building on this, we introduce a family of
text-to-image Masked Generative Models (MaskGen), trained exclusively on open
data while achieving comparable performance to models trained on private data.
We aim to release both the efficient, strong TA-TiTok tokenizers and the
open-data, open-weight MaskGen models to promote broader access and democratize
the field of text-to-image masked generative models.
678734168c1e7b6c4a6e5ff9
Dongwon Kim
claimed_verified
2025-01-16T08:31:43.848Z
64dc5208c38427829de81b16
/avatars/43a08e46a7a78f1e3d1f6645a9b1d26b.svg
Dongwon
user
kdwon
678734168c1e7b6c4a6e5ffa
Ju He
claimed_verified
2025-01-15T08:48:25.249Z
661c9059bcd78151e5c06ea1
https://cdn-avatars.huggingface.co/v1/production/uploads/661c9059bcd78151e5c06ea1/27bfNo1LZeZQ77vWuAa10.png
Ju He
user
turkeyju
678734168c1e7b6c4a6e5ffb
Qihang Yu
admin_assigned
2025-01-15T16:54:50.093Z
677b60e17279b5c57354108b
https://cdn-avatars.huggingface.co/v1/production/uploads/677b60e17279b5c57354108b/YOwDhVf9DkeRjOCOLErb6.png
QihangYu
user
QihangYu
678734168c1e7b6c4a6e5ffc
Chenglin Yang
678734168c1e7b6c4a6e5ffd
Xiaohui Shen
admin_assigned
2025-01-15T16:55:07.851Z
6430aa1b32a732121cd81f98
/avatars/5419f8d6d4d36fa5ac83e30667b9fd99.svg
Xiaohui Shen
user
XiaohuiShen
678734168c1e7b6c4a6e5ffe
Suha Kwak
678734168c1e7b6c4a6e5fff
Liang-Chieh Chen
Democratizing Text-to-Image Masked Generative Models with Compact
  Text-Aware One-Dimensional Tokens
Image tokenizers form the foundation of modern text-to-image generative
models but are notoriously difficult to train. Furthermore, most existing
text-to-image models rely on large-scale, high-quality private datasets, making
them challenging to replicate. In this work, we introduce Text-Aware
Transformer-based 1-Dimensional Tokenizer (TA-TiTok), an efficient and powerful
image tokenizer that can utilize either discrete or continuous 1-dimensional
tokens. TA-TiTok uniquely integrates textual information during the tokenizer
decoding stage (i.e., de-tokenization), accelerating convergence and enhancing
performance. TA-TiTok also benefits from a simplified, yet effective, one-stage
training process, eliminating the need for the complex two-stage distillation
used in previous 1-dimensional tokenizers. This design allows for seamless
scalability to large datasets. Building on this, we introduce a family of
text-to-image Masked Generative Models (MaskGen), trained exclusively on open
data while achieving comparable performance to models trained on private data.
We aim to release both the efficient, strong TA-TiTok tokenizers and the
open-data, open-weight MaskGen models to promote broader access and democratize
the field of text-to-image masked generative models.
TA-TiTok, a text-aware transformer-based tokenizer, integrates textual information during decoding, improving training efficiency and enabling MaskGen models to achieve comparable performance using open datasets.
image tokenizers text-to-image generative models TA-TiTok transformer-based 1-dimensional tokens textual information tokenizer decoding de-tokenization convergence performance one-stage training distillation MaskGen open data open-weight