Tokenize Image as a Set
This paper proposes a fundamentally new paradigm for image generation through
set-based tokenization and distribution modeling. Unlike conventional methods
that serialize images into fixed-position latent codes with a uniform
compression ratio, we introduce an unordered token set representation to
dynamically allocate coding capacity based on regional semantic complexity.
This TokenSet enhances global context aggregation and improves robustness
against local perturbations. To address the critical challenge of modeling
discrete sets, we devise a dual transformation mechanism that bijectively
converts sets into fixed-length integer sequences with summation constraints.
Further, we propose Fixed-Sum Discrete Diffusion--the first framework to
simultaneously handle discrete values, fixed sequence length, and summation
invariance--enabling effective set distribution modeling. Experiments
demonstrate our method's superiority in semantic-aware representation and
generation quality. Our innovations, spanning novel representation and modeling
strategies, advance visual generation beyond traditional sequential token
paradigms. Our code and models are publicly available at
https://github.com/Gengzigang/TokenSet.
67dd38adbf4c007db3bf4b98
Zigang Geng
67dd38adbf4c007db3bf4b99
Mengde Xu
67dd38adbf4c007db3bf4b9a
Han Hu
67dd38adbf4c007db3bf4b9b
Shuyang Gu
Tokenize Image as a Set
This paper proposes a fundamentally new paradigm for image generation through
set-based tokenization and distribution modeling. Unlike conventional methods
that serialize images into fixed-position latent codes with a uniform
compression ratio, we introduce an unordered token set representation to
dynamically allocate coding capacity based on regional semantic complexity.
This TokenSet enhances global context aggregation and improves robustness
against local perturbations. To address the critical challenge of modeling
discrete sets, we devise a dual transformation mechanism that bijectively
converts sets into fixed-length integer sequences with summation constraints.
Further, we propose Fixed-Sum Discrete Diffusion--the first framework to
simultaneously handle discrete values, fixed sequence length, and summation
invariance--enabling effective set distribution modeling. Experiments
demonstrate our method's superiority in semantic-aware representation and
generation quality. Our innovations, spanning novel representation and modeling
strategies, advance visual generation beyond traditional sequential token
paradigms. Our code and models are publicly available at
https://github.com/Gengzigang/TokenSet.
TokenSet uses set-based tokenization and Fixed-Sum Discrete Diffusion to model image distributions, enhancing representation and generation quality by dynamically allocating complexity.
set-based tokenization token set global context aggregation robustness dual transformation mechanism fixed-length integer sequences summation constraints Fixed-Sum Discrete Diffusion semantic-aware representation generation quality