Adaptive Length Image Tokenization via Recurrent Allocation
Current vision systems typically assign fixed-length representations to
images, regardless of the information content. This contrasts with human
intelligence - and even large language models - which allocate varying
representational capacities based on entropy, context and familiarity. Inspired
by this, we propose an approach to learn variable-length token representations
for 2D images. Our encoder-decoder architecture recursively processes 2D image
tokens, distilling them into 1D latent tokens over multiple iterations of
recurrent rollouts. Each iteration refines the 2D tokens, updates the existing
1D latent tokens, and adaptively increases representational capacity by adding
new tokens. This enables compression of images into a variable number of
tokens, ranging from 32 to 256. We validate our tokenizer using reconstruction
loss and FID metrics, demonstrating that token count aligns with image entropy,
familiarity and downstream task requirements. Recurrent token processing with
increasing representational capacity in each iteration shows signs of token
specialization, revealing potential for object / part discovery.
672b801f26c6716f52485c26
Shivam Duggal
672b801f26c6716f52485c27
Phillip Isola
672b801f26c6716f52485c28
Antonio Torralba
672b801f26c6716f52485c29
William T. Freeman
Adaptive Length Image Tokenization via Recurrent Allocation
Current vision systems typically assign fixed-length representations to
images, regardless of the information content. This contrasts with human
intelligence - and even large language models - which allocate varying
representational capacities based on entropy, context and familiarity. Inspired
by this, we propose an approach to learn variable-length token representations
for 2D images. Our encoder-decoder architecture recursively processes 2D image
tokens, distilling them into 1D latent tokens over multiple iterations of
recurrent rollouts. Each iteration refines the 2D tokens, updates the existing
1D latent tokens, and adaptively increases representational capacity by adding
new tokens. This enables compression of images into a variable number of
tokens, ranging from 32 to 256. We validate our tokenizer using reconstruction
loss and FID metrics, demonstrating that token count aligns with image entropy,
familiarity and downstream task requirements. Recurrent token processing with
increasing representational capacity in each iteration shows signs of token
specialization, revealing potential for object / part discovery.
A new method for learning variable-length token representations for 2D images through recursive processing and token specialization is proposed, validated by reconstruction loss and FID metrics.
variable-length token representations encoder-decoder architecture recurrent rollouts 2D tokens 1D latent tokens token specialization image entropy familiarity token count image compression reconstruction loss FID metrics