Heptapod: Language Modeling on Visual Signals
We introduce Heptapod, an image autoregressive model that adheres to the
foundational principles of language modeling. Heptapod employs causal
attention, eliminates reliance on CFG, and eschews the trend
of semantic tokenizers. Our key innovation is next 2D distribution
prediction: a causal Transformer with reconstruction-focused visual tokenizer,
learns to predict the distribution over the entire 2D spatial grid of images at
each timestep. This learning objective unifies the sequential modeling of
autoregressive framework with the holistic self-supervised learning of masked
autoencoding, enabling the model to capture comprehensive image semantics via
generative training. On the ImageNet generation benchmark, Heptapod achieves an
FID of 2.70, significantly outperforming previous causal autoregressive
approaches. We hope our work inspires a principled rethinking of language
modeling on visual signals and beyond.
68e75ac67ae125f9582e6b0c
Yongxin Zhu
claimed_verified
2025-10-09T06:49:41.425Z
637c7d8a88699fba70e1e1ff
/avatars/e7f60fab6ca9ad5ecd320bd7cd51ce2e.svg
yongxinzhu
user
youngsheen
68e75ac67ae125f9582e6b0d
Jiawei Chen
68e75ac67ae125f9582e6b0e
Yuanzhe Chen
68e75ac67ae125f9582e6b0f
Zhuo Chen
68e75ac67ae125f9582e6b10
Dongya Jia
68e75ac67ae125f9582e6b11
Jian Cong
68e75ac67ae125f9582e6b12
Xiaobin Zhuang
68e75ac67ae125f9582e6b13
Yuping Wang
68e75ac67ae125f9582e6b14
Yuxuan Wang
Heptapod: Language Modeling on Visual Signals
We introduce Heptapod, an image autoregressive model that adheres to the
foundational principles of language modeling. Heptapod employs causal
attention, eliminates reliance on CFG, and eschews the trend
of semantic tokenizers. Our key innovation is next 2D distribution
prediction: a causal Transformer with reconstruction-focused visual tokenizer,
learns to predict the distribution over the entire 2D spatial grid of images at
each timestep. This learning objective unifies the sequential modeling of
autoregressive framework with the holistic self-supervised learning of masked
autoencoding, enabling the model to capture comprehensive image semantics via
generative training. On the ImageNet generation benchmark, Heptapod achieves an
FID of 2.70, significantly outperforming previous causal autoregressive
approaches. We hope our work inspires a principled rethinking of language
modeling on visual signals and beyond.
Heptapod, an image autoregressive model using causal attention and next 2D distribution prediction, achieves superior performance on ImageNet generation by combining sequential modeling with holistic self-supervised learning.
causal attention CFG semantic tokenizers next 2D distribution prediction causal Transformer visual tokenizer masked autoencoding FID