HART: Efficient Visual Generation with Hybrid Autoregressive Transformer
We introduce Hybrid Autoregressive Transformer (HART), an autoregressive (AR)
visual generation model capable of directly generating 1024x1024 images,
rivaling diffusion models in image generation quality. Existing AR models face
limitations due to the poor image reconstruction quality of their discrete
tokenizers and the prohibitive training costs associated with generating 1024px
images. To address these challenges, we present the hybrid tokenizer, which
decomposes the continuous latents from the autoencoder into two components:
discrete tokens representing the big picture and continuous tokens representing
the residual components that cannot be represented by the discrete tokens. The
discrete component is modeled by a scalable-resolution discrete AR model, while
the continuous component is learned with a lightweight residual diffusion
module with only 37M parameters. Compared with the discrete-only VAR tokenizer,
our hybrid approach improves reconstruction FID from 2.11 to 0.30 on MJHQ-30K,
leading to a 31% generation FID improvement from 7.85 to 5.38. HART also
outperforms state-of-the-art diffusion models in both FID and CLIP score, with
4.5-7.7x higher throughput and 6.9-13.4x lower MACs. Our code is open sourced
at https://github.com/mit-han-lab/hart.
670e47a7ae6e15b48591a91b
Haotian Tang
670e47a7ae6e15b48591a91c
Yecheng Wu
670e47a7ae6e15b48591a91d
Shang Yang
claimed_verified
2025-02-21T10:00:37.479Z
641d8bacd526196afc12766d
/avatars/73f7b2d86a7bf27940bec2b1f199d71b.svg
Shang Yang
user
Shangy
670e47a7ae6e15b48591a91e
Enze Xie
670e47a7ae6e15b48591a91f
Junsong Chen
claimed_verified
2024-11-21T21:33:26.122Z
645b5b09bc7518912e1f9733
/avatars/4d35f728b41f93881a9b67c337f4d1df.svg
Chen
user
Lawrence-cj
670e47a7ae6e15b48591a920
Junyu Chen
claimed_verified
2024-12-06T09:10:06.175Z
646189cd5dba83471db2af58
/avatars/d5a1549af336cb5f1fa5622250d38a73.svg
JUNYU CHEN
user
cjy2003
670e47a7ae6e15b48591a921
Zhuoyang Zhang
670e47a7ae6e15b48591a922
Han Cai
claimed_verified
2024-10-28T11:25:37.556Z
650e2b14c945dfc9386a7e28
/avatars/0a9be20aa53f4c52a2d1a8b02d4093ea.svg
Han Cai
user
han-cai
670e47a7ae6e15b48591a923
Yao Lu
670e47a7ae6e15b48591a924
Song Han
HART: Efficient Visual Generation with Hybrid Autoregressive Transformer
We introduce Hybrid Autoregressive Transformer (HART), an autoregressive (AR)
visual generation model capable of directly generating 1024x1024 images,
rivaling diffusion models in image generation quality. Existing AR models face
limitations due to the poor image reconstruction quality of their discrete
tokenizers and the prohibitive training costs associated with generating 1024px
images. To address these challenges, we present the hybrid tokenizer, which
decomposes the continuous latents from the autoencoder into two components:
discrete tokens representing the big picture and continuous tokens representing
the residual components that cannot be represented by the discrete tokens. The
discrete component is modeled by a scalable-resolution discrete AR model, while
the continuous component is learned with a lightweight residual diffusion
module with only 37M parameters. Compared with the discrete-only VAR tokenizer,
our hybrid approach improves reconstruction FID from 2.11 to 0.30 on MJHQ-30K,
leading to a 31% generation FID improvement from 7.85 to 5.38. HART also
outperforms state-of-the-art diffusion models in both FID and CLIP score, with
4.5-7.7x higher throughput and 6.9-13.4x lower MACs. Our code is open sourced
at https://github.com/mit-han-lab/hart.
HART, a hybrid autoregressive transformers model, generates 1024x1024 images with high quality and efficiency by combining discrete and continuous tokenization.
Hybrid Autoregressive Transformer HART autoregressive model image generation diffusion models discrete tokenizers continuous tokens autoencoder scalable-resolution discrete AR model lightweight residual diffusion module FID MJHQ-30K CLIP score throughput MACs