JPEG-LM: LLMs as Image Generators with Canonical Codec Representations
Recent work in image and video generation has been adopting the
autoregressive LLM architecture due to its generality and potentially easy
integration into multi-modal systems. The crux of applying autoregressive
training in language generation to visual generation is discretization --
representing continuous data like images and videos as discrete tokens. Common
methods of discretizing images and videos include modeling raw pixel values,
which are prohibitively lengthy, or vector quantization, which requires
convoluted pre-hoc training. In this work, we propose to directly model images
and videos as compressed files saved on computers via canonical codecs (e.g.,
JPEG, AVC/H.264). Using the default Llama architecture without any
vision-specific modifications, we pretrain JPEG-LM from scratch to generate
images (and AVC-LM to generate videos as a proof of concept), by directly
outputting compressed file bytes in JPEG and AVC formats. Evaluation of image
generation shows that this simple and straightforward approach is more
effective than pixel-based modeling and sophisticated vector quantization
baselines (on which our method yields a 31% reduction in FID). Our analysis
shows that JPEG-LM has an especial advantage over vector quantization models in
generating long-tail visual elements. Overall, we show that using canonical
codec representations can help lower the barriers between language generation
and visual generation, facilitating future research on multi-modal
language/image/video LLMs.
66c2a4ae2777c05070a6db3a
Xiaochuang Han
claimed_verified
2024-08-19T22:13:21.157Z
635ea3969f24f6db0a1e2d0b
https://cdn-avatars.huggingface.co/v1/production/uploads/635ea3969f24f6db0a1e2d0b/RxaJKAx4MYMShb58yuUxy.png
Xiaochuang Han
user
xhan77
66c2a4ae2777c05070a6db3b
Marjan Ghazvininejad
claimed_verified
2025-02-24T09:27:55.990Z
660f0fd377a1e2509aa5a679
/avatars/e04ef05bed0bf6cefdc7e3e39674e2f9.svg
Marjan Ghazvininejad
user
mghazvininejad
66c2a4ae2777c05070a6db3c
Pang Wei Koh
66c2a4ae2777c05070a6db3d
Yulia Tsvetkov
JPEG-LM: LLMs as Image Generators with Canonical Codec Representations
Recent work in image and video generation has been adopting the
autoregressive LLM architecture due to its generality and potentially easy
integration into multi-modal systems. The crux of applying autoregressive
training in language generation to visual generation is discretization --
representing continuous data like images and videos as discrete tokens. Common
methods of discretizing images and videos include modeling raw pixel values,
which are prohibitively lengthy, or vector quantization, which requires
convoluted pre-hoc training. In this work, we propose to directly model images
and videos as compressed files saved on computers via canonical codecs (e.g.,
JPEG, AVC/H.264). Using the default Llama architecture without any
vision-specific modifications, we pretrain JPEG-LM from scratch to generate
images (and AVC-LM to generate videos as a proof of concept), by directly
outputting compressed file bytes in JPEG and AVC formats. Evaluation of image
generation shows that this simple and straightforward approach is more
effective than pixel-based modeling and sophisticated vector quantization
baselines (on which our method yields a 31% reduction in FID). Our analysis
shows that JPEG-LM has an especial advantage over vector quantization models in
generating long-tail visual elements. Overall, we show that using canonical
codec representations can help lower the barriers between language generation
and visual generation, facilitating future research on multi-modal
language/image/video LLMs.
Using canonical codec representations (JPEG, AVC) as a direct modeling approach for image and video generation outperforms pixel-based and vector quantization methods, facilitating multi-modal LLMs.
autoregressive LLM architecture discretization vector quantization compressed file bytes JPEG-LM AVC-LM FID long-tail visual elements multi-modal language/image/video LLMs