CMC-Bench: Towards a New Paradigm of Visual Signal Compression
Ultra-low bitrate image compression is a challenging and demanding topic.
With the development of Large Multimodal Models (LMMs), a Cross Modality
Compression (CMC) paradigm of Image-Text-Image has emerged. Compared with
traditional codecs, this semantic-level compression can reduce image data size
to 0.1\% or even lower, which has strong potential applications. However, CMC
has certain defects in consistency with the original image and perceptual
quality. To address this problem, we introduce CMC-Bench, a benchmark of the
cooperative performance of Image-to-Text (I2T) and Text-to-Image (T2I) models
for image compression. This benchmark covers 18,000 and 40,000 images
respectively to verify 6 mainstream I2T and 12 T2I models, including 160,000
subjective preference scores annotated by human experts. At ultra-low bitrates,
this paper proves that the combination of some I2T and T2I models has surpassed
the most advanced visual signal codecs; meanwhile, it highlights where LMMs can
be further optimized toward the compression task. We encourage LMM developers
to participate in this test to promote the evolution of visual signal codec
protocols.
666c7348e0b2d43ff35e7f58
Chunyi Li
666c7348e0b2d43ff35e7f59
Xiele Wu
666c7348e0b2d43ff35e7f5a
Haoning Wu
admin_assigned
2024-06-15T09:13:38.596Z
632c7a0d1d303f5f9acf01b8
https://cdn-avatars.huggingface.co/v1/production/uploads/632c7a0d1d303f5f9acf01b8/T010IFuCp6UaOeIyWhbCk.jpeg
Haoning Wu
user
haoningwu
666c7348e0b2d43ff35e7f5b
Donghui Feng
admin_assigned
2024-06-15T09:13:45.052Z
6498a0c309cea55085307172
/avatars/32c4085ff511a4803f06b1247cebc1cd.svg
Donghui Feng
user
crazysteven
666c7348e0b2d43ff35e7f5c
Zicheng Zhang
admin_assigned
2024-06-15T09:14:30.886Z
6526cc6bab4f5d98382f5603
https://cdn-avatars.huggingface.co/v1/production/uploads/6526cc6bab4f5d98382f5603/NekYYb61I5nt4au6gXsVK.jpeg
Zicheng Zhang
user
zhangzicheng
666c7348e0b2d43ff35e7f5d
Guo Lu
666c7348e0b2d43ff35e7f5e
Xiongkuo Min
666c7348e0b2d43ff35e7f5f
Xiaohong Liu
666c7348e0b2d43ff35e7f60
Guangtao Zhai
admin_assigned
2024-06-15T09:14:58.072Z
65535b125413c1a54e6fb243
/avatars/03bcf1d58865f5406aff49a415e78bdc.svg
Guangtao Zhai
user
GTZhai
666c7348e0b2d43ff35e7f61
Weisi Lin
CMC-Bench: Towards a New Paradigm of Visual Signal Compression
Ultra-low bitrate image compression is a challenging and demanding topic.
With the development of Large Multimodal Models (LMMs), a Cross Modality
Compression (CMC) paradigm of Image-Text-Image has emerged. Compared with
traditional codecs, this semantic-level compression can reduce image data size
to 0.1\% or even lower, which has strong potential applications. However, CMC
has certain defects in consistency with the original image and perceptual
quality. To address this problem, we introduce CMC-Bench, a benchmark of the
cooperative performance of Image-to-Text (I2T) and Text-to-Image (T2I) models
for image compression. This benchmark covers 18,000 and 40,000 images
respectively to verify 6 mainstream I2T and 12 T2I models, including 160,000
subjective preference scores annotated by human experts. At ultra-low bitrates,
this paper proves that the combination of some I2T and T2I models has surpassed
the most advanced visual signal codecs; meanwhile, it highlights where LMMs can
be further optimized toward the compression task. We encourage LMM developers
to participate in this test to promote the evolution of visual signal codec
protocols.
The CMC-Bench benchmark evaluates the performance of Image-to-Text and Text-to-Image models for ultra-low bitrate image compression, showing that certain combinations outperform traditional codecs.
Large Multimodal Models Cross Modality Compression Image-to-Text Text-to-Image CMC-Bench semantic-level compression subjective preference scores