Glyph: Scaling Context Windows via Visual-Text Compression
Large language models (LLMs) increasingly rely on long-context modeling for
tasks such as document understanding, code analysis, and multi-step reasoning.
However, scaling context windows to the million-token level brings prohibitive
computational and memory costs, limiting the practicality of long-context LLMs.
In this work, we take a different perspective-visual context scaling-to tackle
this challenge. Instead of extending token-based sequences, we propose Glyph, a
framework that renders long texts into images and processes them with
vision-language models (VLMs). This approach substantially compresses textual
input while preserving semantic information, and we further design an
LLM-driven genetic search to identify optimal visual rendering configurations
for balancing accuracy and compression. Through extensive experiments, we
demonstrate that our method achieves 3-4x token compression while maintaining
accuracy comparable to leading LLMs such as Qwen3-8B on various long-context
benchmarks. This compression also leads to around 4x faster prefilling and
decoding, and approximately 2x faster SFT training. Furthermore, under extreme
compression, a 128K-context VLM could scale to handle 1M-token-level text
tasks. In addition, the rendered text data benefits real-world multimodal
tasks, such as document understanding. Our code and model are released at
https://github.com/thu-coai/Glyph.
68f6f94824c4489363111924
Jiale Cheng
68f6f94824c4489363111925
Yusen Liu
68f6f94824c4489363111926
Xinyu Zhang
68f6f94824c4489363111927
Yulin Fei
68f6f94824c4489363111928
Wenyi Hong
68f6f94824c4489363111929
Ruiliang Lyu
68f6f94824c448936311192a
Weihan Wang
68f6f94824c448936311192b
Zhe Su
68f6f94824c448936311192c
Xiaotao Gu
68f6f94824c448936311192d
Xiao Liu
68f6f94824c448936311192e
Yushi Bai
68f6f94824c448936311192f
Jie Tang
68f6f94824c4489363111930
Hongning Wang
68f6f94824c4489363111931
Minlie Huang
Glyph: Scaling Context Windows via Visual-Text Compression
Large language models (LLMs) increasingly rely on long-context modeling for
tasks such as document understanding, code analysis, and multi-step reasoning.
However, scaling context windows to the million-token level brings prohibitive
computational and memory costs, limiting the practicality of long-context LLMs.
In this work, we take a different perspective-visual context scaling-to tackle
this challenge. Instead of extending token-based sequences, we propose Glyph, a
framework that renders long texts into images and processes them with
vision-language models (VLMs). This approach substantially compresses textual
input while preserving semantic information, and we further design an
LLM-driven genetic search to identify optimal visual rendering configurations
for balancing accuracy and compression. Through extensive experiments, we
demonstrate that our method achieves 3-4x token compression while maintaining
accuracy comparable to leading LLMs such as Qwen3-8B on various long-context
benchmarks. This compression also leads to around 4x faster prefilling and
decoding, and approximately 2x faster SFT training. Furthermore, under extreme
compression, a 128K-context VLM could scale to handle 1M-token-level text
tasks. In addition, the rendered text data benefits real-world multimodal
tasks, such as document understanding. Our code and model are released at
https://github.com/thu-coai/Glyph.
Glyph compresses long textual inputs into images using vision-language models, achieving significant token compression and improved performance in long-context tasks.
long-context modeling large language models token-based sequences Glyph vision-language models genetic search token compression prefilling decoding SFT training multimodal tasks document understanding