Zero-Shot Vision Encoder Grafting via LLM Surrogates
Vision language models (VLMs) typically pair a modestly sized vision encoder
with a large language model (LLM), e.g., Llama-70B, making the decoder the
primary computational burden during training. To reduce costs, a potential
promising strategy is to first train the vision encoder using a small language
model before transferring it to the large one. We construct small "surrogate
models" that share the same embedding space and representation language as the
large target LLM by directly inheriting its shallow layers. Vision encoders
trained on the surrogate can then be directly transferred to the larger model,
a process we call zero-shot grafting -- when plugged directly into the
full-size target LLM, the grafted pair surpasses the encoder-surrogate pair
and, on some benchmarks, even performs on par with full decoder training with
the target LLM. Furthermore, our surrogate training approach reduces overall
VLM training costs by ~45% when using Llama-70B as the decoder.
6837befa0aa18c6f96f4a9a5
Kaiyu Yue
claimed_verified
2025-06-03T08:50:48.513Z
640d0dbc8036cc2142273a83
https://cdn-avatars.huggingface.co/v1/production/uploads/640d0dbc8036cc2142273a83/cicTWJVqqvQv_DgDucWgY.jpeg
Kaiyu Yue
user
kaiyuyue
6837befa0aa18c6f96f4a9a6
Vasu Singla
6837befa0aa18c6f96f4a9a7
Menglin Jia
6837befa0aa18c6f96f4a9a8
John Kirchenbauer
6837befa0aa18c6f96f4a9a9
Rifaa Qadri
6837befa0aa18c6f96f4a9aa
Zikui Cai
claimed_verified
2025-06-09T10:12:51.628Z
630bc5ae86b8b9904c33e94b
/avatars/b176d9b1691c05cc941409dd6c2b2228.svg
Zikui Cai
user
Zikui
6837befa0aa18c6f96f4a9ab
Abhinav Bhatele
6837befa0aa18c6f96f4a9ac
Furong Huang
6837befa0aa18c6f96f4a9ad
Tom Goldstein
Zero-Shot Vision Encoder Grafting via LLM Surrogates
Vision language models (VLMs) typically pair a modestly sized vision encoder
with a large language model (LLM), e.g., Llama-70B, making the decoder the
primary computational burden during training. To reduce costs, a potential
promising strategy is to first train the vision encoder using a small language
model before transferring it to the large one. We construct small "surrogate
models" that share the same embedding space and representation language as the
large target LLM by directly inheriting its shallow layers. Vision encoders
trained on the surrogate can then be directly transferred to the larger model,
a process we call zero-shot grafting -- when plugged directly into the
full-size target LLM, the grafted pair surpasses the encoder-surrogate pair
and, on some benchmarks, even performs on par with full decoder training with
the target LLM. Furthermore, our surrogate training approach reduces overall
VLM training costs by ~45% when using Llama-70B as the decoder.
The approach of training vision encoders with small surrogate models before transferring them to large language models reduces training costs and enhances performance.
vision language models vision encoder large language model small language model surrogate models embedding space representation language zero-shot grafting