VisCodex: Unified Multimodal Code Generation via Merging Vision and
  Coding Models
Multimodal large language models (MLLMs) have significantly advanced the
integration of visual and textual understanding. However, their ability to
generate code from multimodal inputs remains limited. In this work, we
introduce VisCodex, a unified framework that seamlessly merges vision and
coding language models to empower MLLMs with strong multimodal code generation
abilities. Leveraging a task vector-based model merging technique, we integrate
a state-of-the-art coding LLM into a strong vision-language backbone, while
preserving both visual comprehension and advanced coding skills. To support
training and evaluation, we introduce the Multimodal Coding Dataset (MCD), a
large-scale and diverse collection of 598k samples, including high-quality HTML
code, chart image-code pairs, image-augmented StackOverflow QA, and algorithmic
problems. Furthermore, we propose InfiBench-V, a novel and challenging
benchmark specifically designed to assess models on visually-rich, real-world
programming questions that demand a nuanced understanding of both textual and
visual contexts. Extensive experiments show that VisCodex achieves
state-of-the-art performance among open-source MLLMs and approaches proprietary
models like GPT-4o, highlighting the effectiveness of our model merging
strategy and new datasets.
689da0ffb083e610d741eb02
Lingjie Jiang
689da0ffb083e610d741eb03
Shaohan Huang
claimed_verified
2025-08-28T08:56:52.090Z
632bd2f72d6a805eeb4bc601
/avatars/6e1533e8a599f3068290aa69ac82cab7.svg
HUANG SHAOHAN
user
buaahsh
689da0ffb083e610d741eb04
Xun Wu
689da0ffb083e610d741eb05
Yixia Li
689da0ffb083e610d741eb06
Dongdong Zhang
689da0ffb083e610d741eb07
Furu Wei
VisCodex: Unified Multimodal Code Generation via Merging Vision and
  Coding Models
Multimodal large language models (MLLMs) have significantly advanced the
integration of visual and textual understanding. However, their ability to
generate code from multimodal inputs remains limited. In this work, we
introduce VisCodex, a unified framework that seamlessly merges vision and
coding language models to empower MLLMs with strong multimodal code generation
abilities. Leveraging a task vector-based model merging technique, we integrate
a state-of-the-art coding LLM into a strong vision-language backbone, while
preserving both visual comprehension and advanced coding skills. To support
training and evaluation, we introduce the Multimodal Coding Dataset (MCD), a
large-scale and diverse collection of 598k samples, including high-quality HTML
code, chart image-code pairs, image-augmented StackOverflow QA, and algorithmic
problems. Furthermore, we propose InfiBench-V, a novel and challenging
benchmark specifically designed to assess models on visually-rich, real-world
programming questions that demand a nuanced understanding of both textual and
visual contexts. Extensive experiments show that VisCodex achieves
state-of-the-art performance among open-source MLLMs and approaches proprietary
models like GPT-4o, highlighting the effectiveness of our model merging
strategy and new datasets.
VisCodex integrates vision and coding models to enhance multimodal code generation, achieving top performance using a novel dataset and benchmark.
multimodal large language models MLLMs VisCodex task vector-based model merging coding LLM vision-language backbone Multimodal Coding Dataset MCD InfiBench-V visually-rich programming questions