ScreenCoder: Advancing Visual-to-Code Generation for Front-End
  Automation via Modular Multimodal Agents
Automating the transformation of user interface (UI) designs into front-end
code holds significant promise for accelerating software development and
democratizing design workflows. While recent large language models (LLMs) have
demonstrated progress in text-to-code generation, many existing approaches rely
solely on natural language prompts, limiting their effectiveness in capturing
spatial layout and visual design intent. In contrast, UI development in
practice is inherently multimodal, often starting from visual sketches or
mockups. To address this gap, we introduce a modular multi-agent framework that
performs UI-to-code generation in three interpretable stages: grounding,
planning, and generation. The grounding agent uses a vision-language model to
detect and label UI components, the planning agent constructs a hierarchical
layout using front-end engineering priors, and the generation agent produces
HTML/CSS code via adaptive prompt-based synthesis. This design improves
robustness, interpretability, and fidelity over end-to-end black-box methods.
Furthermore, we extend the framework into a scalable data engine that
automatically produces large-scale image-code pairs. Using these synthetic
examples, we fine-tune and reinforce an open-source VLM, yielding notable gains
in UI understanding and code quality. Extensive experiments demonstrate that
our approach achieves state-of-the-art performance in layout accuracy,
structural coherence, and code correctness. Our code is made publicly available
at https://github.com/leigest519/ScreenCoder.
688abfb98b724c8c7187dd3c
Yilei Jiang
claimed_verified
2025-08-06T19:32:32.414Z
64cf4c662f92537fbcfb08e6
/avatars/a6f77c30178ea0f6757ff8b1bd0dd1c9.svg
jiangyilei
user
Leigest
688abfb98b724c8c7187dd3d
Yaozhi Zheng
688abfb98b724c8c7187dd3e
Yuxuan Wan
688abfb98b724c8c7187dd3f
Jiaming Han
claimed_verified
2025-07-31T08:15:11.876Z
62318c0386753f5f41d0e261
https://cdn-avatars.huggingface.co/v1/production/uploads/62318c0386753f5f41d0e261/xO_5PvOf7lXhQPnQLcmnq.jpeg
Jiaming Han
user
csuhan
688abfb98b724c8c7187dd40
Qunzhong Wang
688abfb98b724c8c7187dd41
Michael R. Lyu
688abfb98b724c8c7187dd42
Xiangyu Yue
ScreenCoder: Advancing Visual-to-Code Generation for Front-End
  Automation via Modular Multimodal Agents
Automating the transformation of user interface (UI) designs into front-end
code holds significant promise for accelerating software development and
democratizing design workflows. While recent large language models (LLMs) have
demonstrated progress in text-to-code generation, many existing approaches rely
solely on natural language prompts, limiting their effectiveness in capturing
spatial layout and visual design intent. In contrast, UI development in
practice is inherently multimodal, often starting from visual sketches or
mockups. To address this gap, we introduce a modular multi-agent framework that
performs UI-to-code generation in three interpretable stages: grounding,
planning, and generation. The grounding agent uses a vision-language model to
detect and label UI components, the planning agent constructs a hierarchical
layout using front-end engineering priors, and the generation agent produces
HTML/CSS code via adaptive prompt-based synthesis. This design improves
robustness, interpretability, and fidelity over end-to-end black-box methods.
Furthermore, we extend the framework into a scalable data engine that
automatically produces large-scale image-code pairs. Using these synthetic
examples, we fine-tune and reinforce an open-source VLM, yielding notable gains
in UI understanding and code quality. Extensive experiments demonstrate that
our approach achieves state-of-the-art performance in layout accuracy,
structural coherence, and code correctness. Our code is made publicly available
at https://github.com/leigest519/ScreenCoder.
A modular multi-agent framework improves UI-to-code generation by integrating vision-language models, hierarchical layout planning, and adaptive prompt-based synthesis, achieving state-of-the-art performance.
vision-language model hierarchical layout adaptive prompt-based synthesis UI-to-code generation multimodal grounding agent planning agent generation agent data engine fine-tuning reinforcement