LayoutNUWA: Revealing the Hidden Layout Expertise of Large Language
  Models
Graphic layout generation, a growing research field, plays a significant role
in user engagement and information perception. Existing methods primarily treat
layout generation as a numerical optimization task, focusing on quantitative
aspects while overlooking the semantic information of layout, such as the
relationship between each layout element. In this paper, we propose LayoutNUWA,
the first model that treats layout generation as a code generation task to
enhance semantic information and harness the hidden layout expertise of large
language models~(LLMs). More concretely, we develop a Code Instruct Tuning
(CIT) approach comprising three interconnected modules: 1) the Code
Initialization (CI) module quantifies the numerical conditions and initializes
them as HTML code with strategically placed masks; 2) the Code Completion (CC)
module employs the formatting knowledge of LLMs to fill in the masked portions
within the HTML code; 3) the Code Rendering (CR) module transforms the
completed code into the final layout output, ensuring a highly interpretable
and transparent layout generation procedure that directly maps code to a
visualized layout. We attain significant state-of-the-art performance (even
over 50\% improvements) on multiple datasets, showcasing the strong
capabilities of LayoutNUWA. Our code is available at
https://github.com/ProjectNUWA/LayoutNUWA.
6509112b623330a3a5104b81
Zecheng Tang
admin_assigned
2023-09-20T13:16:43.387Z
64096ef79e9f790c905b846d
https://cdn-avatars.huggingface.co/v1/production/uploads/64096ef79e9f790c905b846d/hVzw656lXdzbCxTtnheud.jpeg
Zecheng Tang
user
ZetangForward
6509112b623330a3a5104b82
Chenfei Wu
admin_assigned
2023-09-20T14:23:07.118Z
63180f0f49af4d408c35f07f
https://cdn-avatars.huggingface.co/v1/production/uploads/1678864054390-63180f0f49af4d408c35f07f.jpeg
Chenfei Wu
user
Chenfei
6509112b623330a3a5104b83
Juntao Li
admin_assigned
2023-09-20T14:23:22.259Z
60d1d7f94672e8c13e7c2bce
/avatars/be3a3990492ef1c5f3ff33cc043ae166.svg
Juntao Li
user
ljtsuda
6509112b623330a3a5104b84
Nan Duan
LayoutNUWA: Revealing the Hidden Layout Expertise of Large Language
  Models
Graphic layout generation, a growing research field, plays a significant role
in user engagement and information perception. Existing methods primarily treat
layout generation as a numerical optimization task, focusing on quantitative
aspects while overlooking the semantic information of layout, such as the
relationship between each layout element. In this paper, we propose LayoutNUWA,
the first model that treats layout generation as a code generation task to
enhance semantic information and harness the hidden layout expertise of large
language models~(LLMs). More concretely, we develop a Code Instruct Tuning
(CIT) approach comprising three interconnected modules: 1) the Code
Initialization (CI) module quantifies the numerical conditions and initializes
them as HTML code with strategically placed masks; 2) the Code Completion (CC)
module employs the formatting knowledge of LLMs to fill in the masked portions
within the HTML code; 3) the Code Rendering (CR) module transforms the
completed code into the final layout output, ensuring a highly interpretable
and transparent layout generation procedure that directly maps code to a
visualized layout. We attain significant state-of-the-art performance (even
over 50\% improvements) on multiple datasets, showcasing the strong
capabilities of LayoutNUWA. Our code is available at
https://github.com/ProjectNUWA/LayoutNUWA.
LayoutNUWA enhances graphic layout generation by treating it as a code generation task, integrating LLMs to incorporate semantic information and achieve state-of-the-art performance on multiple datasets.
Code Instruct Tuning Code Initialization Code Completion Code Rendering large language models HTML code layout expertise semantic information graphical layout generation