BlenderAlchemy: Editing 3D Graphics with Vision-Language Models
Graphics design is important for various applications, including movie
production and game design. To create a high-quality scene, designers usually
need to spend hours in software like Blender, in which they might need to
interleave and repeat operations, such as connecting material nodes, hundreds
of times. Moreover, slightly different design goals may require completely
different sequences, making automation difficult. In this paper, we propose a
system that leverages Vision-Language Models (VLMs), like GPT-4V, to
intelligently search the design action space to arrive at an answer that can
satisfy a user's intent. Specifically, we design a vision-based edit generator
and state evaluator to work together to find the correct sequence of actions to
achieve the goal. Inspired by the role of visual imagination in the human
design process, we supplement the visual reasoning capabilities of VLMs with
"imagined" reference images from image-generation models, providing visual
grounding of abstract language descriptions. In this paper, we provide
empirical evidence suggesting our system can produce simple but tedious Blender
editing sequences for tasks such as editing procedural materials from text
and/or reference images, as well as adjusting lighting configurations for
product renderings in complex scenes.
663069379edc222ceeaea07e
Ian Huang
admin_assigned
2024-04-30T08:24:29.899Z
639ce51c66106be14370f510
/avatars/74f68f5951fff9c02c7607ff197812aa.svg
Ian Huang
user
ianhuang
663069379edc222ceeaea07f
Guandao Yang
admin_assigned
2024-04-30T08:24:36.330Z
64547ca006728ff79a38e7a5
https://cdn-avatars.huggingface.co/v1/production/uploads/64547ca006728ff79a38e7a5/xfTVQXMSfRv-IbljFpswQ.jpeg
Guandao Yang
user
guandao
663069379edc222ceeaea080
Leonidas Guibas
BlenderAlchemy: Editing 3D Graphics with Vision-Language Models
Graphics design is important for various applications, including movie
production and game design. To create a high-quality scene, designers usually
need to spend hours in software like Blender, in which they might need to
interleave and repeat operations, such as connecting material nodes, hundreds
of times. Moreover, slightly different design goals may require completely
different sequences, making automation difficult. In this paper, we propose a
system that leverages Vision-Language Models (VLMs), like GPT-4V, to
intelligently search the design action space to arrive at an answer that can
satisfy a user's intent. Specifically, we design a vision-based edit generator
and state evaluator to work together to find the correct sequence of actions to
achieve the goal. Inspired by the role of visual imagination in the human
design process, we supplement the visual reasoning capabilities of VLMs with
"imagined" reference images from image-generation models, providing visual
grounding of abstract language descriptions. In this paper, we provide
empirical evidence suggesting our system can produce simple but tedious Blender
editing sequences for tasks such as editing procedural materials from text
and/or reference images, as well as adjusting lighting configurations for
product renderings in complex scenes.
A system using Vision-Language Models and image-generation models automates Blender scene editing by finding the correct sequence of actions to achieve user-defined goals.
Vision-Language Models VLMs GPT-4V vision-based edit generator state evaluator imagined reference images procedural materials lighting configurations product renderings