MagicClay: Sculpting Meshes With Generative Neural Fields
The recent developments in neural fields have brought phenomenal capabilities
to the field of shape generation, but they lack crucial properties, such as
incremental control - a fundamental requirement for artistic work. Triangular
meshes, on the other hand, are the representation of choice for most geometry
related tasks, offering efficiency and intuitive control, but do not lend
themselves to neural optimization. To support downstream tasks, previous art
typically proposes a two-step approach, where first a shape is generated using
neural fields, and then a mesh is extracted for further processing. Instead, in
this paper we introduce a hybrid approach that maintains both a mesh and a
Signed Distance Field (SDF) representations consistently. Using this
representation, we introduce MagicClay - an artist friendly tool for sculpting
regions of a mesh according to textual prompts while keeping other regions
untouched. Our framework carefully and efficiently balances consistency between
the representations and regularizations in every step of the shape
optimization; Relying on the mesh representation, we show how to render the SDF
at higher resolutions and faster. In addition, we employ recent work in
differentiable mesh reconstruction to adaptively allocate triangles in the mesh
where required, as indicated by the SDF. Using an implemented prototype, we
demonstrate superior generated geometry compared to the state-of-the-art, and
novel consistent control, allowing sequential prompt-based edits to the same
mesh for the first time.
65e7faa2ab088a3c36c2b5ec
Amir Barda
admin_assigned
2024-03-06T10:02:08.589Z
65970b381b663b1e536e76c0
/avatars/095dfd84599c74d7600228d8104faac9.svg
Amir Barda
user
Amir001
65e7faa2ab088a3c36c2b5ed
Vladimir G. Kim
65e7faa2ab088a3c36c2b5ee
Noam Aigerman
claimed_verified
2024-07-23T10:05:05.606Z
637ed06c1dbae09191065d0c
/avatars/93bb8b51ca27c89b3ddce1be08f641b3.svg
noam
user
noamaig
65e7faa2ab088a3c36c2b5ef
Amit H. Bermano
65e7faa2ab088a3c36c2b5f0
Thibault Groueix
MagicClay: Sculpting Meshes With Generative Neural Fields
The recent developments in neural fields have brought phenomenal capabilities
to the field of shape generation, but they lack crucial properties, such as
incremental control - a fundamental requirement for artistic work. Triangular
meshes, on the other hand, are the representation of choice for most geometry
related tasks, offering efficiency and intuitive control, but do not lend
themselves to neural optimization. To support downstream tasks, previous art
typically proposes a two-step approach, where first a shape is generated using
neural fields, and then a mesh is extracted for further processing. Instead, in
this paper we introduce a hybrid approach that maintains both a mesh and a
Signed Distance Field (SDF) representations consistently. Using this
representation, we introduce MagicClay - an artist friendly tool for sculpting
regions of a mesh according to textual prompts while keeping other regions
untouched. Our framework carefully and efficiently balances consistency between
the representations and regularizations in every step of the shape
optimization; Relying on the mesh representation, we show how to render the SDF
at higher resolutions and faster. In addition, we employ recent work in
differentiable mesh reconstruction to adaptively allocate triangles in the mesh
where required, as indicated by the SDF. Using an implemented prototype, we
demonstrate superior generated geometry compared to the state-of-the-art, and
novel consistent control, allowing sequential prompt-based edits to the same
mesh for the first time.
A hybrid method combining mesh and SDF representations enables consistent sculpting and text-guided editing of 3D shapes with high resolution rendering and adaptive triangle allocation.
neural fields incremental control triangular meshes Signed Distance Field (SDF) MagicClay shape optimization mesh representation differentiable mesh reconstruction adaptive triangle allocation