LLaMA-Mesh: Unifying 3D Mesh Generation with Language Models
This work explores expanding the capabilities of large language models (LLMs)
pretrained on text to generate 3D meshes within a unified model. This offers
key advantages of (1) leveraging spatial knowledge already embedded in LLMs,
derived from textual sources like 3D tutorials, and (2) enabling conversational
3D generation and mesh understanding. A primary challenge is effectively
tokenizing 3D mesh data into discrete tokens that LLMs can process seamlessly.
To address this, we introduce LLaMA-Mesh, a novel approach that represents the
vertex coordinates and face definitions of 3D meshes as plain text, allowing
direct integration with LLMs without expanding the vocabulary. We construct a
supervised fine-tuning (SFT) dataset enabling pretrained LLMs to (1) generate
3D meshes from text prompts, (2) produce interleaved text and 3D mesh outputs
as required, and (3) understand and interpret 3D meshes. Our work is the first
to demonstrate that LLMs can be fine-tuned to acquire complex spatial knowledge
for 3D mesh generation in a text-based format, effectively unifying the 3D and
text modalities. LLaMA-Mesh achieves mesh generation quality on par with models
trained from scratch while maintaining strong text generation performance.
6736d0f867d01d58999f710b
Zhengyi Wang
admin_assigned
2024-11-15T14:12:25.832Z
634e15aec1ce28f1de91c470
https://cdn-avatars.huggingface.co/v1/production/uploads/634e15aec1ce28f1de91c470/MdtJKgE_fbHydhqESr-DH.png
Zhengyi Wang
user
Zhengyi
6736d0f867d01d58999f710c
Jonathan Lorraine
admin_assigned
2024-11-15T13:49:53.477Z
631b7370bf1351ed2bd0abdc
https://cdn-avatars.huggingface.co/v1/production/uploads/631b7370bf1351ed2bd0abdc/p0ZRMjgp5mt3sT5OhdHsp.png
Jonathan Lorraine
user
lorraine2
6736d0f867d01d58999f710d
Yikai Wang
claimed_verified
2024-11-15T17:25:42.497Z
64206b35e40f66bcd1e3c985
/avatars/a8f830d1c47702f642045399e59c9fe8.svg
Yikai Wang
user
yikaiw
6736d0f867d01d58999f710e
Hang Su
admin_assigned
2024-11-15T13:54:00.109Z
64212f75d039a891a915560a
/avatars/0fa50ce554d4db4d64f83be7c11773e9.svg
Hang Su
user
hang401
6736d0f867d01d58999f710f
Jun Zhu
6736d0f867d01d58999f7110
Sanja Fidler
6736d0f867d01d58999f7111
Xiaohui Zeng
claimed_verified
2024-11-16T19:46:59.604Z
63363cb8a8048332fdc268cd
/avatars/a9bf6ce3394a8b76eade1a7517b37fa1.svg
xiaohui zeng
user
xiaohui2022
LLaMA-Mesh: Unifying 3D Mesh Generation with Language Models
This work explores expanding the capabilities of large language models (LLMs)
pretrained on text to generate 3D meshes within a unified model. This offers
key advantages of (1) leveraging spatial knowledge already embedded in LLMs,
derived from textual sources like 3D tutorials, and (2) enabling conversational
3D generation and mesh understanding. A primary challenge is effectively
tokenizing 3D mesh data into discrete tokens that LLMs can process seamlessly.
To address this, we introduce LLaMA-Mesh, a novel approach that represents the
vertex coordinates and face definitions of 3D meshes as plain text, allowing
direct integration with LLMs without expanding the vocabulary. We construct a
supervised fine-tuning (SFT) dataset enabling pretrained LLMs to (1) generate
3D meshes from text prompts, (2) produce interleaved text and 3D mesh outputs
as required, and (3) understand and interpret 3D meshes. Our work is the first
to demonstrate that LLMs can be fine-tuned to acquire complex spatial knowledge
for 3D mesh generation in a text-based format, effectively unifying the 3D and
text modalities. LLaMA-Mesh achieves mesh generation quality on par with models
trained from scratch while maintaining strong text generation performance.
The work demonstrates the capability of LLMs to generate 3D meshes from text by introducing a novel approach to tokenize 3D mesh data, allowing the unification of 3D and text modalities without expanding the model's vocabulary.
large language models LLMs 3D meshes spatial knowledge textual sources vertex coordinates face definitions LLaMA-Mesh supervised fine-tuning SFT mesh generation interleaved outputs mesh understanding text-based format