3D-LLM: Injecting the 3D World into Large Language Models
Large language models (LLMs) and Vision-Language Models (VLMs) have been
proven to excel at multiple tasks, such as commonsense reasoning. Powerful as
these models can be, they are not grounded in the 3D physical world, which
involves richer concepts such as spatial relationships, affordances, physics,
layout, and so on. In this work, we propose to inject the 3D world into large
language models and introduce a whole new family of 3D-LLMs. Specifically,
3D-LLMs can take 3D point clouds and their features as input and perform a
diverse set of 3D-related tasks, including captioning, dense captioning, 3D
question answering, task decomposition, 3D grounding, 3D-assisted dialog,
navigation, and so on. Using three types of prompting mechanisms that we
design, we are able to collect over 300k 3D-language data covering these tasks.
To efficiently train 3D-LLMs, we first utilize a 3D feature extractor that
obtains 3D features from rendered multi- view images. Then, we use 2D VLMs as
our backbones to train our 3D-LLMs. By introducing a 3D localization mechanism,
3D-LLMs can better capture 3D spatial information. Experiments on ScanQA show
that our model outperforms state-of-the-art baselines by a large margin (e.g.,
the BLEU-1 score surpasses state-of-the-art score by 9%). Furthermore,
experiments on our held-in datasets for 3D captioning, task composition, and
3D-assisted dialogue show that our model outperforms 2D VLMs. Qualitative
examples also show that our model could perform more tasks beyond the scope of
existing LLMs and VLMs. Project Page: : https://vis-www.cs.umass.edu/3dllm/.
64bf5732796f20daadaa7b78
Yining Hong
admin_assigned
2023-07-25T08:27:15.730Z
6431b64df76c34519e93d1ba
/avatars/ea577762b6b4798f87a7a3f1d53d082c.svg
Yining Hong
user
evelynhong
64bf5732796f20daadaa7b79
Haoyu Zhen
admin_assigned
2023-07-25T08:24:41.928Z
6437c7dae282b4a48eaf065e
https://cdn-avatars.huggingface.co/v1/production/uploads/6437c7dae282b4a48eaf065e/AxodKQXyrviTFQRyjnL01.jpeg
Haoyu Zhen
user
anyeZHY
64bf5732796f20daadaa7b7a
Peihao Chen
claimed_verified
2023-07-26T08:43:57.546Z
64bfe895c874dda773303b44
/avatars/a54ddb6d69a73f63f9d848f28abcbff3.svg
PeihaoChen
user
PeihaoChen
64bf5732796f20daadaa7b7b
Shuhong Zheng
admin_assigned
2023-07-25T08:25:54.807Z
64361417a4bd75c62cc1e534
/avatars/b9b43985a0190ccf418c6853f9c3eb16.svg
Shuhong Zheng
user
ShuhongZheng
64bf5732796f20daadaa7b7c
Yilun Du
admin_assigned
2023-07-25T08:26:20.418Z
63c9bd445fdc575773c732fe
/avatars/def472d1ab3fbf751225357c0932ae7e.svg
Yilun Du
user
yilundu
64bf5732796f20daadaa7b7d
Zhenfang Chen
64bf5732796f20daadaa7b7e
Chuang Gan
3D-LLM: Injecting the 3D World into Large Language Models
Large language models (LLMs) and Vision-Language Models (VLMs) have been
proven to excel at multiple tasks, such as commonsense reasoning. Powerful as
these models can be, they are not grounded in the 3D physical world, which
involves richer concepts such as spatial relationships, affordances, physics,
layout, and so on. In this work, we propose to inject the 3D world into large
language models and introduce a whole new family of 3D-LLMs. Specifically,
3D-LLMs can take 3D point clouds and their features as input and perform a
diverse set of 3D-related tasks, including captioning, dense captioning, 3D
question answering, task decomposition, 3D grounding, 3D-assisted dialog,
navigation, and so on. Using three types of prompting mechanisms that we
design, we are able to collect over 300k 3D-language data covering these tasks.
To efficiently train 3D-LLMs, we first utilize a 3D feature extractor that
obtains 3D features from rendered multi- view images. Then, we use 2D VLMs as
our backbones to train our 3D-LLMs. By introducing a 3D localization mechanism,
3D-LLMs can better capture 3D spatial information. Experiments on ScanQA show
that our model outperforms state-of-the-art baselines by a large margin (e.g.,
the BLEU-1 score surpasses state-of-the-art score by 9%). Furthermore,
experiments on our held-in datasets for 3D captioning, task composition, and
3D-assisted dialogue show that our model outperforms 2D VLMs. Qualitative
examples also show that our model could perform more tasks beyond the scope of
existing LLMs and VLMs. Project Page: : https://vis-www.cs.umass.edu/3dllm/.
A new family of 3D-LLMs is introduced to perform 3D-related tasks by leveraging 3D point clouds and features, outperforming state-of-the-art baselines in tasks such as 3D question answering and captioning.
LLMs Vision-Language Models VLMs 3D world 3D point clouds 3D-related tasks captioning dense captioning 3D question answering task decomposition 3D grounding 3D-assisted dialog navigation 3D feature extractor 3D localization mechanism ScanQA BLEU-1 score 3D captioning task composition 3D-assisted dialogue