3D-GRAND: A Million-Scale Dataset for 3D-LLMs with Better Grounding and
  Less Hallucination
The integration of language and 3D perception is crucial for developing
embodied agents and robots that comprehend and interact with the physical
world. While large language models (LLMs) have demonstrated impressive language
understanding and generation capabilities, their adaptation to 3D environments
(3D-LLMs) remains in its early stages. A primary challenge is the absence of
large-scale datasets that provide dense grounding between language and 3D
scenes. In this paper, we introduce 3D-GRAND, a pioneering large-scale dataset
comprising 40,087 household scenes paired with 6.2 million densely-grounded
scene-language instructions. Our results show that instruction tuning with
3D-GRAND significantly enhances grounding capabilities and reduces
hallucinations in 3D-LLMs. As part of our contributions, we propose a
comprehensive benchmark 3D-POPE to systematically evaluate hallucination in
3D-LLMs, enabling fair comparisons among future models. Our experiments
highlight a scaling effect between dataset size and 3D-LLM performance,
emphasizing the critical role of large-scale 3D-text datasets in advancing
embodied AI research. Notably, our results demonstrate early signals for
effective sim-to-real transfer, indicating that models trained on large
synthetic data can perform well on real-world 3D scans. Through 3D-GRAND and
3D-POPE, we aim to equip the embodied AI community with essential resources and
insights, setting the stage for more reliable and better-grounded 3D-LLMs.
Project website: https://3d-grand.github.io
666a643200bacf0820a4c016
Jianing Yang
claimed_verified
2024-06-13T09:31:01.780Z
646afdd535c7a57f936d3ff5
https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/LsDQPftQrnp_Gvanzim4J.jpeg
Jed Yang
user
jedyang97
666a643200bacf0820a4c017
Xuweiyi Chen
admin_assigned
2024-06-13T20:01:02.826Z
634632aaac1cb29fb2ac9f14
https://cdn-avatars.huggingface.co/v1/production/uploads/634632aaac1cb29fb2ac9f14/nGZ2TzKOOcKMAR_NFYKkR.jpeg
Xuweiyi Chen
user
Xuweiyi
666a643200bacf0820a4c018
Nikhil Madaan
admin_assigned
2024-06-13T20:01:09.150Z
646be576942fd7c135dde4f7
https://cdn-avatars.huggingface.co/v1/production/uploads/646be576942fd7c135dde4f7/89f6Vgx_AlaiHM0MjpUdY.jpeg
Nikhil Madaan
user
nmadaan
666a643200bacf0820a4c019
Madhavan Iyengar
admin_assigned
2024-06-13T20:01:14.622Z
6529b8c1789d84df6fe88bec
/avatars/7a435c9ab2cf5d003d1a183229c19b6f.svg
Madhavan Iyengar
user
miyen
666a643200bacf0820a4c01a
Shengyi Qian
admin_assigned
2024-06-13T20:01:20.041Z
6445dc7e219dc6142a6a6861
https://cdn-avatars.huggingface.co/v1/production/uploads/6445dc7e219dc6142a6a6861/Ak_YV8V1NZpkF5AQnJVlJ.png
Shengyi Qian
user
shengyi-qian
666a643200bacf0820a4c01b
David F. Fouhey
666a643200bacf0820a4c01c
Joyce Chai
3D-GRAND: A Million-Scale Dataset for 3D-LLMs with Better Grounding and
  Less Hallucination
The integration of language and 3D perception is crucial for developing
embodied agents and robots that comprehend and interact with the physical
world. While large language models (LLMs) have demonstrated impressive language
understanding and generation capabilities, their adaptation to 3D environments
(3D-LLMs) remains in its early stages. A primary challenge is the absence of
large-scale datasets that provide dense grounding between language and 3D
scenes. In this paper, we introduce 3D-GRAND, a pioneering large-scale dataset
comprising 40,087 household scenes paired with 6.2 million densely-grounded
scene-language instructions. Our results show that instruction tuning with
3D-GRAND significantly enhances grounding capabilities and reduces
hallucinations in 3D-LLMs. As part of our contributions, we propose a
comprehensive benchmark 3D-POPE to systematically evaluate hallucination in
3D-LLMs, enabling fair comparisons among future models. Our experiments
highlight a scaling effect between dataset size and 3D-LLM performance,
emphasizing the critical role of large-scale 3D-text datasets in advancing
embodied AI research. Notably, our results demonstrate early signals for
effective sim-to-real transfer, indicating that models trained on large
synthetic data can perform well on real-world 3D scans. Through 3D-GRAND and
3D-POPE, we aim to equip the embodied AI community with essential resources and
insights, setting the stage for more reliable and better-grounded 3D-LLMs.
Project website: https://3d-grand.github.io
3D-GRAND, a large-scale dataset, enhances 3D-LLM performance and hallucination reduction through instruction tuning and establishes a benchmark for sim-to-real transfer.
large language models 3D-LLMs 3D-GRAND 3D-POPE scene-language instructions grounding capabilities hallucination benchmark sim-to-real transfer