SceneVerse: Scaling 3D Vision-Language Learning for Grounded Scene
  Understanding
3D vision-language grounding, which focuses on aligning language with the 3D
physical environment, stands as a cornerstone in the development of embodied
agents. In comparison to recent advancements in the 2D domain, grounding
language in 3D scenes faces several significant challenges: (i) the inherent
complexity of 3D scenes due to the diverse object configurations, their rich
attributes, and intricate relationships; (ii) the scarcity of paired 3D
vision-language data to support grounded learning; and (iii) the absence of a
unified learning framework to distill knowledge from grounded 3D data. In this
work, we aim to address these three major challenges in 3D vision-language by
examining the potential of systematically upscaling 3D vision-language learning
in indoor environments. We introduce the first million-scale 3D vision-language
dataset, SceneVerse, encompassing about 68K 3D indoor scenes and comprising
2.5M vision-language pairs derived from both human annotations and our scalable
scene-graph-based generation approach. We demonstrate that this scaling allows
for a unified pre-training framework, Grounded Pre-training for Scenes (GPS),
for 3D vision-language learning. Through extensive experiments, we showcase the
effectiveness of GPS by achieving state-of-the-art performance on all existing
3D visual grounding benchmarks. The vast potential of SceneVerse and GPS is
unveiled through zero-shot transfer experiments in the challenging 3D
vision-language tasks. Project website: https://scene-verse.github.io .
65a8ab10f45ee5e5b56d5ca5
Baoxiong Jia
admin_assigned
2024-01-18T08:41:38.451Z
6304b389bad6ce7fc02691d5
/avatars/a762ca59624ce409650165f36b973488.svg
Baoxiong Jia
user
BuzzBeater
65a8ab10f45ee5e5b56d5ca6
Yixin Chen
admin_assigned
2024-01-18T08:42:00.150Z
64c396def6fe448b1ad553d6
/avatars/b2ce4739f42dc00ee974fff7ee1cb301.svg
Yixin Chen
user
YixinChen
65a8ab10f45ee5e5b56d5ca7
Huangyue Yu
admin_assigned
2024-01-18T08:43:13.830Z
65a8cbd4669921943c3c0963
/avatars/c85e1253b8e3ea71eef677d5cd356a55.svg
Yu
user
Huangyueeeee
65a8ab10f45ee5e5b56d5ca8
Yan Wang
claimed_verified
2024-01-18T08:19:33.225Z
65a8cc0516e8e332e7515d7d
/avatars/4a84d28730776962242db6129cf0c4c1.svg
Yan Wang
user
Yannnnnnnn
65a8ab10f45ee5e5b56d5ca9
Xuesong Niu
admin_assigned
2024-01-18T08:43:19.971Z
64620ca8a489ecb6b6b5b4e4
/avatars/9bf8060cb7bc59c0369ddf14c50aa480.svg
Xuesong Niu
user
nxsEdson
65a8ab10f45ee5e5b56d5caa
Tengyu Liu
admin_assigned
2024-01-18T08:43:27.043Z
634379a81bdd3dfa55dcbe82
https://cdn-avatars.huggingface.co/v1/production/uploads/1665366425318-noauth.png
Tengyu Liu
user
oxrider
65a8ab10f45ee5e5b56d5cab
Qing Li
claimed_verified
2024-08-03T08:26:27.371Z
6455f8c2c8f569b995d603c9
/avatars/3cb2cad3ab123887c270234bb6a4ca43.svg
Qing Li
user
li-qing
65a8ab10f45ee5e5b56d5cac
Siyuan Huang
admin_assigned
2024-01-18T08:44:13.905Z
63c7a33121bd95f80ed74652
/avatars/7dd59afea785a2bff0ec2b757abd474e.svg
Siyuan Huang
user
thuhsy
SceneVerse: Scaling 3D Vision-Language Learning for Grounded Scene
  Understanding
3D vision-language grounding, which focuses on aligning language with the 3D
physical environment, stands as a cornerstone in the development of embodied
agents. In comparison to recent advancements in the 2D domain, grounding
language in 3D scenes faces several significant challenges: (i) the inherent
complexity of 3D scenes due to the diverse object configurations, their rich
attributes, and intricate relationships; (ii) the scarcity of paired 3D
vision-language data to support grounded learning; and (iii) the absence of a
unified learning framework to distill knowledge from grounded 3D data. In this
work, we aim to address these three major challenges in 3D vision-language by
examining the potential of systematically upscaling 3D vision-language learning
in indoor environments. We introduce the first million-scale 3D vision-language
dataset, SceneVerse, encompassing about 68K 3D indoor scenes and comprising
2.5M vision-language pairs derived from both human annotations and our scalable
scene-graph-based generation approach. We demonstrate that this scaling allows
for a unified pre-training framework, Grounded Pre-training for Scenes (GPS),
for 3D vision-language learning. Through extensive experiments, we showcase the
effectiveness of GPS by achieving state-of-the-art performance on all existing
3D visual grounding benchmarks. The vast potential of SceneVerse and GPS is
unveiled through zero-shot transfer experiments in the challenging 3D
vision-language tasks. Project website: https://scene-verse.github.io .
A large-scale 3D vision-language dataset and pre-training framework achieve state-of-the-art results in 3D visual grounding tasks.
3D vision-language grounding grounded learning 3D scenes object configurations rich attributes intricate relationships paired 3D vision-language data unified learning framework SceneVerse vision-language pairs scene-graph-based generation Grounded Pre-training for Scenes GPS 3D visual grounding benchmarks zero-shot transfer