Task-oriented Sequential Grounding in 3D Scenes
Grounding natural language in physical 3D environments is essential for the
advancement of embodied artificial intelligence. Current datasets and models
for 3D visual grounding predominantly focus on identifying and localizing
objects from static, object-centric descriptions. These approaches do not
adequately address the dynamic and sequential nature of task-oriented grounding
necessary for practical applications. In this work, we propose a new task:
Task-oriented Sequential Grounding in 3D scenes, wherein an agent must follow
detailed step-by-step instructions to complete daily activities by locating a
sequence of target objects in indoor scenes. To facilitate this task, we
introduce SG3D, a large-scale dataset containing 22,346 tasks with 112,236
steps across 4,895 real-world 3D scenes. The dataset is constructed using a
combination of RGB-D scans from various 3D scene datasets and an automated task
generation pipeline, followed by human verification for quality assurance. We
adapted three state-of-the-art 3D visual grounding models to the sequential
grounding task and evaluated their performance on SG3D. Our results reveal that
while these models perform well on traditional benchmarks, they face
significant challenges with task-oriented sequential grounding, underscoring
the need for further research in this area.
66b5771021975ba10d9af9c5
Zhuofan Zhang
admin_assigned
2024-08-09T10:06:49.372Z
663cac5ba4d7c8c9038997d0
/avatars/f91c5b0f0a1f081b7fa8a37484dc8fbe.svg
Zhang
user
ZhuofanZhang
66b5771021975ba10d9af9c6
Ziyu Zhu
66b5771021975ba10d9af9c7
Pengxiang Li
claimed_verified
2024-08-09T07:47:32.317Z
65d6bd8ebb862d1b367580fe
https://cdn-avatars.huggingface.co/v1/production/uploads/65d6bd8ebb862d1b367580fe/Pf5o-Oxi2IZxJPa-ZgEtV.jpeg
Pengxiang Li
user
PengxiangLi
66b5771021975ba10d9af9c8
Tengyu Liu
admin_assigned
2024-08-09T10:08:22.978Z
634379a81bdd3dfa55dcbe82
https://cdn-avatars.huggingface.co/v1/production/uploads/1665366425318-noauth.png
Tengyu Liu
user
oxrider
66b5771021975ba10d9af9c9
Xiaojian Ma
admin_assigned
2024-08-09T10:08:28.518Z
60dd0e36a15ddd7d2006d2e9
/avatars/8bd98177a79efbf295be8f6457683297.svg
Xiaojian Ma
user
jeasinema
66b5771021975ba10d9af9ca
Yixin Chen
66b5771021975ba10d9af9cb
Baoxiong Jia
admin_assigned
2024-08-09T10:08:55.418Z
6304b389bad6ce7fc02691d5
/avatars/a762ca59624ce409650165f36b973488.svg
Baoxiong Jia
user
BuzzBeater
66b5771021975ba10d9af9cc
Siyuan Huang
admin_assigned
2024-08-09T10:09:11.202Z
63c7a33121bd95f80ed74652
/avatars/7dd59afea785a2bff0ec2b757abd474e.svg
Siyuan Huang
user
thuhsy
66b5771021975ba10d9af9cd
Qing Li
claimed_verified
2024-08-09T14:52:35.788Z
6455f8c2c8f569b995d603c9
/avatars/3cb2cad3ab123887c270234bb6a4ca43.svg
Qing Li
user
li-qing
Task-oriented Sequential Grounding in 3D Scenes
Grounding natural language in physical 3D environments is essential for the
advancement of embodied artificial intelligence. Current datasets and models
for 3D visual grounding predominantly focus on identifying and localizing
objects from static, object-centric descriptions. These approaches do not
adequately address the dynamic and sequential nature of task-oriented grounding
necessary for practical applications. In this work, we propose a new task:
Task-oriented Sequential Grounding in 3D scenes, wherein an agent must follow
detailed step-by-step instructions to complete daily activities by locating a
sequence of target objects in indoor scenes. To facilitate this task, we
introduce SG3D, a large-scale dataset containing 22,346 tasks with 112,236
steps across 4,895 real-world 3D scenes. The dataset is constructed using a
combination of RGB-D scans from various 3D scene datasets and an automated task
generation pipeline, followed by human verification for quality assurance. We
adapted three state-of-the-art 3D visual grounding models to the sequential
grounding task and evaluated their performance on SG3D. Our results reveal that
while these models perform well on traditional benchmarks, they face
significant challenges with task-oriented sequential grounding, underscoring
the need for further research in this area.
A new dataset and task for sequential task-oriented grounding in 3D environments are introduced to advance embodied AI, showcasing challenges for current models in dynamic, instruction-following tasks.
3D visual grounding task-oriented sequential grounding SG3D RGB-D scans automated task generation human verification state-of-the-art 3D visual grounding models