Robin3D: Improving 3D Large Language Model via Robust Instruction Tuning
Recent advancements in 3D Large Language Models (3DLLMs) have highlighted
their potential in building general-purpose agents in the 3D real world, yet
challenges remain due to the lack of high-quality robust instruction-following
data, leading to limited discriminative power and generalization of 3DLLMs. In
this paper, we introduce Robin3D, a powerful 3DLLM trained on large-scale
instruction-following data generated by our novel data engine, Robust
Instruction Generation (RIG) engine. RIG generates two key instruction data: 1)
the Adversarial Instruction-following data, which features mixed negative and
positive samples to enhance the model's discriminative understanding. 2) the
Diverse Instruction-following data, which contains various instruction styles
to enhance model's generalization. As a result, we construct 1 million
instruction-following data, consisting of 344K Adversarial samples, 508K
Diverse samples, and 165K benchmark training set samples. To better handle
these complex instructions, Robin3D first incorporates Relation-Augmented
Projector to enhance spatial understanding, and then strengthens the object
referring and grounding ability through ID-Feature Bonding. Robin3D
consistently outperforms previous methods across five widely-used 3D multimodal
learning benchmarks, without the need for task-specific fine-tuning. Notably,
we achieve a 7.8\% improvement in the grounding task (Multi3DRefer) and a 6.9\%
improvement in the captioning task (Scan2Cap).
66fee5e214b824c86560ce72
Weitai Kang
claimed_verified
2024-10-04T07:31:27.875Z
662050a24360f44332caf535
https://cdn-avatars.huggingface.co/v1/production/uploads/662050a24360f44332caf535/oVLyKvjCxgVV79taeRJPH.png
Weitai Kang
user
weitaikang
66fee5e214b824c86560ce73
Haifeng Huang
admin_assigned
2024-10-07T18:00:02.091Z
64d880e2bab152b247103a28
https://cdn-avatars.huggingface.co/v1/production/uploads/64d880e2bab152b247103a28/2OQV-TO-kGi2XurcdBH5h.jpeg
Haifeng Huang
user
ZzZZCHS
66fee5e214b824c86560ce74
Yuzhang Shang
admin_assigned
2024-10-07T18:00:08.240Z
63a740aec46291f273a95b4c
/avatars/8c19e639e39f93083ab624747f9fd0b3.svg
Yuzhang Shang
user
yuzhang
66fee5e214b824c86560ce75
Mubarak Shah
66fee5e214b824c86560ce76
Yan Yan
Robin3D: Improving 3D Large Language Model via Robust Instruction Tuning
Recent advancements in 3D Large Language Models (3DLLMs) have highlighted
their potential in building general-purpose agents in the 3D real world, yet
challenges remain due to the lack of high-quality robust instruction-following
data, leading to limited discriminative power and generalization of 3DLLMs. In
this paper, we introduce Robin3D, a powerful 3DLLM trained on large-scale
instruction-following data generated by our novel data engine, Robust
Instruction Generation (RIG) engine. RIG generates two key instruction data: 1)
the Adversarial Instruction-following data, which features mixed negative and
positive samples to enhance the model's discriminative understanding. 2) the
Diverse Instruction-following data, which contains various instruction styles
to enhance model's generalization. As a result, we construct 1 million
instruction-following data, consisting of 344K Adversarial samples, 508K
Diverse samples, and 165K benchmark training set samples. To better handle
these complex instructions, Robin3D first incorporates Relation-Augmented
Projector to enhance spatial understanding, and then strengthens the object
referring and grounding ability through ID-Feature Bonding. Robin3D
consistently outperforms previous methods across five widely-used 3D multimodal
learning benchmarks, without the need for task-specific fine-tuning. Notably,
we achieve a 7.8\% improvement in the grounding task (Multi3DRefer) and a 6.9\%
improvement in the captioning task (Scan2Cap).
Robin3D, a 3D Large Language Model trained on adversarial and diverse instruction-following data, enhances spatial understanding and object referencing, leading to improved performance across multiple 3D learning benchmarks without task-specific fine-tuning.
3D Large Language Models 3DLLMs Robust Instruction Generation RIG Adversarial Instruction-following data Diverse Instruction-following data Relation-Augmented Projector ID-Feature Bonding Multi3DRefer Scan2Cap