Video-3D LLM: Learning Position-Aware Video Representation for 3D Scene
  Understanding
The rapid advancement of Multimodal Large Language Models (MLLMs) has
significantly impacted various multimodal tasks. However, these models face
challenges in tasks that require spatial understanding within 3D environments.
Efforts to enhance MLLMs, such as incorporating point cloud features, have been
made, yet a considerable gap remains between the models' learned
representations and the inherent complexity of 3D scenes. This discrepancy
largely stems from the training of MLLMs on predominantly 2D data, which
restricts their effectiveness in comprehending 3D spaces. To address this
issue, in this paper, we propose a novel generalist model, i.e., Video-3D LLM,
for 3D scene understanding. By treating 3D scenes as dynamic videos and
incorporating 3D position encoding into these representations, our Video-3D LLM
aligns video representations with real-world spatial contexts more accurately.
Additionally, we have implemented a maximum coverage sampling technique to
optimize the balance between computational costs and performance efficiency.
Extensive experiments demonstrate that our model achieves state-of-the-art
performance on several 3D scene understanding benchmarks, including ScanRefer,
Multi3DRefer, Scan2Cap, ScanQA, and SQA3D.
675112f1d5d2963818c49068
Duo Zheng
claimed_verified
2024-12-05T09:18:28.356Z
646e2fcaf813cfe153f1af6c
/avatars/2f87d0e5c071000990da29cd744bc03d.svg
Duo Zheng
user
zd11024
675112f1d5d2963818c49069
Shijia Huang
admin_assigned
2024-12-05T10:41:36.198Z
65204fdf377bffdc59602a8d
/avatars/02d200f856f03696b183fd0b59e9f909.svg
Shijia Huang
user
slvjul
675112f1d5d2963818c4906a
Liwei Wang
admin_assigned
2024-12-05T10:41:42.748Z
6497272b6401266eb6752b89
/avatars/d94a011660f4cad5344756027840dc30.svg
Liwei Wang
user
asdfg80
Video-3D LLM: Learning Position-Aware Video Representation for 3D Scene
  Understanding
The rapid advancement of Multimodal Large Language Models (MLLMs) has
significantly impacted various multimodal tasks. However, these models face
challenges in tasks that require spatial understanding within 3D environments.
Efforts to enhance MLLMs, such as incorporating point cloud features, have been
made, yet a considerable gap remains between the models' learned
representations and the inherent complexity of 3D scenes. This discrepancy
largely stems from the training of MLLMs on predominantly 2D data, which
restricts their effectiveness in comprehending 3D spaces. To address this
issue, in this paper, we propose a novel generalist model, i.e., Video-3D LLM,
for 3D scene understanding. By treating 3D scenes as dynamic videos and
incorporating 3D position encoding into these representations, our Video-3D LLM
aligns video representations with real-world spatial contexts more accurately.
Additionally, we have implemented a maximum coverage sampling technique to
optimize the balance between computational costs and performance efficiency.
Extensive experiments demonstrate that our model achieves state-of-the-art
performance on several 3D scene understanding benchmarks, including ScanRefer,
Multi3DRefer, Scan2Cap, ScanQA, and SQA3D.
A Video-3D LLM enhances 3D scene understanding by integrating point cloud features and 3D position encoding into video representations, achieving state-of-the-art performance on benchmarks.
MLLMs 3D environment point cloud features 3D scene understanding Video-3D LLM dynamic videos 3D position encoding maximum coverage sampling