3DGraphLLM: Combining Semantic Graphs and Large Language Models for 3D
  Scene Understanding
A 3D scene graph represents a compact scene model, storing information about
the objects and the semantic relationships between them, making its use
promising for robotic tasks. When interacting with a user, an embodied
intelligent agent should be capable of responding to various queries about the
scene formulated in natural language. Large Language Models (LLMs) are
beneficial solutions for user-robot interaction due to their natural language
understanding and reasoning abilities. Recent methods for creating learnable
representations of 3D scenes have demonstrated the potential to improve the
quality of LLMs responses by adapting to the 3D world. However, the existing
methods do not explicitly utilize information about the semantic relationships
between objects, limiting themselves to information about their coordinates. In
this work, we propose a method 3DGraphLLM for constructing a learnable
representation of a 3D scene graph. The learnable representation is used as
input for LLMs to perform 3D vision-language tasks. In our experiments on
popular ScanRefer, RIORefer, Multi3DRefer, ScanQA, Sqa3D, and Scan2cap
datasets, we demonstrate the advantage of this approach over baseline methods
that do not use information about the semantic relationships between objects.
The code is publicly available at
https://github.com/CognitiveAISystems/3DGraphLLM.
676bbe579484d105b89dba3b
Tatiana Zemskova
claimed_verified
2024-12-30T19:32:37.677Z
6363767e572fd34304f49a67
/avatars/a9fc92b6005d48adf71a45bebf812648.svg
Tatiana Zemskova
user
wingrune
676bbe579484d105b89dba3c
Dmitry Yudin
claimed_verified
2024-12-30T19:32:35.428Z
67585775ab7e701b5754649f
/avatars/7ccaf915b6b409e49d35f5212b8293c3.svg
Dmitry Yudin
user
yuddim
3DGraphLLM: Combining Semantic Graphs and Large Language Models for 3D
  Scene Understanding
A 3D scene graph represents a compact scene model, storing information about
the objects and the semantic relationships between them, making its use
promising for robotic tasks. When interacting with a user, an embodied
intelligent agent should be capable of responding to various queries about the
scene formulated in natural language. Large Language Models (LLMs) are
beneficial solutions for user-robot interaction due to their natural language
understanding and reasoning abilities. Recent methods for creating learnable
representations of 3D scenes have demonstrated the potential to improve the
quality of LLMs responses by adapting to the 3D world. However, the existing
methods do not explicitly utilize information about the semantic relationships
between objects, limiting themselves to information about their coordinates. In
this work, we propose a method 3DGraphLLM for constructing a learnable
representation of a 3D scene graph. The learnable representation is used as
input for LLMs to perform 3D vision-language tasks. In our experiments on
popular ScanRefer, RIORefer, Multi3DRefer, ScanQA, Sqa3D, and Scan2cap
datasets, we demonstrate the advantage of this approach over baseline methods
that do not use information about the semantic relationships between objects.
The code is publicly available at
https://github.com/CognitiveAISystems/3DGraphLLM.
A 3D scene graph representation integrated with large language models enhances user-robot interaction by incorporating semantic relationships between objects, leading to improved performance in 3D vision-language tasks compared to baselines.
3D scene graph learnable representation large language models semantic relationships coordinates ScanRefer RIORefer Multi3DRefer ScanQA Sqa3D Scan2cap 3D vision-language tasks