Exploring the Potential of Encoder-free Architectures in 3D LMMs
Encoder-free architectures have been preliminarily explored in the 2D visual
domain, yet it remains an open question whether they can be effectively applied
to 3D understanding scenarios. In this paper, we present the first
comprehensive investigation into the potential of encoder-free architectures to
overcome the challenges of encoder-based 3D Large Multimodal Models (LMMs).
These challenges include the failure to adapt to varying point cloud
resolutions and the point features from the encoder not meeting the semantic
needs of Large Language Models (LLMs). We identify key aspects for 3D LMMs to
remove the encoder and enable the LLM to assume the role of the 3D encoder: 1)
We propose the LLM-embedded Semantic Encoding strategy in the pre-training
stage, exploring the effects of various point cloud self-supervised losses. And
we present the Hybrid Semantic Loss to extract high-level semantics. 2) We
introduce the Hierarchical Geometry Aggregation strategy in the instruction
tuning stage. This incorporates inductive bias into the LLM early layers to
focus on the local details of the point clouds. To the end, we present the
first Encoder-free 3D LMM, ENEL. Our 7B model rivals the current
state-of-the-art model, ShapeLLM-13B, achieving 55.0%, 50.92%, and 42.7% on the
classification, captioning, and VQA tasks, respectively. Our results
demonstrate that the encoder-free architecture is highly promising for
replacing encoder-based architectures in the field of 3D understanding. The
code is released at https://github.com/Ivan-Tang-3D/ENEL
67aeec91b1bbfb68824df5d1
Yiwen Tang
claimed_verified
2025-02-14T08:00:57.216Z
6552f1ad5d55ccb20e9142a0
/avatars/0e3e80cba64b5ae0bc5638694ac33dbf.svg
Ivan Tang
user
IvanTang
67aeec91b1bbfb68824df5d2
Zoey Guo
admin_assigned
2025-02-14T12:02:40.282Z
642a8302d651bae3c11b72b1
/avatars/4d2d422613e274d80482fed9a7d3f785.svg
Zoey Guo
user
Purple1288
67aeec91b1bbfb68824df5d3
Zhuhao Wang
admin_assigned
2025-02-14T12:02:47.566Z
672489e32fd598f719be33ba
/avatars/63a6edf7bf38957e029fa52c1a7f9061.svg
Zhuhao Wang
user
zhuhaow
67aeec91b1bbfb68824df5d4
Ray Zhang
67aeec91b1bbfb68824df5d5
Qizhi Chen
admin_assigned
2025-02-14T12:03:12.952Z
6535045a910b844786a6642f
/avatars/37a94864a7a348151837b421ea6d77e3.svg
Qizhi Chen
user
Tavish9
67aeec91b1bbfb68824df5d6
Junli Liu
claimed_verified
2025-03-13T08:25:04.055Z
656850a60e4b5ff9d552e825
/avatars/450281494d8e683a60799bb8d4ddfff1.svg
Junli Liu
user
Ideallll
67aeec91b1bbfb68824df5d7
Delin Qu
claimed_verified
2025-02-14T08:00:55.263Z
64daecec888b7e9c400f59b5
https://cdn-avatars.huggingface.co/v1/production/uploads/64daecec888b7e9c400f59b5/f4pfOfWk6jYJX-Nf2-qHn.png
Delin Qu
user
delinqu
67aeec91b1bbfb68824df5d8
Zhigang Wang
67aeec91b1bbfb68824df5d9
Dong Wang
67aeec91b1bbfb68824df5da
Xuelong Li
67aeec91b1bbfb68824df5db
Bin Zhao
Exploring the Potential of Encoder-free Architectures in 3D LMMs
Encoder-free architectures have been preliminarily explored in the 2D visual
domain, yet it remains an open question whether they can be effectively applied
to 3D understanding scenarios. In this paper, we present the first
comprehensive investigation into the potential of encoder-free architectures to
overcome the challenges of encoder-based 3D Large Multimodal Models (LMMs).
These challenges include the failure to adapt to varying point cloud
resolutions and the point features from the encoder not meeting the semantic
needs of Large Language Models (LLMs). We identify key aspects for 3D LMMs to
remove the encoder and enable the LLM to assume the role of the 3D encoder: 1)
We propose the LLM-embedded Semantic Encoding strategy in the pre-training
stage, exploring the effects of various point cloud self-supervised losses. And
we present the Hybrid Semantic Loss to extract high-level semantics. 2) We
introduce the Hierarchical Geometry Aggregation strategy in the instruction
tuning stage. This incorporates inductive bias into the LLM early layers to
focus on the local details of the point clouds. To the end, we present the
first Encoder-free 3D LMM, ENEL. Our 7B model rivals the current
state-of-the-art model, ShapeLLM-13B, achieving 55.0%, 50.92%, and 42.7% on the
classification, captioning, and VQA tasks, respectively. Our results
demonstrate that the encoder-free architecture is highly promising for
replacing encoder-based architectures in the field of 3D understanding. The
code is released at https://github.com/Ivan-Tang-3D/ENEL
An encoder-free architecture, ENEL, demonstrated competitive performance in 3D understanding tasks through LLM-embedded Semantic Encoding and Hierarchical Geometry Aggregation.
encoder-free architectures Large Multimodal Models (LMMs) point cloud resolutions Large Language Models (LLMs) Semantic Encoding strategy point cloud self-supervised losses Hybrid Semantic Loss Hierarchical Geometry Aggregation strategy ENEL classification captioning VQA tasks ShapeLLM-13B