M3: 3D-Spatial MultiModal Memory
We present 3D Spatial MultiModal Memory (M3), a multimodal memory system
designed to retain information about medium-sized static scenes through video
sources for visual perception. By integrating 3D Gaussian Splatting techniques
with foundation models, M3 builds a multimodal memory capable of rendering
feature representations across granularities, encompassing a wide range of
knowledge. In our exploration, we identify two key challenges in previous works
on feature splatting: (1) computational constraints in storing high-dimensional
features for each Gaussian primitive, and (2) misalignment or information loss
between distilled features and foundation model features. To address these
challenges, we propose M3 with key components of principal scene components and
Gaussian memory attention, enabling efficient training and inference. To
validate M3, we conduct comprehensive quantitative evaluations of feature
similarity and downstream tasks, as well as qualitative visualizations to
highlight the pixel trace of Gaussian memory attention. Our approach
encompasses a diverse range of foundation models, including vision-language
models (VLMs), perception models, and large multimodal and language models
(LMMs/LLMs). Furthermore, to demonstrate real-world applicability, we deploy
M3's feature field in indoor scenes on a quadruped robot. Notably, we claim
that M3 is the first work to address the core compression challenges in 3D
feature distillation.
67dccb8ca33f11a56567bd61
Xueyan Zou
67dccb8ca33f11a56567bd62
Yuchen Song
67dccb8ca33f11a56567bd63
Ri-Zhao Qiu
67dccb8ca33f11a56567bd64
Xuanbin Peng
67dccb8ca33f11a56567bd65
Jianglong Ye
67dccb8ca33f11a56567bd66
Sifei Liu
67dccb8ca33f11a56567bd67
Xiaolong Wang
M3: 3D-Spatial MultiModal Memory
We present 3D Spatial MultiModal Memory (M3), a multimodal memory system
designed to retain information about medium-sized static scenes through video
sources for visual perception. By integrating 3D Gaussian Splatting techniques
with foundation models, M3 builds a multimodal memory capable of rendering
feature representations across granularities, encompassing a wide range of
knowledge. In our exploration, we identify two key challenges in previous works
on feature splatting: (1) computational constraints in storing high-dimensional
features for each Gaussian primitive, and (2) misalignment or information loss
between distilled features and foundation model features. To address these
challenges, we propose M3 with key components of principal scene components and
Gaussian memory attention, enabling efficient training and inference. To
validate M3, we conduct comprehensive quantitative evaluations of feature
similarity and downstream tasks, as well as qualitative visualizations to
highlight the pixel trace of Gaussian memory attention. Our approach
encompasses a diverse range of foundation models, including vision-language
models (VLMs), perception models, and large multimodal and language models
(LMMs/LLMs). Furthermore, to demonstrate real-world applicability, we deploy
M3's feature field in indoor scenes on a quadruped robot. Notably, we claim
that M3 is the first work to address the core compression challenges in 3D
feature distillation.
M3, a multimodal memory system combining 3D Gaussian Splatting and foundation models, addresses compression and alignment issues for high-dimensional scene features, enhancing efficiency and real-world applicability.
3D Spatial MultiModal Memory M3 3D Gaussian Splatting multimodal memory feature representations Gaussian primitives principal scene components Gaussian memory attention vision-language models perception models large multimodal models large language models