SOLAMI: Social Vision-Language-Action Modeling for Immersive Interaction
  with 3D Autonomous Characters
Human beings are social animals. How to equip 3D autonomous characters with
similar social intelligence that can perceive, understand and interact with
humans remains an open yet foundamental problem. In this paper, we introduce
SOLAMI, the first end-to-end Social vision-Language-Action (VLA) Modeling
framework for Immersive interaction with 3D autonomous characters.
Specifically, SOLAMI builds 3D autonomous characters from three aspects: (1)
Social VLA Architecture: We propose a unified social VLA framework to generate
multimodal response (speech and motion) based on the user's multimodal input to
drive the character for social interaction. (2) Interactive Multimodal Data: We
present SynMSI, a synthetic multimodal social interaction dataset generated by
an automatic pipeline using only existing motion datasets to address the issue
of data scarcity. (3) Immersive VR Interface: We develop a VR interface that
enables users to immersively interact with these characters driven by various
architectures. Extensive quantitative experiments and user studies demonstrate
that our framework leads to more precise and natural character responses (in
both speech and motion) that align with user expectations with lower latency.
674eadabef38dd7bfb8863cf
Jianping Jiang
674eadabef38dd7bfb8863d0
Weiye Xiao
674eadabef38dd7bfb8863d1
Zhengyu Lin
admin_assigned
2024-12-03T15:18:17.310Z
6391fc80077fee9fbca29790
/avatars/53f89c6de4e91cdcaaa52a295a5f4ea4.svg
Zhengyu Lin
user
hayashiLin
674eadabef38dd7bfb8863d2
Huaizhong Zhang
674eadabef38dd7bfb8863d3
Tianxiang Ren
admin_assigned
2024-12-03T15:18:45.694Z
66652269d22ee88501a25264
/avatars/5680b90d20e179be4e56ceabb3aa145d.svg
Tianxiang Ren
user
rentxray
674eadabef38dd7bfb8863d4
Yang Gao
674eadabef38dd7bfb8863d5
Zhiqian Lin
674eadabef38dd7bfb8863d6
Zhongang Cai
claimed_verified
2024-12-03T09:56:37.337Z
652d06833b5997ed71ce5c46
https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/xZTXEcnEogEmBm_ledJQr.jpeg
Zhongang Cai
user
caizhongang
674eadabef38dd7bfb8863d7
Lei Yang
claimed_verified
2024-12-03T14:40:45.662Z
6626a471430a124253f197c8
/avatars/f5747fdbe495d1296fed9d16d8c95857.svg
yl-1993
user
yl-1993
674eadabef38dd7bfb8863d8
Ziwei Liu
admin_assigned
2024-12-03T15:20:46.734Z
62ab1ac1d48b4d8b048a3473
https://cdn-avatars.huggingface.co/v1/production/uploads/1656826685333-62ab1ac1d48b4d8b048a3473.png
Ziwei Liu
user
liuziwei7
SOLAMI: Social Vision-Language-Action Modeling for Immersive Interaction
  with 3D Autonomous Characters
Human beings are social animals. How to equip 3D autonomous characters with
similar social intelligence that can perceive, understand and interact with
humans remains an open yet foundamental problem. In this paper, we introduce
SOLAMI, the first end-to-end Social vision-Language-Action (VLA) Modeling
framework for Immersive interaction with 3D autonomous characters.
Specifically, SOLAMI builds 3D autonomous characters from three aspects: (1)
Social VLA Architecture: We propose a unified social VLA framework to generate
multimodal response (speech and motion) based on the user's multimodal input to
drive the character for social interaction. (2) Interactive Multimodal Data: We
present SynMSI, a synthetic multimodal social interaction dataset generated by
an automatic pipeline using only existing motion datasets to address the issue
of data scarcity. (3) Immersive VR Interface: We develop a VR interface that
enables users to immersively interact with these characters driven by various
architectures. Extensive quantitative experiments and user studies demonstrate
that our framework leads to more precise and natural character responses (in
both speech and motion) that align with user expectations with lower latency.
SOLAMI is an end-to-end VLA framework for 3D autonomous characters that addresses social interaction by generating multimodal responses and enabling immersive VR interaction.
Social VLA Architecture SynMSI VR interface multimodal response multimodal input social interaction user expectations quantitative experiments user studies