ISDrama: Immersive Spatial Drama Generation through Multimodal Prompting
Multimodal immersive spatial drama generation focuses on creating continuous
multi-speaker binaural speech with dramatic prosody based on multimodal
prompts, with potential applications in AR, VR, and others. This task requires
simultaneous modeling of spatial information and dramatic prosody based on
multimodal inputs, with high data collection costs. To the best of our
knowledge, our work is the first attempt to address these challenges. We
construct MRSDrama, the first multimodal recorded spatial drama dataset,
containing binaural drama audios, scripts, videos, geometric poses, and textual
prompts. Then, we propose ISDrama, the first immersive spatial drama generation
model through multimodal prompting. ISDrama comprises these primary components:
1) Multimodal Pose Encoder, based on contrastive learning, considering the
Doppler effect caused by moving speakers to extract unified pose information
from multimodal prompts. 2) Immersive Drama Transformer, a flow-based
mamba-transformer model that generates high-quality drama, incorporating
Drama-MOE to select proper experts for enhanced prosody and pose control. We
also design a context-consistent classifier-free guidance strategy to
coherently generate complete drama. Experimental results show that ISDrama
outperforms baseline models on objective and subjective metrics. The demos and
dataset are available at https://aaronz345.github.io/ISDramaDemo.
6811ea47f8ca0d9acb45374b
Yu Zhang
claimed_verified
2025-04-30T09:56:39.194Z
66569729ea21cfae5f5797c4
https://cdn-avatars.huggingface.co/v1/production/uploads/66569729ea21cfae5f5797c4/IguwJzljFN3QiEd1bn5BP.jpeg
Yu Zhang
user
AaronZ345
6811ea47f8ca0d9acb45374c
Wenxiang Guo
admin_assigned
2025-04-30T12:22:24.333Z
66692faa83408bb0da40f8f5
/avatars/09f81a8b0bb130c8fa72dbb6526ac4c4.svg
wenxiang guo
user
verstar
6811ea47f8ca0d9acb45374d
Changhao Pan
admin_assigned
2025-04-30T12:22:36.396Z
66568060c6a8cb4e884be331
https://cdn-avatars.huggingface.co/v1/production/uploads/66568060c6a8cb4e884be331/jx8NsxV374oURta6JdTzU.jpeg
PanChanghao
user
DavidPigeon
6811ea47f8ca0d9acb45374e
Zhiyuan Zhu
admin_assigned
2025-04-30T12:22:42.999Z
66984c6a03421b737aa78f24
/avatars/9c92b65469c0ef5f05a4c3c45a6dec97.svg
Zhiyuan Zhu
user
zhiyuanzhu2019
6811ea47f8ca0d9acb45374f
Tao Jin
6811ea47f8ca0d9acb453750
Zhou Zhao
ISDrama: Immersive Spatial Drama Generation through Multimodal Prompting
Multimodal immersive spatial drama generation focuses on creating continuous
multi-speaker binaural speech with dramatic prosody based on multimodal
prompts, with potential applications in AR, VR, and others. This task requires
simultaneous modeling of spatial information and dramatic prosody based on
multimodal inputs, with high data collection costs. To the best of our
knowledge, our work is the first attempt to address these challenges. We
construct MRSDrama, the first multimodal recorded spatial drama dataset,
containing binaural drama audios, scripts, videos, geometric poses, and textual
prompts. Then, we propose ISDrama, the first immersive spatial drama generation
model through multimodal prompting. ISDrama comprises these primary components:
1) Multimodal Pose Encoder, based on contrastive learning, considering the
Doppler effect caused by moving speakers to extract unified pose information
from multimodal prompts. 2) Immersive Drama Transformer, a flow-based
mamba-transformer model that generates high-quality drama, incorporating
Drama-MOE to select proper experts for enhanced prosody and pose control. We
also design a context-consistent classifier-free guidance strategy to
coherently generate complete drama. Experimental results show that ISDrama
outperforms baseline models on objective and subjective metrics. The demos and
dataset are available at https://aaronz345.github.io/ISDramaDemo.
A multimodal spatial drama generation model, ISDrama, leverages contrastive learning and a flow-based transformer to produce high-quality binaural speech with dramatic prosody from multimodal inputs.
multipodal Pose Encoder contrastive learning Immersive Drama Transformer flow-based transformer Drama-MOE context-consistent classifier-free guidance