Taming Text-to-Sounding Video Generation via Advanced Modality Condition
  and Interaction
This study focuses on a challenging yet promising task,
Text-to-Sounding-Video (T2SV) generation, which aims to generate a video with
synchronized audio from text conditions, meanwhile ensuring both modalities are
aligned with text. Despite progress in joint audio-video training, two critical
challenges still remain unaddressed: (1) a single, shared text caption where
the text for video is equal to the text for audio often creates modal
interference, confusing the pretrained backbones, and (2) the optimal mechanism
for cross-modal feature interaction remains unclear. To address these
challenges, we first propose the Hierarchical Visual-Grounded Captioning (HVGC)
framework that generates pairs of disentangled captions, a video caption, and
an audio caption, eliminating interference at the conditioning stage. Based on
HVGC, we further introduce BridgeDiT, a novel dual-tower diffusion transformer,
which employs a Dual CrossAttention (DCA) mechanism that acts as a robust
``bridge" to enable a symmetric, bidirectional exchange of information,
achieving both semantic and temporal synchronization. Extensive experiments on
three benchmark datasets, supported by human evaluations, demonstrate that our
method achieves state-of-the-art results on most metrics. Comprehensive
ablation studies further validate the effectiveness of our contributions,
offering key insights for the future T2SV task. All the codes and checkpoints
will be publicly released.
68e8c3e895e8e6771df38c02
Kaisi Guan
claimed_verified
2025-10-10T08:54:17.605Z
63bbf071d8d676a2299c7d0b
/avatars/4bb1c86ef8651c75b9761afee2865267.svg
Guan
user
Guan123
68e8c3e895e8e6771df38c03
Xihua Wang
68e8c3e895e8e6771df38c04
Zhengfeng Lai
68e8c3e895e8e6771df38c05
Xin Cheng
68e8c3e895e8e6771df38c06
Peng Zhang
68e8c3e895e8e6771df38c07
XiaoJiang Liu
68e8c3e895e8e6771df38c08
Ruihua Song
68e8c3e895e8e6771df38c09
Meng Cao
Taming Text-to-Sounding Video Generation via Advanced Modality Condition
  and Interaction
This study focuses on a challenging yet promising task,
Text-to-Sounding-Video (T2SV) generation, which aims to generate a video with
synchronized audio from text conditions, meanwhile ensuring both modalities are
aligned with text. Despite progress in joint audio-video training, two critical
challenges still remain unaddressed: (1) a single, shared text caption where
the text for video is equal to the text for audio often creates modal
interference, confusing the pretrained backbones, and (2) the optimal mechanism
for cross-modal feature interaction remains unclear. To address these
challenges, we first propose the Hierarchical Visual-Grounded Captioning (HVGC)
framework that generates pairs of disentangled captions, a video caption, and
an audio caption, eliminating interference at the conditioning stage. Based on
HVGC, we further introduce BridgeDiT, a novel dual-tower diffusion transformer,
which employs a Dual CrossAttention (DCA) mechanism that acts as a robust
``bridge" to enable a symmetric, bidirectional exchange of information,
achieving both semantic and temporal synchronization. Extensive experiments on
three benchmark datasets, supported by human evaluations, demonstrate that our
method achieves state-of-the-art results on most metrics. Comprehensive
ablation studies further validate the effectiveness of our contributions,
offering key insights for the future T2SV task. All the codes and checkpoints
will be publicly released.
A novel dual-tower diffusion transformer with a Dual CrossAttention mechanism addresses challenges in Text-to-Sounding-Video generation by disentangling captions and enabling symmetric information exchange.
Hierarchical Visual-Grounded Captioning HVGC BridgeDiT dual-tower diffusion transformer Dual CrossAttention DCA semantic synchronization temporal synchronization