Long-Video Audio Synthesis with Multi-Agent Collaboration
Video-to-audio synthesis, which generates synchronized audio for visual
content, critically enhances viewer immersion and narrative coherence in film
and interactive media. However, video-to-audio dubbing for long-form content
remains an unsolved challenge due to dynamic semantic shifts, temporal
misalignment, and the absence of dedicated datasets. While existing methods
excel in short videos, they falter in long scenarios (e.g., movies) due to
fragmented synthesis and inadequate cross-scene consistency. We propose
LVAS-Agent, a novel multi-agent framework that emulates professional dubbing
workflows through collaborative role specialization. Our approach decomposes
long-video synthesis into four steps including scene segmentation, script
generation, sound design and audio synthesis. Central innovations include a
discussion-correction mechanism for scene/script refinement and a
generation-retrieval loop for temporal-semantic alignment. To enable systematic
evaluation, we introduce LVAS-Bench, the first benchmark with 207
professionally curated long videos spanning diverse scenarios. Experiments
demonstrate superior audio-visual alignment over baseline methods. Project
page: https://lvas-agent.github.io
67d91dadb533888991ade4e1
Yehang Zhang
claimed_verified
2025-03-18T08:07:19.937Z
672c6f3d4c1e2de12c6f174e
https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/mv-8GX6OBBpJLzVvyPYbz.png
Yehang Zhang
user
Buzz-lightyear
67d91dadb533888991ade4e2
Xinli Xu
admin_assigned
2025-03-18T09:35:50.412Z
64b4ab62eec33e27dcd733b5
/avatars/0a9bf220c9a5efe7279f9b287b087d36.svg
Xinli XU
user
Xxlbigbrother
67d91dadb533888991ade4e3
Xiaojie Xu
67d91dadb533888991ade4e4
Li Liu
67d91dadb533888991ade4e5
Yingcong Chen
admin_assigned
2025-03-18T09:36:19.820Z
655cba1d87b67834000590e8
/avatars/3bd43b7c9351f65b8f38f4c8237a0146.svg
Yingcong Chen
user
yingcongchen
Long-Video Audio Synthesis with Multi-Agent Collaboration
Video-to-audio synthesis, which generates synchronized audio for visual
content, critically enhances viewer immersion and narrative coherence in film
and interactive media. However, video-to-audio dubbing for long-form content
remains an unsolved challenge due to dynamic semantic shifts, temporal
misalignment, and the absence of dedicated datasets. While existing methods
excel in short videos, they falter in long scenarios (e.g., movies) due to
fragmented synthesis and inadequate cross-scene consistency. We propose
LVAS-Agent, a novel multi-agent framework that emulates professional dubbing
workflows through collaborative role specialization. Our approach decomposes
long-video synthesis into four steps including scene segmentation, script
generation, sound design and audio synthesis. Central innovations include a
discussion-correction mechanism for scene/script refinement and a
generation-retrieval loop for temporal-semantic alignment. To enable systematic
evaluation, we introduce LVAS-Bench, the first benchmark with 207
professionally curated long videos spanning diverse scenarios. Experiments
demonstrate superior audio-visual alignment over baseline methods. Project
page: https://lvas-agent.github.io
LVAS-Agent, a multi-agent framework, improves video-to-audio synthesis for long-form content through scene segmentation, script generation, sound design, and audio synthesis with a discussion-correction and generation-retrieval loop.
multi-agent framework scene segmentation script generation sound design audio synthesis discussion-correction mechanism generation-retrieval loop temporal-semantic alignment