ThinkSound: Chain-of-Thought Reasoning in Multimodal Large Language
  Models for Audio Generation and Editing
While end-to-end video-to-audio generation has greatly improved, producing
high-fidelity audio that authentically captures the nuances of visual content
remains challenging. Like professionals in the creative industries, such
generation requires sophisticated reasoning about items such as visual
dynamics, acoustic environments, and temporal relationships. We present
ThinkSound, a novel framework that leverages Chain-of-Thought (CoT) reasoning
to enable stepwise, interactive audio generation and editing for videos. Our
approach decomposes the process into three complementary stages: foundational
foley generation that creates semantically coherent soundscapes, interactive
object-centric refinement through precise user interactions, and targeted
editing guided by natural language instructions. At each stage, a multimodal
large language model generates contextually aligned CoT reasoning that guides a
unified audio foundation model. Furthermore, we introduce AudioCoT, a
comprehensive dataset with structured reasoning annotations that establishes
connections between visual content, textual descriptions, and sound synthesis.
Experiments demonstrate that ThinkSound achieves state-of-the-art performance
in video-to-audio generation across both audio metrics and CoT metrics and
excels in out-of-distribution Movie Gen Audio benchmark. The demo page is
available at https://ThinkSound-Project.github.io.
68636ecd588cea0da970c905
Huadai Liu
68636ecd588cea0da970c906
Jialei Wang
68636ecd588cea0da970c907
Kaicheng Luo
68636ecd588cea0da970c908
Wen Wang
68636ecd588cea0da970c909
Qian Chen
68636ecd588cea0da970c90a
Zhou Zhao
68636ecd588cea0da970c90b
Wei Xue
ThinkSound: Chain-of-Thought Reasoning in Multimodal Large Language
  Models for Audio Generation and Editing
While end-to-end video-to-audio generation has greatly improved, producing
high-fidelity audio that authentically captures the nuances of visual content
remains challenging. Like professionals in the creative industries, such
generation requires sophisticated reasoning about items such as visual
dynamics, acoustic environments, and temporal relationships. We present
ThinkSound, a novel framework that leverages Chain-of-Thought (CoT) reasoning
to enable stepwise, interactive audio generation and editing for videos. Our
approach decomposes the process into three complementary stages: foundational
foley generation that creates semantically coherent soundscapes, interactive
object-centric refinement through precise user interactions, and targeted
editing guided by natural language instructions. At each stage, a multimodal
large language model generates contextually aligned CoT reasoning that guides a
unified audio foundation model. Furthermore, we introduce AudioCoT, a
comprehensive dataset with structured reasoning annotations that establishes
connections between visual content, textual descriptions, and sound synthesis.
Experiments demonstrate that ThinkSound achieves state-of-the-art performance
in video-to-audio generation across both audio metrics and CoT metrics and
excels in out-of-distribution Movie Gen Audio benchmark. The demo page is
available at https://ThinkSound-Project.github.io.
ThinkSound uses Chain-of-Thought reasoning to enhance video-to-audio generation through a multimodal large language model and a unified audio foundation model, achieving top performance in audio metrics and CoT metrics.
Chain-of-Thought (CoT) reasoning multimodal large language model unified audio foundation model AudioCoT video-to-audio generation audio metrics CoT metrics out-of-distribution Movie Gen Audio benchmark