AVoCaDO: An Audiovisual Video Captioner Driven by Temporal Orchestration
Audiovisual video captioning aims to generate semantically rich descriptions
with temporal alignment between visual and auditory events, thereby benefiting
both video understanding and generation. In this paper, we present AVoCaDO, a
powerful audiovisual video captioner driven by the temporal orchestration
between audio and visual modalities. We propose a two-stage post-training
pipeline: (1) AVoCaDO SFT, which fine-tunes the model on a newly curated
dataset of 107K high-quality, temporally-aligned audiovisual captions; and (2)
AVoCaDO GRPO, which leverages tailored reward functions to further enhance
temporal coherence and dialogue accuracy while regularizing caption length and
reducing collapse. Experimental results demonstrate that AVoCaDO significantly
outperforms existing open-source models across four audiovisual video
captioning benchmarks, and also achieves competitive performance on the VDC and
DREAM-1K benchmark under visual-only settings.
68eda8c8de1fee572713a6b2
Xinlong Chen
claimed_verified
2025-10-14T07:31:15.388Z
66650d38b52f0890724f3b07
/avatars/c25a365bff4985ebb71c96dd097b804f.svg
Xinlong Chen
user
XinlongChen
68eda8c8de1fee572713a6b3
Yue Ding
68eda8c8de1fee572713a6b4
Weihong Lin
68eda8c8de1fee572713a6b5
Jingyun Hua
claimed_verified
2025-10-14T07:31:10.887Z
61540338e5b9ae6774201e58
https://cdn-avatars.huggingface.co/v1/production/uploads/61540338e5b9ae6774201e58/h_159VrXOlIgu0N0pNgXj.png
jingyun
user
hjy
68eda8c8de1fee572713a6b6
Linli Yao
68eda8c8de1fee572713a6b7
Yang Shi
claimed_verified
2025-10-14T07:31:13.095Z
673c7319d11b1c2e246ead9c
https://cdn-avatars.huggingface.co/v1/production/uploads/673c7319d11b1c2e246ead9c/IjFIO--N7Hm_BOEafhEQv.jpeg
Yang Shi
user
DogNeverSleep
68eda8c8de1fee572713a6b8
Bozhou Li
claimed_verified
2025-10-16T10:38:43.361Z
661e62c6bac5d981f886f77b
/avatars/f1eb51ed4499ca434c8939573dfbd5e2.svg
Bozhou Li
user
zooblastlbz
68eda8c8de1fee572713a6b9
Yuanxing Zhang
68eda8c8de1fee572713a6ba
Qiang Liu
68eda8c8de1fee572713a6bb
Pengfei Wan
68eda8c8de1fee572713a6bc
Liang Wang
68eda8c8de1fee572713a6bd
Tieniu Tan
AVoCaDO: An Audiovisual Video Captioner Driven by Temporal Orchestration
Audiovisual video captioning aims to generate semantically rich descriptions
with temporal alignment between visual and auditory events, thereby benefiting
both video understanding and generation. In this paper, we present AVoCaDO, a
powerful audiovisual video captioner driven by the temporal orchestration
between audio and visual modalities. We propose a two-stage post-training
pipeline: (1) AVoCaDO SFT, which fine-tunes the model on a newly curated
dataset of 107K high-quality, temporally-aligned audiovisual captions; and (2)
AVoCaDO GRPO, which leverages tailored reward functions to further enhance
temporal coherence and dialogue accuracy while regularizing caption length and
reducing collapse. Experimental results demonstrate that AVoCaDO significantly
outperforms existing open-source models across four audiovisual video
captioning benchmarks, and also achieves competitive performance on the VDC and
DREAM-1K benchmark under visual-only settings.
AVoCaDO, an audiovisual video captioner, enhances temporal coherence and dialogue accuracy through a two-stage post-training pipeline, outperforming existing models across multiple benchmarks.
audiovisual video captioning temporal orchestration AVoCaDO AVoCaDO SFT AVoCaDO GRPO reward functions temporal coherence dialogue accuracy caption length collapse VDC DREAM-1K