LLM-AD: Large Language Model based Audio Description System
The development of Audio Description (AD) has been a pivotal step forward in
making video content more accessible and inclusive. Traditionally, AD
production has demanded a considerable amount of skilled labor, while existing
automated approaches still necessitate extensive training to integrate
multimodal inputs and tailor the output from a captioning style to an AD style.
In this paper, we introduce an automated AD generation pipeline that harnesses
the potent multimodal and instruction-following capacities of GPT-4V(ision).
Notably, our methodology employs readily available components, eliminating the
need for additional training. It produces ADs that not only comply with
established natural language AD production standards but also maintain
contextually consistent character information across frames, courtesy of a
tracking-based character recognition module. A thorough analysis on the MAD
dataset reveals that our approach achieves a performance on par with
learning-based methods in automated AD production, as substantiated by a CIDEr
score of 20.5.
663449d7fa10518a897fb08e
Peng Chu
663449d7fa10518a897fb08f
Jiang Wang
663449d7fa10518a897fb090
Andre Abrantes
admin_assigned
2024-05-03T08:01:37.280Z
64b6add720b541022a13e9c4
/avatars/d952f4a3f6f1205eda4d540c7bfd7d1e.svg
Andre Abrantes
user
abrantes
LLM-AD: Large Language Model based Audio Description System
The development of Audio Description (AD) has been a pivotal step forward in
making video content more accessible and inclusive. Traditionally, AD
production has demanded a considerable amount of skilled labor, while existing
automated approaches still necessitate extensive training to integrate
multimodal inputs and tailor the output from a captioning style to an AD style.
In this paper, we introduce an automated AD generation pipeline that harnesses
the potent multimodal and instruction-following capacities of GPT-4V(ision).
Notably, our methodology employs readily available components, eliminating the
need for additional training. It produces ADs that not only comply with
established natural language AD production standards but also maintain
contextually consistent character information across frames, courtesy of a
tracking-based character recognition module. A thorough analysis on the MAD
dataset reveals that our approach achieves a performance on par with
learning-based methods in automated AD production, as substantiated by a CIDEr
score of 20.5.
An automated pipeline using GPT-4V and tracking-based character recognition generates high-quality, contextually consistent audio descriptions without additional training.
GPT-4V multimodal instruction-following tracking-based character recognition MAD dataset CIDEr score