A Video Is Worth 4096 Tokens: Verbalize Story Videos To Understand Them
  In Zero Shot
Multimedia content, such as advertisements and story videos, exhibit a rich
blend of creativity and multiple modalities. They incorporate elements like
text, visuals, audio, and storytelling techniques, employing devices like
emotions, symbolism, and slogans to convey meaning. While previous research in
multimedia understanding has focused mainly on videos with specific actions
like cooking, there is a dearth of large annotated training datasets, hindering
the development of supervised learning models with satisfactory performance for
real-world applications. However, the rise of large language models (LLMs) has
witnessed remarkable zero-shot performance in various natural language
processing (NLP) tasks, such as emotion classification, question-answering, and
topic classification. To bridge this performance gap in multimedia
understanding, we propose verbalizing story videos to generate their
descriptions in natural language and then performing video-understanding tasks
on the generated story as opposed to the original video. Through extensive
experiments on five video-understanding tasks, we demonstrate that our method,
despite being zero-shot, achieves significantly better results than supervised
baselines for video understanding. Further, alleviating a lack of story
understanding benchmarks, we publicly release the first dataset on a crucial
task in computational social science, persuasion strategy identification.
646582770e6c7618f61506c5
Aanisha Bhattacharya
claimed_verified
2024-05-03T15:14:32.432Z
60a7d1ca23ce37179774a210
https://cdn-avatars.huggingface.co/v1/production/uploads/1651500117101-60a7d1ca23ce37179774a210.jpeg
Aanisha Bhattacharyya
user
Aanisha
646582770e6c7618f61506c6
Yaman K Singla
claimed_verified
2023-05-22T09:37:00.156Z
62be9baeae56e75e4d68a289
/avatars/6a9415a05b4c217433bdb1b414ccd221.svg
Yaman
user
yamanksingla
646582770e6c7618f61506c7
Balaji Krishnamurthy
646582770e6c7618f61506c8
Rajiv Ratn Shah
646582770e6c7618f61506c9
Changyou Chen
claimed_verified
2023-05-19T19:03:42.355Z
64678f41946476c5d2170609
/avatars/deadb2c5552b314f3a14c89a9b353769.svg
Changyou Chen
user
cchangyou
A Video Is Worth 4096 Tokens: Verbalize Story Videos To Understand Them
  In Zero Shot
Multimedia content, such as advertisements and story videos, exhibit a rich
blend of creativity and multiple modalities. They incorporate elements like
text, visuals, audio, and storytelling techniques, employing devices like
emotions, symbolism, and slogans to convey meaning. While previous research in
multimedia understanding has focused mainly on videos with specific actions
like cooking, there is a dearth of large annotated training datasets, hindering
the development of supervised learning models with satisfactory performance for
real-world applications. However, the rise of large language models (LLMs) has
witnessed remarkable zero-shot performance in various natural language
processing (NLP) tasks, such as emotion classification, question-answering, and
topic classification. To bridge this performance gap in multimedia
understanding, we propose verbalizing story videos to generate their
descriptions in natural language and then performing video-understanding tasks
on the generated story as opposed to the original video. Through extensive
experiments on five video-understanding tasks, we demonstrate that our method,
despite being zero-shot, achieves significantly better results than supervised
baselines for video understanding. Further, alleviating a lack of story
understanding benchmarks, we publicly release the first dataset on a crucial
task in computational social science, persuasion strategy identification.
The proposed method of verbalizing story videos into natural language descriptions improves zero-shot performance in video understanding tasks, surpassing supervised baselines.
large language models zero-shot performance natural language processing video-understanding tasks zero-shot supervised baselines story-understanding benchmarks persuasion strategy identification