Describe What You See with Multimodal Large Language Models to Enhance
  Video Recommendations
Existing video recommender systems rely primarily on user-defined metadata or
on low-level visual and acoustic signals extracted by specialised encoders.
These low-level features describe what appears on the screen but miss deeper
semantics such as intent, humour, and world knowledge that make clips resonate
with viewers. For example, is a 30-second clip simply a singer on a rooftop, or
an ironic parody filmed amid the fairy chimneys of Cappadocia, Turkey? Such
distinctions are critical to personalised recommendations yet remain invisible
to traditional encoding pipelines. In this paper, we introduce a simple,
recommendation system-agnostic zero-finetuning framework that injects
high-level semantics into the recommendation pipeline by prompting an
off-the-shelf Multimodal Large Language Model (MLLM) to summarise each clip
into a rich natural-language description (e.g. "a superhero parody with
slapstick fights and orchestral stabs"), bridging the gap between raw content
and user intent. We use MLLM output with a state-of-the-art text encoder and
feed it into standard collaborative, content-based, and generative
recommenders. On the MicroLens-100K dataset, which emulates user interactions
with TikTok-style videos, our framework consistently surpasses conventional
video, audio, and metadata features in five representative models. Our findings
highlight the promise of leveraging MLLMs as on-the-fly knowledge extractors to
build more intent-aware video recommenders.
68a581fd38160bfd39426a4f
Marco De Nadai
claimed_verified
2025-08-20T08:51:01.302Z
626193a918176506fab35c62
/avatars/44a34405e4821fa9047cfa635e198f61.svg
Marco De Nadai
user
marcodena
68a581fd38160bfd39426a50
Andreas Damianou
68a581fd38160bfd39426a51
Mounia Lalmas
Describe What You See with Multimodal Large Language Models to Enhance
  Video Recommendations
Existing video recommender systems rely primarily on user-defined metadata or
on low-level visual and acoustic signals extracted by specialised encoders.
These low-level features describe what appears on the screen but miss deeper
semantics such as intent, humour, and world knowledge that make clips resonate
with viewers. For example, is a 30-second clip simply a singer on a rooftop, or
an ironic parody filmed amid the fairy chimneys of Cappadocia, Turkey? Such
distinctions are critical to personalised recommendations yet remain invisible
to traditional encoding pipelines. In this paper, we introduce a simple,
recommendation system-agnostic zero-finetuning framework that injects
high-level semantics into the recommendation pipeline by prompting an
off-the-shelf Multimodal Large Language Model (MLLM) to summarise each clip
into a rich natural-language description (e.g. "a superhero parody with
slapstick fights and orchestral stabs"), bridging the gap between raw content
and user intent. We use MLLM output with a state-of-the-art text encoder and
feed it into standard collaborative, content-based, and generative
recommenders. On the MicroLens-100K dataset, which emulates user interactions
with TikTok-style videos, our framework consistently surpasses conventional
video, audio, and metadata features in five representative models. Our findings
highlight the promise of leveraging MLLMs as on-the-fly knowledge extractors to
build more intent-aware video recommenders.
A zero-finetuning framework uses Multimodal Large Language Models to inject high-level semantics into video recommendations, improving intent-awareness over traditional methods.
Multimodal Large Language Model MLLM natural-language description text encoder collaborative recommenders content-based recommenders generative recommenders MicroLens-100K dataset TikTok-style videos intent-aware video recommenders