Evaluating Podcast Recommendations with Profile-Aware LLM-as-a-Judge
Evaluating personalized recommendations remains a central challenge,
especially in long-form audio domains like podcasts, where traditional offline
metrics suffer from exposure bias and online methods such as A/B testing are
costly and operationally constrained. In this paper, we propose a novel
framework that leverages Large Language Models (LLMs) as offline judges to
assess the quality of podcast recommendations in a scalable and interpretable
manner. Our two-stage profile-aware approach first constructs natural-language
user profiles distilled from 90 days of listening history. These profiles
summarize both topical interests and behavioral patterns, serving as compact,
interpretable representations of user preferences. Rather than prompting the
LLM with raw data, we use these profiles to provide high-level, semantically
rich context-enabling the LLM to reason more effectively about alignment
between a user's interests and recommended episodes. This reduces input
complexity and improves interpretability. The LLM is then prompted to deliver
fine-grained pointwise and pairwise judgments based on the profile-episode
match. In a controlled study with 47 participants, our profile-aware judge
matched human judgments with high fidelity and outperformed or matched a
variant using raw listening histories. The framework enables efficient,
profile-aware evaluation for iterative testing and model selection in
recommender systems.
68a5825738160bfd39426a54
Francesco Fabbri
claimed_verified
2025-08-20T08:50:51.265Z
65eed0a600f1a613dae4d9be
/avatars/a420d0e2c77a482eb5022deeccb2ebf5.svg
Francesco Fabbri
user
frafabbri
68a5825738160bfd39426a55
Gustavo Penha
68a5825738160bfd39426a56
Edoardo D'Amico
68a5825738160bfd39426a57
Alice Wang
68a5825738160bfd39426a58
Marco De Nadai
claimed_verified
2025-08-20T08:50:59.013Z
626193a918176506fab35c62
/avatars/44a34405e4821fa9047cfa635e198f61.svg
Marco De Nadai
user
marcodena
68a5825738160bfd39426a59
Jackie Doremus
68a5825738160bfd39426a5a
Paul Gigioli
68a5825738160bfd39426a5b
Andreas Damianou
68a5825738160bfd39426a5c
Oskar Stal
68a5825738160bfd39426a5d
Mounia Lalmas
Evaluating Podcast Recommendations with Profile-Aware LLM-as-a-Judge
Evaluating personalized recommendations remains a central challenge,
especially in long-form audio domains like podcasts, where traditional offline
metrics suffer from exposure bias and online methods such as A/B testing are
costly and operationally constrained. In this paper, we propose a novel
framework that leverages Large Language Models (LLMs) as offline judges to
assess the quality of podcast recommendations in a scalable and interpretable
manner. Our two-stage profile-aware approach first constructs natural-language
user profiles distilled from 90 days of listening history. These profiles
summarize both topical interests and behavioral patterns, serving as compact,
interpretable representations of user preferences. Rather than prompting the
LLM with raw data, we use these profiles to provide high-level, semantically
rich context-enabling the LLM to reason more effectively about alignment
between a user's interests and recommended episodes. This reduces input
complexity and improves interpretability. The LLM is then prompted to deliver
fine-grained pointwise and pairwise judgments based on the profile-episode
match. In a controlled study with 47 participants, our profile-aware judge
matched human judgments with high fidelity and outperformed or matched a
variant using raw listening histories. The framework enables efficient,
profile-aware evaluation for iterative testing and model selection in
recommender systems.
A novel framework uses Large Language Models to evaluate podcast recommendations by constructing user profiles and providing context for the LLM to make judgments, improving efficiency and interpretability.
Large Language Models profile-aware natural-language user profiles semantic context pointwise judgments pairwise judgments recommender systems