BESPOKE: Benchmark for Search-Augmented Large Language Model
  Personalization via Diagnostic Feedback
Search-augmented large language models (LLMs) have advanced
information-seeking tasks by integrating retrieval into generation, reducing
users' cognitive burden compared to traditional search systems. Yet they remain
insufficient for fully addressing diverse user needs, which requires
recognizing how the same query can reflect different intents across users and
delivering information in preferred forms. While recent systems such as ChatGPT
and Gemini attempt personalization by leveraging user histories, systematic
evaluation of such personalization is under-explored. To address this gap, we
propose BESPOKE, the realistic benchmark for evaluating personalization in
search-augmented LLMs. BESPOKE is designed to be both realistic, by collecting
authentic chat and search histories directly from humans, and diagnostic, by
pairing responses with fine-grained preference scores and feedback. The
benchmark is constructed through long-term, deeply engaged human annotation,
where human annotators contributed their own histories, authored queries with
detailed information needs, and evaluated responses with scores and diagnostic
feedback. Leveraging BESPOKE, we conduct systematic analyses that reveal key
requirements for effective personalization in information-seeking tasks,
providing a foundation for fine-grained evaluation of personalized
search-augmented LLMs. Our code and data are available at
https://augustinlib.github.io/BESPOKE/.
68d6404f8ccd91bdd39ffec6
Hyunseo Kim
claimed_verified
2025-09-26T12:08:44.550Z
65f9083ee8720eb3741524a2
https://cdn-avatars.huggingface.co/v1/production/uploads/65f9083ee8720eb3741524a2/2Ix1Vd0At8S9MF852Hjay.jpeg
Hyunseo Kim
user
hyunseo00
68d6404f8ccd91bdd39ffec7
Sangam Lee
68d6404f8ccd91bdd39ffec8
Kwangwook Seo
68d6404f8ccd91bdd39ffec9
Dongha Lee
BESPOKE: Benchmark for Search-Augmented Large Language Model
  Personalization via Diagnostic Feedback
Search-augmented large language models (LLMs) have advanced
information-seeking tasks by integrating retrieval into generation, reducing
users' cognitive burden compared to traditional search systems. Yet they remain
insufficient for fully addressing diverse user needs, which requires
recognizing how the same query can reflect different intents across users and
delivering information in preferred forms. While recent systems such as ChatGPT
and Gemini attempt personalization by leveraging user histories, systematic
evaluation of such personalization is under-explored. To address this gap, we
propose BESPOKE, the realistic benchmark for evaluating personalization in
search-augmented LLMs. BESPOKE is designed to be both realistic, by collecting
authentic chat and search histories directly from humans, and diagnostic, by
pairing responses with fine-grained preference scores and feedback. The
benchmark is constructed through long-term, deeply engaged human annotation,
where human annotators contributed their own histories, authored queries with
detailed information needs, and evaluated responses with scores and diagnostic
feedback. Leveraging BESPOKE, we conduct systematic analyses that reveal key
requirements for effective personalization in information-seeking tasks,
providing a foundation for fine-grained evaluation of personalized
search-augmented LLMs. Our code and data are available at
https://augustinlib.github.io/BESPOKE/.
BESPOKE is a benchmark for evaluating personalization in search-augmented LLMs using authentic user data and detailed feedback.
search-augmented LLMs personalization information-seeking tasks ChatGPT Gemini user histories benchmark fine-grained preference scores diagnostic feedback human annotation information needs