SynthesizeMe! Inducing Persona-Guided Prompts for Personalized Reward
  Models in LLMs
Recent calls for pluralistic alignment of Large Language Models (LLMs)
encourage adapting models to diverse user preferences. However, most prior work
on personalized reward models heavily rely on additional identity information,
such as demographic details or a predefined set of preference categories. To
this end, we introduce SynthesizeMe, an approach to inducing synthetic user
personas from user interactions for personalized reward modeling. SynthesizeMe
first generates and verifies reasoning to explain user preferences, then
induces synthetic user personas from that reasoning, and finally filters to
informative prior user interactions in order to build personalized prompts for
a particular user. We show that using SynthesizeMe induced prompts improves
personalized LLM-as-a-judge accuracy by 4.4% on Chatbot Arena. Combining
SynthesizeMe derived prompts with a reward model achieves top performance on
PersonalRewardBench: a new curation of user-stratified interactions with
chatbots collected from 854 users of Chatbot Arena and PRISM.
6848609d4fe3b60e21b258ca
Michael J Ryan
extracted_pending
2025-06-10T16:43:10.189Z
63878fa2e40346f68ede7fc4
https://cdn-avatars.huggingface.co/v1/production/uploads/63878fa2e40346f68ede7fc4/MOdQbApvhwpVzPo5FWLVV.jpeg
Michael Ryan
user
MichaelR207
6848609d4fe3b60e21b258cb
Omar Shaikh
6848609d4fe3b60e21b258cc
Aditri Bhagirath
6848609d4fe3b60e21b258cd
Daniel Frees
6848609d4fe3b60e21b258ce
William Held
6848609d4fe3b60e21b258cf
Diyi Yang
SynthesizeMe! Inducing Persona-Guided Prompts for Personalized Reward
  Models in LLMs
Recent calls for pluralistic alignment of Large Language Models (LLMs)
encourage adapting models to diverse user preferences. However, most prior work
on personalized reward models heavily rely on additional identity information,
such as demographic details or a predefined set of preference categories. To
this end, we introduce SynthesizeMe, an approach to inducing synthetic user
personas from user interactions for personalized reward modeling. SynthesizeMe
first generates and verifies reasoning to explain user preferences, then
induces synthetic user personas from that reasoning, and finally filters to
informative prior user interactions in order to build personalized prompts for
a particular user. We show that using SynthesizeMe induced prompts improves
personalized LLM-as-a-judge accuracy by 4.4% on Chatbot Arena. Combining
SynthesizeMe derived prompts with a reward model achieves top performance on
PersonalRewardBench: a new curation of user-stratified interactions with
chatbots collected from 854 users of Chatbot Arena and PRISM.
SynthesizeMe generates personalized prompts from user interactions to enhance the accuracy of LLMs in judging chatbot performances.
natural language models large language models personalized reward models user personas user interactions reasoning prompts LLM-as-a-judge Chatbot Arena PersonalRewardBench PRISM