FSPO: Few-Shot Preference Optimization of Synthetic Preference Data in
  LLMs Elicits Effective Personalization to Real Users
Effective personalization of LLMs is critical for a broad range of
user-interfacing applications such as virtual assistants and content curation.
Inspired by the strong in-context learning capabilities of LLMs, we propose
Few-Shot Preference Optimization (FSPO), which reframes reward modeling as a
meta-learning problem. Under this framework, an LLM learns to quickly adapt to
a user via a few labeled preferences from that user, constructing a
personalized reward function for them. Additionally, since real-world
preference data is scarce and challenging to collect at scale, we propose
careful design choices to construct synthetic preference datasets for
personalization, generating over 1M synthetic personalized preferences using
publicly available LLMs. In particular, to successfully transfer from synthetic
data to real users, we find it crucial for the data to exhibit both high
diversity and coherent, self-consistent structure. We evaluate FSPO on
personalized open-ended generation for up to 1,500 synthetic users across
across three domains: movie reviews, pedagogical adaptation based on
educational background, and general question answering, along with a controlled
human study. Overall, FSPO achieves an 87% Alpaca Eval winrate on average in
generating responses that are personalized to synthetic users and a 72% winrate
with real human users in open-ended question answering.
67c01972d63ea6742473aa2a
Anikait Singh
extracted_pending
2025-02-27T07:51:17.284Z
6511ee845b7e52b0251fdee9
https://cdn-avatars.huggingface.co/v1/production/uploads/6511ee845b7e52b0251fdee9/hTIwiIYBGOVnIrxtpri83.png
Anikait Singh
user
Asap7772
67c01972d63ea6742473aa2b
Sheryl Hsu
67c01972d63ea6742473aa2c
Kyle Hsu
67c01972d63ea6742473aa2d
Eric Mitchell
67c01972d63ea6742473aa2e
Stefano Ermon
67c01972d63ea6742473aa2f
Tatsunori Hashimoto
67c01972d63ea6742473aa30
Archit Sharma
67c01972d63ea6742473aa31
Chelsea Finn
FSPO: Few-Shot Preference Optimization of Synthetic Preference Data in
  LLMs Elicits Effective Personalization to Real Users
Effective personalization of LLMs is critical for a broad range of
user-interfacing applications such as virtual assistants and content curation.
Inspired by the strong in-context learning capabilities of LLMs, we propose
Few-Shot Preference Optimization (FSPO), which reframes reward modeling as a
meta-learning problem. Under this framework, an LLM learns to quickly adapt to
a user via a few labeled preferences from that user, constructing a
personalized reward function for them. Additionally, since real-world
preference data is scarce and challenging to collect at scale, we propose
careful design choices to construct synthetic preference datasets for
personalization, generating over 1M synthetic personalized preferences using
publicly available LLMs. In particular, to successfully transfer from synthetic
data to real users, we find it crucial for the data to exhibit both high
diversity and coherent, self-consistent structure. We evaluate FSPO on
personalized open-ended generation for up to 1,500 synthetic users across
across three domains: movie reviews, pedagogical adaptation based on
educational background, and general question answering, along with a controlled
human study. Overall, FSPO achieves an 87% Alpaca Eval winrate on average in
generating responses that are personalized to synthetic users and a 72% winrate
with real human users in open-ended question answering.
FSPO uses meta-learning to adapt LLMs to users with few preferences, constructs synthetic preference datasets, and achieves high win rates in personalized open-ended generation.
Few-Shot Preference Optimization meta-learning reward modeling personalized reward function synthetic preference datasets high diversity coherent structure Alpaca Eval open-ended generation movie reviews pedagogical adaptation question answering