PEARL: Personalizing Large Language Model Writing Assistants with
  Generation-Calibrated Retrievers
Powerful large language models have facilitated the development of writing
assistants that promise to significantly improve the quality and efficiency of
composition and communication. However, a barrier to effective assistance is
the lack of personalization in LLM outputs to the author's communication style
and specialized knowledge. In this paper, we address this challenge by
proposing PEARL, a retrieval-augmented LLM writing assistant personalized with
a generation-calibrated retriever. Our retriever is trained to select historic
user-authored documents for prompt augmentation, such that they are likely to
best personalize LLM generations for a user request. We propose two key
novelties for training our retriever: 1) A training data selection method that
identifies user requests likely to benefit from personalization and documents
that provide that benefit; and 2) A scale-calibrating KL-divergence objective
that ensures that our retriever closely tracks the benefit of a document for
personalized generation. We demonstrate the effectiveness of PEARL in
generating personalized workplace social media posts and Reddit comments.
Finally, we showcase the potential of a generation-calibrated retriever to
double as a performance predictor and further improve low-quality generations
via LLM chaining.
6555ab778094dce8237364d2
Sheshera Mysore
admin_assigned
2023-11-16T10:06:10.703Z
62258434518cb976dadeaabd
https://cdn-avatars.huggingface.co/v1/production/uploads/1646747907864-62258434518cb976dadeaabd.jpeg
Sheshera Mysore
user
Sheshera
6555ab778094dce8237364d3
Zhuoran Lu
6555ab778094dce8237364d4
Mengting Wan
claimed_verified
2024-03-27T08:46:39.220Z
660328d9738bdf26e72dd454
/avatars/7af51be1cc1dfaaa3a78e0b546d97cf8.svg
Mengting Wan
user
mengtingwan
6555ab778094dce8237364d5
Longqi Yang
admin_assigned
2023-11-16T10:06:39.703Z
64d048b197ca59bcf726bdf2
/avatars/6384aa7558382cad60ee94d9e08d6182.svg
Longqi Yang
user
ylongqi
6555ab778094dce8237364d6
Steve Menezes
6555ab778094dce8237364d7
Tina Baghaee
6555ab778094dce8237364d8
Emmanuel Barajas Gonzalez
6555ab778094dce8237364d9
Jennifer Neville
6555ab778094dce8237364da
Tara Safavi
extracted_pending
2023-11-16T05:41:12.149Z
63ffc744cafc9d54985fdf4f
/avatars/0820cad83df012182cdf4094a7dcf744.svg
Tara Safavi
user
tararootcake
PEARL: Personalizing Large Language Model Writing Assistants with
  Generation-Calibrated Retrievers
Powerful large language models have facilitated the development of writing
assistants that promise to significantly improve the quality and efficiency of
composition and communication. However, a barrier to effective assistance is
the lack of personalization in LLM outputs to the author's communication style
and specialized knowledge. In this paper, we address this challenge by
proposing PEARL, a retrieval-augmented LLM writing assistant personalized with
a generation-calibrated retriever. Our retriever is trained to select historic
user-authored documents for prompt augmentation, such that they are likely to
best personalize LLM generations for a user request. We propose two key
novelties for training our retriever: 1) A training data selection method that
identifies user requests likely to benefit from personalization and documents
that provide that benefit; and 2) A scale-calibrating KL-divergence objective
that ensures that our retriever closely tracks the benefit of a document for
personalized generation. We demonstrate the effectiveness of PEARL in
generating personalized workplace social media posts and Reddit comments.
Finally, we showcase the potential of a generation-calibrated retriever to
double as a performance predictor and further improve low-quality generations
via LLM chaining.
large language models writing assistants communication style specialized knowledge PEARL retrieval-augmented generation-calibrated retriever prompt augmentation user-authored documents KL-divergence objective personalized generation workplace social media posts Reddit comments LLM chaining