User-LLM: Efficient LLM Contextualization with User Embeddings
Large language models (LLMs) have revolutionized natural language processing.
However, effectively incorporating complex and potentially noisy user
interaction data remains a challenge. To address this, we propose User-LLM, a
novel framework that leverages user embeddings to contextualize LLMs. These
embeddings, distilled from diverse user interactions using self-supervised
pretraining, capture latent user preferences and their evolution over time. We
integrate these user embeddings with LLMs through cross-attention and
soft-prompting, enabling LLMs to dynamically adapt to user context. Our
comprehensive experiments on MovieLens, Amazon Review, and Google Local Review
datasets demonstrate significant performance gains across various tasks.
Notably, our approach outperforms text-prompt-based contextualization on long
sequence tasks and tasks that require deep user understanding while being
computationally efficient. We further incorporate Perceiver layers to
streamline the integration between user encoders and LLMs, reducing
computational demands.
65d6cc965e971572da052bf5
Lin Ning
65d6cc965e971572da052bf6
Luyang Liu
claimed_verified
2024-02-24T14:36:58.415Z
65d8fa4e16cebdd689fa5587
/avatars/9646f22ae16653af47ea38af0839ae7b.svg
Luyang Liu
user
luyangl
65d6cc965e971572da052bf7
Jiaxing Wu
65d6cc965e971572da052bf8
Neo Wu
65d6cc965e971572da052bf9
Devora Berlowitz
65d6cc965e971572da052bfa
Sushant Prakash
65d6cc965e971572da052bfb
Bradley Green
65d6cc965e971572da052bfc
Shawn O'Banion
65d6cc965e971572da052bfd
Jun Xie
User-LLM: Efficient LLM Contextualization with User Embeddings
Large language models (LLMs) have revolutionized natural language processing.
However, effectively incorporating complex and potentially noisy user
interaction data remains a challenge. To address this, we propose User-LLM, a
novel framework that leverages user embeddings to contextualize LLMs. These
embeddings, distilled from diverse user interactions using self-supervised
pretraining, capture latent user preferences and their evolution over time. We
integrate these user embeddings with LLMs through cross-attention and
soft-prompting, enabling LLMs to dynamically adapt to user context. Our
comprehensive experiments on MovieLens, Amazon Review, and Google Local Review
datasets demonstrate significant performance gains across various tasks.
Notably, our approach outperforms text-prompt-based contextualization on long
sequence tasks and tasks that require deep user understanding while being
computationally efficient. We further incorporate Perceiver layers to
streamline the integration between user encoders and LLMs, reducing
computational demands.
User-LLM framework uses user embeddings and Perceiver layers to improve LLM performance on long sequence tasks and user understanding efficiently.
User embeddings self-supervised pretraining cross-attention soft-prompting Perceiver layers