Capturing Individual Human Preferences with Reward Features
Andre Barreto, Vincent Dumoulin, Yiran Mao, Mark Rowland, Nicolas Perez-Nieves, Bobak Shahriari, Yann Dauphin, Doina Precup, Hugo Larochelle
Reinforcement learning from human feedback usually models preferences using a reward model that does not distinguish between people. We argue that this is unlikely to be a good design choice in contexts with high potential for disagreement, like in the training of large language models. We formalise and analyse the problem of learning a reward model that can be specialised to a user. Using the principle of empirical risk minimisation, we derive a probably approximately correct bound showing the dependency of the approximation error not only on the number of training examples, but also on the number of human raters who provided feedback on them. We also put forward a formal argument supporting the intuition that adaptive reward models should be beneficial when there is considerable disagreement among users. Building on our theoretical findings, we propose a concrete architecture for an adaptive reward model. Our approach leverages the observation that individual preferences can be captured as a linear combination of a set of general reward features. We show how to learn such features and subsequently use them to quickly adapt the reward model to a specific individual, even if their preferences are not reflected in the training data. We present experiments with large language models illustrating our theoretical results and comparing the proposed architecture with a non-adaptive baseline. As predicted by the theory, the benefits provided by our model increase with the number of raters and the heterogeneity of their preferences. We also show how our model compare favourably to adaptive counterparts, including models that do in-context personalisation.