Insights from the Inverse: Reconstructing LLM Training Goals Through
  Inverse RL
Large language models (LLMs) trained with Reinforcement Learning from Human
Feedback (RLHF) have demonstrated remarkable capabilities, but their underlying
reward functions and decision-making processes remain opaque. This paper
introduces a novel approach to interpreting LLMs by applying inverse
reinforcement learning (IRL) to recover their implicit reward functions. We
conduct experiments on toxicity-aligned LLMs of varying sizes, extracting
reward models that achieve up to 80.40% accuracy in predicting human
preferences. Our analysis reveals key insights into the non-identifiability of
reward functions, the relationship between model size and interpretability, and
potential pitfalls in the RLHF process. We demonstrate that IRL-derived reward
models can be used to fine-tune new LLMs, resulting in comparable or improved
performance on toxicity benchmarks. This work provides a new lens for
understanding and improving LLM alignment, with implications for the
responsible development and deployment of these powerful systems.
6710e1e2786f6ba453d1787f
Jared Joselowitz
claimed_verified
2024-11-15T09:42:56.439Z
63471d1fc849ba78f285a59f
/avatars/e2445fb517427360e6c2adc8c683c244.svg
Jared Joselowitz
user
jaredjoss
6710e1e2786f6ba453d17880
Arjun Jagota
6710e1e2786f6ba453d17881
Satyapriya Krishna
claimed_verified
2024-10-17T11:23:44.325Z
6186fef1b1085ab638324e7f
https://cdn-avatars.huggingface.co/v1/production/uploads/6186fef1b1085ab638324e7f/BL6_WJCkxB-BatBUBilT8.jpeg
Satya
user
skrishna
6710e1e2786f6ba453d17882
Sonali Parbhoo
Insights from the Inverse: Reconstructing LLM Training Goals Through
  Inverse RL
Large language models (LLMs) trained with Reinforcement Learning from Human
Feedback (RLHF) have demonstrated remarkable capabilities, but their underlying
reward functions and decision-making processes remain opaque. This paper
introduces a novel approach to interpreting LLMs by applying inverse
reinforcement learning (IRL) to recover their implicit reward functions. We
conduct experiments on toxicity-aligned LLMs of varying sizes, extracting
reward models that achieve up to 80.40% accuracy in predicting human
preferences. Our analysis reveals key insights into the non-identifiability of
reward functions, the relationship between model size and interpretability, and
potential pitfalls in the RLHF process. We demonstrate that IRL-derived reward
models can be used to fine-tune new LLMs, resulting in comparable or improved
performance on toxicity benchmarks. This work provides a new lens for
understanding and improving LLM alignment, with implications for the
responsible development and deployment of these powerful systems.
Applying inverse reinforcement learning to large language models trained with RLHF recovers their implicit reward functions, offering insights into model interpretability and improving performance on toxicity benchmarks.
Reinforcement Learning from Human Feedback (RLHF) inverse reinforcement learning (IRL) reward functions toxicity-aligned LLMs fine-tuning model interpretability LLM alignment