Maximizing Alignment with Minimal Feedback: Efficiently Learning Rewards
  for Visuomotor Robot Policy Alignment
Visuomotor robot policies, increasingly pre-trained on large-scale datasets,
promise significant advancements across robotics domains. However, aligning
these policies with end-user preferences remains a challenge, particularly when
the preferences are hard to specify. While reinforcement learning from human
feedback (RLHF) has become the predominant mechanism for alignment in
non-embodied domains like large language models, it has not seen the same
success in aligning visuomotor policies due to the prohibitive amount of human
feedback required to learn visual reward functions. To address this limitation,
we propose Representation-Aligned Preference-based Learning (RAPL), an
observation-only method for learning visual rewards from significantly less
human preference feedback. Unlike traditional RLHF, RAPL focuses human feedback
on fine-tuning pre-trained vision encoders to align with the end-user's visual
representation and then constructs a dense visual reward via feature matching
in this aligned representation space. We first validate RAPL through simulation
experiments in the X-Magical benchmark and Franka Panda robotic manipulation,
demonstrating that it can learn rewards aligned with human preferences, more
efficiently uses preference data, and generalizes across robot embodiments.
Finally, our hardware experiments align pre-trained Diffusion Policies for
three object manipulation tasks. We find that RAPL can fine-tune these policies
with 5x less real human preference data, taking the first step towards
minimizing human feedback while maximizing visuomotor robot policy alignment.
67595a8662950d6eeb940516
Ran Tian
claimed_verified
2024-12-12T08:22:35.613Z
6535786a24c3063696804cdf
/avatars/2e59a26b691a9ab9825a015c7d47f9aa.svg
Ran (Thomas) Tian
user
thomasrantian
67595a8662950d6eeb940517
Yilin Wu
67595a8662950d6eeb940518
Chenfeng Xu
admin_assigned
2024-12-11T14:03:46.829Z
654ca71d5255ee86711b52c5
/avatars/52bf00fd74c8db5643c4daa185c678e6.svg
Chenfeng Xu
user
chenfengx
67595a8662950d6eeb940519
Masayoshi Tomizuka
67595a8662950d6eeb94051a
Jitendra Malik
admin_assigned
2024-12-11T14:03:37.137Z
65369a95605a07338de78ab0
https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/sGFjOjLT2akN-sn5beVWL.jpeg
Jitendra Malik 
user
jitendra1995
67595a8662950d6eeb94051b
Andrea Bajcsy
Maximizing Alignment with Minimal Feedback: Efficiently Learning Rewards
  for Visuomotor Robot Policy Alignment
Visuomotor robot policies, increasingly pre-trained on large-scale datasets,
promise significant advancements across robotics domains. However, aligning
these policies with end-user preferences remains a challenge, particularly when
the preferences are hard to specify. While reinforcement learning from human
feedback (RLHF) has become the predominant mechanism for alignment in
non-embodied domains like large language models, it has not seen the same
success in aligning visuomotor policies due to the prohibitive amount of human
feedback required to learn visual reward functions. To address this limitation,
we propose Representation-Aligned Preference-based Learning (RAPL), an
observation-only method for learning visual rewards from significantly less
human preference feedback. Unlike traditional RLHF, RAPL focuses human feedback
on fine-tuning pre-trained vision encoders to align with the end-user's visual
representation and then constructs a dense visual reward via feature matching
in this aligned representation space. We first validate RAPL through simulation
experiments in the X-Magical benchmark and Franka Panda robotic manipulation,
demonstrating that it can learn rewards aligned with human preferences, more
efficiently uses preference data, and generalizes across robot embodiments.
Finally, our hardware experiments align pre-trained Diffusion Policies for
three object manipulation tasks. We find that RAPL can fine-tune these policies
with 5x less real human preference data, taking the first step towards
minimizing human feedback while maximizing visuomotor robot policy alignment.
Representation-Aligned Preference-based Learning (RAPL) fine-tunes visual encoders using less human feedback to align visuomotor robot policies with user preferences in simulations and real-world tasks.
reinforcement learning from human feedback (RLHF) Representation-Aligned Preference-based Learning (RAPL) observation-only method visual rewards fine-tuning vision encoders feature matching X-Magical benchmark Franka Panda Diffusion Policies real human preference data