Steering Your Generalists: Improving Robotic Foundation Models via Value
  Guidance
Large, general-purpose robotic policies trained on diverse demonstration
datasets have been shown to be remarkably effective both for controlling a
variety of robots in a range of different scenes, and for acquiring broad
repertoires of manipulation skills. However, the data that such policies are
trained on is generally of mixed quality -- not only are human-collected
demonstrations unlikely to perform the task perfectly, but the larger the
dataset is, the harder it is to curate only the highest quality examples. It
also remains unclear how optimal data from one embodiment is for training on
another embodiment. In this paper, we present a general and broadly applicable
approach that enhances the performance of such generalist robot policies at
deployment time by re-ranking their actions according to a value function
learned via offline RL. This approach, which we call Value-Guided Policy
Steering (V-GPS), is compatible with a wide range of different generalist
policies, without needing to fine-tune or even access the weights of the
policy. We show that the same value function can improve the performance of
five different state-of-the-art policies with different architectures, even
though they were trained on distinct datasets, attaining consistent performance
improvement on multiple robotic platforms across a total of 12 tasks. Code and
videos can be found at: https://nakamotoo.github.io/V-GPS
671ae55d82e53a391ca1707e
Mitsuhiko Nakamoto
671ae55d82e53a391ca1707f
Oier Mees
671ae55d82e53a391ca17080
Aviral Kumar
claimed_verified
2025-02-11T15:34:18.785Z
67315a324d2eab8035de786a
https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/INH5_PlcJYfYcGkyibjgP.png
Aviral Kumar
user
aviralku
671ae55d82e53a391ca17081
Sergey Levine
Steering Your Generalists: Improving Robotic Foundation Models via Value
  Guidance
Large, general-purpose robotic policies trained on diverse demonstration
datasets have been shown to be remarkably effective both for controlling a
variety of robots in a range of different scenes, and for acquiring broad
repertoires of manipulation skills. However, the data that such policies are
trained on is generally of mixed quality -- not only are human-collected
demonstrations unlikely to perform the task perfectly, but the larger the
dataset is, the harder it is to curate only the highest quality examples. It
also remains unclear how optimal data from one embodiment is for training on
another embodiment. In this paper, we present a general and broadly applicable
approach that enhances the performance of such generalist robot policies at
deployment time by re-ranking their actions according to a value function
learned via offline RL. This approach, which we call Value-Guided Policy
Steering (V-GPS), is compatible with a wide range of different generalist
policies, without needing to fine-tune or even access the weights of the
policy. We show that the same value function can improve the performance of
five different state-of-the-art policies with different architectures, even
though they were trained on distinct datasets, attaining consistent performance
improvement on multiple robotic platforms across a total of 12 tasks. Code and
videos can be found at: https://nakamotoo.github.io/V-GPS
Value-Guided Policy Steering (V-GPS) enhances the performance of generalist robotic policies at deployment by re-ranking their actions using an offline RL learned value function, applicable across different policy architectures and robotic platforms.
offline RL Value-Guided Policy Steering V-GPS generalist robot policies robotic policies deployment time re-ranking value function robotic platforms manipulation skills