PEEK: Guiding and Minimal Image Representations for Zero-Shot
  Generalization of Robot Manipulation Policies
Robotic manipulation policies often fail to generalize because they must
simultaneously learn where to attend, what actions to take, and how to execute
them. We argue that high-level reasoning about where and what can be offloaded
to vision-language models (VLMs), leaving policies to specialize in how to act.
We present PEEK (Policy-agnostic Extraction of Essential Keypoints), which
fine-tunes VLMs to predict a unified point-based intermediate representation:
1. end-effector paths specifying what actions to take, and 2. task-relevant
masks indicating where to focus. These annotations are directly overlaid onto
robot observations, making the representation policy-agnostic and transferable
across architectures. To enable scalable training, we introduce an automatic
annotation pipeline, generating labeled data across 20+ robot datasets spanning
9 embodiments. In real-world evaluations, PEEK consistently boosts zero-shot
generalization, including a 41.4x real-world improvement for a 3D policy
trained only in simulation, and 2-3.5x gains for both large VLAs and small
manipulation policies. By letting VLMs absorb semantic and visual complexity,
PEEK equips manipulation policies with the minimal cues they need--where, what,
and how. Website at https://peek-robot.github.io/.
68d4213ab1702be9357f4148
Jesse Zhang
claimed_verified
2025-09-25T07:11:22.180Z
62cf279e3200bfd438e80941
https://cdn-avatars.huggingface.co/v1/production/uploads/62cf279e3200bfd438e80941/4Sd_qWMqliNz02Y33SWHn.jpeg
Jesse Zhang
user
jesbu1
68d4213ab1702be9357f4149
Marius Memmel
68d4213ab1702be9357f414a
Kevin Kim
68d4213ab1702be9357f414b
Dieter Fox
68d4213ab1702be9357f414c
Jesse Thomason
68d4213ab1702be9357f414d
Fabio Ramos
68d4213ab1702be9357f414e
Erdem Bıyık
68d4213ab1702be9357f414f
Abhishek Gupta
68d4213ab1702be9357f4150
Anqi Li
PEEK: Guiding and Minimal Image Representations for Zero-Shot
  Generalization of Robot Manipulation Policies
Robotic manipulation policies often fail to generalize because they must
simultaneously learn where to attend, what actions to take, and how to execute
them. We argue that high-level reasoning about where and what can be offloaded
to vision-language models (VLMs), leaving policies to specialize in how to act.
We present PEEK (Policy-agnostic Extraction of Essential Keypoints), which
fine-tunes VLMs to predict a unified point-based intermediate representation:
1. end-effector paths specifying what actions to take, and 2. task-relevant
masks indicating where to focus. These annotations are directly overlaid onto
robot observations, making the representation policy-agnostic and transferable
across architectures. To enable scalable training, we introduce an automatic
annotation pipeline, generating labeled data across 20+ robot datasets spanning
9 embodiments. In real-world evaluations, PEEK consistently boosts zero-shot
generalization, including a 41.4x real-world improvement for a 3D policy
trained only in simulation, and 2-3.5x gains for both large VLAs and small
manipulation policies. By letting VLMs absorb semantic and visual complexity,
PEEK equips manipulation policies with the minimal cues they need--where, what,
and how. Website at https://peek-robot.github.io/.
PEEK fine-tunes vision-language models to predict essential keypoints for robotic manipulation, enhancing zero-shot generalization across different policies and robot embodiments.
vision-language models VLMs end-effector paths task-relevant masks policy-agnostic automatic annotation pipeline zero-shot generalization 3D policy VLAs manipulation policies