VER: Vision Expert Transformer for Robot Learning via Foundation
  Distillation and Dynamic Routing
Pretrained vision foundation models (VFMs) advance robotic learning via rich
visual representations, yet individual VFMs typically excel only in specific
domains, limiting generality across tasks. Distilling multiple VFMs into a
unified representation for policy can mitigate this limitation but often yields
inflexible task-specific feature selection and requires costly full re-training
to incorporate robot-domain knowledge. We propose VER, a Vision Expert
transformer for Robot learning. During pretraining, VER distills multiple VFMs
into a vision expert library. It then fine-tunes only a lightweight routing
network (fewer than 0.4% of parameters) to dynamically select task-relevant
experts from the pretrained library for downstream robot tasks. We further
introduce Patchwise Expert Routing with Curriculum Top-K Annealing to improve
both flexibility and precision of dynamic expert selection. Moreover, VER
supports parameter-efficient finetuning for scalable expert utilization and
adaptive robot-domain knowledge integration. Across 17 diverse robotic tasks
and multiple policy heads, VER achieves state-of-the-art performance. We find
that VER reduces large-norm outliers in task-irrelevant regions (e.g.,
background) and concentrates on task-critical regions. Visualizations and codes
can be found in https://yixiaowang7.github.io/ver_page/.
68ede69ede1fee572713a9cc
Yixiao Wang
68ede69ede1fee572713a9cd
Mingxiao Huo
68ede69ede1fee572713a9ce
Zhixuan Liang
claimed_verified
2025-10-18T04:50:19.961Z
662a471e94baa018b00c0f5c
/avatars/62a67a2ee6e4b9a7124f8b02b9b3f280.svg
Zhixuan Liang
user
Liang-ZX
68ede69ede1fee572713a9cf
Yushi Du
68ede69ede1fee572713a9d0
Lingfeng Sun
68ede69ede1fee572713a9d1
Haotian Lin
68ede69ede1fee572713a9d2
Jinghuan Shang
68ede69ede1fee572713a9d3
Chensheng Peng
68ede69ede1fee572713a9d4
Mohit Bansal
68ede69ede1fee572713a9d5
Mingyu Ding
68ede69ede1fee572713a9d6
Masayoshi Tomizuka
VER: Vision Expert Transformer for Robot Learning via Foundation
  Distillation and Dynamic Routing
Pretrained vision foundation models (VFMs) advance robotic learning via rich
visual representations, yet individual VFMs typically excel only in specific
domains, limiting generality across tasks. Distilling multiple VFMs into a
unified representation for policy can mitigate this limitation but often yields
inflexible task-specific feature selection and requires costly full re-training
to incorporate robot-domain knowledge. We propose VER, a Vision Expert
transformer for Robot learning. During pretraining, VER distills multiple VFMs
into a vision expert library. It then fine-tunes only a lightweight routing
network (fewer than 0.4% of parameters) to dynamically select task-relevant
experts from the pretrained library for downstream robot tasks. We further
introduce Patchwise Expert Routing with Curriculum Top-K Annealing to improve
both flexibility and precision of dynamic expert selection. Moreover, VER
supports parameter-efficient finetuning for scalable expert utilization and
adaptive robot-domain knowledge integration. Across 17 diverse robotic tasks
and multiple policy heads, VER achieves state-of-the-art performance. We find
that VER reduces large-norm outliers in task-irrelevant regions (e.g.,
background) and concentrates on task-critical regions. Visualizations and codes
can be found in https://yixiaowang7.github.io/ver_page/.
VER, a Vision Expert Transformer, dynamically selects task-relevant experts from a pretrained vision expert library, achieving state-of-the-art performance across diverse robotic tasks with parameter-efficient fine-tuning.
pretrained vision foundation models VFMs unified representation policy routing network Patchwise Expert Routing Curriculum Top-K Annealing parameter-efficient fine-tuning expert utilization adaptive robot-domain knowledge integration