OTTER: A Vision-Language-Action Model with Text-Aware Visual Feature
  Extraction
Vision-Language-Action (VLA) models aim to predict robotic actions based on
visual observations and language instructions. Existing approaches require
fine-tuning pre-trained visionlanguage models (VLMs) as visual and language
features are independently fed into downstream policies, degrading the
pre-trained semantic alignments. We propose OTTER, a novel VLA architecture
that leverages these existing alignments through explicit, text-aware visual
feature extraction. Instead of processing all visual features, OTTER
selectively extracts and passes only task-relevant visual features that are
semantically aligned with the language instruction to the policy transformer.
This allows OTTER to keep the pre-trained vision-language encoders frozen.
Thereby, OTTER preserves and utilizes the rich semantic understanding learned
from large-scale pre-training, enabling strong zero-shot generalization
capabilities. In simulation and real-world experiments, OTTER significantly
outperforms existing VLA models, demonstrating strong zeroshot generalization
to novel objects and environments. Video, code, checkpoints, and dataset:
https://ottervla.github.io/.
67d1bc433c18d49c52723521
Huang Huang
67d1bc433c18d49c52723522
Fangchen Liu
67d1bc433c18d49c52723523
Letian Fu
extracted_pending
2025-03-12T16:54:28.791Z
64908d6b0c18343a09361b93
/avatars/2fa44344fe760f68f19de775b76f45b8.svg
Max (Letian) Fu
user
mlfu7
67d1bc433c18d49c52723524
Tingfan Wu
67d1bc433c18d49c52723525
Mustafa Mukadam
67d1bc433c18d49c52723526
Jitendra Malik
67d1bc433c18d49c52723527
Ken Goldberg
67d1bc433c18d49c52723528
Pieter Abbeel
OTTER: A Vision-Language-Action Model with Text-Aware Visual Feature
  Extraction
Vision-Language-Action (VLA) models aim to predict robotic actions based on
visual observations and language instructions. Existing approaches require
fine-tuning pre-trained visionlanguage models (VLMs) as visual and language
features are independently fed into downstream policies, degrading the
pre-trained semantic alignments. We propose OTTER, a novel VLA architecture
that leverages these existing alignments through explicit, text-aware visual
feature extraction. Instead of processing all visual features, OTTER
selectively extracts and passes only task-relevant visual features that are
semantically aligned with the language instruction to the policy transformer.
This allows OTTER to keep the pre-trained vision-language encoders frozen.
Thereby, OTTER preserves and utilizes the rich semantic understanding learned
from large-scale pre-training, enabling strong zero-shot generalization
capabilities. In simulation and real-world experiments, OTTER significantly
outperforms existing VLA models, demonstrating strong zeroshot generalization
to novel objects and environments. Video, code, checkpoints, and dataset:
https://ottervla.github.io/.
OTTER selectively extracts task-relevant semantic-aligned visual features for VLA models, preserving pre-trained vision-language encoders and enhancing zero-shot generalization.
vision-language-action models VLA pre-trained vision-language models VLMs downstream policies semantic alignments explicit text-aware visual feature extraction policy transformer zero-shot generalization simulations real-world experiments