Provable Ordering and Continuity in Vision-Language Pretraining for Generalizable Embodied Agents
Zhizhen Zhang, Lei Zhu, Zhen Fang, Zi Huang, Yadan Luo
Pre-training vision-language representations on human action videos has emergedas a promising approach to reduce reliance on large-scale expert demonstrationsfor training embodied agents. However, prior methods often employ time con-trastive learning based on goal-reaching heuristics, progressively aligning languageinstructions from the initial to the final frame. This overemphasis on future framescan result in erroneous vision-language associations, as actions may terminateearly or include irrelevant moments in the end. To address this issue, we proposeAction Temporal Coherence Learning (AcTOL) to learn ordered and continuousvision-language representations without rigid goal-based constraint. AcTOL treatsa video as a continuous trajectory where it (1) contrasts semantic differences be-tween frames to reflect their natural ordering, and (2) imposes a local Brownianbridge constraint to ensure smooth transitions across intermediate frames. Exten-sive imitation learning experiments on both simulated and real robots show that thepretrained features significantly enhance downstream manipulation tasks with highrobustness to different linguistic styles of instructions, offering a viable pathwaytoward generalized embodied agents. We provide source code and demo videos inthe supplemental material for reference.