DeeR-VLA: Dynamic Inference of Multimodal Large Language Models for
  Efficient Robot Execution
MLLMs have demonstrated remarkable comprehension and reasoning capabilities
with complex language and visual data. These advances have spurred the vision
of establishing a generalist robotic MLLM proficient in understanding complex
human instructions and accomplishing various embodied tasks. However,
developing MLLMs for real-world robots is challenging due to the typically
limited computation and memory capacities available on robotic platforms. In
contrast, the inference of MLLMs involves storing billions of parameters and
performing tremendous computation, imposing significant hardware demands. In
our paper, we propose a Dynamic Early-Exit Framework for Robotic
Vision-Language-Action Model (DeeR-VLA, or simply DeeR) that automatically
adjusts the size of the activated MLLM based on each situation at hand. The
approach leverages a multi-exit architecture in MLLMs, which allows the model
to terminate processing once a proper size of the model has been activated for
a specific situation, thus avoiding further redundant computation.
Additionally, we develop novel algorithms that establish early-termination
criteria for DeeR, conditioned on predefined demands such as average
computational cost (i.e., power consumption), as well as peak computational
consumption (i.e., latency) and GPU memory usage. These enhancements ensure
that DeeR operates efficiently under varying resource constraints while
maintaining competitive performance. On the CALVIN robot manipulation
benchmark, DeeR demonstrates significant reductions in computational costs of
LLM by 5.2-6.5x and GPU memory of LLM by 2-6x without compromising performance.
Code and checkpoints are available at https://github.com/yueyang130/DeeR-VLA.
672ad61870734f59f1bdb3ea
Yang Yue
claimed_verified
2025-05-07T08:10:13.804Z
649d475111592b1a765ac1a3
https://cdn-avatars.huggingface.co/v1/production/uploads/649d475111592b1a765ac1a3/rjORJjErJq-mthghan08U.jpeg
Yang Yue
user
Yang130
672ad61870734f59f1bdb3eb
Yulin Wang
672ad61870734f59f1bdb3ec
Bingyi Kang
672ad61870734f59f1bdb3ed
Yizeng Han
672ad61870734f59f1bdb3ee
Shenzhi Wang
claimed_verified
2024-11-15T09:37:13.354Z
6486dde1f74857df3f1a5828
https://cdn-avatars.huggingface.co/v1/production/uploads/6486dde1f74857df3f1a5828/FgE80CpalBO5qqArdfwxA.jpeg
Shenzhi Wang
user
shenzhi-wang
672ad61870734f59f1bdb3ef
Shiji Song
672ad61870734f59f1bdb3f0
Jiashi Feng
672ad61870734f59f1bdb3f1
Gao Huang
DeeR-VLA: Dynamic Inference of Multimodal Large Language Models for
  Efficient Robot Execution
MLLMs have demonstrated remarkable comprehension and reasoning capabilities
with complex language and visual data. These advances have spurred the vision
of establishing a generalist robotic MLLM proficient in understanding complex
human instructions and accomplishing various embodied tasks. However,
developing MLLMs for real-world robots is challenging due to the typically
limited computation and memory capacities available on robotic platforms. In
contrast, the inference of MLLMs involves storing billions of parameters and
performing tremendous computation, imposing significant hardware demands. In
our paper, we propose a Dynamic Early-Exit Framework for Robotic
Vision-Language-Action Model (DeeR-VLA, or simply DeeR) that automatically
adjusts the size of the activated MLLM based on each situation at hand. The
approach leverages a multi-exit architecture in MLLMs, which allows the model
to terminate processing once a proper size of the model has been activated for
a specific situation, thus avoiding further redundant computation.
Additionally, we develop novel algorithms that establish early-termination
criteria for DeeR, conditioned on predefined demands such as average
computational cost (i.e., power consumption), as well as peak computational
consumption (i.e., latency) and GPU memory usage. These enhancements ensure
that DeeR operates efficiently under varying resource constraints while
maintaining competitive performance. On the CALVIN robot manipulation
benchmark, DeeR demonstrates significant reductions in computational costs of
LLM by 5.2-6.5x and GPU memory of LLM by 2-6x without compromising performance.
Code and checkpoints are available at https://github.com/yueyang130/DeeR-VLA.
A Dynamic Early-Exit Framework for Robotic Vision-Language-Action Models optimizes computation and memory usage for MLLMs in robotics without sacrificing performance.
MLLMs Dynamic Early-Exit Framework DeeR-VLA multi-exit architecture early-termination criteria CALVIN robot manipulation benchmark