On Speculative Decoding for Multimodal Large Language Models
Inference with Multimodal Large Language Models (MLLMs) is slow due to their
large-language-model backbone which suffers from memory bandwidth bottleneck
and generates tokens auto-regressively. In this paper, we explore the
application of speculative decoding to enhance the inference efficiency of
MLLMs, specifically the LLaVA 7B model. We show that a language-only model can
serve as a good draft model for speculative decoding with LLaVA 7B, bypassing
the need for image tokens and their associated processing components from the
draft model. Our experiments across three different tasks show that speculative
decoding can achieve a memory-bound speedup of up to 2.37times using a 115M
parameter language model that we trained from scratch. Additionally, we
introduce a compact LLaVA draft model incorporating an image adapter, which
shows marginal performance gains in image captioning while maintaining
comparable results in other tasks.
661dfac00d0c77a84194522d
Mukul Gagrani
661dfac00d0c77a84194522e
Raghavv Goel
claimed_verified
2024-04-18T19:35:08.743Z
649b6eb2f7cc759ab756adaf
/avatars/a0f7c7345dd653887122200fbe375c2b.svg
Raghavv Goel
user
RaghavvGoel
661dfac00d0c77a84194522f
Wonseok Jeon
admin_assigned
2024-04-18T11:04:51.108Z
64b8096b08e2452d18ef1a28
/avatars/6ccb615abe479c33872e44a8c8792db1.svg
Wonseok Jeon
user
jeonwonseok0125
661dfac00d0c77a841945230
Junyoung Park
661dfac00d0c77a841945231
Mingu Lee
661dfac00d0c77a841945232
Christopher Lott
On Speculative Decoding for Multimodal Large Language Models
Inference with Multimodal Large Language Models (MLLMs) is slow due to their
large-language-model backbone which suffers from memory bandwidth bottleneck
and generates tokens auto-regressively. In this paper, we explore the
application of speculative decoding to enhance the inference efficiency of
MLLMs, specifically the LLaVA 7B model. We show that a language-only model can
serve as a good draft model for speculative decoding with LLaVA 7B, bypassing
the need for image tokens and their associated processing components from the
draft model. Our experiments across three different tasks show that speculative
decoding can achieve a memory-bound speedup of up to 2.37times using a 115M
parameter language model that we trained from scratch. Additionally, we
introduce a compact LLaVA draft model incorporating an image adapter, which
shows marginal performance gains in image captioning while maintaining
comparable results in other tasks.
Speculative decoding enhances the inference efficiency of Multimodal Large Language Models (MLLMs), specifically LLaVA 7B, by using a language-only draft model and a compact LLaVA model with an image adapter.
Multimodal Large Language Models MLLMs speculative decoding LLaVA 7B draft model image adapter image captioning memory-bound speedup