Speculate Deep and Accurate: Lossless and Training-Free Acceleration for Offloaded LLMs via Substitute Speculative Decoding
Pei-Shuo Wang, Jian-Jia Chen, Chun-Che Yang, Chi-Chih Chang, Ning-Chi Huang, Mohamed Abdelfattah, Kai-Chiang Wu
The immense model sizes of large language models (LLMs) challenge deployment on memory-limited consumer GPUs.    Model compression and parameter offloading are common strategies to address the memory limitation. While compression may degrade quality, offloading maintains quality but suffers from slow inference.    Speculative decoding presents a promising avenue to accelerate parameter offloading, utilizing a fast draft model to propose multiple draft tokens, which are then verified by the target LLM in parallel with a single forward pass. This reduces the time-consuming data transfers in forward passes involving offloaded weight transfers.    Existing methods often rely on pretrained weights of the same family, but actually require additional training to align with custom-trained models in reality.    Some other approaches involving draft model training often yield only modest speedups. This limitation arises from insufficient alignment with the target model, which prevents achieving higher token acceptance lengths.    To address these challenges and realize greater speedups, we propose SubSpec, a lossless and training-free approach to accelerate parameter offloading. SubSpec constructs a highly aligned draft model by generating low-bit quantized substitute layers from offloaded target LLM portions. Additionally, our method shares remaining GPU-resident layers and the KV-Cache. These sharing strategies further reduce memory overhead and enhance alignment.     SubSpec achieves a high average acceptance length, delivering 9.1$\times$ speedup for Qwen2.5 7B on MT-Bench (8GB VRAM limit) and an average of 12.5$\times$ speedup for Qwen2.5 32B on popular generation benchmarks (24GB VRAM limit). The code will be made publicly available.