BiTA: Bi-Directional Tuning for Lossless Acceleration in Large Language
  Models
Large language models (LLMs) commonly employ autoregressive generation during
inference, leading to high memory bandwidth demand and consequently extended
latency. To mitigate this inefficiency, we present Bi-directional Tuning for
lossless Acceleration (BiTA), an innovative method expediting LLMs via
streamlined semi-autoregressive generation and draft verification. Inspired by
the concept of prompt tuning, we enhance LLMs with a parameter-efficient design
called bi-directional tuning for the capability in semi-autoregressive
generation. Employing efficient tree-based decoding, the models perform draft
candidate generation and verification in parallel, ensuring outputs identical
to their autoregressive counterparts under greedy sampling. BiTA serves as a
lightweight plug-in module, seamlessly boosting the inference efficiency of
existing LLMs without requiring additional assistance models or incurring
significant extra memory costs. Applying the proposed BiTA, LLaMA-2-70B-Chat
achieves a 2.7times speedup on the MT-Bench benchmark. Extensive experiments
confirm our method surpasses state-of-the-art acceleration techniques.
65b07cd7baebc9e3715336af
Feng Lin
claimed_verified
2024-01-24T12:18:49.541Z
6500215baf1e40214375ec7e
/avatars/622f3e6db1915cfd3d35586f1f457cab.svg
Feng Lin
user
linfeng93
65b07cd7baebc9e3715336b0
Hanling Yi
admin_assigned
2024-01-24T10:15:01.806Z
658b8d72e2db8a8e253f79db
/avatars/36337ce7fa25626bc293b133e29d706a.svg
Hanling Yi
user
ANTMAN2
65b07cd7baebc9e3715336b1
Hongbin Li
65b07cd7baebc9e3715336b2
Yifan Yang
65b07cd7baebc9e3715336b3
Xiaotian Yu
65b07cd7baebc9e3715336b4
Guangming Lu
65b07cd7baebc9e3715336b5
Rong Xiao
BiTA: Bi-Directional Tuning for Lossless Acceleration in Large Language
  Models
Large language models (LLMs) commonly employ autoregressive generation during
inference, leading to high memory bandwidth demand and consequently extended
latency. To mitigate this inefficiency, we present Bi-directional Tuning for
lossless Acceleration (BiTA), an innovative method expediting LLMs via
streamlined semi-autoregressive generation and draft verification. Inspired by
the concept of prompt tuning, we enhance LLMs with a parameter-efficient design
called bi-directional tuning for the capability in semi-autoregressive
generation. Employing efficient tree-based decoding, the models perform draft
candidate generation and verification in parallel, ensuring outputs identical
to their autoregressive counterparts under greedy sampling. BiTA serves as a
lightweight plug-in module, seamlessly boosting the inference efficiency of
existing LLMs without requiring additional assistance models or incurring
significant extra memory costs. Applying the proposed BiTA, LLaMA-2-70B-Chat
achieves a 2.7times speedup on the MT-Bench benchmark. Extensive experiments
confirm our method surpasses state-of-the-art acceleration techniques.
BiTA, a parameter-efficient bi-directional tuning method for LLMs, accelerates inference via parallel draft candidate generation and verification without additional memory costs.
autoregressive generation semi-autoregressive generation draft verification prompt tuning parameter-efficient tuning tree-based decoding MT-Bench benchmark