Towards Fast Multilingual LLM Inference: Speculative Decoding and
  Specialized Drafters
Large language models (LLMs) have revolutionized natural language processing
and broadened their applicability across diverse commercial applications.
However, the deployment of these models is constrained by high inference time
in multilingual settings. To mitigate this challenge, this paper explores a
training recipe of an assistant model in speculative decoding, which are
leveraged to draft and-then its future tokens are verified by the target LLM.
We show that language-specific draft models, optimized through a targeted
pretrain-and-finetune strategy, substantially brings a speedup of inference
time compared to the previous methods. We validate these models across various
languages in inference time, out-of-domain speedup, and GPT-4o evaluation.
667a2585e737163dcb22f1df
Euiin Yi
admin_assigned
2024-06-25T08:15:32.506Z
66027cead56cd39a44992631
/avatars/3c8b679bc72f30c547782579e9592bb1.svg
Euiin Yi
user
euiin
667a2585e737163dcb22f1e0
Taehyeon Kim
claimed_verified
2024-06-25T07:33:36.931Z
639f32952e13e54dcbb9bf8d
/avatars/a797b72363e06cd2222165493bf7e39c.svg
Taehyeon Kim
user
Kthyeon
667a2585e737163dcb22f1e1
Hongseok Jeung
667a2585e737163dcb22f1e2
Du-Seong Chang
admin_assigned
2024-06-25T08:15:43.224Z
653b44214d105d9696cf5e7a
/avatars/907509576817ec2199a07d9b42af9c42.svg
Du-Seong Chang
user
Du-Seong
667a2585e737163dcb22f1e3
Se-Young Yun
Towards Fast Multilingual LLM Inference: Speculative Decoding and
  Specialized Drafters
Large language models (LLMs) have revolutionized natural language processing
and broadened their applicability across diverse commercial applications.
However, the deployment of these models is constrained by high inference time
in multilingual settings. To mitigate this challenge, this paper explores a
training recipe of an assistant model in speculative decoding, which are
leveraged to draft and-then its future tokens are verified by the target LLM.
We show that language-specific draft models, optimized through a targeted
pretrain-and-finetune strategy, substantially brings a speedup of inference
time compared to the previous methods. We validate these models across various
languages in inference time, out-of-domain speedup, and GPT-4o evaluation.
A speculative decoding approach with language-specific draft models speeds up inference time for large language models in multilingual settings.
large language models speculative decoding draft models pretrain-and-finetune strategy GPT-4o evaluation