S-LoRA: Serving Thousands of Concurrent LoRA Adapters
The "pretrain-then-finetune" paradigm is commonly adopted in the deployment
of large language models. Low-Rank Adaptation (LoRA), a parameter-efficient
fine-tuning method, is often employed to adapt a base model to a multitude of
tasks, resulting in a substantial collection of LoRA adapters derived from one
base model. We observe that this paradigm presents significant opportunities
for batched inference during serving. To capitalize on these opportunities, we
present S-LoRA, a system designed for the scalable serving of many LoRA
adapters. S-LoRA stores all adapters in the main memory and fetches the
adapters used by the currently running queries to the GPU memory. To
efficiently use the GPU memory and reduce fragmentation, S-LoRA proposes
Unified Paging. Unified Paging uses a unified memory pool to manage dynamic
adapter weights with different ranks and KV cache tensors with varying sequence
lengths. Additionally, S-LoRA employs a novel tensor parallelism strategy and
highly optimized custom CUDA kernels for heterogeneous batching of LoRA
computation. Collectively, these features enable S-LoRA to serve thousands of
LoRA adapters on a single GPU or across multiple GPUs with a small overhead.
Compared to state-of-the-art libraries such as HuggingFace PEFT and vLLM (with
naive support of LoRA serving), S-LoRA can improve the throughput by up to 4
times and increase the number of served adapters by several orders of
magnitude. As a result, S-LoRA enables scalable serving of many task-specific
fine-tuned models and offers the potential for large-scale customized
fine-tuning services.
6549c8e17ab20d807df276dd
Ying Sheng
admin_assigned
2023-11-07T14:45:44.239Z
6455b969654d8bccae50736c
https://cdn-avatars.huggingface.co/v1/production/uploads/6455b969654d8bccae50736c/WsK0_sbQSVeosi-fh-O_U.jpeg
Ying Sheng
user
ying1123
6549c8e17ab20d807df276de
Shiyi Cao
admin_assigned
2023-11-07T14:45:51.245Z
64ebbae6895a36ab28de811a
https://cdn-avatars.huggingface.co/v1/production/uploads/64ebbae6895a36ab28de811a/gBiaQP4paS4L13eu-yRm7.jpeg
Shiyi Cao
user
eva98
6549c8e17ab20d807df276df
Dacheng Li
admin_assigned
2023-11-07T14:45:59.515Z
63715b25ffc0489ed7d1f415
https://cdn-avatars.huggingface.co/v1/production/uploads/63715b25ffc0489ed7d1f415/xZJepbs0LRqFbW1knnBKR.jpeg
Dacheng Li
user
DachengLi
6549c8e17ab20d807df276e0
Coleman Hooper
admin_assigned
2023-11-07T14:46:06.278Z
647d650483cdb93baf77dd8f
/avatars/03a076088efb009a5fe5f27033569320.svg
Coleman Hooper
user
chooper1
6549c8e17ab20d807df276e1
Nicholas Lee
claimed_verified
2024-07-15T09:02:30.645Z
656a4547996819a82836852d
/avatars/a5b522c18fced97140a156ff3e88d2e8.svg
Nicholas Lee
user
nicholaslee
6549c8e17ab20d807df276e2
Shuo Yang
claimed_verified
2025-05-27T08:00:36.002Z
642b970ceb31218a5f204a29
/avatars/582287f477bbb1a0842787145e375fd3.svg
andy-yang
user
andy-yang
6549c8e17ab20d807df276e3
Christopher Chou
admin_assigned
2023-11-07T14:47:13.582Z
62fb0555e8c9c532aa7c6e5d
https://cdn-avatars.huggingface.co/v1/production/uploads/62fb0555e8c9c532aa7c6e5d/AUds_M-PA7TCrj3gQ15IN.jpeg
Christopher Chou
user
BabyChou
6549c8e17ab20d807df276e4
Banghua Zhu
admin_assigned
2023-11-07T14:47:24.274Z
647b8885aba7062fe5c32000
/avatars/184e6199e20ad63b3de12436bdd00a00.svg
Banghua Zhu
user
banghua
6549c8e17ab20d807df276e5
Lianmin Zheng
6549c8e17ab20d807df276e6
Kurt Keutzer
admin_assigned
2023-11-07T14:47:49.327Z
6251bf4b183aa4266924ad91
https://cdn-avatars.huggingface.co/v1/production/uploads/1678041834400-6251bf4b183aa4266924ad91.jpeg
Kurt Keutzer
user
kurtkeutzer
6549c8e17ab20d807df276e7
Joseph E. Gonzalez
admin_assigned
2023-11-07T14:48:03.399Z
645d2e8401f4eaab2a0878ce
/avatars/1273c5fb607b4b622a746a42692fa632.svg
Joseph E. Gonzalez
user
ProfJoeyG
6549c8e17ab20d807df276e8
Ion Stoica
S-LoRA: Serving Thousands of Concurrent LoRA Adapters
The "pretrain-then-finetune" paradigm is commonly adopted in the deployment
of large language models. Low-Rank Adaptation (LoRA), a parameter-efficient
fine-tuning method, is often employed to adapt a base model to a multitude of
tasks, resulting in a substantial collection of LoRA adapters derived from one
base model. We observe that this paradigm presents significant opportunities
for batched inference during serving. To capitalize on these opportunities, we
present S-LoRA, a system designed for the scalable serving of many LoRA
adapters. S-LoRA stores all adapters in the main memory and fetches the
adapters used by the currently running queries to the GPU memory. To
efficiently use the GPU memory and reduce fragmentation, S-LoRA proposes
Unified Paging. Unified Paging uses a unified memory pool to manage dynamic
adapter weights with different ranks and KV cache tensors with varying sequence
lengths. Additionally, S-LoRA employs a novel tensor parallelism strategy and
highly optimized custom CUDA kernels for heterogeneous batching of LoRA
computation. Collectively, these features enable S-LoRA to serve thousands of
LoRA adapters on a single GPU or across multiple GPUs with a small overhead.
Compared to state-of-the-art libraries such as HuggingFace PEFT and vLLM (with
naive support of LoRA serving), S-LoRA can improve the throughput by up to 4
times and increase the number of served adapters by several orders of
magnitude. As a result, S-LoRA enables scalable serving of many task-specific
fine-tuned models and offers the potential for large-scale customized
fine-tuning services.
S-LoRA is a system that allows for efficient and scalable serving of numerous LoRA adapters using a unified memory pool, tensor parallelism, and custom CUDA kernels.
Low-Rank Adaptation (LoRA) parameter-efficient fine-tuning S-LoRA Unified Paging tensor parallelism custom CUDA kernels heterogeneous batching LoRA computation