SiRA: Sparse Mixture of Low Rank Adaptation
Parameter Efficient Tuning has been an prominent approach to adapt the Large
Language Model to downstream tasks. Most previous works considers adding the
dense trainable parameters, where all parameters are used to adapt certain
task. We found this less effective empirically using the example of LoRA that
introducing more trainable parameters does not help. Motivated by this we
investigate the importance of leveraging "sparse" computation and propose SiRA:
sparse mixture of low rank adaption. SiRA leverages the Sparse Mixture of
Expert(SMoE) to boost the performance of LoRA. Specifically it enforces the top
k experts routing with a capacity limit restricting the maximum number of
tokens each expert can process. We propose a novel and simple expert dropout on
top of gating network to reduce the over-fitting issue. Through extensive
experiments, we verify SiRA performs better than LoRA and other mixture of
expert approaches across different single tasks and multitask settings.
6555ab2638985840b36fb3c8
Yun Zhu
6555ab2638985840b36fb3c9
Nevan Wichers
6555ab2638985840b36fb3ca
Chu-Cheng Lin
admin_assigned
2023-11-16T10:15:10.635Z
6137b1b20317b0a5c10877c6
/avatars/8eccd696bffab67e431aed200e96af14.svg
Chu-Cheng Lin
user
kitsing
6555ab2638985840b36fb3cb
Xinyi Wang
6555ab2638985840b36fb3cc
Tianlong Chen
6555ab2638985840b36fb3cd
Lei Shu
6555ab2638985840b36fb3ce
Han Lu
6555ab2638985840b36fb3cf
Canoee Liu
admin_assigned
2023-11-16T10:17:20.424Z
63449f6acbc4ea2ab7405361
https://cdn-avatars.huggingface.co/v1/production/uploads/1665441596687-noauth.jpeg
Canoee Liu
user
Canoee
6555ab2638985840b36fb3d0
Liangchen Luo
admin_assigned
2023-11-16T10:17:27.627Z
645a9ab321ab438e732a7b8d
https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/bNp7XtXqPQoauh7CU5B7b.jpeg
Liangchen Luo
user
luolc
6555ab2638985840b36fb3d1
Jindong Chen
admin_assigned
2023-11-16T10:14:46.481Z
635242b8b3678a43742baec6
/avatars/fc9f6d3922c76ab902cb11ed23554c54.svg
chenjindong
user
chenjindong
6555ab2638985840b36fb3d2
Lei Meng
SiRA: Sparse Mixture of Low Rank Adaptation
Parameter Efficient Tuning has been an prominent approach to adapt the Large
Language Model to downstream tasks. Most previous works considers adding the
dense trainable parameters, where all parameters are used to adapt certain
task. We found this less effective empirically using the example of LoRA that
introducing more trainable parameters does not help. Motivated by this we
investigate the importance of leveraging "sparse" computation and propose SiRA:
sparse mixture of low rank adaption. SiRA leverages the Sparse Mixture of
Expert(SMoE) to boost the performance of LoRA. Specifically it enforces the top
k experts routing with a capacity limit restricting the maximum number of
tokens each expert can process. We propose a novel and simple expert dropout on
top of gating network to reduce the over-fitting issue. Through extensive
experiments, we verify SiRA performs better than LoRA and other mixture of
expert approaches across different single tasks and multitask settings.
SiRA, a sparse mixture of low-rank adaptation with expert dropout and capacity limits, improves performance over LoRA and other mixture of expert methods in various tasks.
parameter efficient tuning large language model dense trainable parameters LoRA sparse computation SiRA sparse mixture of low rank adaption Sparse Mixture of Experts SMoE expert dropout gating network multitask settings