Configurable Foundation Models: Building LLMs from a Modular Perspective
Advancements in LLMs have recently unveiled challenges tied to computational
efficiency and continual scalability due to their requirements of huge
parameters, making the applications and evolution of these models on devices
with limited computation resources and scenarios requiring various abilities
increasingly cumbersome. Inspired by modularity within the human brain, there
is a growing tendency to decompose LLMs into numerous functional modules,
allowing for inference with part of modules and dynamic assembly of modules to
tackle complex tasks, such as mixture-of-experts. To highlight the inherent
efficiency and composability of the modular approach, we coin the term brick to
represent each functional module, designating the modularized structure as
configurable foundation models. In this paper, we offer a comprehensive
overview and investigation of the construction, utilization, and limitation of
configurable foundation models. We first formalize modules into emergent bricks
- functional neuron partitions that emerge during the pre-training phase, and
customized bricks - bricks constructed via additional post-training to improve
the capabilities and knowledge of LLMs. Based on diverse functional bricks, we
further present four brick-oriented operations: retrieval and routing, merging,
updating, and growing. These operations allow for dynamic configuration of LLMs
based on instructions to handle complex tasks. To verify our perspective, we
conduct an empirical analysis on widely-used LLMs. We find that the FFN layers
follow modular patterns with functional specialization of neurons and
functional neuron partitions. Finally, we highlight several open issues and
directions for future research. Overall, this paper aims to offer a fresh
modular perspective on existing LLM research and inspire the future creation of
more efficient and scalable foundational models.
66db3dfca46deac8798e3c95
Chaojun Xiao
claimed_verified
2024-09-09T06:55:38.495Z
608f6d72283d0a8d7be9d1f9
/avatars/7f499a37019359a3c488ba6cc11751fc.svg
Chaojun XIAO
user
xcjthu
66db3dfca46deac8798e3c96
Zhengyan Zhang
admin_assigned
2024-09-09T07:52:09.262Z
65654ed2219af7f841640f27
/avatars/e6904b3479fc5e65ea1f752919ca8290.svg
Zhengyan Zhang
user
ZhengyanZhang
66db3dfca46deac8798e3c97
Chenyang Song
admin_assigned
2024-09-09T07:52:39.751Z
64c09684e56520a63d35ec87
/avatars/86a338e8d122a66a94143bbb9bf3ebf8.svg
Chenyang Song
user
Raincleared
66db3dfca46deac8798e3c98
Dazhi Jiang
admin_assigned
2024-09-09T07:54:28.930Z
65ba471ad88a65abb9328ee2
/avatars/956238ce5034091e64d026b0272c4400.svg
Dazhi Jiang
user
thuzhizhi
66db3dfca46deac8798e3c99
Feng Yao
claimed_verified
2024-09-09T06:55:40.389Z
64c8b2c5c547ed5243e14a6e
/avatars/96d4a9010f96001c8cff235915926390.svg
Feng Yao
user
fengyao1909
66db3dfca46deac8798e3c9a
Xu Han
66db3dfca46deac8798e3c9b
Xiaozhi Wang
admin_assigned
2024-09-09T07:54:55.877Z
648c48e7a47a0850f9458b67
/avatars/47d71d80f9901313feb0199c37296389.svg
Xiaozhi Wang
user
wangxz098
66db3dfca46deac8798e3c9c
Shuo Wang
66db3dfca46deac8798e3c9d
Yufei Huang
66db3dfca46deac8798e3c9e
Guanyu Lin
admin_assigned
2024-09-09T07:56:03.905Z
6586a7a84f9d2b955ee21b0d
/avatars/4f55c26fe62a12aab8f1413387d2e75a.svg
Guanyu Lin
user
cmulgy
66db3dfca46deac8798e3c9f
Yingfa Chen
admin_assigned
2024-09-09T07:56:20.870Z
6144e4667f2544bb450787b2
https://cdn-avatars.huggingface.co/v1/production/uploads/6144e4667f2544bb450787b2/5wSDDqJbI4TGtBLP9IvjY.png
Yingfa Chen
user
chen-yingfa
66db3dfca46deac8798e3ca0
Weilin Zhao
admin_assigned
2024-09-09T07:56:32.800Z
64b8ff3d95bd42c770878042
/avatars/564a4dccdf9e5b813a99979b0ef58183.svg
Weilin Zhao
user
Achazwl
66db3dfca46deac8798e3ca1
Yuge Tu
admin_assigned
2024-09-09T07:56:40.581Z
637f3220e0d5fd3199579458
https://cdn-avatars.huggingface.co/v1/production/uploads/637f3220e0d5fd3199579458/uaPvnmGqRcScz9RgoUcVY.png
Yuge Tu
user
tuyuge
66db3dfca46deac8798e3ca2
Zexuan Zhong
admin_assigned
2024-09-09T07:56:46.564Z
6514dd906e0d30ed7e0df37e
/avatars/893fb3fe2db1b684c2d9801d9f00aa57.svg
Zexuan Zhong
user
a3616001
66db3dfca46deac8798e3ca3
Ao Zhang
admin_assigned
2024-09-09T07:57:03.318Z
649a9133225ce03f1c82e0b2
/avatars/f3702a46a0a310d0292dc690794e7cc9.svg
Ao Zhang
user
AoZhang
66db3dfca46deac8798e3ca4
Chenglei Si
admin_assigned
2024-09-09T07:57:09.916Z
6144c0aa8a3e54bb3ecef8ea
https://cdn-avatars.huggingface.co/v1/production/uploads/1631895645405-noauth.jpeg
Chenglei Si
user
CLS
66db3dfca46deac8798e3ca5
Khai Hao Moo
66db3dfca46deac8798e3ca6
Chenyang Zhao
66db3dfca46deac8798e3ca7
Huimin Chen
66db3dfca46deac8798e3ca8
Yankai Lin
admin_assigned
2024-09-09T07:57:44.913Z
657a651e1433ea7d44de6397
/avatars/ccfc76f94595a38ff4a80f77c911eabf.svg
Yankai Lin
user
lyk423
66db3dfca46deac8798e3ca9
Zhiyuan Liu
admin_assigned
2024-09-09T07:58:01.998Z
6472f1f7485a7c8e1bcf507b
/avatars/79dca137583875191ea883b193b630b7.svg
Zhiyuan Liu
user
zibuyu9
66db3dfca46deac8798e3caa
Jingbo Shang
admin_assigned
2024-09-09T07:58:13.783Z
660655119e3555d648f6c6b5
/avatars/ae1e2c97a08be39b77a9f1a5c2a718ef.svg
Jingbo Shang
user
shangjingbo
66db3dfca46deac8798e3cab
Maosong Sun
Configurable Foundation Models: Building LLMs from a Modular Perspective
Advancements in LLMs have recently unveiled challenges tied to computational
efficiency and continual scalability due to their requirements of huge
parameters, making the applications and evolution of these models on devices
with limited computation resources and scenarios requiring various abilities
increasingly cumbersome. Inspired by modularity within the human brain, there
is a growing tendency to decompose LLMs into numerous functional modules,
allowing for inference with part of modules and dynamic assembly of modules to
tackle complex tasks, such as mixture-of-experts. To highlight the inherent
efficiency and composability of the modular approach, we coin the term brick to
represent each functional module, designating the modularized structure as
configurable foundation models. In this paper, we offer a comprehensive
overview and investigation of the construction, utilization, and limitation of
configurable foundation models. We first formalize modules into emergent bricks
- functional neuron partitions that emerge during the pre-training phase, and
customized bricks - bricks constructed via additional post-training to improve
the capabilities and knowledge of LLMs. Based on diverse functional bricks, we
further present four brick-oriented operations: retrieval and routing, merging,
updating, and growing. These operations allow for dynamic configuration of LLMs
based on instructions to handle complex tasks. To verify our perspective, we
conduct an empirical analysis on widely-used LLMs. We find that the FFN layers
follow modular patterns with functional specialization of neurons and
functional neuron partitions. Finally, we highlight several open issues and
directions for future research. Overall, this paper aims to offer a fresh
modular perspective on existing LLM research and inspire the future creation of
more efficient and scalable foundational models.
The paper explores a modular approach for large language models (LLMs) by decomposing them into functional modules (bricks) to enhance computational efficiency and scalability.
LLMs computational efficiency continual scalability parameters modularizing functional modules mixture-of-experts configurable foundation models emergent bricks customized bricks retrieval and routing merging updating growing FFN layers functional specialization functional neuron partitions