Rethinking Compression: Reduced Order Modelling of Latent Features in
  Large Language Models
Due to the substantial scale of Large Language Models (LLMs), the direct
application of conventional compression methodologies proves impractical. The
computational demands associated with even minimal gradient updates present
challenges, particularly on consumer-grade hardware. This paper introduces an
innovative approach for the parametric and practical compression of LLMs based
on reduced order modelling, which entails low-rank decomposition within the
feature space and re-parameterization in the weight space. Notably, this
compression technique operates in a layer-wise manner, obviating the need for a
GPU device and enabling the compression of billion-scale models within
stringent constraints of both memory and time. Our method represents a
significant advancement in model compression by leveraging matrix
decomposition, demonstrating superior efficacy compared to the prevailing
state-of-the-art structured pruning method.
65791dea8628ec00e9129ce3
Arnav Chavan
claimed_verified
2023-12-13T08:32:56.799Z
648979dd308f19bc7c42899f
/avatars/6b7bde1d453f0328fcd5a86cfd9eb3b2.svg
Arnav Chavan
user
Arnav0400
65791dea8628ec00e9129ce4
Nahush Lele
claimed_verified
2023-12-13T08:32:52.611Z
65794432821f4394980cd5e3
/avatars/be5896fe21672949b7ffade6dd06e812.svg
Nahush Lele
user
Nahush-26
65791dea8628ec00e9129ce5
Deepak Gupta
Rethinking Compression: Reduced Order Modelling of Latent Features in
  Large Language Models
Due to the substantial scale of Large Language Models (LLMs), the direct
application of conventional compression methodologies proves impractical. The
computational demands associated with even minimal gradient updates present
challenges, particularly on consumer-grade hardware. This paper introduces an
innovative approach for the parametric and practical compression of LLMs based
on reduced order modelling, which entails low-rank decomposition within the
feature space and re-parameterization in the weight space. Notably, this
compression technique operates in a layer-wise manner, obviating the need for a
GPU device and enabling the compression of billion-scale models within
stringent constraints of both memory and time. Our method represents a
significant advancement in model compression by leveraging matrix
decomposition, demonstrating superior efficacy compared to the prevailing
state-of-the-art structured pruning method.
This paper presents a new compression technique for Large Language Models (LLMs) using reduced order modeling and low-rank decomposition that operates in a layer-wise manner without a GPU, achieving better results than structured pruning.
Large Language Models (LLMs) reduced order modeling low-rank decomposition weight space layer-wise compression matrix decomposition structured pruning