COSPADI: Compressing LLMs via Calibration-Guided Sparse Dictionary
  Learning
Post-training compression of large language models (LLMs) largely relies on
low-rank weight approximation, which represents each column of a weight matrix
in a shared low-dimensional subspace. While this is a computationally efficient
strategy, the imposed structural constraint is rigid and can lead to a
noticeable model accuracy drop. In this work, we propose CoSpaDi (Compression
via Sparse Dictionary Learning), a novel training-free compression framework
that replaces low-rank decomposition with a more flexible structured sparse
factorization in which each weight matrix is represented with a dense
dictionary and a column-sparse coefficient matrix. This formulation enables a
union-of-subspaces representation: different columns of the original weight
matrix are approximated in distinct subspaces spanned by adaptively selected
dictionary atoms, offering greater expressiveness than a single invariant
basis. Crucially, CoSpaDi leverages a small calibration dataset to optimize the
factorization such that the output activations of compressed projection layers
closely match those of the original ones, thereby minimizing functional
reconstruction error rather than mere weight approximation. This data-aware
strategy preserves better model fidelity without any fine-tuning under
reasonable compression ratios. Moreover, the resulting structured sparsity
allows efficient sparse-dense matrix multiplication and is compatible with
post-training quantization for further memory and latency gains. We evaluate
CoSpaDi across multiple Llama and Qwen models under per-layer and per-group
settings at 20-50\% compression ratios, demonstrating consistent superiority
over state-of-the-art data-aware low-rank methods both in accuracy and
perplexity. Our results establish structured sparse dictionary learning as a
powerful alternative to conventional low-rank approaches for efficient LLM
deployment.
68da98cfd2bf1f4b15ec7137
Dmitriy Shopkhoev
68da98cfd2bf1f4b15ec7138
Denis Makhov
68da98cfd2bf1f4b15ec7139
Magauiya Zhussip
68da98cfd2bf1f4b15ec713a
Ammar Ali
claimed_verified
2025-09-30T06:36:16.248Z
6166db59f78a267701a78c2a
/avatars/8784efc36f67719e9455b1f081340ed9.svg
Ammar Ali
user
ammarali32
68da98cfd2bf1f4b15ec713b
Stamatios Lefkimmiatis
COSPADI: Compressing LLMs via Calibration-Guided Sparse Dictionary
  Learning
Post-training compression of large language models (LLMs) largely relies on
low-rank weight approximation, which represents each column of a weight matrix
in a shared low-dimensional subspace. While this is a computationally efficient
strategy, the imposed structural constraint is rigid and can lead to a
noticeable model accuracy drop. In this work, we propose CoSpaDi (Compression
via Sparse Dictionary Learning), a novel training-free compression framework
that replaces low-rank decomposition with a more flexible structured sparse
factorization in which each weight matrix is represented with a dense
dictionary and a column-sparse coefficient matrix. This formulation enables a
union-of-subspaces representation: different columns of the original weight
matrix are approximated in distinct subspaces spanned by adaptively selected
dictionary atoms, offering greater expressiveness than a single invariant
basis. Crucially, CoSpaDi leverages a small calibration dataset to optimize the
factorization such that the output activations of compressed projection layers
closely match those of the original ones, thereby minimizing functional
reconstruction error rather than mere weight approximation. This data-aware
strategy preserves better model fidelity without any fine-tuning under
reasonable compression ratios. Moreover, the resulting structured sparsity
allows efficient sparse-dense matrix multiplication and is compatible with
post-training quantization for further memory and latency gains. We evaluate
CoSpaDi across multiple Llama and Qwen models under per-layer and per-group
settings at 20-50\% compression ratios, demonstrating consistent superiority
over state-of-the-art data-aware low-rank methods both in accuracy and
perplexity. Our results establish structured sparse dictionary learning as a
powerful alternative to conventional low-rank approaches for efficient LLM
deployment.
CoSpaDi, a training-free compression framework, uses structured sparse dictionary learning to compress large language models with better accuracy and efficiency compared to low-rank methods.
low-rank weight approximation CoSpaDi structured sparse factorization dense dictionary column-sparse coefficient matrix union-of-subspaces representation dictionary atoms calibration dataset functional reconstruction error sparse-dense matrix multiplication post-training quantization per-layer compression per-group compression accuracy perplexity