Quantizing Large Language Models for Code Generation: A Differentiated
  Replication
Large Language Models (LLMs) have shown an impressive capability in code
generation and, specifically, to automatically implement requirements described
in natural language. The LLM effectiveness generally increases with its size:
The higher the number of LLM's trainable parameters the better its ability to
implement code. However, when it comes to deploying LLM-based code generators,
larger LLMs pose significant challenges related to their memory (and,
consequently, carbon) footprint. A previous work by Wei et al. proposed to
leverage quantization techniques to reduce the memory footprint of LLM-based
code generators without substantially degrading their effectiveness. In short,
they studied LLMs featuring up to 16B parameters, quantizing their precision
from floating point 32 bits down to int 8 bits and showing their limited impact
on code generation performance. Given the fast pace at which LLM capabilities
and quantization techniques are evolving, in this work we present a
differentiated replication of the work by Wei et al. in which we consider (i)
on the one side, more recent and larger code-related LLMs, of up to 34B
parameters; (ii) the latest advancements in model quantization techniques,
which allow pushing the compression to the extreme quantization level of 2 bits
per model parameter and; (iii) different types of calibration datasets to guide
the quantization process, including code-specific ones. Our empirical
evaluation reveals that the new frontier for LLM quantization is 4-bit
precision, resulting in an average memory footprint reduction of 70% compared
to the original model without observing any significant decrease in
performance. Additionally, when the quantization becomes even more extreme (3
and 2 bits), a code-specific calibration dataset helps to limit the loss of
performance.
67d27d4c6fbfb8ee21886b60
Alessandro Giagnorio
claimed_verified
2025-03-13T08:23:50.840Z
663486a1f64712540644cb68
https://cdn-avatars.huggingface.co/v1/production/uploads/663486a1f64712540644cb68/YZFR41ERY6UrC6rCC6Nan.jpeg
Alessandro
user
Devy1
67d27d4c6fbfb8ee21886b61
Antonio Mastropaolo
67d27d4c6fbfb8ee21886b62
Saima Afrin
67d27d4c6fbfb8ee21886b63
Massimiliano Di Penta
extracted_pending
2025-03-13T06:38:05.744Z
63f378004745321de351a554
/avatars/3be79f0f6eb0bcfdc9f31b46d2bafc14.svg
Max Di Penta
user
mdiipenta
67d27d4c6fbfb8ee21886b64
Gabriele Bavota
claimed_verified
2025-05-06T16:53:43.585Z
6638bea59e57161faac814e7
/avatars/91375b88945af50e51b7229a789a31b8.svg
Gabriele Bavota
user
gbavota
Quantizing Large Language Models for Code Generation: A Differentiated
  Replication
Large Language Models (LLMs) have shown an impressive capability in code
generation and, specifically, to automatically implement requirements described
in natural language. The LLM effectiveness generally increases with its size:
The higher the number of LLM's trainable parameters the better its ability to
implement code. However, when it comes to deploying LLM-based code generators,
larger LLMs pose significant challenges related to their memory (and,
consequently, carbon) footprint. A previous work by Wei et al. proposed to
leverage quantization techniques to reduce the memory footprint of LLM-based
code generators without substantially degrading their effectiveness. In short,
they studied LLMs featuring up to 16B parameters, quantizing their precision
from floating point 32 bits down to int 8 bits and showing their limited impact
on code generation performance. Given the fast pace at which LLM capabilities
and quantization techniques are evolving, in this work we present a
differentiated replication of the work by Wei et al. in which we consider (i)
on the one side, more recent and larger code-related LLMs, of up to 34B
parameters; (ii) the latest advancements in model quantization techniques,
which allow pushing the compression to the extreme quantization level of 2 bits
per model parameter and; (iii) different types of calibration datasets to guide
the quantization process, including code-specific ones. Our empirical
evaluation reveals that the new frontier for LLM quantization is 4-bit
precision, resulting in an average memory footprint reduction of 70% compared
to the original model without observing any significant decrease in
performance. Additionally, when the quantization becomes even more extreme (3
and 2 bits), a code-specific calibration dataset helps to limit the loss of
performance.
Replication of LLM quantization strategies reveals that 4-bit precision can reduce memory footprint by 70% for large code-related LLMs with up to 34B parameters without significant performance degradation.
Large Language Models (LLMs) code generation trainable parameters memory footprint carbon footprint quantization techniques precision calibration datasets