A Refined Analysis of Massive Activations in LLMs
Motivated in part by their relevance for low-precision training and
quantization, massive activations in large language models (LLMs) have recently
emerged as a topic of interest. However, existing analyses are limited in
scope, and generalizability across architectures is unclear. This paper helps
address some of these gaps by conducting an analysis of massive activations
across a broad range of LLMs, including both GLU-based and non-GLU-based
architectures. Our findings challenge several prior assumptions, most
importantly: (1) not all massive activations are detrimental, i.e. suppressing
them does not lead to an explosion of perplexity or a collapse in downstream
task performance; (2) proposed mitigation strategies such as Attention KV bias
are model-specific and ineffective in certain cases. We consequently
investigate novel hybrid mitigation strategies; in particular pairing Target
Variance Rescaling (TVR) with Attention KV bias or Dynamic Tanh (DyT)
successfully balances the mitigation of massive activations with preserved
downstream model performance in the scenarios we investigated. Our code is
available at: https://github.com/bluorion-com/refine_massive_activations.
67ea01e3d13d75fc155fa69d
Louis Owen
claimed_verified
2025-03-31T08:11:39.976Z
6071c4b270e11b30cfcfd7a3
https://cdn-avatars.huggingface.co/v1/production/uploads/6071c4b270e11b30cfcfd7a3/-1ekCBzSTpqxkkul0bgmI.jpeg
Louis Owen
user
louisowen6
67ea01e3d13d75fc155fa69e
Nilabhra Roy Chowdhury
claimed_verified
2025-03-31T08:11:37.938Z
645a0d3dd6648853107c5fdc
/avatars/1e3b6a4f5ce81a707ba7cbdf81631091.svg
Nilabhra Roy Chowdhury
user
nilabhra
67ea01e3d13d75fc155fa69f
Abhay Kumar
claimed_verified
2025-03-31T08:11:35.757Z
62cd4b03c5cc157be82f0b56
/avatars/351e963c1c763d507ae78cbcd62966a3.svg
Abhay kumar
user
akanyaani
67ea01e3d13d75fc155fa6a0
Fabian Güra
claimed_verified
2025-04-03T13:33:11.338Z
65e4be59e8b017ee1310a1b6
/avatars/c3f7cdf5d0859cb80bfb2b970a675dfa.svg
Fabian
user
gueraf
A Refined Analysis of Massive Activations in LLMs
Motivated in part by their relevance for low-precision training and
quantization, massive activations in large language models (LLMs) have recently
emerged as a topic of interest. However, existing analyses are limited in
scope, and generalizability across architectures is unclear. This paper helps
address some of these gaps by conducting an analysis of massive activations
across a broad range of LLMs, including both GLU-based and non-GLU-based
architectures. Our findings challenge several prior assumptions, most
importantly: (1) not all massive activations are detrimental, i.e. suppressing
them does not lead to an explosion of perplexity or a collapse in downstream
task performance; (2) proposed mitigation strategies such as Attention KV bias
are model-specific and ineffective in certain cases. We consequently
investigate novel hybrid mitigation strategies; in particular pairing Target
Variance Rescaling (TVR) with Attention KV bias or Dynamic Tanh (DyT)
successfully balances the mitigation of massive activations with preserved
downstream model performance in the scenarios we investigated. Our code is
available at: https://github.com/bluorion-com/refine_massive_activations.
An analysis of massive activations in large language models shows that suppression is not always detrimental and proposes hybrid mitigation strategies that balance activation management with model performance.
low-precision training quantization massive activations large language models LLMs GLU-based non-GLU-based perplexity downstream task performance Attention KV bias Target Variance Rescaling TVR Dynamic Tanh DyT