Spectral Scaling Laws in Language Models: How Effectively Do
  Feed-Forward Networks Use Their Latent Space?
As large language models (LLMs) scale, the question is not only how large
they become, but how much of their capacity is effectively utilized. Existing
scaling laws relate model size to loss, yet overlook how components exploit
their latent space. We study feed-forward networks (FFNs) and recast width
selection as a spectral utilization problem. Using a lightweight diagnostic
suite -- Hard Rank (participation ratio), Soft Rank (Shannon rank), Spectral
Concentration, and the composite Spectral Utilization Index (SUI) -- we
quantify how many latent directions are meaningfully activated across LLaMA,
GPT-2, and nGPT families. Our key finding is an asymmetric spectral scaling
law: soft rank follows an almost perfect power law with FFN width, while hard
rank grows only sublinearly and with high variance. This asymmetry suggests
that widening FFNs mostly adds low-energy tail directions, while dominant-mode
subspaces saturate early. Moreover, at larger widths, variance further
collapses into a narrow subspace, leaving much of the latent space
under-utilized. These results recast FFN width selection as a principled
trade-off between tail capacity and dominant-mode capacity, offering concrete
guidance for inference-efficient LLM design.
68df3f9edf49fb0df1e03d1c
Nandan Kumar Jha
claimed_verified
2025-10-03T12:29:24.444Z
670ec3f6db1a6bcfe832e0a6
https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/-mZNLeLJoXzkwPgYO38lF.png
Nandan Kumar Jha
user
nandan523
68df3f9edf49fb0df1e03d1d
Brandon Reagen
Spectral Scaling Laws in Language Models: How Effectively Do
  Feed-Forward Networks Use Their Latent Space?
As large language models (LLMs) scale, the question is not only how large
they become, but how much of their capacity is effectively utilized. Existing
scaling laws relate model size to loss, yet overlook how components exploit
their latent space. We study feed-forward networks (FFNs) and recast width
selection as a spectral utilization problem. Using a lightweight diagnostic
suite -- Hard Rank (participation ratio), Soft Rank (Shannon rank), Spectral
Concentration, and the composite Spectral Utilization Index (SUI) -- we
quantify how many latent directions are meaningfully activated across LLaMA,
GPT-2, and nGPT families. Our key finding is an asymmetric spectral scaling
law: soft rank follows an almost perfect power law with FFN width, while hard
rank grows only sublinearly and with high variance. This asymmetry suggests
that widening FFNs mostly adds low-energy tail directions, while dominant-mode
subspaces saturate early. Moreover, at larger widths, variance further
collapses into a narrow subspace, leaving much of the latent space
under-utilized. These results recast FFN width selection as a principled
trade-off between tail capacity and dominant-mode capacity, offering concrete
guidance for inference-efficient LLM design.
Research on large language models reveals an asymmetric spectral scaling law in feed-forward networks, indicating that increasing width primarily adds low-energy directions while dominant modes saturate early, leading to underutilized latent space.
feed-forward networks FFNs Hard Rank Soft Rank Spectral Concentration Spectral Utilization Index SUI latent space spectral scaling law dominant-mode subspaces tail capacity dominant-mode capacity inference-efficient LLM design