Unlocking Out-of-Distribution Generalization in Transformers via
  Recursive Latent Space Reasoning
Systematic, compositional generalization beyond the training distribution
remains a core challenge in machine learning -- and a critical bottleneck for
the emergent reasoning abilities of modern language models. This work
investigates out-of-distribution (OOD) generalization in Transformer networks
using a GSM8K-style modular arithmetic on computational graphs task as a
testbed. We introduce and explore a set of four architectural mechanisms aimed
at enhancing OOD generalization: (i) input-adaptive recurrence; (ii)
algorithmic supervision; (iii) anchored latent representations via a discrete
bottleneck; and (iv) an explicit error-correction mechanism. Collectively,
these mechanisms yield an architectural approach for native and scalable latent
space reasoning in Transformer networks with robust algorithmic generalization
capabilities. We complement these empirical results with a detailed mechanistic
interpretability analysis that reveals how these mechanisms give rise to robust
OOD generalization abilities.
68f2802537ed56c2c926f973
Awni Altabaa
claimed_verified
2025-10-18T04:47:36.310Z
637abd9114076b808c6a2f4f
/avatars/7fa8b275b4347dfeab73384e2aa72ea4.svg
Awni Altabaa
user
awni00
68f2802537ed56c2c926f974
Siyu Chen
68f2802537ed56c2c926f975
John Lafferty
68f2802537ed56c2c926f976
Zhuoran Yang
claimed_verified
2025-10-20T12:13:06.370Z
646d769cda8e99940b71928e
/avatars/acee495a23362aa39b3d3e75c9afd967.svg
Zhuoran Yang
user
zhuoranyang
Unlocking Out-of-Distribution Generalization in Transformers via
  Recursive Latent Space Reasoning
Systematic, compositional generalization beyond the training distribution
remains a core challenge in machine learning -- and a critical bottleneck for
the emergent reasoning abilities of modern language models. This work
investigates out-of-distribution (OOD) generalization in Transformer networks
using a GSM8K-style modular arithmetic on computational graphs task as a
testbed. We introduce and explore a set of four architectural mechanisms aimed
at enhancing OOD generalization: (i) input-adaptive recurrence; (ii)
algorithmic supervision; (iii) anchored latent representations via a discrete
bottleneck; and (iv) an explicit error-correction mechanism. Collectively,
these mechanisms yield an architectural approach for native and scalable latent
space reasoning in Transformer networks with robust algorithmic generalization
capabilities. We complement these empirical results with a detailed mechanistic
interpretability analysis that reveals how these mechanisms give rise to robust
OOD generalization abilities.
Transformer networks are enhanced with four architectural mechanisms to improve out-of-distribution generalization and algorithmic reasoning.
Transformer networks out-of-distribution (OOD) generalization GSM8K-style modular arithmetic input-adaptive recurrence algorithmic supervision anchored latent representations discrete bottleneck explicit error-correction mechanism latent space reasoning algorithmic generalization