Length Generalization via Auxiliary Tasks
Pranjal Awasthi, Anupam Gupta, Ravi Kumar
Length generalization, the ability of sequence models to  generalize to sequences longer than those encountered during  training, remains a fundamental challenge for Transformers,  especially in tasks requiring algorithmic reasoning. Existing  theoretical understanding of length generalization is limited, often providing  asymptotic results or focusing on specific problem classes or  architectural variants; empirical solutions frequently rely on  ad-hoc and often fragile techniques.  In this work we introduce a novel framework for analyzing and  proving length generalization bounds under specified, verifiable assumptions. A key  outcome of the theory is the identification of a natural set of  auxiliary tasks, intricately related to the primary task structure,  such that strong performance on these auxiliary tasks, alongside the  primary task, provably guarantees length generalization within the framework.  This  motivates a multi-task training procedure that explicitly optimizes  performance on both the primary and the identified auxiliary tasks.  Empirical evaluations on a variety of synthetic benchmarks  known to be challenging for length generalization, including sequence  sorting, and reversal, demonstrate that the proposed method yields  significant improvements in generalization to substantially longer  sequences.