CoT Information: Improved Sample Complexity under Chain-of-Thought Supervision
Awni Altabaa, Omar Montasser, John Lafferty
Chain-of-thought (CoT) supervision has emerged as a powerful empirical technique in large language models. This paper develops a statistical theory of learning under CoT supervision. Central to the theory is a quantity called the CoT information measure, which quantifies the additional discriminative power gained from the chain-of-thought for distinguishing hypotheses with different end-to-end behaviors. The main theoretical results demonstrate how CoT supervision can yield significantly faster learning rates compared to standard end-to-end supervision, with both upper bounds and information-theoretic lower bounds on sample complexity in terms of the CoT information.