Scaling Code-Assisted Chain-of-Thoughts and Instructions for Model
  Reasoning
Reasoning capability is pivotal for Large Language Models (LLMs) to solve
complex tasks, yet achieving reliable and scalable reasoning remains
challenging. While Chain-of-Thought (CoT) prompting has become a mainstream
approach, existing methods often suffer from uncontrolled generation,
insufficient quality, and limited diversity in reasoning paths. Recent efforts
leverage code to enhance CoT by grounding reasoning in executable steps, but
such methods are typically constrained to predefined mathematical problems,
hindering scalability and generalizability. In this work, we propose Caco
(Code-Assisted Chain-of-ThOught), a novel framework that automates the
synthesis of high-quality, verifiable, and diverse instruction-CoT reasoning
data through code-driven augmentation. Unlike prior work, Caco first fine-tunes
a code-based CoT generator on existing math and programming solutions in a
unified code format, then scales the data generation to a large amount of
diverse reasoning traces. Crucially, we introduce automated validation via code
execution and rule-based filtering to ensure logical correctness and structural
diversity, followed by reverse-engineering filtered outputs into natural
language instructions and language CoTs to enrich task adaptability. This
closed-loop process enables fully automated, scalable synthesis of reasoning
data with guaranteed executability. Experiments on our created Caco-1.3M
dataset demonstrate that Caco-trained models achieve strong competitive
performance on mathematical reasoning benchmarks, outperforming existing strong
baselines. Further analysis reveals that Caco's code-anchored verification and
instruction diversity contribute to superior generalization across unseen
tasks. Our work establishes a paradigm for building self-sustaining,
trustworthy reasoning systems without human intervention.
68e5cffd975ac4c405ef210e
Honglin Lin
claimed_verified
2025-10-24T16:14:48.240Z
640d99628512ec51d7ef71c7
https://cdn-avatars.huggingface.co/v1/production/uploads/640d99628512ec51d7ef71c7/fcBkqnxfxuuuZTqfN_BGy.jpeg
Honglin Lin
user
LHL3341
68e5cffd975ac4c405ef210f
Qizhi Pei
68e5cffd975ac4c405ef2110
Xin Gao
68e5cffd975ac4c405ef2111
Zhuoshi Pan
68e5cffd975ac4c405ef2112
Yu Li
68e5cffd975ac4c405ef2113
Juntao Li
68e5cffd975ac4c405ef2114
Conghui He
68e5cffd975ac4c405ef2115
Lijun Wu
claimed_verified
2025-10-08T11:49:05.353Z
643e60d96db6ba8c5ee177ad
/avatars/73ac7740e462ba0b53a2f2480d9f1e3e.svg
Lijun Wu
user
apeters
Scaling Code-Assisted Chain-of-Thoughts and Instructions for Model
  Reasoning
Reasoning capability is pivotal for Large Language Models (LLMs) to solve
complex tasks, yet achieving reliable and scalable reasoning remains
challenging. While Chain-of-Thought (CoT) prompting has become a mainstream
approach, existing methods often suffer from uncontrolled generation,
insufficient quality, and limited diversity in reasoning paths. Recent efforts
leverage code to enhance CoT by grounding reasoning in executable steps, but
such methods are typically constrained to predefined mathematical problems,
hindering scalability and generalizability. In this work, we propose Caco
(Code-Assisted Chain-of-ThOught), a novel framework that automates the
synthesis of high-quality, verifiable, and diverse instruction-CoT reasoning
data through code-driven augmentation. Unlike prior work, Caco first fine-tunes
a code-based CoT generator on existing math and programming solutions in a
unified code format, then scales the data generation to a large amount of
diverse reasoning traces. Crucially, we introduce automated validation via code
execution and rule-based filtering to ensure logical correctness and structural
diversity, followed by reverse-engineering filtered outputs into natural
language instructions and language CoTs to enrich task adaptability. This
closed-loop process enables fully automated, scalable synthesis of reasoning
data with guaranteed executability. Experiments on our created Caco-1.3M
dataset demonstrate that Caco-trained models achieve strong competitive
performance on mathematical reasoning benchmarks, outperforming existing strong
baselines. Further analysis reveals that Caco's code-anchored verification and
instruction diversity contribute to superior generalization across unseen
tasks. Our work establishes a paradigm for building self-sustaining,
trustworthy reasoning systems without human intervention.
Caco, a code-assisted chain-of-thought framework, automates the generation of high-quality, verifiable, and diverse reasoning data, enhancing the performance of large language models on mathematical reasoning tasks.
Chain-of-Thought CoT prompting code-based CoT generator code-driven augmentation automated validation code execution rule-based filtering reverse-engineering natural language instructions language CoTs Caco-1.3M dataset mathematical reasoning benchmarks self-sustaining trustworthy reasoning systems