Satori-SWE: Evolutionary Test-Time Scaling for Sample-Efficient Software
  Engineering
Language models (LMs) perform well on standardized coding benchmarks but
struggle with real-world software engineering tasks such as resolving GitHub
issues in SWE-Bench, especially when model parameters are less than 100B. While
smaller models are preferable in practice due to their lower computational
cost, improving their performance remains challenging. Existing approaches
primarily rely on supervised fine-tuning (SFT) with high-quality data, which is
expensive to curate at scale. An alternative is test-time scaling: generating
multiple outputs, scoring them using a verifier, and selecting the best one.
Although effective, this strategy often requires excessive sampling and costly
scoring, limiting its practical application. We propose Evolutionary Test-Time
Scaling (EvoScale), a sample-efficient method that treats generation as an
evolutionary process. By iteratively refining outputs via selection and
mutation, EvoScale shifts the output distribution toward higher-scoring
regions, reducing the number of samples needed to find correct solutions. To
reduce the overhead from repeatedly sampling and selection, we train the model
to self-evolve using reinforcement learning (RL). Rather than relying on
external verifiers at inference time, the model learns to self-improve the
scores of its own generations across iterations. Evaluated on
SWE-Bench-Verified, EvoScale enables our 32B model, Satori-SWE-32B, to match or
exceed the performance of models with over 100B parameters while using a few
samples. Code, data, and models will be fully open-sourced.
68390da8f527444e97c4ab95
Guangtao Zeng
68390da8f527444e97c4ab96
Maohao Shen
extracted_pending
2025-05-30T01:45:13.275Z
6553c985a7aded0380b5f928
https://cdn-avatars.huggingface.co/v1/production/uploads/6553c985a7aded0380b5f928/tAJnmYvmaAse4fbHq0CjA.jpeg
Maohao Shen
user
maohaos2
68390da8f527444e97c4ab97
Delin Chen
68390da8f527444e97c4ab98
Zhenting Qi
68390da8f527444e97c4ab99
Subhro Das
68390da8f527444e97c4ab9a
Dan Gutfreund
68390da8f527444e97c4ab9b
David Cox
68390da8f527444e97c4ab9c
Gregory Wornell
68390da8f527444e97c4ab9d
Wei Lu
68390da8f527444e97c4ab9e
Zhang-Wei Hong
68390da8f527444e97c4ab9f
Chuang Gan
Satori-SWE: Evolutionary Test-Time Scaling for Sample-Efficient Software
  Engineering
Language models (LMs) perform well on standardized coding benchmarks but
struggle with real-world software engineering tasks such as resolving GitHub
issues in SWE-Bench, especially when model parameters are less than 100B. While
smaller models are preferable in practice due to their lower computational
cost, improving their performance remains challenging. Existing approaches
primarily rely on supervised fine-tuning (SFT) with high-quality data, which is
expensive to curate at scale. An alternative is test-time scaling: generating
multiple outputs, scoring them using a verifier, and selecting the best one.
Although effective, this strategy often requires excessive sampling and costly
scoring, limiting its practical application. We propose Evolutionary Test-Time
Scaling (EvoScale), a sample-efficient method that treats generation as an
evolutionary process. By iteratively refining outputs via selection and
mutation, EvoScale shifts the output distribution toward higher-scoring
regions, reducing the number of samples needed to find correct solutions. To
reduce the overhead from repeatedly sampling and selection, we train the model
to self-evolve using reinforcement learning (RL). Rather than relying on
external verifiers at inference time, the model learns to self-improve the
scores of its own generations across iterations. Evaluated on
SWE-Bench-Verified, EvoScale enables our 32B model, Satori-SWE-32B, to match or
exceed the performance of models with over 100B parameters while using a few
samples. Code, data, and models will be fully open-sourced.
EvoScale, an evolutionary and reinforcement learning-based method, enhances small language models' performance on real-world software engineering tasks by iteratively improving and refining outputs.
supervised fine-tuning test-time scaling Evolutionary Test-Time Scaling EvoScale reinforcement learning SWE-Bench-Verified