Sharing is Caring: Efficient LM Post-Training with Collective RL
  Experience Sharing
Post-training language models (LMs) with reinforcement learning (RL) can
enhance their complex reasoning capabilities without supervised fine-tuning, as
demonstrated by DeepSeek-R1-Zero. However, effectively utilizing RL for LMs
requires significant parallelization to scale-up inference, which introduces
non-trivial technical challenges (e.g. latency, memory, and reliability)
alongside ever-growing financial costs. We present Swarm sAmpling Policy
Optimization (SAPO), a fully decentralized and asynchronous RL post-training
algorithm. SAPO is designed for decentralized networks of heterogenous compute
nodes, where each node manages its own policy model(s) while "sharing" rollouts
with others in the network; no explicit assumptions about latency, model
homogeneity, or hardware are required and nodes can operate in silo if desired.
As a result, the algorithm avoids common bottlenecks in scaling RL
post-training while also allowing (and even encouraging) new possibilities. By
sampling rollouts "shared" across the network, it enables "Aha moments" to
propagate, thereby bootstrapping the learning process. In this paper we show
SAPO achieved cumulative reward gains of up to 94% in controlled experiments.
We also share insights from tests on a network with thousands of nodes
contributed by Gensyn community members running the algorithm on diverse
hardware and models during an open-source demo.
68c221ac29b8ec9932cd08b9
Jeffrey Amico
68c221ac29b8ec9932cd08ba
Gabriel Passamani Andrade
claimed_verified
2025-09-12T16:15:03.904Z
67a2bf2fe6242abd8d559af2
/avatars/67046c518d3303ff8417c419c75ab1cf.svg
Gabriel P Andrade
user
gpandrad
68c221ac29b8ec9932cd08bb
John Donaghy
claimed_verified
2025-09-12T16:14:48.571Z
67e8761654cc82de7b632159
https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/9anqjWVQEBFRG7lyX-rcA.png
john donaghy
user
johnnyd-gensyn
68c221ac29b8ec9932cd08bc
Ben Fielding
claimed_verified
2025-09-12T16:15:06.915Z
66d252ec8a438492b0d6e4ce
/avatars/4f0fc7fbe8afae463741f786ea774e95.svg
Ben
user
benfielding
68c221ac29b8ec9932cd08bd
Tristin Forbus
claimed_verified
2025-09-12T16:14:58.166Z
688927e35c2ae7707c5450b3
https://cdn-avatars.huggingface.co/v1/production/uploads/688927e35c2ae7707c5450b3/FbNgXqO6DZ3zua_TvUfZj.jpeg
t forbus
user
tforbus
68c221ac29b8ec9932cd08be
Harry Grieve
claimed_verified
2025-09-12T16:14:42.188Z
659c5e12bf855af488cfb545
https://cdn-avatars.huggingface.co/v1/production/uploads/659c5e12bf855af488cfb545/KZ83UExB_pxLbpdbghiiV.jpeg
H
user
h-grieve
68c221ac29b8ec9932cd08bf
Semih Kara
claimed_verified
2025-09-13T14:56:33.424Z
689a0dd01db955f4e89167ce
https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/CvLYwnlXHPQ_n83leKcZR.png
Semih Kara
user
semihkara
68c221ac29b8ec9932cd08c0
Jari Kolehmainen
68c221ac29b8ec9932cd08c1
Yihua Lou
68c221ac29b8ec9932cd08c2
Christopher Nies
68c221ac29b8ec9932cd08c3
Edward Phillip Flores Nuño
claimed_verified
2025-09-12T16:14:45.664Z
68c2ec4d6f98def012e6ac56
https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/Ixr76J05-F4rLHkrcYzUX.png
Edward Phillip Flores Nuño
user
archaengel
68c221ac29b8ec9932cd08c4
Diogo Ortega
claimed_verified
2025-09-12T16:14:51.743Z
681206b096da70073b8f32f1
/avatars/6903e7374c188f68eaac167d5cd0f035.svg
Diogo Ortega
user
diortega
68c221ac29b8ec9932cd08c5
Shikhar Rastogi
claimed_verified
2025-09-12T16:14:54.627Z
6387a1d43811d7ebe6e2f187
/avatars/5989f81f319b909300d3480fb2cda5e2.svg
Shikhar Rastogi
user
shikharras
68c221ac29b8ec9932cd08c6
Austin Virts
claimed_verified
2025-09-12T16:15:01.095Z
684af1e3afaa9597bf6edacd
https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/V-kJrw2MkDcP677Sx0549.png
Austin V
user
palmtreecoffee
68c221ac29b8ec9932cd08c7
Matthew J. Wright
Sharing is Caring: Efficient LM Post-Training with Collective RL
  Experience Sharing
Post-training language models (LMs) with reinforcement learning (RL) can
enhance their complex reasoning capabilities without supervised fine-tuning, as
demonstrated by DeepSeek-R1-Zero. However, effectively utilizing RL for LMs
requires significant parallelization to scale-up inference, which introduces
non-trivial technical challenges (e.g. latency, memory, and reliability)
alongside ever-growing financial costs. We present Swarm sAmpling Policy
Optimization (SAPO), a fully decentralized and asynchronous RL post-training
algorithm. SAPO is designed for decentralized networks of heterogenous compute
nodes, where each node manages its own policy model(s) while "sharing" rollouts
with others in the network; no explicit assumptions about latency, model
homogeneity, or hardware are required and nodes can operate in silo if desired.
As a result, the algorithm avoids common bottlenecks in scaling RL
post-training while also allowing (and even encouraging) new possibilities. By
sampling rollouts "shared" across the network, it enables "Aha moments" to
propagate, thereby bootstrapping the learning process. In this paper we show
SAPO achieved cumulative reward gains of up to 94% in controlled experiments.
We also share insights from tests on a network with thousands of nodes
contributed by Gensyn community members running the algorithm on diverse
hardware and models during an open-source demo.
Swarm sAmpling Policy Optimization (SAPO) is a decentralized and asynchronous RL algorithm that enhances post-training language models without supervised fine-tuning, achieving significant reward gains and scalability across diverse hardware.
reinforcement learning post-training language models DeepSeek-R1-Zero Swarm sAmpling Policy Optimization SAPO decentralized networks heterogenous compute nodes policy models rollouts cumulative reward gains