Mixtures of Experts Unlock Parameter Scaling for Deep RL
The recent rapid progress in (self) supervised learning models is in large
part predicted by empirical scaling laws: a model's performance scales
proportionally to its size. Analogous scaling laws remain elusive for
reinforcement learning domains, however, where increasing the parameter count
of a model often hurts its final performance. In this paper, we demonstrate
that incorporating Mixture-of-Expert (MoE) modules, and in particular Soft MoEs
(Puigcerver et al., 2023), into value-based networks results in more
parameter-scalable models, evidenced by substantial performance increases
across a variety of training regimes and model sizes. This work thus provides
strong empirical evidence towards developing scaling laws for reinforcement
learning.
65cc30b751c1738a55e94ba9
Johan Obando-Ceron
claimed_verified
2024-02-14T08:41:41.148Z
6478f30ea68454566353ef95
https://cdn-avatars.huggingface.co/v1/production/uploads/6478f30ea68454566353ef95/hX5NiVPyQbK8TIBKnT4J1.jpeg
Johan Samir Obando Ceron
user
johanobandoc
65cc30b751c1738a55e94baa
Ghada Sokar
65cc30b751c1738a55e94bab
Timon Willi
65cc30b751c1738a55e94bac
Clare Lyle
admin_assigned
2024-02-14T10:41:45.967Z
6313f69d54e6e5d9f0fb9f10
/avatars/12114d479bb6dc394d29f988944f6d47.svg
Clare Lyle
user
justclarifying
65cc30b751c1738a55e94bad
Jesse Farebrother
admin_assigned
2024-02-14T10:42:01.932Z
64b85c41cf14c2fabe8edd37
/avatars/900d99b3b24aac5c98fb1f50eef8e825.svg
Jesse Farebrother
user
jessefarebro
65cc30b751c1738a55e94bae
Jakob Foerster
65cc30b751c1738a55e94baf
Gintare Karolina Dziugaite
65cc30b751c1738a55e94bb0
Doina Precup
65cc30b751c1738a55e94bb1
Pablo Samuel Castro
admin_assigned
2024-02-14T10:42:53.652Z
6303fabc7373aacccd87ddcf
https://cdn-avatars.huggingface.co/v1/production/uploads/1661205153867-noauth.png
Pablo Samuel Castro
user
pcastr
Mixtures of Experts Unlock Parameter Scaling for Deep RL
The recent rapid progress in (self) supervised learning models is in large
part predicted by empirical scaling laws: a model's performance scales
proportionally to its size. Analogous scaling laws remain elusive for
reinforcement learning domains, however, where increasing the parameter count
of a model often hurts its final performance. In this paper, we demonstrate
that incorporating Mixture-of-Expert (MoE) modules, and in particular Soft MoEs
(Puigcerver et al., 2023), into value-based networks results in more
parameter-scalable models, evidenced by substantial performance increases
across a variety of training regimes and model sizes. This work thus provides
strong empirical evidence towards developing scaling laws for reinforcement
learning.
Incorporating Soft MoE modules into value-based networks enhances scalability and performance across different training regimes and model sizes in reinforcement learning.
Mixture-of-Expert (MoE) Soft MoEs value-based networks scaling laws reinforcement learning