SimBa: Simplicity Bias for Scaling Up Parameters in Deep Reinforcement
  Learning
Recent advances in CV and NLP have been largely driven by scaling up the
number of network parameters, despite traditional theories suggesting that
larger networks are prone to overfitting. These large networks avoid
overfitting by integrating components that induce a simplicity bias, guiding
models toward simple and generalizable solutions. However, in deep RL,
designing and scaling up networks have been less explored. Motivated by this
opportunity, we present SimBa, an architecture designed to scale up parameters
in deep RL by injecting a simplicity bias. SimBa consists of three components:
(i) an observation normalization layer that standardizes inputs with running
statistics, (ii) a residual feedforward block to provide a linear pathway from
the input to output, and (iii) a layer normalization to control feature
magnitudes. By scaling up parameters with SimBa, the sample efficiency of
various deep RL algorithms-including off-policy, on-policy, and unsupervised
methods-is consistently improved. Moreover, solely by integrating SimBa
architecture into SAC, it matches or surpasses state-of-the-art deep RL methods
with high computational efficiency across DMC, MyoSuite, and HumanoidBench.
These results demonstrate SimBa's broad applicability and effectiveness across
diverse RL algorithms and environments.
670f6e287d1b0373c6419900
Hojoon Lee
670f6e287d1b0373c6419901
Dongyoon Hwang
claimed_verified
2024-10-16T09:43:08.864Z
67061bf00cc840f4462a0705
/avatars/0fab53fa981f166483aa8ad61172d32d.svg
Dongyoon
user
godnpeter
670f6e287d1b0373c6419902
Donghu Kim
670f6e287d1b0373c6419903
Hyunseung Kim
670f6e287d1b0373c6419904
Jun Jet Tai
670f6e287d1b0373c6419905
Kaushik Subramanian
670f6e287d1b0373c6419906
Peter R. Wurman
670f6e287d1b0373c6419907
Jaegul Choo
670f6e287d1b0373c6419908
Peter Stone
670f6e287d1b0373c6419909
Takuma Seno
SimBa: Simplicity Bias for Scaling Up Parameters in Deep Reinforcement
  Learning
Recent advances in CV and NLP have been largely driven by scaling up the
number of network parameters, despite traditional theories suggesting that
larger networks are prone to overfitting. These large networks avoid
overfitting by integrating components that induce a simplicity bias, guiding
models toward simple and generalizable solutions. However, in deep RL,
designing and scaling up networks have been less explored. Motivated by this
opportunity, we present SimBa, an architecture designed to scale up parameters
in deep RL by injecting a simplicity bias. SimBa consists of three components:
(i) an observation normalization layer that standardizes inputs with running
statistics, (ii) a residual feedforward block to provide a linear pathway from
the input to output, and (iii) a layer normalization to control feature
magnitudes. By scaling up parameters with SimBa, the sample efficiency of
various deep RL algorithms-including off-policy, on-policy, and unsupervised
methods-is consistently improved. Moreover, solely by integrating SimBa
architecture into SAC, it matches or surpasses state-of-the-art deep RL methods
with high computational efficiency across DMC, MyoSuite, and HumanoidBench.
These results demonstrate SimBa's broad applicability and effectiveness across
diverse RL algorithms and environments.
SimBa improves sample efficiency and computational efficiency in deep RL by scaling up parameters with simplicity bias components.
deep RL observation normalization layer residual feedforward block layer normalization sample efficiency off-policy on-policy unsupervised methods SimBa SAC DMC MyoSuite HumanoidBench