Stop Regressing: Training Value Functions via Classification for
  Scalable Deep RL
Value functions are a central component of deep reinforcement learning (RL).
These functions, parameterized by neural networks, are trained using a mean
squared error regression objective to match bootstrapped target values.
However, scaling value-based RL methods that use regression to large networks,
such as high-capacity Transformers, has proven challenging. This difficulty is
in stark contrast to supervised learning: by leveraging a cross-entropy
classification loss, supervised methods have scaled reliably to massive
networks. Observing this discrepancy, in this paper, we investigate whether the
scalability of deep RL can also be improved simply by using classification in
place of regression for training value functions. We demonstrate that value
functions trained with categorical cross-entropy significantly improves
performance and scalability in a variety of domains. These include: single-task
RL on Atari 2600 games with SoftMoEs, multi-task RL on Atari with large-scale
ResNets, robotic manipulation with Q-transformers, playing Chess without
search, and a language-agent Wordle task with high-capacity Transformers,
achieving state-of-the-art results on these domains. Through careful analysis,
we show that the benefits of categorical cross-entropy primarily stem from its
ability to mitigate issues inherent to value-based RL, such as noisy targets
and non-stationarity. Overall, we argue that a simple shift to training value
functions with categorical cross-entropy can yield substantial improvements in
the scalability of deep RL at little-to-no cost.
65e95673f42775fd671b4f54
Jesse Farebrother
claimed_verified
2024-03-07T09:13:35.336Z
64b85c41cf14c2fabe8edd37
/avatars/900d99b3b24aac5c98fb1f50eef8e825.svg
Jesse Farebrother
user
jessefarebro
65e95673f42775fd671b4f55
Jordi Orbay
65e95673f42775fd671b4f56
Quan Vuong
admin_assigned
2024-03-07T10:07:58.506Z
63e1d6687fbb6ae4d4f4d980
/avatars/4959319e69d69e03a31ff4be326e602a.svg
Quan Vuong
user
quintonx
65e95673f42775fd671b4f57
Adrien Ali Taïga
admin_assigned
2024-03-07T10:08:08.469Z
63694e1dbf21b20c568d9bb9
/avatars/9ed8c8a76ae60de26dede86d298bd64f.svg
Adrien Ali Taiga
user
aalitaiga
65e95673f42775fd671b4f58
Yevgen Chebotar
admin_assigned
2024-03-07T10:08:17.760Z
64c93f9c9471c54a9438162c
https://cdn-avatars.huggingface.co/v1/production/uploads/64c93f9c9471c54a9438162c/bYGUqFsF67VuAuXecjxjX.jpeg
Yevgen Chebotar
user
ychebotar
65e95673f42775fd671b4f59
Ted Xiao
65e95673f42775fd671b4f5a
Alex Irpan
65e95673f42775fd671b4f5b
Sergey Levine
65e95673f42775fd671b4f5c
Pablo Samuel Castro
admin_assigned
2024-03-07T10:09:03.406Z
6303fabc7373aacccd87ddcf
https://cdn-avatars.huggingface.co/v1/production/uploads/1661205153867-noauth.png
Pablo Samuel Castro
user
pcastr
65e95673f42775fd671b4f5d
Aleksandra Faust
admin_assigned
2024-03-07T10:09:10.317Z
64a3b828225b2121f6b3beb3
https://cdn-avatars.huggingface.co/v1/production/uploads/64a3b828225b2121f6b3beb3/44AgAQENc7cJMOWcoK4g_.jpeg
Aleksandra Faust
user
sandraorion
65e95673f42775fd671b4f5e
Aviral Kumar
admin_assigned
2024-03-07T10:10:04.363Z
64879130cc0e54f272b5dbcc
/avatars/ead1211a56a4e5c14cb8f6b2ea59c0e6.svg
Aviral Kumar
user
aviralkumar
65e95673f42775fd671b4f5f
Rishabh Agarwal
extracted_pending
2024-03-07T05:53:55.930Z
647875e37b370854241b89a8
/avatars/fd25f7e5960e4e132b9ee689f114820b.svg
Rishabh Agarwal
user
agarwl
Stop Regressing: Training Value Functions via Classification for
  Scalable Deep RL
Value functions are a central component of deep reinforcement learning (RL).
These functions, parameterized by neural networks, are trained using a mean
squared error regression objective to match bootstrapped target values.
However, scaling value-based RL methods that use regression to large networks,
such as high-capacity Transformers, has proven challenging. This difficulty is
in stark contrast to supervised learning: by leveraging a cross-entropy
classification loss, supervised methods have scaled reliably to massive
networks. Observing this discrepancy, in this paper, we investigate whether the
scalability of deep RL can also be improved simply by using classification in
place of regression for training value functions. We demonstrate that value
functions trained with categorical cross-entropy significantly improves
performance and scalability in a variety of domains. These include: single-task
RL on Atari 2600 games with SoftMoEs, multi-task RL on Atari with large-scale
ResNets, robotic manipulation with Q-transformers, playing Chess without
search, and a language-agent Wordle task with high-capacity Transformers,
achieving state-of-the-art results on these domains. Through careful analysis,
we show that the benefits of categorical cross-entropy primarily stem from its
ability to mitigate issues inherent to value-based RL, such as noisy targets
and non-stationarity. Overall, we argue that a simple shift to training value
functions with categorical cross-entropy can yield substantial improvements in
the scalability of deep RL at little-to-no cost.
Using categorical cross-entropy for value function training in deep reinforcement learning improves scalability and performance across various tasks compared to traditional regression methods.
deep reinforcement learning value functions neural networks mean squared error regression Transformers supervised learning cross-entropy classification loss SoftMoEs ResNets Q-transformers robotic manipulation language-agent Wordle task Atari 2600 games categorical cross-entropy