Use the Online Network If You Can: Towards Fast and Stable Reinforcement
  Learning
The use of target networks is a popular approach for estimating value
functions in deep Reinforcement Learning (RL). While effective, the target
network remains a compromise solution that preserves stability at the cost of
slowly moving targets, thus delaying learning. Conversely, using the online
network as a bootstrapped target is intuitively appealing, albeit well-known to
lead to unstable learning. In this work, we aim to obtain the best out of both
worlds by introducing a novel update rule that computes the target using the
MINimum estimate between the Target and Online network, giving rise to our
method, MINTO. Through this simple, yet effective modification, we show that
MINTO enables faster and stable value function learning, by mitigating the
potential overestimation bias of using the online network for bootstrapping.
Notably, MINTO can be seamlessly integrated into a wide range of value-based
and actor-critic algorithms with a negligible cost. We evaluate MINTO
extensively across diverse benchmarks, spanning online and offline RL, as well
as discrete and continuous action spaces. Across all benchmarks, MINTO
consistently improves performance, demonstrating its broad applicability and
effectiveness.
68e95030b12ca14cb8a16640
Ahmed Hendawy
claimed_verified
2025-10-11T13:27:59.549Z
66337c054a7779b3b0beddf6
https://cdn-avatars.huggingface.co/v1/production/uploads/66337c054a7779b3b0beddf6/-oB6XNXKOfrtkFP3SjdtN.jpeg
Ahmed Hendawy
user
ahmedhendawy19
68e95030b12ca14cb8a16641
Henrik Metternich
68e95030b12ca14cb8a16642
Théo Vincent
68e95030b12ca14cb8a16643
Mahdi Kallel
68e95030b12ca14cb8a16644
Jan Peters
68e95030b12ca14cb8a16645
Carlo D'Eramo
Use the Online Network If You Can: Towards Fast and Stable Reinforcement
  Learning
The use of target networks is a popular approach for estimating value
functions in deep Reinforcement Learning (RL). While effective, the target
network remains a compromise solution that preserves stability at the cost of
slowly moving targets, thus delaying learning. Conversely, using the online
network as a bootstrapped target is intuitively appealing, albeit well-known to
lead to unstable learning. In this work, we aim to obtain the best out of both
worlds by introducing a novel update rule that computes the target using the
MINimum estimate between the Target and Online network, giving rise to our
method, MINTO. Through this simple, yet effective modification, we show that
MINTO enables faster and stable value function learning, by mitigating the
potential overestimation bias of using the online network for bootstrapping.
Notably, MINTO can be seamlessly integrated into a wide range of value-based
and actor-critic algorithms with a negligible cost. We evaluate MINTO
extensively across diverse benchmarks, spanning online and offline RL, as well
as discrete and continuous action spaces. Across all benchmarks, MINTO
consistently improves performance, demonstrating its broad applicability and
effectiveness.
MINTO, a novel update rule using the minimum estimate between target and online networks, enhances stable and faster value function learning in deep reinforcement learning.
target networks value functions deep Reinforcement Learning online network bootstrapped target MINTO value-based algorithms actor-critic algorithms overestimation bias online RL offline RL discrete action spaces continuous action spaces