Exploring Landscapes for Better Minima along Valleys
Tong Zhao, Jiacheng Li, Yuanchang Zhou, Guangming Tan, Weile Jia
Finding lower and better-generalized minima is crucial. However, most existing optimizers stop searching the parameter space once they reach a local minimum. Given the complex geometric properties of the loss landscape, it is difficult to guarantee that such a point is the lowest or provides the best generalization. To address this, we propose a novel optimization strategy: after reaching a local minimum, the optimizer inclines to continue exploring along valleys (areas with low gradient norms) in the landscape to search for other possible local minima. This approach increases the likelihood of finding a lower and flatter local minimum, which is often associated with better generalization. We analyze the relationship between the flat local minima and generalization and provide a proof of convergence for the proposed method ALTO, in both convex and non-convex scenarios. Finally, we demonstrate its effectiveness in a typical training scenario, large minibatch training. Our testing results show that the accuracy (generalization) of ALTO is 2.5% higher than the of the current state-of-the-art method when using a large batch size on a variety of tasks averagely. This work potentially opens a new research direction in the design of optimization algorithms.