Title: Understanding adaptive methods for non-convex optimization

Abstract: Optimization methods which do per-feature adaptive scaling like AdaGrad, Adam, RMSProp, etc are critical for training deep learning models today. However, there is little theoretical understanding of why these methods are effective in practice. In this talk, I will present results from three recent research projects investigating the effectiveness of adaptive methods: (a) an understanding of the convergence properties of methods like Adam and RMSProp, specifically, in what circumstances these methods can fail to converge, (b) a new adaptive optimizer, Yogi, based on the previous analysis which has better performance than Adam in many challenging machine learning tasks, and (c) an understanding of how to properly tune Adam and RMSProp so that these methods escape saddle points and reach (near) second-order critical points.

Bio: Satyen Kale is a research scientist at Google Research working in the New York office. His current research is the design of efficient and practical algorithms for fundamental problems in Machine Learning and Optimization. More specifically, he is interested in decision making under uncertainty, statistical learning theory, combinatorial optimization, and convex optimization techniques such as linear and semidefinite programming. His research has been recognized with several awards: a best paper award at ICML 2015, a best paper award at ICLR 2018, and a best student paper award at COLT 2018. He was a program chair of COLT 2017 and ALT 2019.