Adam optimizer
November 19, 2022 ⚊ 1 Min read ⚊ Views 102 ⚊ EDUCATION
Adam stands for Adaptive Moment Estimation, is another method that computes adaptive learning rates for each parameter. In addition to storing an exponentially decaying average of past squared gradients like Adadelta and RMSprop.
Adam also keeps an exponentially decaying average of past gradients, similar to momentum.
Adam can be viewed as a combination of Adagrad and RMSprop, (Adagrad) which works well on sparse gradients and (RMSProp) which works well in online and nonstationary settings respectively.
Tags: Adam, explain, Python
