Adam Optimizer
Know a great resource on Adam Optimization? Send it to us at contactbackprop@gmail.com!
Papers
Adam: A Method for Stochastic Optimization
Original Paper introducing the Adam Optimizer
Decoupled Weight Decay Regularization
A proposed improvement to Adam
On the convergence of Adam and beyond
Another proposed improvement to Adam
Videos
A quick tour of different optimization algorithms used in deep learning, ending with Adam
Although it has a couple typos, Andrew Ng’s explanation of the Adam Optimizer is good
Adam Optimization from scratch in python
A short overview of the adam optimization algorithm with python implementation
Adam Optimizer for Neural Network
A walkthrough of the adam optimizer
Posts
An overview of gradient descent optimization algorithms
An explanation of Adam in a post on optimization algorithms by Sebastian Ruder
A towardsdatascience blog post explaining and analyzing Adam by Vitaly Bushaev
Code Examples
A code/explanation side-by-side of the Adam Optimizer by labml.ai
Github repo with simple and clear example of adam optimizer
Classifying MNIST dataset using a CNN with Adam optimizer - Pytorch
A clear pytorch example of the Adam optimizer in action
Classifying MNIST dataset using a MLP with Adam optimizer - Tensorflow
A clear tensorflow keras example of the Adam optimizer in action
Textbook Sections
Deep Learning Chapter 8.5.3 Adam (Page 305)
Section on Adam from Ian Goodfellow’s Textbook
Slides
Adam - A Method for Stochastic Optimization
Some slides looking at Adam and related topics by Nadav Cohen
Know a great resource on Adam Optimization? Send it to us at contactbackprop@gmail.com!