Papers

Adam: A Method for Stochastic Optimization

• Original Paper introducing the Adam Optimizer

Decoupled Weight Decay Regularization

• A proposed improvement to Adam

On the convergence of Adam and beyond

• Another proposed improvement to Adam

Videos

Optimizers - Explained

• A quick tour of different optimization algorithms used in deep learning, ending with Adam

Adam Optimization Algorithm

• Although it has a couple typos, Andrew Ng’s explanation of the Adam Optimizer is good

Adam Optimization from scratch in python

• A short overview of the adam optimization algorithm with python implementation

Adam Optimizer for Neural Network

• A walkthrough of the adam optimizer

Posts

An overview of gradient descent optimization algorithms

• An explanation of Adam in a post on optimization algorithms by Sebastian Ruder

Adam

• A towardsdatascience blog post explaining and analyzing Adam by Vitaly Bushaev

Code Examples

Adam Optimizer | labml.ai

• A code/explanation side-by-side of the Adam Optimizer by labml.ai

Adam Optimizer from Scratch

• Github repo with simple and clear example of adam optimizer

• Another example

Classifying MNIST dataset using a CNN with Adam optimizer - Pytorch

• A clear pytorch example of the Adam optimizer in action

Classifying MNIST dataset using a MLP with Adam optimizer - Tensorflow

• A clear tensorflow keras example of the Adam optimizer in action

APIs

Tensorflow:

• tf.keras.optimizers.Adam

• tf.compact.v1.train.AdamOptimizer

Pytorch:

• torch.optim page with torch.optim.adam

• Source code for torch.optim.adam

Textbook Sections

Deep Learning Chapter 8.5.3 Adam (Page 305)

• Section on Adam from Ian Goodfellow’s Textbook

Slides

Adam - A Method for Stochastic Optimization

• Some slides looking at Adam and related topics by Nadav Cohen