Adam Optimizer

Know a great resource on Adam Optimization? Send it to us at contactbackprop@gmail.com!

Tags: Backpropagation, Gradient Descent Optimization

Papers

Adam: A Method for Stochastic Optimization

  • Original Paper introducing the Adam Optimizer


Decoupled Weight Decay Regularization

  • A proposed improvement to Adam


On the convergence of Adam and beyond

  • Another proposed improvement to Adam

Videos

Optimizers - Explained

  • A quick tour of different optimization algorithms used in deep learning, ending with Adam


Adam Optimization Algorithm

  • Although it has a couple typos, Andrew Ng’s explanation of the Adam Optimizer is good


Adam Optimization from scratch in python

  • A short overview of the adam optimization algorithm with python implementation


Adam Optimizer for Neural Network

  • A walkthrough of the adam optimizer

Posts

An overview of gradient descent optimization algorithms

  • An explanation of Adam in a post on optimization algorithms by Sebastian Ruder


Adam

  • A towardsdatascience blog post explaining and analyzing Adam by Vitaly Bushaev

Code Examples

Adam Optimizer | labml.ai

  • A code/explanation side-by-side of the Adam Optimizer by labml.ai


Adam Optimizer from Scratch


Classifying MNIST dataset using a CNN with Adam optimizer - Pytorch

  • A clear pytorch example of the Adam optimizer in action


Classifying MNIST dataset using a MLP with Adam optimizer - Tensorflow

  • A clear tensorflow keras example of the Adam optimizer in action

Textbook Sections

Deep Learning Chapter 8.5.3 Adam (Page 305)

  • Section on Adam from Ian Goodfellow’s Textbook

Slides

Adam - A Method for Stochastic Optimization

  • Some slides looking at Adam and related topics by Nadav Cohen

Know a great resource on Adam Optimization? Send it to us at contactbackprop@gmail.com!