Visualizations
Scalar Dynamics: GF / HB / NAG
Theory
Matrix calculus
Convex sets
Affine set
Convex set
Conic set
Projection
Convex function
Conjugate set
Conjugate function
Dual norm
Subgradient and subdifferential
Optimality conditions. KKT
Convex optimization problem
Duality
Rates of convergence
Methods
Line search
Binary search
Golden search
Inexact Line Search
Successive parabolic interpolation
Zero order methods
Bee algorithm
Nelder–Mead
Simulated annealing
First order methods
Gradient descent
Heavy Ball Method
Subgradient descent
Projected subgradient descent
Mirror descent
Stochastic gradient descent
Stochastic average gradient
ADAM: A Method for Stochastic Optimization
Lookahead Optimizer:
k
steps forward,
1
step back
Proximal gradient method
Adaptive metric methods
Newton method
Quasi Newton methods
Conjugate gradients
Natural gradient descent
Linear Programming and simplex algorithm
Automatic differentiation
Exercises
Matrix calculus
Convex sets
Projection
Separation
Conjugate sets
Convex functions
Subgradient and subdifferential
Conjugate functions
General optimization problems
Duality
Rates of convergence
Line search
CVXPY library
Automatic differentiation
Zero order methods
First order methods
Uncategorized
Applications
A^*
algorithm for path finding
Deep learning
Minimum volume ellipsoid
Knapsack problem
Linear least squares
Maximum likelihood estimation
Neural network Lipschitz constant
Neural Network Loss Surface Visualization
Principal component analysis
Rendezvous problem
Travelling salesman problem
Total variation in-painting
Two way partitioning problem
Benchmarks
CNN on FashionMNIST
Linear Least Squares
Materials
Tutorials
Quick start to the Colab
Visualizations
Conjugate Gradient for Quadratics
Chebyshev’s acceleration of gradient descent
Condition number and gradient descent
GF / AGF / GD
We have Double Descent at home!
Fokker–Planck Density Evolution
Gradient descent for linear regression
L1 sparsity: subgradient vs proximal
Lipschitz continuity
Step Size Selection in Gradient Descent via Line Search
Multidimensional scaling
Nesterov Flow: Infinite Path Length
Local convergence of Newton method
Simple 3D visualization of gradient descent
QR Algorithm
Scalar Dynamics: GF / HB / NAG
Adaptive gradient algorithms visualization
Why stochastic gradient descent does not converge?
SGD intuition in a scalar case
Visualizations
Scalar Dynamics: GF / HB / NAG
Scalar Dynamics: GF / HB / NAG
Your browser does not support the video tag.
Code