Now we have only first order information from the oracle.

Materials

• Visualization of optimization algorithms.

Table of contents

• Gradient descent
• Subgradient descent
• Projected subgradient descent
• Mirror descent
• Stochastic gradient descent
• Stochastic average gradient
• ADAM: A Method for Stochastic Optimization
• Lookahead Optimizer: $k$ steps forward, $1$ step back
• Shampoo: Preconditioned Stochastic Tensor Optimization