First order methods

Now we have only first order information from the oracle.

Gradient descent

Original Cauchy paper

Subgradient descent

Рассматривается классическая задача выпуклой оптимизации:

Projected subgradient descent

Suppose, we are to solve the following problem:

Mirror descent

Метод зеркального спуска является естественным обобщением метода проекции субградиента в случае обобщения l_2 нормы на более общий случай какой-то функции расстояния.

Stochastic gradient descent

Suppose, our target function is the sum of functions.

Stochastic average gradient

A classical problem of minimizing finite sum of the smooth and convex functions was considered.

ADAM: A Method for Stochastic Optimization

Adam is the stochastic first order optimization algorithm, that uses historical information about stochastic gradients and incorporates it in attempt to estimate second…

Lookahead Optimizer: k steps forward, 1 step back

The lookahead method provides an interesting way to accelerate and stabilize algorithms of stochastic gradient descent family. The main idea is quite simple:

1 Materials

Visualization of optimization algorithms.