It is known, that antigradient is the direction of the steepest descent of the function at point . However, we can introduce another concept for choosing the best direction of function decreasing.
Given and a point . Define as the set of points with distance to . Here we presume the existence of a distance function .
Than, we can define another steepest descent direction in terms of minimizer of function on a sphere:
Let us assume that the distance is defined locally by some metric :
Let us also consider first order Taylor approximation of a function near the point :
Now we can explicitly pose a problem of finding , as it was stated above.
Using it can be written as:
Using Lagrange multipliers method, we can easily conclude, that the answer is:
Which means, that new direction of steepest descent is nothing else, but .
Indeed, if the space is isotropic and , we immediately have gradient descent formula, while Newton method uses local Hessian as a metric matrix.