Automatic differentiation

Calculate the gradient of a Taylor series of a \cos (x) using autograd library:

import autograd.numpy as np # Thinly-wrapped version of Numpy 
from autograd import grad 

def taylor_cosine(x): # Taylor approximation to cosine function 
  # Your np code here
  return ans

In the following code for the gradient descent for linear regression change the manual gradient computation to the PyTorch/jax autograd way. Compare those two approaches in time.

In order to do this, set the tolerance rate for the function value \varepsilon = 10^{-9}. Compare the total time required to achieve the specified value of the function for analytical and automatic differentiation. Perform measurements for different values of n from np.logspace(1,4).

For each n value carry out at least 3 runs.

import numpy as np 

# Compute every step manually

# Linear regression
# f = w * x 

# here : f = 2 * x
X = np.array([1, 2, 3, 4], dtype=np.float32)
Y = np.array([2, 4, 6, 8], dtype=np.float32)

w = 0.0

# model output
def forward(x):
    return w * x

# loss = MSE
def loss(y, y_pred):
    return ((y_pred - y)**2).mean()

# J = MSE = 1/N * (w*x - y)**2
# dJ/dw = 1/N * 2x(w*x - y)
def gradient(x, y, y_pred):
    return np.dot(2*x, y_pred - y).mean()

print(f'Prediction before training: f(5) = {forward(5):.3f}')

# Training
learning_rate = 0.01
n_iters = 20

for epoch in range(n_iters):
    # predict = forward pass
    y_pred = forward(X)

    # loss
    l = loss(Y, y_pred)

    # calculate gradients
    dw = gradient(X, Y, y_pred)

    # update weights
    w -= learning_rate * dw

    if epoch % 2 == 0:
        print(f'epoch {epoch+1}: w = {w:.3f}, loss = {l:.8f}')

print(f'Prediction after training: f(5) = {forward(5):.3f}')

Calculate the 4th derivative of hyperbolic tangent function using Jax autograd.
Compare analytic and autograd (with any framework) approach for the hessian of:

f(x) = \dfrac{1}{2}x^TAx + b^Tx + c
Compare analytic and autograd (with any framework) approach for the gradient of:

f(X) = tr(AXB)
Compare analytic and autograd (with any framework) approach for the gradient and hessian of:

f(x) = \dfrac{1}{2} \|Ax - b\|^2_2
Compare analytic and autograd (with any framework) approach for the gradient and hessian of:

f(x) = \ln \left( 1 + \exp\langle a,x\rangle\right)
You will work with the following function for this exercise,

f(x,y)=e^{−\left(sin(x)−cos(y)\right)^2}

Draw the computational graph for the function. Note, that it should contain only primitive operations - you need to do it automatically - jax example, PyTorch example - you can google/find your own way to visualise it.
Compare analytic and autograd (with any framework) approach for the gradient of:

f(X) = - \log \det X
Suppose, we have the following function f(x) = \frac{1}{2}\|x\|^2, select a random point x_0 \in \mathbb{B}^{1000} = \{0 \leq x_i \leq 1 \mid \forall i\}. Consider 10 steps of the gradient descent starting from the point x_0:

x_{k+1} = x_k - \alpha_k \nabla f(x_k)

Your goal in this problem is to write the function, that takes 10 scalar values \alpha_i and return the result of the gradient descent on function L = f(x_{10}). And optimize this function using gradient descent on \alpha \in \mathbb{R}^{10}. Suppose, \alpha_0 = \mathbb{1}^{10}.

\alpha_{k+1} = \alpha_k - \beta \frac{\partial L}{\partial \alpha}

\frac{\partial L}{\partial \alpha} should be computed at each step using automatic differentiation. Choose any \beta and the number of steps your need. Describe obtained results.
Compare analytic and autograd (with any framework) approach for the gradient and hessian of:

f(x) = x^\top x x^\top x