# Automatic differentiation

- Calculate the gradient of a Taylor series of a $\cos (x)$ using
`autograd`

library:`import autograd.numpy as np # Thinly-wrapped version of Numpy from autograd import grad def taylor_cosine(x): # Taylor approximation to cosine function # Your np code here return ans`

In the following code for the gradient descent for linear regression change the manual gradient computation to the PyTorch/jax autograd way. Compare those two approaches in time.

In order to do this, set the tolerance rate for the function value \(\varepsilon = 10^{-9}\). Compare the total time required to achieve the specified value of the function for analytical and automatic differentiation. Perform measurements for different values of \(n\) from

`np.logspace(1,4)`

.For each \(n\) value carry out at least 3 runs.

`import numpy as np # Compute every step manually # Linear regression # f = w * x # here : f = 2 * x X = np.array([1, 2, 3, 4], dtype=np.float32) Y = np.array([2, 4, 6, 8], dtype=np.float32) w = 0.0 # model output def forward(x): return w * x # loss = MSE def loss(y, y_pred): return ((y_pred - y)**2).mean() # J = MSE = 1/N * (w*x - y)**2 # dJ/dw = 1/N * 2x(w*x - y) def gradient(x, y, y_pred): return np.dot(2*x, y_pred - y).mean() print(f'Prediction before training: f(5) = {forward(5):.3f}') # Training learning_rate = 0.01 n_iters = 20 for epoch in range(n_iters): # predict = forward pass y_pred = forward(X) # loss l = loss(Y, y_pred) # calculate gradients dw = gradient(X, Y, y_pred) # update weights w -= learning_rate * dw if epoch % 2 == 0: print(f'epoch {epoch+1}: w = {w:.3f}, loss = {l:.8f}') print(f'Prediction after training: f(5) = {forward(5):.3f}')`

- Calculate the 4th derivative of hyperbolic tangent function using
`Jax`

autograd. Compare analytic and autograd (with any framework) approach for the hessian of:

\[f(x) = \dfrac{1}{2}x^TAx + b^Tx + c\]Compare analytic and autograd (with any framework) approach for the gradient of:

\[f(X) = tr(AXB)\]Compare analytic and autograd (with any framework) approach for the gradient and hessian of:

\[f(x) = \dfrac{1}{2} \|Ax - b\|^2_2\]Compare analytic and autograd (with any framework) approach for the gradient and hessian of:

\[f(x) = \ln \left( 1 + \exp\langle a,x\rangle\right)\]