# Matrix calculus

1. Find the derivatives of $f(x) = Ax, \quad \nabla_x f(x) = ?, \nabla_A f(x) = ?$
2. Find $\nabla f(x)$, if $f(x) = c^Tx$.
3. Find $\nabla f(x)$, if $f(x) = \dfrac{1}{2}x^TAx + b^Tx + c$.
4. Find $\nabla f(x), f''(x)$, if $f(x) = -e^{-x^Tx}$.
5. Find the gradient $\nabla f(x)$ and hessian $f''(x)$, if $f(x) = \dfrac{1}{2} \|Ax - b\|^2_2$.
6. Find $\nabla f(x)$, if $f(x) = \|x\|_2 , x \in \mathbb{R}^p \setminus \{0\}$.
7. Find $\nabla f(x)$, if $f(x) = \|Ax\|_2 , x \in \mathbb{R}^p \setminus \{0\}$.
8. Find $\nabla f(x), f''(x)$, if $f(x) = \dfrac{-1}{1 + x^\top x}$.
9. Find $f'(X)$, if $f(X) = \det X$

Note: here under $f'(X)$ assumes first order approximation of $f(X)$ using Taylor series: $f(X + \Delta X) \approx f(X) + \mathbf{tr}(f'(X)^\top \Delta X)$

10. Find $f''(X)$, if $f(X) = \log \det X$

Note: here under $f''(X)$ assumes second order approximation of $f(X)$ using Taylor series: $f(X + \Delta X) \approx f(X) + \mathbf{tr}(f'(X)^\top \Delta X) + \frac{1}{2}\mathbf{tr}(\Delta X^\top f''(X) \Delta X)$

11. Find gradient and hessian of $f : \mathbb{R}^n \to \mathbb{R}$, if:

12. What is the gradient, Jacobian, Hessian? Is there any connection between those three definitions?
13. Calculate: $\dfrac{\partial }{\partial X} \sum \text{eig}(X), \;\;\dfrac{\partial }{\partial X} \prod \text{eig}(X), \;\;\dfrac{\partial }{\partial X}\text{tr}(X), \;\; \dfrac{\partial }{\partial X} \text{det}(X)$
14. Calculate the Frobenious norm derivative: $\dfrac{\partial}{\partial X}\|X\|_F^2$
15. Calculate the gradient of the softmax regression $\nabla_\theta L$ in binary case ($K = 2$) $n$ - dimensional objects: $h_\theta(x) = \begin{bmatrix} P(y = 1 | x; \theta) \\ P(y = 2 | x; \theta) \\ \vdots \\ P(y = K | x; \theta) \end{bmatrix} = \frac{1}{ \sum_{j=1}^{K}{\exp(\theta^{(j)\top} x) }} \begin{bmatrix} \exp(\theta^{(1)\top} x ) \\ \exp(\theta^{(2)\top} x ) \\ \vdots \\ \exp(\theta^{(K)\top} x ) \\ \end{bmatrix}$

16. Find $\nabla f(X)$, if $f(X) = \text{tr } AX$
17. Find $\nabla f(X)$, if $f(X) = \langle S, X\rangle - \log \det X$
18. Find $\nabla f(X)$, if $f(X) = \ln \langle Ax, x\rangle, A \in \mathbb{S^n_{++}}$
19. Find the gradient $\nabla f(x)$ and hessian $f''(x)$, if

20. Find the gradient $\nabla f(x)$ and hessian $f''(x)$, if $f(x) = \frac{1}{3}\|x\|_2^3$
21. Calculate $\nabla f(X)$, if $f(X) = \| AX - B\|_F, X \in \mathbb{R}^{k \times n}, A \in \mathbb{R}^{m \times k}, B \in \mathbb{R}^{m \times n}$
22. Calculate the derivatives of the loss function with respect to parameters $\frac{\partial L}{\partial W}, \frac{\partial L}{\partial b}$ for the single object $x_i$ (or, $n = 1$)
23. Find the gradient $\nabla f(x)$ and hessian $f''(x)$, if $f(x) = \langle x, x\rangle^{\langle x, x\rangle}, x \in \mathbb{R}^p \setminus \{0\}$
24. Find the gradient $\nabla f(x)$ and hessian $f''(x)$, if $f(x) = \frac{\langle Ax, x\rangle}{\|x\|_2^2}, x \in \mathbb{R}^p \setminus \{0\}, A \in \mathbb{S}^n$
25. Find the gradient $\nabla f(x)$ and hessian $f''(x)$, if $f(x) = \frac{1}{2}\|A - xx^\top\|^2_F, A \in \mathbb{S}^n$
26. Find the gradient $\nabla f(x)$ and hessian $f''(x)$, if $f(x) = \|xx^\top\|_2$