Matrix Calculus

Gradients of vectors and matrices, and backpropagation.

Matrix Calculus

How to differentiate functions of vectors and matrices, and how backpropagation uses those rules to compute gradients in neural networks.

Derivatives of vector functions

Gradient – For a scalar function f(x)f(x) of a vector x∈Rnx \in \mathbb{R}^n, the gradient ∇f\nabla f is the vector of partial derivatives ∂f/∂xi\partial f/\partial x_i. It points in the direction of steepest increase. For a scalar function of a matrix XX, the gradient is the matrix of partial derivatives with the same shape as XX.

Jacobian – For a vector function y=f(x)y = f(x) with y∈Rmy \in \mathbb{R}^m, the Jacobian J∈Rm×nJ \in \mathbb{R}^{m \times n} collects every partial derivative: Jij=∂yi/∂xjJ_{ij} = \partial y_i/\partial x_j.

Hessian – For a scalar ff, the Hessian H∈Rn×nH \in \mathbb{R}^{n \times n} is the matrix of second derivatives, Hij=∂2f/∂xi∂xjH_{ij} = \partial^2 f/\partial x_i\partial x_j. It describes curvature and gives the second-order Taylor expansion

f(x+δ)≈f(x)+∇f⊤δ+12δ⊤Hδf(x + \delta) \approx f(x) + \nabla f^\top\delta + \tfrac12\delta^\top H\delta

Shape rule – Gradients have the same shape as the variable. This is the fastest check on any result: if ∂L/∂W\partial L/\partial W doesn't have the shape of WW, something is transposed.

Common gradients

Here aa, bb are constant vectors and AA is a constant matrix.

FunctionGradient
a⊤xa^\top xaa
x⊤Axx^\top Ax(A+A⊤)x(A + A^\top)x, which is 2Ax2Ax when AA is symmetric
∥Ax−b∥2\lVert Ax - b\rVert^2 (squared Euclidean length)2A⊤(Ax−b)2A^\top(Ax - b)
tr⁡(AX)\operatorname{tr}(AX), with respect to XX (trace = sum of diagonal)A⊤A^\top
tr⁡(X⊤AX)\operatorname{tr}(X^\top AX)(A+A⊤)X(A + A^\top)X
log⁡det⁡X\log\det XX−⊤X^{-\top} (inverse transpose)

The Hessian of x⊤Axx^\top Ax is A+A⊤A + A^\top. The derivative of an inverse is d(X−1)=−X−1(dX)X−1d(X^{-1}) = -X^{-1}(dX)X^{-1}.

Backpropagation

Chain rule – If LL depends on xx through y=f(x)y = f(x), then ∂L/∂x=J⊤(∂L/∂y)\partial L/\partial x = J^\top(\partial L/\partial y), where JJ is the Jacobian of ff.

Backpropagation – A network computes a scalar loss LL through a chain of simple operations. Backpropagation applies the chain rule from the loss backwards, one operation at a time. Write zˉ=∂L/∂z\bar z = \partial L/\partial z for the gradient of the loss with respect to an intermediate value zz, called its adjoint. Each operation only needs a rule for turning the adjoint of its output into adjoints of its inputs.

Forward operationBackward rule
Matrix product Y=XWY = XWXˉ=YˉW⊤\bar X = \bar YW^\top, Wˉ=X⊤Yˉ\bar W = X^\top\bar Y
Bias y=x+by = x + b, with bb added to every row of a batchbˉ=\bar b = sum of yˉ\bar y over the batch
Elementwise y=ϕ(x)y = \phi(x)xˉ=yˉ⊙ϕ′(x)\bar x = \bar y \odot \phi'(x), where ⊙\odot is elementwise product
Elementwise product y=x1⊙x2y = x_1 \odot x_2xˉ1=yˉ⊙x2\bar x_1 = \bar y \odot x_2
A value used in two placesadd the gradients from both uses
Prefix product yi=∏j≤ixjy_i = \prod_{j \le i} x_jxˉk=∑i≥kyˉi yi/xk\bar x_k = \sum_{i \ge k}\bar y_i\, y_i/x_k, computable with a reverse cumulative sum

Activation derivatives – Sigmoid σ(x)=1/(1+e−x)\sigma(x) = 1/(1 + e^{-x}) has σ′=σ(1−σ)\sigma' = \sigma(1 - \sigma). tanh⁡′=1−tanh⁡2\tanh' = 1 - \tanh^2. ReLU⁡(x)=max⁡(0,x)\operatorname{ReLU}(x) = \max(0, x) has derivative 11 for x>0x > 0 and 00 otherwise.

Example: a two-layer network on a batch of BB inputs X∈RB×dX \in \mathbb{R}^{B \times d} computes H=ReLU⁡(XW1)H = \operatorname{ReLU}(XW_1) with W1∈Rd×hW_1 \in \mathbb{R}^{d \times h}, then Z=HW2Z = HW_2 with W2∈Rh×kW_2 \in \mathbb{R}^{h \times k}. Given Zˉ∈RB×k\bar Z \in \mathbb{R}^{B \times k}: Wˉ2=H⊤Zˉ\bar W_2 = H^\top\bar Z has shape h×kh \times k, Hˉ=ZˉW2⊤\bar H = \bar ZW_2^\top has shape B×hB \times h, then Wˉ1=X⊤(Hˉ⊙1[XW1>0])\bar W_1 = X^\top(\bar H \odot \mathbf{1}[XW_1 > 0]) has shape d×hd \times h. Every shape matches its variable.

Softmax and cross-entropy

Softmax – Turns a vector of scores z∈RKz \in \mathbb{R}^K into probabilities:

pi=softmax⁡(z)i=ezi∑jezjp_i = \operatorname{softmax}(z)_i = \frac{e^{z_i}}{\sum_j e^{z_j}}

Its Jacobian is ∂pi/∂zj=pi(δij−pj)\partial p_i/\partial z_j = p_i(\delta_{ij} - p_j), where δij\delta_{ij} is 11 if i=ji = j and 00 otherwise. In matrix form that is diag⁡(p)−pp⊤\operatorname{diag}(p) - pp^\top.

Cross-entropy loss – For a correct class yy, the loss is L=−log⁡pyL = -\log p_y. Combined with softmax, the gradient is simply

∂L∂z=p−ey\frac{\partial L}{\partial z} = p - e_y

where eye_y is the one-hot vector with a 11 at position yy.

The same "prediction minus target" gradient appears for sigmoid with binary cross-entropy, and for a linear model with squared error.

Log-sum-exp trick – Exponentials overflow for large scores. Subtracting the maximum m=max⁡izim = \max_i z_i fixes this without changing the result:

logsumexp⁡(z)=m+log⁡∑iezi−m,log⁡py=zy−logsumexp⁡(z)\operatorname{logsumexp}(z) = m + \log\sum_i e^{z_i - m}, \qquad \log p_y = z_y - \operatorname{logsumexp}(z)

Compute log⁡py\log p_y this way, never as the log of a softmax output, which can underflow to log⁡0\log 0.

Normalization layers

Layer normalization – Rescales each input vector x∈Rdx \in \mathbb{R}^d to mean zero and variance one: x^=(x−μ)/σ\hat x = (x - \mu)/\sigma, with μ\mu the mean of the entries and σ=var⁡(x)+ϵ\sigma = \sqrt{\operatorname{var}(x) + \epsilon} for a small ϵ\epsilon. Its backward pass is

xˉ=1σ(x^ˉ−mean⁡(x^ˉ)−x^⋅mean⁡(x^ˉ⊙x^))\bar x = \frac{1}{\sigma}\Big(\bar{\hat x} - \operatorname{mean}(\bar{\hat x}) - \hat x\cdot\operatorname{mean}(\bar{\hat x} \odot \hat x)\Big)

The result has no component along the all-ones direction or along x^\hat x, the two directions that normalization removes.

Checking gradients

Finite differences – Estimate each partial derivative with a centred difference, where eie_i is the ii-th unit vector:

∂f∂xi≈f(x+hei)−f(x−hei)2h\frac{\partial f}{\partial x_i} \approx \frac{f(x + he_i) - f(x - he_i)}{2h}

Use h≈10−5h \approx 10^{-5} in 64-bit floating point, and compare to the analytic gradient by relative error, which should be below about 10−610^{-6}.