Gradients of vectors and matrices, and backpropagation.
How to differentiate functions of vectors and matrices, and how backpropagation uses those rules to compute gradients in neural networks.
Derivatives of vector functions
Gradient – For a scalar function f(x) of a vector x∈Rn, the gradient ∇f is the vector of partial derivatives ∂f/∂xi. It points in the direction of steepest increase. For a scalar function of a matrix X, the gradient is the matrix of partial derivatives with the same shape as X.
Jacobian – For a vector function y=f(x) with y∈Rm, the Jacobian J∈Rm×n collects every partial derivative: Jij=∂yi/∂xj.
Hessian – For a scalar f, the Hessian H∈Rn×n is the matrix of second derivatives, Hij=∂2f/∂xi∂xj. It describes curvature and gives the second-order Taylor expansion
f(x+δ)≈f(x)+∇f⊤δ+21δ⊤Hδ
Shape rule – Gradients have the same shape as the variable. This is the fastest check on any result: if ∂L/∂W doesn't have the shape of W, something is transposed.
Common gradients
Here a, b are constant vectors and A is a constant matrix.
Function
Gradient
a⊤x
a
x⊤Ax
(A+A⊤)x, which is 2Ax when A is symmetric
∥Ax−b∥2 (squared Euclidean length)
2A⊤(Ax−b)
tr(AX), with respect to X (trace = sum of diagonal)
A⊤
tr(X⊤AX)
(A+A⊤)X
logdetX
X−⊤ (inverse transpose)
💡
The Hessian of x⊤Ax is A+A⊤. The derivative of an inverse is d(X−1)=−X−1(dX)X−1.
Backpropagation
Chain rule – If L depends on x through y=f(x), then ∂L/∂x=J⊤(∂L/∂y), where J is the Jacobian of f.
Backpropagation – A network computes a scalar loss L through a chain of simple operations. Backpropagation applies the chain rule from the loss backwards, one operation at a time. Write zˉ=∂L/∂z for the gradient of the loss with respect to an intermediate value z, called its adjoint. Each operation only needs a rule for turning the adjoint of its output into adjoints of its inputs.
Forward operation
Backward rule
Matrix product Y=XW
Xˉ=YˉW⊤, Wˉ=X⊤Yˉ
Bias y=x+b, with b added to every row of a batch
bˉ= sum of yˉ over the batch
Elementwise y=ϕ(x)
xˉ=yˉ⊙ϕ′(x), where ⊙ is elementwise product
Elementwise product y=x1⊙x2
xˉ1=yˉ⊙x2
A value used in two places
add the gradients from both uses
Prefix product yi=∏j≤ixj
xˉk=∑i≥kyˉiyi/xk, computable with a reverse cumulative sum
Activation derivatives – Sigmoid σ(x)=1/(1+e−x) has σ′=σ(1−σ). tanh′=1−tanh2. ReLU(x)=max(0,x) has derivative 1 for x>0 and 0 otherwise.
Example: a two-layer network on a batch of B inputs X∈RB×d computes H=ReLU(XW1) with W1∈Rd×h, then Z=HW2 with W2∈Rh×k. Given Zˉ∈RB×k: Wˉ2=H⊤Zˉ has shape h×k, Hˉ=ZˉW2⊤ has shape B×h, then Wˉ1=X⊤(Hˉ⊙1[XW1>0]) has shape d×h. Every shape matches its variable.
Softmax and cross-entropy
Softmax – Turns a vector of scores z∈RK into probabilities:
pi=softmax(z)i=∑jezjezi
Its Jacobian is ∂pi/∂zj=pi(δij−pj), where δij is 1 if i=j and 0 otherwise. In matrix form that is diag(p)−pp⊤.
Cross-entropy loss – For a correct class y, the loss is L=−logpy. Combined with softmax, the gradient is simply
∂z∂L=p−ey
where ey is the one-hot vector with a 1 at position y.
💡
The same "prediction minus target" gradient appears for sigmoid with binary cross-entropy, and for a linear model with squared error.
Log-sum-exp trick – Exponentials overflow for large scores. Subtracting the maximum m=maxizi fixes this without changing the result:
Compute logpy this way, never as the log of a softmax output, which can underflow to log0.
Normalization layers
Layer normalization – Rescales each input vector x∈Rd to mean zero and variance one: x^=(x−μ)/σ, with μ the mean of the entries and σ=var(x)+ϵ for a small ϵ. Its backward pass is
xˉ=σ1(x^ˉ−mean(x^ˉ)−x^⋅mean(x^ˉ⊙x^))
💡
The result has no component along the all-ones direction or along x^, the two directions that normalization removes.
Checking gradients
Finite differences – Estimate each partial derivative with a centred difference, where ei is the i-th unit vector:
∂xi∂f≈2hf(x+hei)−f(x−hei)
Use h≈10−5 in 64-bit floating point, and compare to the analytic gradient by relative error, which should be below about 10−6.