Handout #6 — Backprop in Practice

Handout #5 derived a backward sweep for a layered network. Now we apply the same idea to general computation graphs, follow local and upstream gradients, and see how automatic differentiation implements the sweep. The code reading makes these ideas concrete in a small autograd engine.

1Backprop is the chain rule on a graph

Slides 111–136 · Computation graphs and local/upstream gradients

There is no new mathematics in backpropagation. It is the chain rule from Handout #1 §1.3, applied to a composition of functions, organized so that nothing gets computed twice. The organizing idea is to stop thinking of a network as a formula and start thinking of it as a computation graph: one node per elementary operation, edges carrying values forward.

Take a concrete function and build its graph:

Forward pass: walk left to right, storing each intermediate value. Backward pass: walk right to left, and at every node apply the same rule —

import torch

x = torch.tensor([1.0], requires_grad=True)
y = torch.tensor([3.0], requires_grad=True)
z = torch.tensor([2.0], requires_grad=True)

f = torch.log(x) + torch.exp(y) * z
f.backward()
x.grad, y.grad, z.grad     # tensor([1.]), tensor([40.1711]), tensor([20.0855])

2Why backwards?

The chain rule does not tell you which direction to evaluate in. You could go left to right, computing $\partial(\text{everything})/\partial x$ as you sweep forward — that is forward-mode differentiation, and it is perfectly valid. So why does everyone go backwards?

Count the passes. For a function $f : \mathbb{R}^n \to \mathbb{R}^m$:

Now look at what we actually need. Training minimizes a scalar loss with respect to every parameter: $n$ is in the billions, $m = 1$. Reverse mode gets the whole gradient in one backward pass; forward mode would need billions of them. That asymmetry — and nothing else — is why backpropagation runs backwards.

The price is memory. Every intermediate value the backward pass will need must be kept from the forward pass — that is what $\partial h_3/\partial z = h_2$ was telling us in §1. Activation memory grows with batch size, while parameters, gradients, and optimizer state also consume memory; which component dominates depends on the model and training setup. Gradient checkpointing trades recomputation for reduced saved-activation memory.

3What autograd actually does

PyTorch builds the graph from §1 for you, at runtime, as your forward code executes. There is no compilation step and no special syntax: ordinary Python arithmetic on tensors records itself.

import torch

w = torch.tensor([2.0], requires_grad=True)     # a leaf we want gradients for
x = torch.tensor([3.0])                         # data: no requires_grad, no gradient

y = w * x                                       # y remembers: "I came from a mul"
loss = (y - 1.0) ** 2                           # ...and so on, node by node

print(y.grad_fn)        # <MulBackward0 ...>    -- the recorded operation
print(loss.requires_grad)   # True              -- inherited from w

loss.backward()         # walk the recorded graph, right to left
print(w.grad)           # tensor([30.]) -- d/dw (wx-1)^2 = 2(wx-1)x = 2(5)(3)
print(x.grad)           # None -- x never asked for gradients

Four things in there deserve emphasis.

• requires_grad marks what you want gradients for. It In normal grad mode, differentiable operations on tensors requiring gradients are recorded. Non-differentiable operations and disabled grad mode are exceptions. Trainable model parameters usually require gradients; data usually does not, unless we explicitly want input derivatives.
• grad_fn is the graph. Each non-leaf tensor holds a reference to the operation that produced it and to its inputs. "Building the graph" is not a separate phase — it is the forward pass.
• .backward() normally releases saved tensors. A second traversal that needs those saved values raises an error unless the first call retained them with retain_graph=True. Some simple graphs need no saved values and can be traversed again; do not rely on that exception. Ordinary training builds a fresh graph on the next forward pass.
• .grad accumulates. It is added to, never replaced — hence optimizer.zero_grad(), and Handout #1 Exercise 1.5c.

model.eval()
with torch.no_grad():                    # do not record this forward pass
    val_loss = F.cross_entropy(model(x_val), y_val)

3.1  The loop, annotated

Handout #4 §1.4 introduced minibatch gradient descent. Here is its training loop, with every line now accounted for:

for x_batch, y_batch in dataloader:
    optimizer.zero_grad()              # .grad accumulates -- clear last step's
    logits = model(x_batch)            # forward: compute AND record the graph
    loss = F.cross_entropy(logits, y_batch)     # scalar loss -- implicit backward seed is 1
    loss.backward()                    # accumulate gradients in participating parameters
    optimizer.step()                   # apply optimizer update (plain SGD: theta -= lr * grad)

The order is forced. backward() needs the graph, so it must follow the forward pass; step() needs the gradients, so it must follow backward(); and gradients must be cleared between updates unless accumulation is intentional. Clearing at the end of a step also works if the first step starts clear. These are the core operations; practical loops can add clipping, accumulation, and other controls.

import torch
w = torch.tensor(2.0, requires_grad=True)
(w ** 2).backward()
print(w.grad.item())
(3 * w).backward()
print(w.grad.item())
w.grad = None
(3 * w).backward()
print(w.grad.item())
assert w.item() == 2.0 and w.grad.item() == 3.0

3.2  Why is this tensor's .grad None?

A derivative can flow through a tensor without being stored in that tensor's .grad attribute. With backward(), PyTorch normally accumulates gradients into participating leaf tensors that have requires_grad=True. A leaf requiring gradients was created directly, rather than by a recorded operation. An intermediate produced by a recorded operation is a non-leaf; call retain_grad() on it before backward if you want its gradient stored for inspection.

Predict which gradients below are 18, 6, zero, or absent. This example requests an input gradient deliberately, to make the distinction visible.

import torch

x = torch.tensor(2.0, requires_grad=True)
unused = torch.tensor(7.0, requires_grad=True)
zero = torch.tensor(0.0, requires_grad=True)
h = 3 * x
assert x.is_leaf and not h.is_leaf
assert h.requires_grad and h.grad_fn is not None
assert not h.retains_grad             # by default, h.grad would not be stored
h.retain_grad()
loss = 0.5 * h.square() + zero.square()
loss.backward()
assert x.grad.item() == 18.0
assert h.grad.item() == 6.0
assert zero.grad.item() == 0.0
assert unused.grad is None

# A new leaf made from detached values does not reconnect to the source.
source = torch.tensor(2.0, requires_grad=True)
isolated = source.detach().clone().requires_grad_()
isolated.square().backward()
assert isolated.grad.item() == 4.0 and source.grad is None

# detach shares storage; clone gives independent storage.
# No backward is pending for these tensors.
original = torch.tensor([2.0], requires_grad=True)
shared = original.detach()
snapshot = original.detach().clone()
shared.add_(3)
assert original.item() == 5.0 and snapshot.item() == 2.0
print("Leaf, retained, zero, unused, and detached gradients checked.")

torch.autograd.grad(loss, inputs), used in Handout #5's checks, returns the requested derivatives rather than accumulating them in those inputs' .grad buffers. Do not interpret an empty buffer after that call as a failed derivative calculation. See the leaf/non-leaf tutorial and detach documentation.

3.3  Backward through tensor operations

The scalar rule “upstream times local” becomes a Jacobian-transpose product. If $y=f(x)\in\mathbb R^m$ and $x\in\mathbb R^n$, an upstream column $v=\nabla_yL$ produces

This is also called a vector–Jacobian product when written as the row $v^\top J_f$. Backward computes this product without constructing the full Jacobian. A scalar loss uses the implicit seed $\partial L/\partial L=1$. For a multi-element output, y.backward(v) supplies the seed explicitly: for fixed $v$, it differentiates $v^\top y$. It does not return every row of the Jacobian.

import torch

x = torch.tensor([2.0, 3.0], requires_grad=True)
y = torch.stack([x[0] * x[1], x[0] + x[1]])
v = torch.tensor([2.0, -1.0])
# J = [[3, 2], [1, 1]], so J.T @ v = [5, 3].
y.backward(v)
torch.testing.assert_close(x.grad, torch.tensor([5.0, 3.0]))

# A fresh graph: the explicitly weighted scalar gives the same result.
x2 = x.detach().clone().requires_grad_()
y2 = torch.stack([x2[0] * x2[1], x2[0] + x2[1]])
(v * y2).sum().backward()
torch.testing.assert_close(x2.grad, x.grad)
print("Explicit seed and weighted scalar loss agree:", x.grad)

For common tensor operations, the product has a simple form. In the table, $G$ is the upstream gradient, and every input gradient has the shape of that input.

Broadcast in forward, sum in backward. Reduce the axes over which an input was reused, preserving singleton dimensions where needed to recover its original shape. A forward mean over selected axes divides by their combined size, not necessarily by the batch size.

import torch

# The mean has six inputs, although the batch size is only two.
X = torch.arange(6.0).reshape(2, 3).requires_grad_()
b = torch.zeros(3, requires_grad=True)
Y = X + b
Y.mean().backward()
torch.testing.assert_close(X.grad, torch.full_like(X, 1 / 6))
torch.testing.assert_close(b.grad, torch.full_like(b, 1 / 3))

# Build a fresh graph and clear accumulated leaf gradients.
X.grad = b.grad = None
(X + b).sum().backward()
torch.testing.assert_close(X.grad, torch.ones_like(X))
torch.testing.assert_close(b.grad, torch.full_like(b, 2))
print("Mean/sum and broadcast gradients have the expected scales.")

See the backward API for explicit seeds and graph retention.

4Reading real code: micrograd

Everything above is now in your head. The rest of your career it will be in someone else's repository, written by people who were not trying to teach you anything. Being able to open an unfamiliar file, find the four lines that matter, and recognize them as something you already understand is a skill in itself — and it is learnable. Each handout from here on ends with one short reading of real, widely used code.

The one for backprop is Karpathy's micrograd: a scalar-valued autograd engine in about 150 lines, plus a small neural-net library on top of it. It is a real, working reverse-mode autodiff implementation — the same algorithm PyTorch runs, with the tensors, the kernels, the device handling and the twenty years of performance work stripped away. What is left is §1–§3 of this handout, in Python you can read in one sitting.

What to assess as you read. Not "do I understand every line" — you will, and that is not the point. The question is whether you can point at the place in the file where each idea from this handout lives: the computation graph, the local derivatives, gradient accumulation, the reverse topological traversal, and the parameter interface. The four subsections below walk those five, then §4.5 hands the file back to you.

4.1  The graph is built by the forward pass

There is no graph data structure anywhere in micrograd, and no function that constructs one. Every node builds itself, at the moment the operation that produced it is executed:

engine.py · Value.__init__

def __init__(self, data, _children=(), _op=''):
    self.data = data
    self.grad = 0
    # internal variables used for autograd graph construction
    self._backward = lambda: None
    self._prev = set(_children)
    self._op = _op # the op that produced this node, for graphviz / debugging / etc

That is the whole node: a number (data), a slot for its gradient (grad, initialized to zero and not to None — remember that), the set of nodes it was computed from (_prev), and a closure (_backward) that knows how to push a gradient one step further back. The default _backward is a no-op, which is exactly right for a leaf: an input has no children to send anything to.

_prev is the edge set of §1's diagram. It points backwards, from a node to its inputs, because backwards is the only direction the algorithm ever needs to walk. This is PyTorch's grad_fn from §3, in nine lines: _op is the string PyTorch would render as MulBackward0, and _backward is the function behind it.

4.2  Local derivatives, and why +=

Now the operations. Each one does two things: compute the output value, and install the closure that will later distribute the output's gradient to the inputs.

engine.py · Value.__mul__

def __mul__(self, other):
    other = other if isinstance(other, Value) else Value(other)
    out = Value(self.data * other.data, (self, other), '*')

    def _backward():
        self.grad += other.data * out.grad
        other.grad += self.data * out.grad
    out._backward = _backward

    return out

Read the two lines inside _backward against §1's rule. out.grad is the upstream gradient $\partial L/\partial \text{out}$; other.data is the local derivative $\partial \text{out}/\partial \text{self}$, which for a product is just the other operand. Their product is the downstream gradient. Downstream = upstream × local is not a slogan about backprop; it is line 1 of _backward, in every op in the file. Check the others and you will find nothing new — __add__ passes out.grad through unchanged (local derivative $1$), __pow__ multiplies by $n x^{n-1}$, and relu multiplies by (out.data > 0), a Python bool doing duty as the $\{0,1\}$ gate.

The += is the part to slow down on. It is gradient accumulation — §3's claim that .grad adds rather than overwrites, and the reason your training loop needs zero_grad(). Here you can see precisely why it must be a sum. Consider $y = x \cdot x + x$. The node x is reached along three separate edges, and each one delivers its own contribution: two from the product, one from the sum. The correct derivative $2x + 1$ only appears if they are added up.

4.3  The traversal: backward()

Eleven lines, and it is the entire algorithm of §2:

engine.py · Value.backward

def backward(self):

    # topological order all of the children in the graph
    topo = []
    visited = set()
    def build_topo(v):
        if v not in visited:
            visited.add(v)
            for child in v._prev:
                build_topo(child)
            topo.append(v)
    build_topo(self)

    # go one variable at a time and apply the chain rule to get its gradient
    self.grad = 1
    for v in reversed(topo):
        v._backward()

build_topo is a depth-first post-order walk: a node is appended only after all of its children are, so topo runs inputs-first and ends with the node you called .backward() on. Then reversed(topo) walks it root-first. That ordering is the correctness condition of the whole method — a node's _backward may only run once out.grad is final, meaning every consumer of that node has already contributed. Reverse topological order is exactly the order that guarantees it.

self.grad = 1 is the scalar seed: $\partial L/\partial L = 1$. For a tensor output, an explicit seed specifies the weighted combination to differentiate, as in §3.3. That is PyTorch's loss.backward() versus t.backward(gradient=...), and it is §2's $m = 1$ argument reduced to an assignment.

Finally, count the work: visited ensures each node is appended once, and the loop calls each _backward once. One pass, one visit per node — linear in the size of the graph, which is the answer to §5's Q1 sitting in plain sight.

4.4  How a model exposes its parameters

nn.py adds no calculus whatsoever — it is 60 lines of bookkeeping over Value. But it answers a question Handout #3's training loop skipped: when you write optimizer.step(), how does anything know which numbers are the model's?

nn.py · the parameter interface (__repr__ and constructors elided; everything else is verbatim)

class Module:
    def zero_grad(self):
        for p in self.parameters():
            p.grad = 0
    def parameters(self):
        return []

class Neuron(Module):
    def __init__(self, nin, nonlin=True):
        self.w = [Value(random.uniform(-1,1)) for _ in range(nin)]
        self.b = Value(0)
        self.nonlin = nonlin
    def __call__(self, x):
        act = sum((wi*xi for wi,xi in zip(self.w, x)), self.b)
        return act.relu() if self.nonlin else act
    def parameters(self):
        return self.w + [self.b]

class Layer(Module):
    def parameters(self):
        return [p for n in self.neurons for p in n.parameters()]

class MLP(Module):
    def parameters(self):
        return [p for layer in self.layers for p in layer.parameters()]

One contract — parameters() returns a flat list of leaf Values — and it composes by recursion: a layer's parameters are its neurons' concatenated, a network's are its layers'. zero_grad is then a two-line loop defined once, on the base class, and inherited by everything. This is torch.nn.Module in miniature: model.parameters(), the thing you hand to your optimizer, is the same recursive flattening, and PyTorch's version differs mainly in doing the registration automatically (via __setattr__) instead of making you write the list.

Two details worth catching in __call__, both easy to read past. The sum(..., self.b) uses the bias as the start value of the sum, which is how $b$ enters the graph as an ordinary addend rather than a special case. And nonlin exists because MLP.__init__ sets nonlin = i != len(nouts)-1: every layer gets a ReLU except the last. That is not a stylistic choice — a ReLU on the output would clamp every negative prediction to zero and, worse, hand back a zero gradient for all of them.

4.5  Now do something to it

Show all answers

def exp(self):
    out = Value(math.exp(self.data), (self,), 'exp')
    def _backward():
        self.grad += out.data * out.grad      # d/dx e^x = e^x -- reuse out.data
    out._backward = _backward
    return out

def log(self):
    out = Value(math.log(self.data), (self,), 'log')
    def _backward():
        self.grad += (1 / self.data) * out.grad
    out._backward = _backward
    return out

from micrograd.engine import Value

x = Value(2.0)
a = x * x
y = a * a
# y = x^4; the correct derivative at x=2 is 32.
y.backward()
assert a.grad == 8.0 and x.grad == 32.0
y.backward()
assert a.grad == 16.0 and x.grad == 96.0
print("After two traversals:", a.grad, x.grad)

# For this graph, clear every node, not only the input.
for node in (x, a, y):
    node.grad = 0.0
y.backward()
assert a.grad == 8.0 and x.grad == 32.0

5Check yourself

Show all answers

6Optional further reading

Strictly optional — nothing here is assumed in class or on a quiz.

• 3Blue1Brown, Backpropagation, intuitively and Backpropagation calculus — these are the assigned reading for Session 5, and if §1 did not land, watch them before anything else.
• Karpathy, The spelled-out intro to neural networks and backpropagation — two and a half hours in which micrograd is written from an empty file. If §4 was the part of this handout you most wanted more of, this is the more.
• Karpathy, Yes you should understand backprop — a short argument for why treating .backward() as a black box will eventually cost you, with the failure modes named. Read it after §3.
• PyTorch, A Gentle Introduction to torch.autograd — the official version of §3, with more on graph mechanics.
• Rumelhart, Hinton & Williams, Learning representations by back-propagating errors (1986) — four pages, and now readable: you have everything you need.