Handout #3 — From Counting to Learning

Handout #2 ended at a wall: counting gives you an excellent estimate for contexts you have seen and nothing at all for the rest, and every repair was a rule someone designed by hand. This handout takes the way out. Stop counting; fit a function instead — and then work out what kind of function, and how you fit it.

1Language modeling as a learning problem

Here is the reframing that opens the rest of the course. Everything so far treated estimation as counting. Treat it instead as fitting a function.

We want a function $f_\theta$ that maps a context to a distribution over the vocabulary. The training data is free — Handout #1 §2.1 — one example per position in the corpus. The loss is cross-entropy, which §5 just told us is the thing we want to minimize anyway. That is an ordinary supervised learning problem.

1.1  A fixed-window neural language model

The simplest version keeps the Markov assumption and changes only how the conditional is computed. Fix a window — say four words. Discard anything earlier. Look up a vector (an embedding) for each of the four words, hand the collection to a neural network with parameters $\Theta$, and have it emit a distribution over the whole vocabulary. Train $\Theta$ to put high probability on the word that actually came next.

Note what has and has not changed. The window is still finite, so long-range dependencies are still invisible — this model does not solve Handout #2 §1.3's computer … crashed problem. What it solves is sparsity.

1.2  Why representations beat counting

That last row is the whole argument. A count table has no notion that the dog slept resembles the cat slept: they are different keys, and what you learn about one tells you nothing about the other. Represent words as vectors and the context as a function of those vectors, and similarity comes for free — if dog and cat end up with nearby vectors because they occur in similar places, a prediction learned for one context transfers to the other automatically.

Generalization replaces smoothing. Not as a patch, but as a structural property. And notice the parameter count: a neural LM's size does not grow with $V^n$, so the exponential wall in Handout #2 §3.2 simply is not there. That is why the 2003 paper mattered, and why every model in this course descends from it.

2What a neural network is

§1 left a box in the diagram labelled "neural network" and an obligation to explain it. Strip away the mystique and the object is unglamorous: a neural network is a function built by alternating matrix multiplications with a simple nonlinear function. That is the whole definition. Everything else — depth, architecture, attention — is a choice about which matrices and in what order.

2.1  One unit

Start with a single unit. It takes a vector $x \in \mathbb{R}^d$, scores it against a weight vector $w$, adds a bias $b$, and passes the result through a nonlinear activation $\sigma$:

The inner product is the operation from Handout #1 §1.1, doing exactly what §1.2 said an inner product does: measuring alignment. A unit asks "how much does this input look like the pattern $w$?", shifts the answer by $b$, and squashes it.

2.2  A layer is a matrix

Now put $m$ units side by side. Each has its own weight vector, so stack them as rows of a matrix $W \in \mathbb{R}^{m \times d}$, collect the biases into $b \in \mathbb{R}^m$, and all $m$ units compute at once:

This is why the whole field runs on matrix multiplication, and why Handout #1 §1.4 insisted you be able to count its cost: a layer is $O(md)$, and $\sigma$ is applied element-wise, so it is free by comparison. The hardware story in Handout #1 §4 is a story about making this one line fast.

A feedforward network (or multi-layer perceptron, MLP) is that line repeated, each layer feeding the next:

Note the last layer usually has no activation — for language modeling its output is the logits, and softmax comes after. The vector $\theta = \{W_1, b_1, \ldots, W_L, b_L\}$ is everything the model has learned, and "a model with 7 billion parameters" is a statement about the total size of these matrices.

2.3  The activations you will meet

3A brief history

Handout #1 §4 asked why, if neural networks are so old, almost everything happened recently. Here is the same story at closer range — and note the shape of it: two long winters, each ended by a technical fix that had been available for years.

The pattern worth extracting: twice, the ideas were sound and the field concluded they were not, because the surrounding conditions were missing. That is a reason to be careful about what you infer from a method's current failure — and equally, a reason not to assume every currently-failing method is merely waiting for more compute.

4Background: the algebra you will actually use

Handout #1 §1.1 covered the operations. This section is about the one thing that trips everyone up in practice: shapes.

4.1  Batching, and why everything has a leading dimension

§2.2 wrote a layer as $h = \sigma(Wx + b)$ for a single input $x$. You never do that. GPUs are efficient because they do the same operation to many inputs at once, so we stack $B$ examples into a matrix $X \in \mathbb{R}^{B \times d}$ and compute

Note the transpose, and note that $b \in \mathbb{R}^m$ is added to every row by broadcasting. In this course activations almost always carry the shape (batch, sequence, dim): $B$ independent documents, $T$ positions each, $d$ numbers per position.

import torch
import torch.nn as nn

B, T, d_in, d_out = 32, 128, 768, 3072

x = torch.randn(B, T, d_in)          # (32, 128, 768)
layer = nn.Linear(d_in, d_out)       # holds W: (3072, 768) and b: (3072,)
h = torch.relu(layer(x))             # (32, 128, 3072)

print(layer.weight.shape, layer.bias.shape, h.shape)
# nn.Linear applies to the LAST dimension and leaves the leading ones alone,
# so the same layer works for (d_in,), (B, d_in), or (B, T, d_in).

scores = torch.randn(32, 128)     # (batch, sequence) -- one score per position
mask   = torch.randn(128)         # (sequence,)
scores + mask                     # fine: broadcasts over batch

bad = scores + torch.randn(32)    # RuntimeError -- 128 vs 32
worse = scores + torch.randn(32, 1)   # NO ERROR. Broadcast over sequence.
                                      # Every position in a document got the
                                      # same value. Trains, silently worse.

5Background: optimization

We have a function $f_\theta$ and a loss. Training means finding $\theta$ that makes the loss small. There is no closed form — unlike the counting in Handout #2 §3, where the answer was a ratio of integers — so we search, by repeatedly taking a small step downhill.

5.1  Gradient descent

The gradient $\nabla_\theta \mathcal{L}$ points in the direction of steepest increase. So step the other way:

That single line is the whole algorithm, and $\eta$ — the learning rate — is the most consequential number you will choose. The widget below is a 1-D loss surface with one parameter, which is enough to see everything that matters.

5.2  Stochastic gradient descent

One problem remains. $\mathcal{L}$ is a sum over the whole training set, so one exact gradient step requires a pass over every token you have. At web scale that is absurd — you would get a handful of steps per week.

The fix is to estimate the gradient from a random minibatch of $B$ examples instead of all $N$:

This is the same Monte Carlo move as Handout #2 §5's cross-entropy question: replace an expectation with an average over a sample. The estimate is noisy, and that is an acceptable trade — a slightly wrong direction computed a thousand times faster wins easily. The noise even helps, by shaking the optimizer out of the shallow minima §5.1 warned about.

optimizer = torch.optim.SGD(model.parameters(), lr=0.1)

for x_batch, y_batch in dataloader:      # each pass = one minibatch
    optimizer.zero_grad()                # .grad accumulates -- clear it first
    logits = model(x_batch)              # forward
    loss = F.cross_entropy(logits, y_batch)
    loss.backward()                      # gradients into every .grad  (Session 4)
    optimizer.step()                     # theta <- theta - lr * grad

Those five lines are the training loop for every model in this course. What changes later is the model, the optimizer (SGD becomes Adam), and the scale — never the shape of the loop. The one line not yet explained is loss.backward(), which is Session 4's entire subject.

6Check yourself

Show all answers

7Optional further reading

Strictly optional — nothing here is assumed in class or on a quiz.

• 3Blue1Brown, Neural Networks — the first two videos cover §2 and §5 visually and are the single best use of forty minutes if this handout felt abstract. (The backpropagation videos are assigned reading for Session 4.)
• Goodfellow, Bengio & Courville, Deep Learning, Ch. 6 "Deep Feedforward Networks" — the textbook treatment of §2, including a proper discussion of why depth helps.
• Bengio, Ducharme, Vincent & Jauvin, A Neural Probabilistic Language Model (2003) — the model in §1.1, in the authors' own words. Very readable, and short.
• Rumelhart, Hinton & Williams, Learning representations by back-propagating errors (1986) — four pages that ended the first AI winter. Read it after Session 4.