Handout #1 — Foundations & Prerequisites

1. $0\cdot 3 + 1\cdot 2 + 2\cdot 4 = \mathbf{10}$
2. $0\cdot 1 + 1\cdot 2 + 2\cdot(-1) = \mathbf{0}$. The two vectors are orthogonal, which is exactly what a zero inner product means — and it is why we will use the dot product as a similarity score.
3. $2\begin{bmatrix}3\\3\\6\end{bmatrix} = \begin{bmatrix}\mathbf{6}\\\mathbf{6}\\\mathbf{12}\end{bmatrix}$
4. $\sqrt{0^2+1^2+2^2} = \sqrt{5} \approx 2.236$
5. $\begin{bmatrix}0&1&2\end{bmatrix}$ — a row vector. The transpose does not change the data, only which axis it lies along; that distinction is what makes $x^\top y$ a scalar and $x y^\top$ a matrix.
6. $Ax = \begin{bmatrix}3\cdot0+2\cdot1+2\cdot2\\1\cdot0+3\cdot1+1\cdot2\\1\cdot0+1\cdot1+3\cdot2\end{bmatrix} = \begin{bmatrix}\mathbf{6}\\\mathbf{5}\\\mathbf{7}\end{bmatrix}$
7. $x^\top(Ax) = \begin{bmatrix}0&1&2\end{bmatrix}\begin{bmatrix}6\\5\\7\end{bmatrix} = 0 + 5 + 14 = \mathbf{19}$

1. True — this is the definition.
2. True — and the reason $\lVert\cdot\rVert^2$ is so much more pleasant to differentiate than $\lVert\cdot\rVert$.
3. False, and not even the same type: the left side is a $1\times1$ scalar, the right side is an $n \times n$ matrix (the outer product). Shape-checking catches this immediately.
4. False. Note $(x-y)^\top(y-x) = -(x-y)^\top(x-y) = -\lVert x-y\rVert^2$, so the right side is off by a sign: the correct identity is $\lVert x-y \rVert^2 = \lVert x \rVert^2 - 2x^\top y + \lVert y \rVert^2$.
5. False — matrix multiplication does not commute. This is why the order of layers matters.
6. True — distributivity holds.
7. True — the transpose reverses the order. You will use this constantly when deriving backward passes.
8. True. Both sides are the same scalar: transposing a $1\times1$ matrix leaves it alone, and $(x^\top A y)^\top = y^\top A^\top x$.
9. True — entry $(i,j)$ of $A^\top A$ is the inner product of columns $i$ and $j$, which is $1$ when $i=j$ and $0$ otherwise precisely when the columns are orthonormal.

Not invertible. $\det(B) = 0$ — and you can see why without computing it: the three rows sum to the zero vector, so they are linearly dependent, and $B\mathbf{1} = 0$.

Diagonalizable anyway. Singularity and diagonalizability are independent properties: the first asks whether an eigenvalue is $0$, the second whether there are enough independent eigenvectors. The eigenvalues are $\lambda_1 = 3,\ \lambda_2 = 1,\ \lambda_3 = 0$, with eigenvectors

$$ v_1 = \begin{bmatrix}1\\-2\\1\end{bmatrix},\quad v_2 = \begin{bmatrix}1\\0\\-1\end{bmatrix},\quad v_3 = \begin{bmatrix}1\\1\\1\end{bmatrix}. $$

Three distinct eigenvalues force three independent eigenvectors, so with $P = \begin{bmatrix}v_1 & v_2 & v_3\end{bmatrix}$ we get $P^{-1} B P = D = \operatorname{diag}(3, 1, 0)$.

($B$ is symmetric, so the spectral theorem guaranteed a real orthogonal diagonalization before we computed anything — the eigenvectors above are mutually orthogonal.)

1. Count outcomes over the 36 equally likely pairs. Difference $\geq 3$: the pairs $(1,4),(1,5),(1,6),(2,5),(2,6),(3,6)$ and their six mirrors, so $12$ outcomes. Doubles: $6$ outcomes. These are disjoint, so

$$ \mathbb{E}[X] = \tfrac{12}{36}\cdot 15 + \tfrac{6}{36}\cdot 5 = 5 + \tfrac{5}{6} = \tfrac{35}{6} \approx \mathbf{\$5.83}. $$

A fair price is the expected payout: it is the price at which the game has zero expected value to either side.

2. Inclusion–exclusion: $P(A \cup B) = P(A) + P(B) - P(A,B)$, so $0.95 = 0.4 + P(B) - 0 \Rightarrow P(B) = \mathbf{0.55}$.

3. Independence means $P(A,B) = P(A)P(B) = 0.4\,P(B)$, so

$$ 0.95 = 0.4 + P(B) - 0.4P(B) = 0.4 + 0.6\,P(B) \;\Longrightarrow\; P(B) = \frac{0.55}{0.6} = \frac{11}{12} \approx \mathbf{0.9167}. $$

Worth pausing on: the same two marginals are consistent with very different $P(B)$ depending on the dependence structure you assume. "Independent" is a modelling assumption, not a default.

First, collapse the two-stage experiment. Each trial independently produces a head with probability $p = 0.5(0.3) + 0.5(0.9) = 0.6$, so $X \sim \text{Binomial}(3, 0.6)$. The coin choice is re-randomized every trial, which is what keeps the trials independent — if you picked one coin once and flipped it three times, the trials would be correlated and the variance would be larger.

1. $\mathbb{E}[X] = np = 3(0.6) = \mathbf{1.8}$.

2. $\operatorname{Var}[X] = np(1-p) = 3(0.6)(0.4) = \mathbf{0.72}$. Via the definition, as a check: $\mathbb{E}[X^2] = \sum_k \binom{3}{k}p^k(1-p)^{3-k}k^2 = 3.96$, and $3.96 - 1.8^2 = 3.96 - 3.24 = 0.72$. ✓

3. $Y$ is a nonlinear function of $X$, so you must average over the distribution — $\mathbb{E}[1/(2+X)] \neq 1/(2 + \mathbb{E}[X])$. With $P(X=k) = \binom{3}{k}(0.6)^k(0.4)^{3-k}$:

$$ \mathbb{E}[Y] = 0.0320 + 0.0960 + 0.1080 + 0.0432 = \frac{349}{1250} = \mathbf{0.2792}. $$

(For comparison, $1/(2 + \mathbb{E}[X]) = 1/3.8 \approx 0.2632$ — close, but not equal. Jensen's inequality tells you which way the gap must go, since $1/(2+x)$ is convex.)

No. The identity $\operatorname{Var}[\sum X_i] = \sum \operatorname{Var}[X_i]$ requires the summands to be uncorrelated. In general

$$ \operatorname{Var}[X_1 + X_2] = \operatorname{Var}[X_1] + \operatorname{Var}[X_2] + 2\operatorname{Cov}[X_1, X_2]. $$

For two independent draws the covariance is $0$ and you get $2\sigma^2$. But $X + X$ is not two draws — it is one draw used twice, so $\operatorname{Cov}[X, X] = \operatorname{Var}[X] = \sigma^2$ and the formula gives $\sigma^2 + \sigma^2 + 2\sigma^2 = 4\sigma^2$. Consistent. A random variable is emphatically not independent of itself.

This distinction is not a trick question. "Are these two samples independent?" is exactly the question behind duplicated training data, contaminated evaluation sets, and why averaging $k$ samples from one model reduces variance far less than you would hope.

1. $f'(x) = 8x - 3$.

2. $f(x) = x - x^2$, so $f'(x) = 1 - 2x$.

3. The sigmoid satisfies $p'(x) = p(x)\bigl(1 - p(x)\bigr)$ — worth memorizing. Then

$$ f'(x) = 1 - \frac{p'(x)}{p(x)} = 1 - \bigl(1 - p(x)\bigr) = \mathbf{p(x)}. $$

1. $\begin{bmatrix}2x_1 & e^{x_2}\end{bmatrix}$
2. $e^{x_1 + x_2x_3}\begin{bmatrix}1 & x_3 & x_2\end{bmatrix}$ — one chain rule, three partial derivatives of the exponent.
3. $a^\top$. Linear functions have constant gradients; this is the vector version of $\frac{d}{dx}(ax) = a$.
4. In general $\nabla \mathbf{x}^\top A \mathbf{x} = \mathbf{x}^\top (A + A^\top)$. Here $A$ is symmetric, so this is $2\mathbf{x}^\top A = \begin{bmatrix}4x_1 - 2x_2 & -2x_1 + 2x_2\end{bmatrix}$.
5. $\mathbf{x}^\top$. This is why the $\tfrac{1}{2}$ is conventionally there — it cancels the $2$ — and why $L_2$ regularization contributes a term proportional to the weights themselves ("weight decay").

(Whether a gradient is a row or a column vector is a convention. Pick one and be consistent; the shapes will tell you if you have slipped.)

Write $\sigma(z) = (1 + e^{-z})^{-1}$ and apply the chain rule:

$$ \frac{d\sigma}{dz} = -(1+e^{-z})^{-2}\cdot(-e^{-z}) = \frac{e^{-z}}{(1+e^{-z})^{2}}. $$

Now rewrite $e^{-z}$ in terms of $\sigma$. From $\sigma = 1/(1+e^{-z})$ we get $1 + e^{-z} = 1/\sigma$, so $e^{-z} = \dfrac{1-\sigma}{\sigma}$. Substituting:

$$ \frac{d\sigma}{dz} = \frac{(1-\sigma)/\sigma}{(1/\sigma)^{2}} = \frac{1-\sigma}{\sigma}\cdot\sigma^{2} = \sigma(z)\bigl(1-\sigma(z)\bigr). $$

The trick worth keeping is the second step: expressing the derivative in terms of the function's own output. It is why a sigmoid layer's backward pass needs only the value it already computed going forward, and the same trick works for $\tanh$ and for softmax.

1. Differentiate each component with respect to each variable:

$$ J_f(x,y,z) = \begin{bmatrix} 2x & 1 & 0 \\ y & x & 0 \\ 0 & -z^2/y^2 & 2z/y \end{bmatrix}, \qquad J_f\!\left(1, \tfrac12, -1\right) = \begin{bmatrix} 2 & 1 & 0 \\ \tfrac12 & 1 & 0 \\ 0 & -4 & -4 \end{bmatrix}. $$

Check the two entries that are easy to get wrong: at $y = \tfrac12$, $-z^2/y^2 = -1/\tfrac14 = -4$, and $2z/y = -2/\tfrac12 = -4$.

2. Diagonal, with $\sigma(z_i)\bigl(1-\sigma(z_i)\bigr)$ down the diagonal and zeros everywhere else — because output $i$ depends on input $i$ alone.

That matters enormously. A general $d \times d$ Jacobian costs $O(d^2)$ to store and $O(d^2)$ to multiply by (Handout §1.4). A diagonal one is $d$ numbers, and multiplying by it is an element-wise product — $O(d)$. So an element-wise activation is essentially free in the backward pass too, which is a large part of why networks are built from element-wise nonlinearities rather than something that mixes coordinates.

Softmax is the exception: it normalizes across coordinates, so its Jacobian is not diagonal.

1. $O(n \log n)$ — $\log n$ levels of merging, $O(n)$ work per level.
2. $O(n)$. A single pass keeping the top three; no sort needed. (In general, the $k$-th largest is $O(n)$ for fixed $k$.)
3. $O(\log n)$ by binary search.
4. $O(1)$ on average, $O(n)$ worst case when every key collides.
5. $O(nd)$ — $n$ output entries, each a $d$-term inner product.
6. $O(d^2)$ — dominated by the $A\mathbf{x}$ product; the final inner product is only $O(d)$.
7. $O(mnd)$ — $md$ output entries, each an $n$-term inner product.

1. Dividing through by $e^0 = 1$:

$$ \left[\frac{e^2}{1 + e^1 + e^2},\; \frac{e^1}{1 + e^1 + e^2},\; \frac{1}{1 + e^1 + e^2}\right] \approx [0.665,\; 0.245,\; 0.090]. $$

2. Identical. Softmax is shift-invariant: adding a constant $c$ to every logit multiplies numerator and denominator by $e^{c}$, which cancels. Since $[10002, 10001, 10000] = [2,1,0] + 10000$, the outputs match exactly.

This is not a curiosity — it is a numerical necessity. $e^{10002}$ overflows a 64-bit float, so every real implementation subtracts $\max_i z_i$ from the logits first, which changes nothing mathematically and everything practically. It also means logits are only meaningful up to an additive constant: a model that outputs $[2,1,0]$ and one that outputs $[10002,10001,10000]$ are the same model.

3. Exactly $1$, by construction — the denominator is the sum of the numerators. Softmax always returns a valid distribution, which is why an output that is negative or exceeds 1 means the softmax is missing, not broken.

$\mathbf{-\log(0.1)} \approx 2.303$ nats.

Cross-entropy on one example is $-\log p(\text{correct class})$ — see the identity above. Ten classes at equal probability means $p = 0.1$ for whichever class is correct, so each instance contributes $-\log(0.1)$. The question asks for the average, and the average of 20 identical numbers is that number. $-20\log(0.1)$ is the sum, and it is the trap: read whether a loss is summed or averaged before you compare it to anything.

Worth memorizing as a sanity check: an untrained classifier over $V$ classes starts at a loss of about $\log V$. If your training run begins somewhere else, something is wrong before the first gradient step — with $V = 50{,}000$ you should see roughly $\log(50000) \approx 10.8$ nats. This is the same fact that will reappear in Handout #2 as "a uniform model has perplexity $V$".

1. 25. The first layer's weight is $5 \times 3 = 15$ numbers, the second's is $2 \times 5 = 10$, and bias=False means no bias vectors. ReLU has no parameters.

2. (4, 2) — nn.Linear maps the last dimension and leaves the leading ones alone, so a batch of 4 gives 4 rows of 2 logits. The loss is a scalar: cross_entropy reduces over the batch by default (reduction="mean").

3. zero_grad() → backward() → step(). (Equivalently, clearing at the end of the previous iteration.) The one ordering that is definitely wrong is step() before backward() — you would be applying last iteration's gradients.

Omitting zero_grad does not crash. PyTorch accumulates into .grad, so each step would use the sum of every gradient seen so far: the effective step size grows without bound and training diverges for no visible reason. This is the silent-failure mode flagged in the autograd snippet above, and it is worth recognizing from the symptom alone.

Self-supervised learning still has a supervised objective — real $(x, y)$ pairs with a real loss and real gradients. The pairs are constructed automatically rather than annotated. Classical unsupervised methods (clustering, PCA) have no target to predict at all. The practical consequence: self-supervision inherits everything we know about optimizing supervised losses, which is why it scaled and clustering did not.

The second. The definition has three parts, and all three do work:

1. predictive models of the world — the objective is prediction, §2.2;
2. trained on large amounts of freely available unlabeled data — the labels are manufactured, §2.1, which is what makes the scale possible;
3. tightly connected to real-world tasks we care about — §2.4's argument that predicting text forces useful knowledge.

Note why the last option is wrong, since it is the tempting one: self-supervision is not supervised learning with fewer labels. It has as many training targets as there are tokens — it just does not need a human to write them.

From the factors, not the factorization. The identity is exact; the approximation is that a finite network $f_\theta$ with finite training data can only imperfectly represent each conditional $p(x_t \mid x_{<t})$. An n-gram model additionally does assume independence, by truncating the conditioning context to the previous $n-1$ words — that is a second, separate error, and Handout #2 §1.3 takes it up.

No to both. "Pretrained" describes a workflow — a small model pretrained on one narrow corpus and fine-tuned for one task is pretrained but foundational to nothing. "Foundation model" describes a role in an ecosystem, and a model could in principle occupy that role having been trained with human labels (ImageNet-pretrained backbones played roughly that role for years). The terms describe different kinds of property, which is exactly why they are not interchangeable.

There is no single right answer here, and the point of the question is the second half. A defensible case: high-quality text is the constraint, since compute keeps growing while the stock of human-written text does not — which is why synthetic data, data curation and multiple epochs are all live research areas. An equally defensible case: compute, since every published scaling result says more of it still helps. What should move you is evidence, e.g. results showing that additional compute on fixed data has stopped paying, or that synthetic data substitutes cleanly for human text. Be able to say what would change your mind; we will revisit this when we cover scaling laws.

The causal model's factors compose into a single valid joint distribution over sequences, by the chain rule — so sampling token by token, feeding each choice back in, draws from that joint. A masked model only ever learned conditionals of the form $p(x_i \mid x_{\neq i})$, each conditioned on both sides. There is no ordering of those conditionals whose product is a coherent joint, so iterative unmasking is a heuristic, not a sampler.

The objective rewards plausible continuations, not true ones. A fluent, well-formed wrong answer scores well on the training distribution because confident text is what the corpus contains; text is not annotated with whether it was correct. Nothing in $\sum_t \log p_\theta(x_t \mid x_{<t})$ prefers a truthful answer over a probable-looking one, and nothing prefers admitting ignorance — human writing rarely does. Later sessions on alignment are largely about adding the signal the pretraining objective does not contain.