Math & Stats Foundations

Say you're building the arrival estimate for a delivery app. You have three past orders: 1 km took 12 minutes, 2 km took 15, and 3 km took 20.

You guess a rule: minutes = 3 × km + 5. That's a model with two numbers in it, a slope and an intercept. It predicts 8, 11 and 14 minutes, and every one of those is too short. What now?

Three questions come up, in this order:

• How wrong is it, as one number? You can't tell whether a change helped unless you can score it. Square each miss and average: (16 + 16 + 36) / 3 ≈ 22.67.
• Which way should the 3 and the 5 move? You could nudge each one and watch the score, but a model with a billion parameters would need a billion extra evaluations per step.
• How far should they move? Too little and training crawls. Too far and the next guess is worse.

A quieter question hides behind the first. Why square the misses at all? If the target were yes or no ("will this order be late?"), would squaring still be a good score? It isn't, and seeing why takes probability.

Each question belongs to a branch of math you may have met separately. Linear algebra holds the data and computes every prediction at once. Calculus finds the direction that lowers the score. Probability says what "wrong" should mean for your kind of target, and information theory turns that into a loss. Optimization decides the step, and explains why some problems take three steps and others take three hundred.

The same stages train a linear regression in scikit-learn and a GPT-style language model. Section 01 lays out the loop, sections 02 to 06 build each piece on the delivery model with numbers you can check on paper, and section 07 covers what goes wrong on data the model hasn't seen.

01One loop, five toolkits

Here's the loop, run once on the delivery model. Every number in it gets derived later.

1.1 One training step

Why does the score have to be a single number?

Because "downhill" needs one hill. Score a model on accuracy and speed at once and a change can help one and hurt the other, with no single direction to move in. So training collapses its goals into one scalar, the loss. If you care about two things, you add them with a weight, and picking that weight is your call.

1.2 Where these ideas came from

None of this was invented for deep learning. The field borrowed each piece from older math.

02Linear algebra: holding the data and the model

Three orders fit in your head. A million don't, and a GPU is only fast on numbers laid out in regular arrays. Linear algebra is the bookkeeping for those arrays, plus a few facts about them that turn out to control training.

2.1 Vectors, matrices, and one product

A vector is a list of numbers, a matrix is a table of them, and a tensor has any number of axes. A batch of colour images is a 4-D tensor of shape (batch, height, width, channels).

Here's the delivery data as a matrix $X$, one row per order, times the parameters $\theta$:

Each prediction is a dot product, one row times $\theta$, multiplied entry by entry and summed. The column of ones folds the intercept into the product, so it's one more weight and every formula below treats both parameters alike.

2.2 Norms: the size of a miss

Predictions minus targets gives the error vector $e = (-4, -4, -6)$. A norm measures a vector's size, and two come up constantly:

• L2, $\|e\|_2 = \sqrt{\sum_i e_i^2}$, the ordinary length: $\sqrt{68} \approx 8.25$.
• L1, $\|e\|_1 = \sum_i |e_i|$, the sum of absolute values: 14.

Mean squared error is the squared L2 norm over the number of examples, $68/3 \approx 22.67$. Put the parameters inside the same norms and you get L2 and L1 regularization; section 04 shows where each comes from.

Dot products also measure alignment. Divided by both lengths, $\frac{a\cdot b}{\|a\|\,\|b\|}$ is the cosine similarity, from −1 to 1. A projection uses the same number to ask how much of one vector lies along another. Attention (08) is built on this: a large query-key dot product means "this token is relevant to me".

2.3 Eigenvectors, SVD and PCA

Most matrices rotate and stretch. An eigenvector is a direction a matrix only stretches, $Av = \lambda v$, and the eigenvalue $\lambda$ is the stretch factor. For the delivery data, the matrix that matters later is $X^\top X$:

(A 2×2 matrix's eigenvalues come from its trace, 17, and determinant, $14\cdot3 - 6\cdot6 = 6$.) One direction is stretched 46 times more than the other. Section 06 shows that ratio is why this tiny model needs hundreds of steps.

SVD extends the idea to any matrix: $A = U\Sigma V^\top$, a rotation, a scaling, and another rotation. PCA is SVD on mean-centred data. The top eigenvectors of the covariance matrix are the directions the data varies along most, so projecting onto the first few gives the best linear summary in fewer dimensions.

03Calculus: which way is downhill

You have a score and two knobs. Calculus tells you how the score changes when you turn each knob slightly, without trying every setting.

3.1 Derivatives and the gradient

A derivative is a local slope. You can measure one: raise $w$ from 3 to 3.001 and the loss goes from 22.66667 to 22.64667. That's −0.02 per 0.001, so the loss falls about 20 for each unit of $w$. The exact version, with $e_i = wx_i + b - y_i$ and $L = \frac{1}{n}\sum_i e_i^2$:

A partial derivative holds every other input fixed. Stack them and you have the gradient, $\nabla L = (-20, -9.33)$. In matrix form it's $\frac{2}{n}X^\top e$, and $X^\top e = (-30, -14)$ gives the same numbers.

Why walk against the gradient?

The gradient points where the loss rises fastest, so its negative is where it falls fastest, at least for a small step. Both components are negative here, so both parameters should go up. That fits: every prediction was too short.

3.2 One gradient step by hand

The update subtracts the gradient times a learning rate $\eta$. With $\eta = 0.05$:

The predictions become 9.47, 13.47 and 17.47, and the loss falls from 22.67 to 5.06. One step removed 78% of the error. That pace doesn't last:

Watch the slope: it hits the best value, 4.0, after one step, climbs to 4.74, and takes hundreds of steps to come back. The parameters are coupled: while the intercept is too low, a steeper slope is the fastest way to lengthen predictions. Section 06 explains the slow tail.

3.3 The chain rule is backpropagation

A neural network is formulas nested inside formulas, and the chain rule, $\frac{d}{dx}f(g(x)) = f'(g(x))\cdot g'(x)$, says how to differentiate through the nesting. For one weight, through whatever sits between it and the loss, it's a product of local derivatives:

Here $\hat y$ is the output and $z$ the pre-activation of the layer $w$ lives in. Each layer in between adds one factor. On the delivery model it looks like this:

Why keep the forward values around?

Every local derivative needs them: the first backward step needs $e$, and a sigmoid layer's derivative needs its own output. Caching them keeps the backward pass cheap. It's also why training needs far more memory than inference, since every layer's activations must survive until the backward pass reaches them.

3.4 Jacobians, Hessians and curvature

For a function that outputs a vector, the derivative is a matrix, the Jacobian. One derivative further, the matrix of second derivatives is the Hessian, and it describes curvature: all eigenvalues positive is a bowl, mixed signs a saddle, all negative a peak.

The delivery loss has a constant Hessian, $H = \frac{2}{n}X^\top X = \begin{bmatrix} 9.33 & 4 \\ 4 & 2 \end{bmatrix}$, with eigenvalues of about 11.09 and 0.24. Both positive, so it's one bowl, but a long, narrow one.

3.5 Why deep chains vanish or explode

The chain rule multiplies one factor per layer. Factors a bit below 1 shrink the product toward zero on its way back to the first layers; factors above 1 blow it up.

A sigmoid's derivative is never larger than 0.25, so across ten sigmoid layers the activation derivatives alone multiply a gradient by at most $0.25^{10} \approx 9.5 \times 10^{-7}$. A factor of 1.5 per layer gives $1.5^{10} \approx 58$ instead. That's why plain deep networks and long unrolled RNNs were hard to train. Residual connections, careful initialization and normalization are the fixes, covered in 04 · Neural Network Fundamentals.

04Probability: what "wrong" should mean

Squaring the misses was a guess, if a pretty reasonable one. Probability tells you when it's the right guess, what to use when it isn't, and how to reason about a model's outputs once it exists.

4.1 A loss is a likelihood in disguise

Suppose the real delivery time is your line plus Gaussian noise with spread $\sigma$. Then the likelihood is how probable the observed times are under a given $\theta$, and maximum likelihood estimation (MLE) picks the $\theta$ that makes them most probable. The negative log of one Gaussian density is:

The second term has no $\theta$ in it. So mean squared error is maximum likelihood under Gaussian noise, and that's the assumption you make whenever you use it.

Why take the log?

A dataset's likelihood is a product, one probability per example. A thousand examples at 0.5 each give $0.5^{1000} \approx 9.3 \times 10^{-302}$, near the bottom of what a 64-bit float can hold, and two thousand underflow to zero. The log turns the product into a sum, $1000 \log 0.5 \approx -693$, that differentiates term by term. Log only increases, so the maximum doesn't move.

4.2 Yes-or-no targets

Now ask "will this order be late?". The label is 0 or 1, and the model outputs a probability through a sigmoid, $p = \sigma(z) = 1/(1+e^{-z})$. Gaussian noise around a 0/1 label makes no sense. The natural model is a Bernoulli, and its negative log-likelihood is:

That is binary cross-entropy, which falls out of maximum likelihood once the target is binary. Section 05 shows how differently it behaves from squared error when the model is confidently wrong.

4.3 The distributions you'll meet

A random variable maps outcomes to numbers, and its distribution says how likely each value is.

4.4 Expectation, variance, covariance

Expectation is a probability-weighted average. Variance, $\mathrm{Var}(X) = \mathbb{E}[(X-\mathbb{E}[X])^2]$, is spread around it. Covariance is positive when two variables move together and negative when they move apart.

On the delivery data, distance has variance $\frac{1+0+1}{3} \approx 0.67$, and its covariance with time is $\frac{(-1)(-3.67) + 0 + (1)(4.33)}{3} \approx 2.67$. Divide: $2.67/0.67 = 4$, the best-fit slope. Least-squares regression is covariance over variance. Correlation divides covariance by both standard deviations to remove the units; here it's about 0.99.

4.5 Bayes' theorem and the base rate

Conditional probability, $P(A\mid B) = P(A \cap B)/P(B)$, is how likely A is once you know B happened. Bayes' theorem turns one conditional into the other, and it's where intuition fails most often.

Bayes' theorem is that count written as a formula. The numerator is $0.9 \times 0.01 = 0.009$, the fraud that gets flagged; the extra term below it is $0.05 \times 0.99 = 0.0495$, the false alarms.

• P(θ) the prior, belief before the evidence: 1% fraud.
• P(D|θ) the likelihood of the evidence if the hypothesis holds: a 90% flag rate.
• P(θ|D) the posterior, the updated belief you want.
• P(D) the evidence, summed over every hypothesis: 5.85% of transactions get flagged. Over a model's parameters it is usually intractable, which is why approximate Bayesian inference exists.

Why so low, when the model catches 90%?

Legitimate transactions outnumber fraud 99 to 1, and 5% of a large pile beats 90% of a small one. A flag is 18 times likelier on fraud (0.9 against 0.05), which turns prior odds of 1 to 99 into 18 to 99. Raise the prior to 10% fraud and the same flag means 67%. The model didn't change; the population did.

4.6 MLE, MAP, and where regularization comes from

What people call "training" is one of three amounts of Bayes' theorem:

• MLE ignores the prior and maximizes $P(D\mid\theta)$. Unregularized loss minimization is MLE.
• MAP (maximum a posteriori) maximizes $P(D\mid\theta)P(\theta)$. A Gaussian prior on the weights gives L2 regularization; a Laplace prior gives L1.
• Full Bayesian inference keeps the whole posterior, so predictions carry uncertainty, at a much higher computing cost.

The L2 link takes two lines. A zero-mean Gaussian prior with spread $\tau$ has $-\log P(\theta) = \frac{\|\theta\|^2}{2\tau^2} + \text{const}$. Add it to the negative log-likelihood from 4.1 and multiply through by $2\sigma^2$:

That's weight decay with $\lambda = \sigma^2/\tau^2$. A narrower prior, a stronger belief that weights are small, means a bigger penalty.

4.7 Intervals and tests

A confidence interval is a range of plausible values built from data at a chosen confidence level. Hypothesis testing asks whether an effect is surprising enough to reject a "nothing's going on" baseline.

05Information theory: pricing surprise

Section 04 said the loss for a yes/no target is $-\log p$ of the right answer. Information theory explains why that's a natural price for being wrong, and how to compare two distributions.

5.1 Entropy: average surprise

Shannon measured the surprise of an outcome with probability $p$ as $-\log p$. A certain outcome costs nothing; a one-in-a-hundred outcome costs about 6.6 bits. Entropy is the average surprise, $H(p) = -\sum_i p_i \log p_i$:

• A coin that always lands heads: 0 bits.
• A 90/10 coin: $-(0.9\log_2 0.9 + 0.1\log_2 0.1) \approx$ 0.47 bits.
• A fair coin: 1 bit, the maximum for two outcomes.

With $\log_2$ the unit is bits. Deep learning libraries use the natural log, so their losses are in nats (one nat is about 1.44 bits), and the minimum doesn't move.

5.2 Cross-entropy and KL divergence

Cross-entropy $H(p,q)$ is the average surprise when outcomes come from $p$ but you score them with your model's $q$. The extra over $H(p)$ is the KL divergence. On the 90/10 coin with a model that believes 80/20, $H(p,q) \approx 0.522$ bits against $H(p) \approx 0.469$, so the model's mistake costs 0.053 bits per flip.

$H(p)$ is a property of the data that no model changes. So minimizing cross-entropy and minimizing KL are the same optimization problem, which is the case for cross-entropy as a loss.

KL is asymmetric: on the same coins, $D_{KL}(q\|p) \approx 0.064$ bits, where the other direction gave 0.053. So it is not a true distance, and which side you call "true" changes what gets penalized. The project in section 09 makes that visible.

Both run through modern ML. GPT-style pretraining is cross-entropy on the next token. KL is the VAE's pull toward its prior (12), the penalty that keeps an RLHF policy near its reference model (15), and the target in knowledge distillation.

5.3 Cross-entropy against squared error at p = 0.01

Back to the late-delivery classifier. An order was late ($y = 1$) and the model said $p = 0.01$. The model is confidently wrong, so which loss pushes harder? Differentiate each with respect to the score $z$, using the sigmoid's derivative $\frac{dp}{dz} = p(1-p)$. Cross-entropy first, then squared error:

At $p = 0.01$, cross-entropy gives $-0.99$. Squared error gives $2(-0.99)(0.01)(0.99) \approx -0.0196$. Cross-entropy pushes roughly 50 times harder, exactly when the model most needs it.

Squared error's gradient is largest at $p = 1/3$ (about −0.30) and fades toward both ends. Cross-entropy's gradient is the size of the miss. With a softmax over many classes it has the same form, $\partial L/\partial z = \hat{y} - y$.

5.4 Computing it without overflowing

Softmax, $\hat y_i = e^{z_i} / \sum_j e^{z_j}$, uses exponentials. Feed it scores of 1000 and 999 and $e^{1000}$ overflows: a 64-bit float tops out near $e^{709.8}$, a 32-bit float near $e^{88.7}$.

Shifting every score by the same amount doesn't change softmax, so subtract the largest first. (1000, 999) becomes (0, −1), giving probabilities of 0.731 and 0.269. Log-softmax does this and takes the log in one fused step, returning −0.313 and −1.313 directly, so it never computes $\log 0$.

5.5 Mutual information

Mutual information, $I(X;Y)$, is how much observing $Y$ reduces your uncertainty about $X$, and it's zero for independent variables. A good representation, roughly speaking, keeps high mutual information with what you care about and drops nuisance variation. Self-supervised objectives (03) are different ways of chasing that.

06Optimization: taking good steps

Calculus gives a direction but doesn't say how far to walk, or what to do when the direction keeps changing. The delivery model already shows each problem.

6.1 How big a step?

On a bowl-shaped loss, the error along each eigenvector of the Hessian is multiplied by $(1 - \eta\lambda)$ every step. If that factor's size is below 1 the error shrinks; above 1 it grows.

The steep direction of the delivery loss has $\lambda \approx 11.09$. At $\eta = 0.17$ the factor is about −0.89: the sign flips every step (the zigzag) but the size shrinks. At $\eta = 0.19$ it's about −1.11, and the error grows 11% per step. Divergence starts at $\eta = 2/\lambda_{max} \approx 0.180$.

Why not pick a tiny learning rate and wait?

Because the flattest direction sets the wait. Its eigenvalue is 0.24, so at $\eta = 0.05$ the error there shrinks by only $1 - 0.05 \times 0.24 \approx 0.988$ per step, roughly 83 steps per factor of e. The steepest direction caps the learning rate, and the flattest one sets the pace.

6.2 Why the valley zigzags

The ratio of largest to smallest curvature is the condition number: $11.09/0.24 \approx 46$, the same 46 as in $X^\top X$. A large one means a long, narrow valley, and 46 is already a bit of one. The gradient there points mostly across the valley, toward the steep walls, so each step overshoots sideways and barely moves forward.

Here's what it costs on the delivery model, counting steps until the loss is within 0.01 of its best:

Same data, same final predictions. Centring the input changed only the shape of the bowl. Normalization layers apply a similar rescaling to activations inside a network (04).

6.3 Stochastic gradients and momentum

With a billion examples, one exact gradient per step is far too slow. Stochastic gradient descent (SGD) estimates it from a random minibatch: noisy, much cheaper, and right on average. The noise seems to help a bit, too, by jostling parameters out of shallow irregularities.

Plain SGD treats each step independently, which is what makes the zigzag. Momentum steps along a running sum of past gradients:

With the usual $\beta = 0.9$, velocity along the valley floor, where the gradient keeps its sign, builds toward $1/(1-\beta) = 10$ gradients' worth. Across the valley the sign flips each step and the terms cancel. Momentum is a low-pass filter on the gradient.

6.4 Adam and AdamW

Adam keeps an average of the gradient and of its square, and uses the second to scale each parameter's step:

$m_t$ is the momentum term. $v_t$ tracks how large each parameter's gradient has been lately, so dividing by its root gives large gradients smaller steps. Both averages start at zero, and the hats correct for that early bias.

Take the first step on the delivery model with $\beta_1 = 0.9$, $\beta_2 = 0.999$. After one gradient, $\hat m_1 = 0.1\,g / 0.1 = g$ and $\hat v_1 = g^2$, so the step is $\eta\, g/|g|$: exactly $\eta$ per parameter, whatever the gradient's size. At $\eta = 0.1$, Adam raises both slope and intercept by 0.1. Plain gradient descent would move them by 2.0 and 0.93. Adam has discarded the scale difference between parameters, which is much of why it's forgiving on badly scaled problems.

AdamW changes one thing: standard L2 adds $\lambda\theta$ to the gradient before the adaptive division, which warps the regularization strength per parameter. AdamW subtracts $\eta\lambda\theta$ from the weights directly, and it is the default in most modern recipes.

Why not always use Adam?

Adam tolerates a poorly tuned learning rate, which matters when a model is too big for much hyperparameter search. That's why transformers and LLMs default to it. But well-tuned SGD with momentum often matches or beats it on final accuracy for some vision architectures, and why that happens is still argued about. The practical trade: Adam gets a working model faster; tuned SGD sometimes gets a slightly better one.

6.5 Local minima and saddle points

The delivery loss is convex, one bowl, so going downhill always finds the global minimum. Neural network losses are non-convex, with no such guarantee, but that matters less than it seems. A true local minimum needs every Hessian eigenvalue positive, and across millions of directions that's rare. Saddle points, curving up in some directions and down in others, are far more common.

07Generalization, and what breaks

Everything so far fitted three orders. The fourth is the one that matters, and a model can fit the first three perfectly and still miss it badly.

7.1 A perfect fit is the wrong goal

The best line has a loss of 0.222. The curve $\text{minutes} = \text{km}^2 + 11$ passes through all three orders exactly (12, 15, 20), so its loss is zero.

Ask both about a 10 km order: the line says $4 \times 10 + 7.67 \approx 48$ minutes, the curve says 111. Three points can't say which is right, though the line is probably the safer bet. The curve has enough freedom to fit whatever those points happened to be, noise included, and the next equation calls that freedom variance.

7.2 The bias-variance decomposition

• Bias is error from a model too simple for the pattern, wrong the same way even with infinite data, and it looks like underfitting.
• Variance is error from a model too sensitive to its training sample, so predictions swing when you retrain on fresh data, and it looks like overfitting.
• σ² the noise in the data itself. No model gets below it.

7.3 Regularization is one trade

Weight decay, dropout, early stopping, augmentation and a smaller architecture all make the same bet: a little more bias for a larger cut in variance. So training loss usually gets slightly worse while validation loss improves. An interviewer is usually checking whether you see that trade, and no single technique matters as much.

7.4 Calibration

A model can be accurate and still overconfident, and modern deep nets probably are, straight out of training. If its "90% confident" predictions are right 70% of the time, those scores can't drive a threshold or a hand-off to a human. That's miscalibration, and accuracy can't see it.

Temperature scaling is the cheap fix: divide the logits by one scalar, fitted on held-out data, before the softmax. It doesn't change which class ranks first, only how confident the model sounds. 25 · Evaluation, Reliability & Safety goes further.

7.5 Symptom, cause, fix

Most training failures show up first in the loss curve, and each row traces back to an earlier section.

08Summary

1. Training is one loop. Predict with a matrix product, score with a loss, differentiate with the chain rule, step against the gradient.
2. The loss is one number because "downhill" needs one direction. Weighing two goals is your decision.
3. Gradients are cheap. For squared error it's $\frac{2}{n}X^\top e$; in a network, backprop multiplies local derivatives and reuses the forward pass.
4. A loss is a noise assumption. Squared error is maximum likelihood under Gaussian noise; binary cross-entropy is maximum likelihood for a yes/no label.
5. Cross-entropy keeps its gradient when the model is confidently wrong, about 50 times squared error's at $p = 0.01$.
6. Base rates dominate Bayes. A detector that catches 90% of fraud at a 5% false-alarm rate is right 15% of the time it fires when fraud is 1%.
7. Regularization is a prior. L2 is MAP with a Gaussian prior; L1 is a Laplace prior.
8. Curvature sets the learning rate. Gradient descent diverges above $2/\lambda_{max}$, and the flattest direction sets the pace.
9. Conditioning is often a preprocessing problem. Centring one input took the delivery model from 90 steps to 3.
10. A perfect training fit proves little. Test error is bias, variance and noise, and every regularizer trades the first for the second.

09Build this

Section 05 says KL is asymmetric and that the direction you pick changes what gets penalized, which is easy to nod at and hard to believe. Two plots settle it.

10Interview questions

11Go deeper

●Now write it yourself

Reading the derivation and being able to produce it are different skills. These are Deep-ML problems that exercise what this page covers — each one is checked against real test cases, not multiple choice.

Matched to this page from Deep-ML's catalogue of 1,380 problems. More at deep-ml.com, and Where to practise covers the other platforms and what each one trains.

12Related pages