Neural Network Fundamentals

01Intuition

Backpropagation answers one question, asked for every weight, at every step of training: how much is this specific weight to blame for the error?

A forward pass is a long chain of simple, differentiable steps. Multiply by a weight matrix, add a bias, squash through a non-linearity, repeat. Feed in an input and every layer transforms it a little, until you get a prediction.

Compare that prediction to the true answer and you get one number: the loss. "You were this wrong."

Computing that number was never the hard part. The hard part is credit assignment. For a weight sitting three layers deep, which direction should it move, and by how much, to make that one number smaller next time?

Brute force is hopeless: you would nudge one weight, rerun the entire forward pass, see what changed, and repeat for every weight, which does not scale to a million parameters.

Concretely, a 2-layer network is $x \to$ linear + activation $\to h \to$ linear + activation $\to \hat y \to$ loss $L$.

To find $\partial L / \partial W^{[1]}$, the gradient for the first layer's weights, you do not analyze layer 1 in isolation. You take the gradient signal that already arrived at the hidden layer, computed a moment ago while working out layer 2, and multiply by one more local derivative.

Gradients flow backward like water through a pipe network, picking up a multiplicative factor at every junction. This is also why they can shrink toward zero or blow up over many junctions. More on that in Tradeoffs.

A single neuron can't compute XOR, the function that is 1 only when exactly one input is 1, but two hidden neurons can. The weights below are hand-set to show that it works, and training would find something equivalent.

import numpy as np

def relu(z): return np.maximum(0, z)

X = np.array([[0, 0], [0, 1], [1, 0], [1, 1]], dtype=float)
W1 = np.array([[1.0, 1.0], [1.0, 1.0]])   # both hidden units add inputs
b1 = np.array([0.0, -1.0])                # the second fires only above 1
W2 = np.array([1.0, -2.0])                # output = h1 - 2 * h2

out = relu(X @ W1 + b1) @ W2
for x, o in zip(X, out):
    print(f"inputs {x.astype(int)} -> {o:.0f}")

02Timeline

03Architecture: forward pass, then backward pass through the same graph

A multilayer perceptron (MLP) is nothing more than this pattern, stacked: linear transformation → non-linear activation, repeated.

The universal approximation theorem (Cybenko, 1989; Hornik, 1991) says a single hidden layer, given enough neurons and a non-linear activation, can approximate any continuous function on a compact input domain to arbitrary precision.

The theorem is an existence proof and gives no training recipe. It says nothing about:

• How many neurons "enough" is. It can be exponentially many.
• Whether gradient descent will ever find those weights.

In practice depth matters, even though the theorem only requires width. A deep network can represent the same function with far fewer total neurons than a shallow one. Each layer composes on features the previous layer already built (edges → textures → parts → objects) where a shallow network would represent the whole function flat in one layer.

Real networks almost never stack raw linear+activation layers indefinitely. Past a modest depth, plain stacking becomes very hard to optimize (see Why, next).

The fix that made 50-, 100-, even 1000-layer networks trainable is the residual block: normalize, transform, and add the result back onto a running residual stream and does not overwrite it. This is the same residual stream that runs through every Transformer block running through every Transformer block (08).

Take shapes and cost concretely. A linear layer maps (batch, in_features) → (batch, out_features) via a weight matrix of shape (in_features, out_features) plus a bias. That is in_features × out_features + out_features parameters, regardless of batch size.

A useful rule of thumb from the scaling-law literature: total training compute is roughly $6 \times N \times D$, where $N$ is parameter count and $D$ is training tokens seen. It breaks down as:

• Forward pass: about $2N$ FLOPs, one multiply and one add per parameter.
• Backward pass: roughly twice the forward pass again, since it propagates gradients with respect to both the weights and the activations.

Two separate memory costs follow from that. Parameters set your floor for weights. Activations, which you must keep around for the backward pass, set a second cost that scales with batch size and depth, and it is often the larger of the two. This is why gradient accumulation and activation checkpointing exist (more in section 07 and in 23).

04The equations

Backprop through a 2-layer MLP

That is the forward pass. The backward pass computes a gradient $\delta$ at each layer's pre-activation, propagates it back through one linear map, and stops:

• δ the "error signal" arriving at a layer's pre-activation — everything the loss cares about that flows through this point, collapsed into one vector.
• f', g' is the local derivative of that layer's own activation function, evaluated at the value it output on the forward pass. This is why the forward pass's intermediate values have to be cached — the backward pass needs them.
• (W^[2])ᵀδ^[2] how the error signal at layer 2 propagates back through layer 2's weights to become "blame" felt by layer 1's output — this is the literal mechanism of credit assignment.
• δ(a)ᵀ an outer product: every weight $W_{ij}$ gets the product of "how wrong was output $i$" and "how active was input $j$" — a weight only gets blamed in proportion to how much it was actually used.

Normalization

BatchNorm computes statistics per feature, across the batch:

LayerNorm uses the same formula, but $\mu$ and $\sigma^2$ range over each example's own features and never touch the batch dimension. RMSNorm goes one step further and drops mean-centering entirely, normalizing only by root-mean-square magnitude:

No batch statistics, no running mean, and in the original formulation no bias term or mean subtraction. Cheaper to compute than LayerNorm, and empirically about as effective. That combination is why it is the default in most modern LLMs.

05Why these specific choices

Why ReLU beat sigmoid/tanh. Sigmoid and tanh squash their input into a bounded range. For large positive or negative inputs the function is nearly flat, so its derivative is close to zero.

Chain a dozen of those together. The gradient reaching early layers is a product of a dozen near-zero numbers. It vanishes, and early layers stop learning almost entirely.

ReLU, $f(z)=\max(0,z)$, has a derivative of exactly 1 for any positive input, so it does not saturate or shrink gradients, and they pass through unchanged wherever the unit is active. This is why it unlocked deeper networks.

It has a cost: for negative inputs ReLU's derivative is exactly 0, so a unit that ends up permanently negative on every training example receives no gradient and dies. Two patches keep ReLU's advantage while fixing that:

• LeakyReLU. A small non-zero slope for negative inputs.
• GELU / SiLU. Smooth, with non-zero gradient almost everywhere.

Why GELU/SiLU in modern transformers. GELU is $x\cdot\Phi(x)$ and SiLU (Swish) is $x\cdot\sigma(x)$. Both multiply the input by a smooth gate: close to 0 for very negative inputs, close to 1 for very positive ones.

Unlike ReLU, the transition is smooth, and the function dips slightly negative just below zero and does not clamp hard at zero. That smoothness gives better-behaved gradients at the scale transformers are trained at, and empirically these activations outperform ReLU in that regime. Hence BERT, GPT-family models, and most modern transformer feed-forward blocks default to GELU or a gated SiLU variant (SwiGLU).

Why residual connections and not just a deeper plain network. He et al. (2015) showed something that looks paradoxical: a plain 56-layer network had higher training error than an 18-layer one. That is worse error on the data it was trained on, and overfitting does not explain it.

A deeper plain network is strictly more expressive, since it could always reproduce the shallow one by setting extra layers to the identity. So this is an optimization failure and not a capacity failure: plain SGD struggles to find that identity mapping across many stacked non-linear layers.

A residual block computes $f(x)+x$ in place of $f(x)$, so the identity mapping becomes the default whenever $f$ contributes little. The network starts at identity and only learns a useful deviation from it. This is what unlocked networks past ~20 layers.

06Normalization and regularization: what breaks

What you normalize over (batch, features, or a group of channels) is not a stylistic detail. It is the answer to a very common interview trap question: why do transformers use LayerNorm/RMSNorm and not BatchNorm?

Failure modes that show up

• Dead ReLUs. A unit whose pre-activation is negative for every input in the training set receives zero gradient forever. A large learning-rate spike or an unlucky initialization can push a unit there, and it never recovers. Symptom: a growing fraction of activations reading exactly zero. Fixes: lower learning rate, He initialization, LeakyReLU/GELU/SiLU in place of plain ReLU.
• Vanishing / exploding gradients in RNNs. An RNN reuses the same weight matrix at every timestep, so the gradient flowing back through $T$ timesteps is roughly that matrix raised to the $T$-th power. Dominant eigenvalue below 1 and the gradient shrinks toward zero over long sequences. Above 1 and it grows exponentially, which can produce a NaN loss within a few hundred steps.
• Gradient clipping handles the exploding side. It caps the global gradient norm at a threshold before the optimizer step, which limits update magnitude without changing its direction. It is cheap, standard (Pascanu et al., 2013), and still routinely used when training any recurrent or otherwise deep, unstable model.
• Exploding logits. Unnormalized final-layer logits that grow very large before the softmax push cross-entropy toward numerical overflow, and make the loss landscape near that point extremely sharp. Handled in practice with the log-sum-exp trick, and in some modern LLMs with an explicit logit soft-cap.

Regularization: dropout, early stopping, weight decay

Normalization stabilises optimisation, while Regularization deliberately makes training harder so the network cannot memorise its way to a low loss. Page 01 frames the tradeoff; here is what the three most common levers do to the network.

• Dropout. During training, zero out each unit independently with probability $p$, then rescale the survivors by $1/(1-p)$ so the expected activation is unchanged. At evaluation time nothing is dropped. The effect is that no unit can rely on any particular other unit being present, which stops co-adaptation, where a group of units only works as a committee and none is individually meaningful.
• Early stopping. Track validation loss every epoch, keep the weights from the best one, and stop once it has not improved for a set number of epochs. It costs nothing and it is the only technique here that needs no hyperparameter tuning beyond the patience value.
• Weight decay. It adds a penalty proportional to the squared magnitude of the weights, which pulls them toward zero unless the data pushes back. This is $L_2$ regularization, and in a Bayesian reading it is a Gaussian prior centred on zero.

A practical ordering is to always use early stopping, since it is free, then add weight decay, because it is one number and it usually helps, and then add dropout when the model still overfits after both, and expect to need less of it than older tutorials suggest: large models trained on large corpora are often regularised adequately by the data itself.

07Build this

Section 05 makes a claim that sounds wrong. A deep plain network can have higher training error than a shallow one, on the same data. The loss curve is more convincing than the sentence.

Where this runs in production

Two of the tradeoffs above compound into a real constraint. A 7B-parameter model's weights, gradients, and optimizer state already consume tens of gigabytes in FP32, before a single activation is stored.

Three levers, each attacking a different part of that:

• Mixed precision. Store most of the forward and backward computation in FP16 or BF16. Half the memory, and much higher throughput on GPU tensor cores. Keep a master copy of the weights in FP32 so small updates are not lost to rounding, and add loss scaling to stop small gradients underflowing to zero. Loss scaling matters more for FP16's narrow exponent range than for BF16's.
• Gradient accumulation. This handles the batch-size side. If the batch you want does not fit, run several smaller micro-batches forward and backward, sum their gradients, and take one optimizer step after enough have accumulated. Mathematically close to training at the larger batch size, paid for with extra passes and not extra memory.
• Gradient clipping. Keeps the mixed-precision, accumulated updates numerically stable.

Together those are the standard recipe that lets a handful of memory-constrained GPUs train a model that would otherwise need far more hardware. The rest of the toolbox (sharding, offloading, activation checkpointing) gets a full treatment in 23.

08Interview questions

09Go deeper

●Now write it yourself

Reading the derivation and being able to produce it are different skills. These are Deep-ML problems that exercise what this page covers — each one is checked against real test cases, not multiple choice.

Matched to this page from Deep-ML's catalogue of 1,380 problems. More at deep-ml.com, and Where to practise covers the other platforms and what each one trains.

10Related pages