Reinforcement Learning

01Intuition

Supervised learning tells you the right answer for every example. RL only tells you, much later, whether things went well.

An RL agent is a trial-and-error learner operating under exactly this constraint. It observes a state, picks an action, the environment responds with a reward and a new state, and the loop repeats.

There's no teacher whispering the correct action at every step. The agent has to discover, purely from the pattern of rewards it eventually receives, which actions tend to lead somewhere good. That's a fundamentally different learning problem from anything in supervised learning. It's why RL needs its own vocabulary: states, actions, rewards, policies, value functions.

Once that vocabulary is in place, algorithms like Q-learning and PPO are just different, principled ways of attacking the same credit-assignment problem.

A reinforcement learning agent has to balance trying new things against repeating what has worked. Three slot machines win 20%, 50% and 80% of the time, and the agent doesn't know that.

import numpy as np

rng = np.random.default_rng(1)
true = np.array([0.2, 0.5, 0.8])       # win rate of 3 slot machines

def run(eps, steps=5000):
    est, n, total = np.zeros(3), np.zeros(3), 0
    for _ in range(steps):
        a = rng.integers(3) if rng.random() < eps else est.argmax()
        r = float(rng.random() < true[a])
        n[a] += 1
        est[a] += (r - est[a]) / n[a]  # running average of this machine
        total += r
    return total / steps

for eps in (0.0, 0.1, 1.0):
    print(f"epsilon {eps:.1f}: average reward {run(eps):.3f}")

02Timeline

03The agent-environment loop, and the vocabulary built on it

Every RL problem is framed the same way. An agent sits in an environment. At each time step it observes a state $s$, chooses an action $a$ according to its policy, and the environment returns a reward $r$ and a next state $s'$. That's the entire loop, repeated thousands or millions of times during training.

The MDP formalism

Formally this loop is a Markov decision process (MDP), defined by five pieces:

• $S$, a set of states.
• $A$, a set of actions.
• $P(s'|s,a)$, a transition function giving the probability of landing in state $s'$ after taking action $a$ in state $s$.
• $R(s,a,s')$, a reward function.
• $\gamma \in [0,1)$, a discount factor.

"Markov" means the future depends only on the current state and not on the full history that led there, so the state is assumed to already summarize everything relevant.

The discount factor needs some intuition on top of the symbol. A reward received right now is worth more than the identical reward received ten steps from now, for two separate reasons.

• Uncertainty. The world between now and ten steps from now might change, the episode might end early, you might never collect that later reward. A bird in the hand.
• Convergence. In a task that can run forever (no natural end point), summing an infinite sequence of undiscounted rewards can diverge to infinity, which makes it meaningless to compare one policy against another.

Multiplying reward $k$ steps in the future by $\gamma^k$ fixes both problems at once. It encodes "sooner is better" and it guarantees the infinite sum converges to a finite number as long as $\gamma < 1$, since it becomes a geometric series.

Value functions, Q-functions, and advantage

Once you're discounting future reward, you need a way to ask "how good is this situation, in total, going forward?" That's what a value function is.

$V(s)$ is the expected discounted sum of all future rewards, starting from state $s$, assuming the agent acts optimally (or according to some fixed policy) from here on. It answers "how good is this state to be in?": a single number summarizing the entire rest of the episode.

$Q(s,a)$ answers a more specific question: how good is it to take this particular action $a$ in this state $s$, and then act optimally after that? The difference between $V$ and $Q$ is exactly one decision. $V(s)$ already assumes you'll pick the best action, while $Q(s,a)$ fixes the first action for you and lets the value function take over afterward.

So Q-learning can learn a full control policy just from Q-values: the best action in any state is $\arg\max_a Q(s,a)$.

Advantage combines the two: $A(s,a) = Q(s,a) - V(s)$. It's literally "how much better than average is this specific action, compared to what I'd get by just following my usual policy in this state?" A positive advantage means this action beat the state's baseline expectation; a negative advantage means it underperformed it.

Advantage matters enormously in practice because policy-gradient methods use it (or an estimate of it) to decide how hard to push the policy toward or away from an action. Pushing based on raw reward is noisy; pushing based on "better or worse than expected" is a much cleaner signal, which is exactly the motivation behind actor-critic methods below.

04The Bellman equation, and PPO's clipped objective

Almost every value-based RL algorithm exploits one recursive fact about value functions, called the Bellman equation. That includes dynamic programming, Q-learning, TD learning, and DQN.

• V(s) the value of the current state: the expected discounted sum of all future reward, starting here.
• r the immediate reward received for the transition out of state $s$.
• γ the discount factor: how much less a unit of reward is worth one step in the future, versus right now.
• V(s') the value of wherever the agent ends up next, computed by exactly the same function, applied one step later.
• 𝔼[·] an expectation over the randomness in the transition (which next state you land in) and, for $Q$, over the policy's choice of action.

Dynamic programming applies this equation exactly, sweeping over all states. Q-learning and TD learning apply an approximate, sampled version of it, updating one estimate at a time from lived experience, where no exact model of the environment is available.

The Bellman equation is the backbone of value-based methods. Policy-gradient methods rest on a different result, and it is worth seeing where it comes from, because the whole of PPO is built on top of it.

PPO's clipped surrogate objective, conceptually

Policy gradients want to increase the probability of actions that led to high advantage and decrease the probability of actions that led to low advantage. The naive way to do this computes a ratio between the new policy's probability of an action and the old policy's probability of that same action, and scales the gradient step by that ratio times the advantage.

The problem: nothing stops a single update from moving the policy enormously if the estimated advantage is large. And a policy that has moved too far in one update can collapse. It starts producing actions so different from what generated the training data that the whole learning process destabilizes and never recovers.

• r_t(θ) the probability ratio between the new policy and the old policy for the action taken: how much the policy wants to shift toward or away from this action.
• A_t the estimated advantage of that action. Positive means push toward it, negative means push away.
• clip(·, 1-ε, 1+ε) caps the ratio so it can't move more than $\epsilon$ (typically 0.1–0.2) away from 1 in either direction. The new policy is not allowed to become more than roughly 20% more or less likely to take this action in a single update.
• min(·, ·) takes the more pessimistic of the unclipped and clipped objective, which means clipping only ever removes the incentive to keep pushing once you've already moved far enough. It never creates an incentive to push further.

The whole point in one sentence: take a policy-improving step, but clip it so the new policy can't move too far from the old policy in a single update. It's a cheap, first-order stand-in for what TRPO (PPO's more complicated predecessor) enforced with an explicit, expensive constraint on how different the new and old policies are allowed to be.

05Why bootstrapping wins, and why PPO clips

Two "why" questions come up constantly, and both have a clean answer.

Why does TD/Bellman-style bootstrapping usually beat waiting for the full Monte Carlo return? The two differ in what they use as the learning target.

• Monte Carlo waits for an entire episode to finish, then uses the actual observed total reward. That target is unbiased, since it is the real quantity you're trying to estimate and not an approximation of it. It is also extremely noisy, because it depends on every random action and every random environment transition for the rest of the episode.
• Temporal-difference (TD) doesn't wait and bootstraps instead, using the immediate reward plus its own current estimate of the value of the next state, exactly as the Bellman equation describes. That estimate is biased (it depends on a value function that's still being learned and is currently wrong), but it has far lower variance, because it only depends on one step of randomness instead of an entire trajectory's worth.

In practice, lower variance usually wins. TD methods learn faster and more stably from the same amount of experience. TD also has a structural advantage Monte Carlo cannot offer: it works in continuing tasks that never terminate, since it never needs to wait for an episode to end at all.

Why does PPO's clipping specifically matter? Without it, a single large policy-gradient update (triggered by, say, one unusually high-advantage rollout) can shove the policy into a region of action-space it has never explored well.

Once there, the next batch of rollouts is collected under this new, poorly-understood policy, the advantage estimates for that batch are unreliable, and the next update compounds the damage. This failure mode happens in practice.

An RL run can look like it's training fine and then collapse in a handful of updates with no way back. The very data being used to compute the next gradient step is generated by the policy that just got wrecked.

Clipping prevents the update from moving the probability ratio more than $\epsilon$ away from 1, which caps how far any single step can push the policy.

That gives most of the stability guarantee TRPO achieved with an explicit, expensive trust-region constraint, but computed with a simple min-and-clip on scalar ratios and with no constrained optimization problem. PPO displaced almost everything before it because it offers close to TRPO's stability at a small fraction of the implementation complexity.

06Families of methods, and where each one fails

Actor-critic is best understood as the direct fusion of the other two ideas. Two components, two jobs:

• The actor is the policy, improved by a policy-gradient update. It uses the advantage the way REINFORCE uses raw return.
• The critic is a learned value function, used in place of the raw, noisy Monte Carlo return to judge how good each action was. It supplies a low-variance advantage estimate the way TD learning does.

PPO is, precisely, an actor-critic method with a clipped policy-gradient update for the actor.

Where training breaks

• Sparse reward environments. If the agent almost never receives a nonzero reward (reach the end of a long maze, win a long game), then for a long stretch of training every action looks equally (un)informative, and gradient-based learning has almost nothing to learn from. This is the credit-assignment problem at its worst, and it is why reward shaping, curiosity-driven exploration bonuses, and hierarchical RL exist as whole subfields.
• Reward hacking. The agent optimizes what you told it to reward, which may differ from what you wanted. Think of a boat-racing agent that spins in circles collecting the same power-up over and over instead of finishing the race, because that scores higher under the literal reward function. Any time a reward function is a proxy for the real goal, and it almost always is, the optimizer will find the gap between the proxy and the goal if the gap is exploitable.
• Offline RL's distribution-shift trap. Offline RL means learning a policy from a fixed, previously-collected dataset with no further interaction with the real environment at training time. The danger is that the learned policy can drift toward actions the data-collecting policy rarely or never took. For those actions, the training data gives no reliable signal about what would happen. The value function's estimate there can be badly wrong and badly overoptimistic, with no way to catch the error since there's no environment left to test against.

07Build this

Why DQN needs a replay buffer and a target network

Interviewers often ask why each fix exists, and each one solves a distinct instability.

Training a neural network with plain gradient descent assumes roughly independent, identically distributed training examples. But an agent playing through an environment produces a highly correlated stream. Consecutive frames of the same game look nearly identical. Training on them in that order badly biases the network toward whatever situation the agent happens to be in right now, then violently biases it toward the next situation.

A replay buffer breaks this correlation: store past transitions (state, action, reward, next state) in a large buffer, and train on randomly sampled batches from it and not on the live stream. It also improves sample efficiency, since each transition can be reused in many updates instead of being seen once and discarded.

Separately, Q-learning's update target is itself built from the network's own current Q-value estimate at the next state. You're regressing the network toward a target that's computed by the same network you're currently updating. Every gradient step changes the network, which changes the target, which changes what "correct" means for the very next step. The target keeps moving, so training can oscillate or diverge and may never settle.

DQN fixes this by keeping a separate target network, a lagging copy of the main network's weights that's only updated (or slowly blended in) every so many steps. The target used for each update comes from this frozen copy, so the regression target stays stable for a while, giving the main network something fixed to converge toward before the target itself shifts.

Both of those fixes are repairs to a method that already works in a table, with no network anywhere in it. Print that table after every episode and the Bellman equation stops being an equation.

Where this runs in production

The most common place PPO shows up in a 2026 interview is RLHF. A reward model, trained on human preference comparisons, stands in for the reward function. The language model itself is the policy. Generating a response token-by-token is the sequence of actions.

PPO takes the update step described above (improve the policy, but clip so it doesn't move too far from the previous policy in one step) with a KL penalty added on top, to keep the fine-tuned model from drifting too far from the original supervised-fine-tuned model it started from.

Page 10 covers RLHF's full pipeline in depth: reward modeling, the KL penalty, RLHF vs. DPO. What matters here is that everything RLHF does with PPO applies the MDP, advantage, and clipped-objective machinery on this page directly, with no new machinery at the LLM layer.

08Interview questions

09Go deeper

●Now write it yourself

Reading the derivation and being able to produce it are different skills. These are Deep-ML problems that exercise what this page covers — each one is checked against real test cases, not multiple choice.

Matched to this page from Deep-ML's catalogue of 1,380 problems. More at deep-ml.com, and Where to practise covers the other platforms and what each one trains.

10Related pages

Other techniques for this problem

A scoped slice of the full Technique Map — every technique this page covers, grouped by what it solves.