RAG, Agents & Reasoning

01Intuition

RAG is an open-book exam. An agent runs a loop where a plain model makes a single forward pass.

Agents solve an entirely different problem. A plain LLM call is one forward pass: prompt in, text out, done. An agent wraps that single call in a loop and keeps going until the task is finished or it gives up.

• Think. The model reasons in text about what it should do next.
• Act. It emits a structured call to a tool: search the web, query a database, hit an internal API, run code.
• Observe. The tool's real output goes back into the model's context before the next think step.

The loop is what lets an agent recover from a bad first guess. A plan-then-execute-blindly system commits to a plan up front and just runs it. A ReAct-style agent looks at what happened after each action and corrects course. That's closer to how you'd debug something yourself than to how you'd write out a to-do list and never look up again.

Retrieval-augmented generation starts by finding the documents most relevant to a question. Here the score is the cosine similarity between word-count vectors, the crudest version of the idea.

import numpy as np

docs = ["the refund policy allows returns within 30 days",
        "our office is open monday to friday",
        "shipping takes five business days"]
query = "how many days for returns"

vocab = sorted({w for d in docs + [query] for w in d.split()})
def vec(s): return np.array([s.split().count(w) for w in vocab], float)
def cos(a, b): return a @ b / (np.linalg.norm(a) * np.linalg.norm(b))

scores = [cos(vec(query), vec(d)) for d in docs]
for d, s in zip(docs, scores):
    print(f"{s:.2f}  {d}")
print("retrieved:", docs[int(np.argmax(scores))])

02Timeline

03Architecture: how the data flows

The RAG pipeline, end to end

Before any query arrives, there's an offline ingestion step. Take a corpus of documents, split each one into chunks of a few hundred to a couple thousand tokens, embed each chunk, and store the vectors in an index.

Chunk size is a real tradeoff and no hyperparameter to set once and forget:

• Too small and you lose surrounding context. A sentence about "the deductible" is useless in isolation if no nearby text says which insurance plan it belongs to.
• Too large and you dilute relevance. The embedding for a 3-page chunk averages over everything in it, so a highly specific question matches it only weakly even when the one relevant sentence is buried inside.

Most production systems land on a few hundred tokens per chunk, with some overlap between consecutive chunks. The overlap avoids severing a sentence or an idea across a boundary.

At query time the pipeline below runs. The query is embedded the same way the documents were. A cheap first-pass retrieval pulls back a candidate set, say the top 50–100 by similarity. A reranking step re-scores that much smaller set with a more expensive, more accurate model and keeps the true top few. Only then does the LLM see them and generate an answer conditioned on that retrieved text.

Reference figureThe same pipeline drawn as an end-to-end production flow — ingestion/chunking on one side, query/retrieval/generation on the other. Source: Pinecone, Retrieval-Augmented Generation.

Dense vs. sparse retrieval, and why hybrid usually wins

Dense retrieval embeds the query and every chunk into the same vector space and ranks by cosine similarity. It's good at semantic matching. "How do I get my money back" retrieves a chunk about "refund policy" despite sharing not a single word, because the embedding model learned they mean similar things.

Sparse retrieval (classically BM25, a weighted keyword-overlap scoring function) matches on exact terms. It's good at precisely what dense retrieval is bad at: rare tokens, product codes, error messages, acronyms, and proper nouns that an embedding model tends to blur together with semantically-similar-but-wrong neighbors.

A query containing an exact SKU or an error code like ERR_429 is a case sparse nails and dense can easily miss, because embeddings compress meaning and lose exact-token precision doing it.

Hybrid search runs both and combines the scores, commonly via reciprocal rank fusion or a weighted blend. It tends to beat either alone for the same reason ensembling generally helps: dense and sparse make different kinds of mistakes, so together they cover more ground than either does by itself.

Rerankers: why a second, more expensive pass

The retrieval step above uses a bi-encoder. The query and every document are embedded independently, and similarity is a single cosine computation between two fixed vectors. That's what makes it fast enough for millions of chunks: document embeddings are precomputed once, offline, and at query time you only embed the query and run a nearest-neighbor search.

Independence is also its weakness. The document embedding was computed with no knowledge of what the query would eventually be, so it's necessarily a compressed, query-agnostic summary.

A cross-encoder reranker removes that independence. It takes the query and one candidate document together, as a single joint input, and attends across both to produce one relevance score. That joint attention is why it's more accurate: the model can notice fine-grained interactions between this specific query and this specific document that two separately-computed vectors cannot represent.

The catch is cost: it runs once per query-document pair, so it doesn't scale to the full corpus. A cross-encoder over a million chunks per query is far too slow for interactive use.

Hence the two-stage pattern: cheap bi-encoder retrieval casts a wide net over the whole corpus, and an expensive cross-encoder then reranks only the small candidate set that survived. You get most of the accuracy of exhaustive cross-encoder scoring at a tiny fraction of the cost.

Smarter retrieval: query rewriting, HyDE, and Graph RAG

The raw user question is often a bad search query. Three techniques attack that from different angles.

• Query rewriting / multi-query retrieval. The LLM generates one or several reformulations of the question, retrieves for each, and merges the results. This helps when the user's phrasing is ambiguous, underspecified, or worded very differently from how the answer appears in the source documents.
• HyDE (Hypothetical Document Embeddings). Instead of embedding the question directly, have the LLM generate a hypothetical answer and embed that. A plausible-looking answer is written in the same style and vocabulary as the real answer documents, so it lands closer to them in embedding space than the question itself does. Questions and answers are often phrased so differently that embedding the question undersells how relevant the right document is.
• Graph RAG. Replaces flat chunk retrieval with retrieval over a knowledge graph of entities and relationships extracted from the corpus. Useful when answers depend on multi-hop relationships between entities ("which suppliers does the company that acquired X use?") that a single flat chunk is unlikely to contain, but that a graph traversal can walk directly.

Agent architecture: planner, executor, memory, tools

Strip away the framework-specific naming and almost every agent system has the same four pieces:

• Planner. Decides what to do next, given the current state and the goal.
• Executor. Carries out the chosen action, usually a tool call.
• Memory. Holds state across steps (its own subsection below).
• Tools. The agent's only way to affect or observe anything outside its own text generation: search, code execution, internal APIs, a calculator, another model.

The pattern tying planning and execution together is ReAct (Reason + Act). Instead of planning the whole task up front and executing it blindly, the model alternates step by step.

It produces a short reasoning trace about what it needs and why ("I need the customer's order status, I should check the orders API"), emits an action, receives an observation, and loops. Each new observation folds into the next reasoning step, adjusting or abandoning the original plan when it's contradicted.

Tool use / function calling

Tool use is what makes "Act" in the loop above possible at all. The model is given a set of tool definitions in its context: names, descriptions, and expected arguments, typically as a JSON schema. Instead of only producing free-form prose, it can emit a structured call matching one of those schemas, like get_order_status(order_id="A123").

Multi-agent systems: when they help, when they're overhead

Splitting a task across multiple specialized agents helps when the subtasks decompose cleanly, and each one benefits from a different context window, tool set, or specialization.

A "researcher" agent with web search and a "coder" agent with a code sandbox rarely need to see each other's intermediate scratch work. Keeping their contexts separate stops one agent's noise polluting the other's reasoning. It also helps when subtasks run in parallel and coordination overhead is small next to the work being parallelized.

But multi-agent is not automatically "more sophisticated" or "more capable." Often it's just more moving parts.

A single well-prompted agent with good tools and a clean context frequently outperforms an over-engineered pipeline on tasks that don't decompose cleanly. Every extra agent-to-agent handoff is a place for information to get lost, restated badly, or misinterpreted: the same lossy-summarization problem as a long game of telephone.

Coordination has a real cost too. More agents means more prompts to maintain, more failure surface, and often higher latency and token spend for a task one loop could have handled directly.

The honest interview answer: reach for multiple agents when the decomposition is real — different tools, different context needs, genuine parallelism. Not by default, and not because it looks more advanced.

Agent memory: three tiers

Agent memory comes in three tiers, and interviews like to hear them named separately:

• Short-term memory. The context window itself, including the running ReAct trace of thoughts, actions, and observations for this task. Fast and free to access, but bounded, and gone once the context is cleared.
• Episodic memory. The history of this session or task: what's been tried already, what failed, what the user already said. Usually a running log that gets summarized or pruned as it outgrows the context.
• Semantic memory. Durable knowledge that persists across sessions entirely, such as user preferences, facts learned earlier, and prior decisions. Frequently implemented with the exact same retrieval stack as RAG: embed and store facts, then retrieve the relevant ones into context when needed, rather than keeping everything in the prompt at all times.

Reflection, self-critique, and code-execution agents

Reflection is a useful add-on to the base ReAct loop. After producing an answer or finishing a sub-task, the model (or a separate verifier model) critiques its own output against the retrieved evidence or the stated goal before finalizing it. Some errors get caught before they reach the user and not after.

Code-execution agents take tool use to its most general form. Instead of a fixed menu of hand-written tools, the agent writes and runs arbitrary code in a sandbox. That turns "any computation expressible in code" into an available action, beyond the specific functions someone pre-registered.

Both patterns trade extra latency and cost for a real reduction in a specific class of errors. They refine the core think/act/observe loop and do not replace it.

Where the vectors live

Section 04 reduces retrieval to cosine similarity over an approximate nearest-neighbour index. A vector database is what wraps that index in the things a real system needs: persistence, metadata filtering, incremental updates and deletes. The names you will be asked about differ mainly in how much infrastructure they assume.

• FAISS. A library and not a service: Meta's ANN implementation runs in-process, with no server and no metadata layer. It is the right answer for a single machine and a fixed corpus, and it is what several of the managed options run underneath.
• Chroma. Embedded and developer-facing, closest to "FAISS plus metadata and persistence with almost no setup". Good for prototypes and small production systems.
• Pinecone. Fully managed and hosted. You trade control and per-query cost for not operating an index yourself, which matters most when the corpus changes constantly.
• pgvector. An extension that keeps vectors in Postgres beside the relational data. Frequently the correct choice, and frequently overlooked, because most teams already run Postgres and most corpora are far smaller than they assume.

Orchestration frameworks

Everything on this page so far is a control-flow pattern: retrieve, then generate; or think, act, observe, repeat. LangChain and LangGraph package those patterns so you write less glue.

• LangChain supplies the components and the connectors: document loaders, text splitters, retriever and vector-store adapters, prompt templates, and a uniform interface across model providers. Its value is that swapping an embedding model or a vector store becomes a configuration change.
• LangGraph models an agent as an explicit state graph instead of a chain. Nodes are steps, edges are transitions, and a conditional edge decides where to go next. Because the graph is a real object, you get cycles, checkpointing, resuming a run mid-way, and a place to require human approval before a step executes. A chain abstraction cannot express that.

Use them for what they are. The frameworks are worth it for the connector surface and, in LangGraph's case, for durable multi-step state. They are not worth it as a substitute for understanding the loop, and a retrieval pipeline you cannot debug without the framework's tracing is one you do not yet understand. Build the pipeline in section 07 by hand first; the abstractions make more sense once you know what they are abstracting.

04The one equation

RAG and agents are architectural and systems topics more than equation-heavy ones. The one piece of math worth having cold is the similarity metric almost every dense retriever is built on: cosine similarity between a query embedding and a document embedding.

• q, d the query and document embedding vectors, produced by the same embedding model so they live in the same space and are comparable.
• q · d the dot product — large when the two vectors point in similar directions with large magnitude.
• ∥q∥ ∥d∥ dividing by both norms removes the effect of vector length entirely, leaving only the angle between them — two embeddings pointing the same direction score 1 regardless of how "big" either vector is, which is what you want: relevance shouldn't depend on document length or embedding magnitude.

Retrieval at scale reduces to computing this score between the query and every stored document embedding, then returning the highest-scoring few. In practice that runs on an approximate nearest-neighbor index such as HNSW, since exact search over millions of vectors per query doesn't scale.

05Why this approach

The obvious alternative to RAG is: just fine-tune the model on the new knowledge. For information that changes often (pricing, inventory, this week's internal docs, anything with a "last updated" date) fine-tuning loses on every axis that matters in production.

• Cost. Retraining every time a document changes is expensive and slow. Updating a vector index with a new or edited document is cheap and close to instant.
• Freshness. A fine-tuned model is stale the moment anything changes again, and you're back to retraining. A retrieval index reflects whatever's in it right now.
• Provenance and citability. A fine-tuned model bakes a fact into its weights with no record of where it came from, so it can't show its source or let you verify the answer. A RAG system can quote the exact retrieved passage an answer came from, which matters enormously for trust, debugging, and compliance.

Fine-tuning still has a real role. Teaching a model a new skill, format, or behavior pattern is a different problem from giving it new facts, and RAG doesn't help with the former.

The obvious alternative to ReAct's interleaving is plan-then-execute: have the model write out a full multi-step plan up front, then run every step without looking back. It's faster and simpler when it works.

It's also brittle exactly when it matters most. Real actions produce real, sometimes surprising results: an API call fails, a search comes back empty, a tool returns something the plan didn't anticipate. A system that can't look at that result before deciding its next step will keep executing a plan that's already invalid.

Interleaving reasoning with acting means every step incorporates the newest, truest information available: the actual output of the previous action, and not a stale prediction made before any of it was known.

06Tradeoffs, evaluation, and what breaks

Evaluating RAG and agents

It is harder than evaluating a classifier: "correct" is fuzzy, and several independent things can fail.

• Retrieval: precision@k and recall@k. Precision asks what fraction of the top-k retrieved chunks are relevant. Recall asks what fraction of all the relevant chunks that exist made it into the top-k. A system can have great generation and still fail purely because the right chunk never got retrieved.
• Generation: groundedness. Is the final answer supported by the retrieved text, or did the model produce something plausible-sounding that no passage backs? This matters more than plain fluency, because a fluent but ungrounded answer is the hallucination RAG is meant to prevent.
• Agents: task success rate and tool-call success rate. Did the multi-step task get completed correctly, end to end? Did individual tool calls use valid arguments and get parsed correctly? These are the closest things to a north-star metric here.

Both agent metrics usually need human judgment or a separate LLM-as-judge system. There's rarely a single ground-truth string to compare against, the way there is for a classification label.

Failure modes

• Confidently wrong from real evidence. Retrieval can return a chunk that's topically related but wrong for the specific question, and the model, being fluent and not skeptical, synthesizes it into a confident, well-cited-looking answer anyway. Retrieval doesn't eliminate hallucination; it changes its shape, from "wrong from memory" to "wrong from a misleading source," which is harder to catch because it arrives with the appearance of a citation.
• Chunking loses critical context. A chunk boundary that splits a table from its header, or a caveat from the sentence it qualifies, can make an individually-retrieved chunk actively misleading even though the full source document was correct.
• Loops that never terminate, or take unsafe actions. Without an explicit step budget or a clear stopping condition, a ReAct-style loop can spin: repeatedly re-trying a failing tool call, or oscillating between two plans, burning time and tokens without converging. Worse, an agent with a powerful tool (delete a record, send an email, spend money) and insufficient guardrails can execute an action off a flawed intermediate reasoning step, with real and hard-to-reverse consequences.

07Build this

A working RAG pipeline teaches you less than a broken one. The point of building this is to watch the model answer confidently from a passage you know is wrong.

Where this runs in production

A support agent for an e-commerce company combines both systems directly. When a customer asks about their order, the planner recognizes it needs two different capabilities:

• General policy knowledge. RAG over the internal knowledge base: return policy, shipping timelines.
• Account-specific, real-time state. A tool call to the actual orders API, since no static document contains "where is order #48213 right now."

It retrieves the relevant policy chunk, calls get_order_status(order_id=...) against a live backend, observes the returned status, and composes a final answer citing both the retrieved passage and the live API result.

This is the pattern the "why" section argued for: retrieval for knowledge that's written down and reasonably stable, and a tool call for state that's live and can't be pre-indexed, combined inside one ReAct-style loop and not forced through either mechanism alone.

08Interview questions

09Go deeper

●Now write it yourself

Reading the derivation and being able to produce it are different skills. These are Deep-ML problems that exercise what this page covers — each one is checked against real test cases, not multiple choice.

Matched to this page from Deep-ML's catalogue of 1,380 problems. More at deep-ml.com, and Where to practise covers the other platforms and what each one trains.

10Related pages