Scientific & Structured AI

01Why this is a different game

Look at the diagram before the prose. Every architecture on this page is a choice about how far right to sit.

Vision and language models are trained on billions of examples that cost nothing to collect. Scientific data does not work like that.

One density-functional-theory energy is CPU-hours. One crystal structure is a lab. One protein structure was months of crystallography. The dataset you would like does not exist and cannot be bought.

What you have instead is knowledge about the problem that a web-scale engineer does not have:

• Exact symmetries. Translate a molecule, rotate it, or renumber its atoms. The energy does not change. This holds exactly.
• Conservation laws. Forces are the negative gradient of an energy. Mass and momentum are conserved. A model can be built so these hold by construction.
• Cheap verifiers. A predicted structure can be scored against a held-out experiment. A forecast is scored by tomorrow. A proposed equation can be evaluated.

Inductive bias is the name for assumptions a model carries before it sees any data. Convolution assumes translation matters less than local pattern. Attention assumes any token may matter to any other.

02Invariance and equivariance, precisely

Two words that get swapped casually. The difference decides what your model is allowed to output.

Let $g$ be a transformation from some group: a rotation, a translation, a relabelling of atoms. A function $f$ is invariant if the transformation makes no difference at all, and equivariant if the output transforms the same way the input did.

Energy is invariant. Force is equivariant. Rotate the molecule and the force vector rotates with it.

The groups that matter here are small and concrete:

• E(3): rotations, translations and reflections of 3D space. SE(3) is the same without reflections, which matters for chirality.
• Permutation of identical atoms or of graph nodes. Renumbering the atoms cannot change the energy.
• Gauge and scale symmetries in physics problems, which is why a PDE residual holds at every point rather than only where you sampled.

The cheapest route to invariance is to throw the pose away. Feed the network pairwise distances instead of coordinates and rotation becomes structurally impossible to detect. Here is that, in nine lines.

import numpy as np
rng = np.random.default_rng(0)

X = rng.normal(0, 1, (5, 3))            # a 5-atom point cloud
c, s = np.cos(0.7), np.sin(0.7)         # 0.7 rad about z, ~40 degrees
R = np.array([[c, -s, 0.], [s, c, 0.], [0., 0., 1.]])
Xr = X @ R.T                            # the same molecule, turned

W1 = rng.normal(0, .4, (15, 8))         # 15 = 5 atoms x 3 coordinates
V1 = rng.normal(0, .4, (10, 8))         # 10 = the pairwise distances
W2 = rng.normal(0, .4, 8)               # shared output layer

def coord_mlp(P):                       # reads raw xyz, the naive way
    return np.tanh(P.reshape(-1) @ W1) @ W2

def dist_mlp(P):                        # reads |ri - rj| instead
    D = np.linalg.norm(P[:, None, :] - P[None, :, :], axis=-1)
    return np.tanh(D[np.triu_indices(5, 1)] @ V1) @ W2

print("coords", tuple(X.shape), " distances", (10,))
print("coord_mlp  %.4f -> %.4f" % (coord_mlp(X), coord_mlp(Xr)))
print("dist_mlp   %.4f -> %.4f" % (dist_mlp(X), dist_mlp(Xr)))
print("dist_mlp shifted by %.2e" % abs(dist_mlp(X) - dist_mlp(Xr)))

That trick has a price. Distances are scalars, so a distance network can only ever produce scalars. Ask it for a force vector and it has no way to point.

Equivariant networks solve that by carrying geometric objects through the layers rather than stripping them. Features are typed by how they transform: scalars, vectors, and higher-order tensors, written as irreducible representations of the rotation group.

Tensor Field Networks introduced the construction for 3D point clouds. Layers combine features with spherical harmonics through tensor products, and the type arithmetic guarantees the output rotates correctly. The e3nn library is the standard implementation.

03What the symmetry buys, measured

The argument is quantitative, so it should be measured rather than asserted. Two seconds of CPU is enough.

Fix a target that depends only on geometry. Train three models on the same 32 molecules: one on raw coordinates, one on raw coordinates with rotation augmentation, one on pairwise distances. Test on fresh molecules in arbitrary orientations.

import torch, torch.nn as nn
torch.manual_seed(0)

def rot(P):                            # a random 3D rotation per sample
    Q, _ = torch.linalg.qr(torch.randn(len(P), 3, 3))
    return P @ (Q * torch.sign(torch.det(Q))[:, None, None]).mT

D    = lambda P: torch.cdist(P, P)                 # (n, 4, 4)
dist = lambda P: D(P).flatten(1)                   # (n, 16), invariant
flat = lambda P: P.reshape(len(P), -1)             # (n, 12), not
y_of = lambda P: torch.exp(-D(P)).triu(1).sum((1, 2))[:, None]
Ptr = torch.randn(32, 4, 3)            # only 32 training molecules
Pte = rot(torch.randn(600, 4, 3))      # fresh ones, arbitrary poses
ytr, yte = y_of(Ptr), y_of(Pte)

def fit(feat, aug):
    net = nn.Sequential(nn.Linear(feat(Ptr).shape[1], 64), nn.Tanh(),
                        nn.Linear(64, 1))
    opt = torch.optim.Adam(net.parameters(), lr=0.02)
    for _ in range(1500):
        loss = ((net(feat(rot(Ptr) if aug else Ptr)) - ytr) ** 2).mean()
        opt.zero_grad(); loss.backward(); opt.step()
    return ((net(feat(Pte)) - yte) ** 2).mean().sqrt().item()

print("train", tuple(Ptr.shape), "| std(y) %.3f" % yte.std())
print("coords        test RMSE %.3f" % fit(flat, False))
print("coords + aug  test RMSE %.3f" % fit(flat, True))
print("distances     test RMSE %.3f" % fit(dist, False))

That gap is the field's central selling point, and it shows up in real systems. The NequIP paper reports matching or beating existing interatomic potentials with up to three orders of magnitude fewer training data.

Note also what augmentation did in the run above. It helped a lot and it did not close the gap, at a fixed number of steps.

04Molecules: graphs, point clouds and forces

A molecule is a graph with 3D coordinates attached. Which of those two things you use decides what you can predict.

The 2D route treats the molecule as a graph of atoms and bonds and runs message passing over it. Page 18 covers that machinery. It is permutation-equivariant by construction and it never sees geometry.

For many property-prediction tasks that is enough, and the baseline to beat is older than any of it. Extended-connectivity fingerprints hash local substructures into a bit vector, feed it to gradient boosting, and win more often than the literature admits.

The 3D route uses coordinates, and the architectures form a clean ladder:

• SchNet uses continuous-filter convolutions over interatomic distances. Invariant, scalars only.
• DimeNet adds bond angles, because distances alone cannot distinguish some geometries.
• EGNN keeps coordinates as a feature that is updated by relative-position vectors, giving E(n) equivariance with almost no machinery.
• NequIP and MACE go fully equivariant with tensor features. MACE raises the body order of each message, so one layer captures many-body interactions.

The largest application is the machine-learned interatomic potential. Density functional theory gives accurate energies at a cost that makes long molecular dynamics impossible. A learned potential fits DFT energies and forces, then runs simulations orders of magnitude faster.

Where this goes wrong is extrapolation. A potential fitted near equilibrium meets configurations no training point resembled, and molecular dynamics will find them.

A low force error on a held-out benchmark is not evidence that a 100-nanosecond trajectory stays stable. Those are different claims, and only the second one matters for the simulation you wanted.

05Protein structure prediction, mechanism first

Page 37 covers what AlphaFold changed and what it did not. This section is about how it works.

A protein is a chain of amino acids that folds into a shape, and the shape determines function. Predicting that shape from sequence was open for fifty years.

AlphaFold 2 combines several priors that are all worth naming separately:

• Coevolution. The input is not one sequence but a multiple sequence alignment of evolutionary relatives. Residue pairs that mutate together are probably in contact. Evolution did the labelling.
• Two coupled representations. The Evoformer maintains an MSA representation and a residue-pair representation, and passes information between them repeatedly.
• Geometric consistency. Triangle multiplicative updates constrain edge $ij$ using edges $ik$ and $jk$. Distances must obey the triangle inequality, so the update enforces something true rather than merely learned.
• Equivariant structure. Each residue gets a local frame. Invariant Point Attention reasons about points in those frames, so the module is equivariant to global rotation and translation.
• Recycling. The whole network is run again on its own output, three times, which is cheap depth.

It is trained end to end against a frame-aligned point error, and it emits a per-residue confidence, pLDDT, alongside the coordinates.

Protein language models take a different bet. ESM-2 is a masked language model trained on sequences alone, and ESMFold predicts structure from a single sequence with no alignment. It is much faster and generally less accurate on hard targets, which tells you how much the MSA was contributing.

AlphaFold 3 replaced the structure module with a diffusion module over raw atom coordinates, dropped the explicit equivariance, and extended coverage to ligands, nucleic acids and complexes.

06Designing molecules and proteins

Prediction scores a structure somebody gave you. Design produces one. The symmetry argument returns, in a sharper form.

Page 12 has the diffusion machinery. The scientific version changes one thing: the denoiser is equivariant.

The constraint has a purpose: if the noise is isotropic and the denoising network is equivariant, the learned distribution over structures is invariant to rotation. The model never spends capacity on orientation, because every orientation is already the same sample. Equivariant Diffusion for Molecule Generation in 3D is the reference construction.

Protein design runs as a pipeline rather than one model:

1. RFdiffusion generates a backbone, optionally conditioned on a binding site or a motif to scaffold.
2. ProteinMPNN solves the inverse problem: given that backbone, which amino-acid sequence folds into it?
3. AlphaFold is then run as an in-silico filter. Does the designed sequence fold back to the requested backbone?
4. The wet lab expresses the survivors and measures whether they do anything.

Step three is the interesting one. It is a cheap verifier bolted onto a generator, which is the same pattern page 37 traces across every field that has worked.

The honest caveat is that the wet-lab hit rate is the only number that counts, it varies enormously by target class, and a generator's reported validity rate tells you nothing about it.

07Physics-informed neural networks

If you know the governing equation, you can put it in the loss. It is elegant and useful, and frequently the wrong tool.

Represent the solution as a network $u_\theta(t, x)$. Because the network is differentiable, you can evaluate the differential operator on it with autodiff, exactly and at any point. The residual of the PDE becomes a loss term.

Three terms: fit the observations, respect the boundary condition $\mathcal{B}$, and drive the PDE residual $\mathcal{N}[u_\theta]$ to zero. The residual is evaluated at collocation points you choose, not at data points. There is no mesh.

The payoff is that physics constrains the solution where no data exists. That is easiest to see by removing the data and watching what happens.

import torch, torch.nn as nn
torch.manual_seed(0)

net = lambda: nn.Sequential(nn.Linear(1, 32), nn.Tanh(),
                            nn.Linear(32, 32), nn.Tanh(),
                            nn.Linear(32, 1))

td = torch.linspace(0, 2, 12)[:, None]     # data exists only on [0, 2]
ud = torch.sin(td)                         # truth: u' = cos t, u(0) = 0
tc = torch.linspace(0, 8, 200)[:, None].requires_grad_(True)
tt = torch.linspace(2, 8, 61)[:, None]     # scored where no data exists

def train(physics):
    f = net()
    opt = torch.optim.Adam(f.parameters(), lr=0.01)
    for _ in range(4000):
        loss = ((f(td) - ud) ** 2).mean()
        if physics:                        # residual of the ODE itself
            u = f(tc)
            du, = torch.autograd.grad(u, tc, torch.ones_like(u),
                                      create_graph=True)
            loss = loss + ((du - torch.cos(tc)) ** 2).mean()
        opt.zero_grad(); loss.backward(); opt.step()
    return (f(tt) - torch.sin(tt)).abs().max().item()

print("data", tuple(td.shape), "collocation", tuple(tc.shape))
print("data loss only   max |error| on [2,8]: %.4f" % train(False))
print("+ physics residual                   : %.4f" % train(True))

That demo is also a fair warning, because the ODE there is trivial. Real problems behave worse, and the failure modes are well documented:

• Competing loss terms. Data, boundary and residual terms have different scales and different gradients. The weights $\lambda$ are hyperparameters, and a badly set one lets the network satisfy the residual while ignoring the boundary.
• Trivial minimisers. Many residuals are minimised by $u = 0$. If the data and boundary terms are weak, that is where the optimiser goes.
• Spectral bias. Networks learn low frequencies first, so stiff and multiscale problems converge slowly or not at all.
• Non-convex optimisation replacing a linear solve. For a problem a finite-element solver handles, you have swapped an exact method for gradient descent.

08Neural operators: learning the solver, not the solution

A PINN is trained for one problem instance. Change the initial condition and you retrain. Operators fix exactly that.

A neural operator learns a mapping between function spaces. Give it an initial condition or a coefficient field as a function, and it returns the solution as a function. Training costs a lot of solved instances. Inference is a forward pass.

Two designs dominate:

• Fourier Neural Operator. Each layer transforms to Fourier space, multiplies a learned complex weight per mode, truncates the high modes, and transforms back. The FFT makes it a global convolution, so information crosses the domain in one layer.
• DeepONet. A branch network encodes the input function sampled at fixed sensors. A trunk network encodes the query location. The output is their inner product, a structure that follows a universal approximation theorem for operators.

The property worth understanding is discretisation invariance. Because the FNO's weights live on Fourier modes rather than grid cells, the same trained model can be evaluated on a finer grid than it was trained on. It learned an operator, not an array.

The same idea at scale is what weather models are. GraphCast and GenCast learn the map from one atmospheric state to the next, and page 37 has the deployment evidence and the skill numbers.

The costs are real. You need a large corpus of solved instances, which means running the classical solver many times first. The surrogate has no error bar, and it degrades silently off distribution, which is the regime you most wanted it for.

09Symbolic regression, and outputs you can check

Everything above outputs numbers. This outputs an equation, which a scientist can read and an algebra system can verify.

Symbolic regression searches the space of mathematical expressions for one that fits the data. The search space is discrete, combinatorial and enormous. Three approaches to it:

• Genetic programming. Mutate and recombine expression trees, keep what fits. PySR is the practical modern implementation.
• Physics-motivated decomposition. AI Feynman tests for properties that shrink the problem, including dimensional analysis, symmetry and separability, then recurses on the smaller pieces.
• Sparse regression. SINDy fixes a library of candidate terms and selects a sparse combination. This turns search into a regression with an $\ell_1$ penalty, which is dramatically more tractable.

The output is not one equation but a Pareto front trading accuracy against complexity. You pick the point. That choice is a scientific judgement and the method does not make it for you.

Be clear about the limits. Symbolic regression scales badly past a handful of variables, and it finds an expression that fits your data which may not be a law of nature.

The same shape, in two other fields

Look at what symbolic regression, theorem proving and program synthesis have in common. A vast discrete search space, and a checker that costs almost nothing: evaluate the expression, compile the Lean proof, run the tests.

That combination is where current AI is strongest. It is the argument page 36 makes for mathematics and page 37 makes for industry. Where the checker is a wet-lab experiment instead, throughput on candidates stops being the bottleneck.

10The loop around the model

In science the model rarely gets a fixed dataset. It gets to ask for the next experiment, which changes what a good model is.

Bayesian optimisation is the standard machinery for optimising something expensive. Fit a probabilistic surrogate, usually a Gaussian process, then choose the next point by maximising an acquisition function over it.

• Expected improvement scores how much a point is likely to beat the current best.
• Upper confidence bound adds a tunable multiple of the predicted standard deviation to the mean.
• Thompson sampling draws one function from the posterior and takes its argmax, so uncertainty drives exploration directly.

Active learning is the same idea aimed at the model rather than the objective. Label the points the model is least sure about. For neural potentials the usual uncertainty estimate is the disagreement across a small ensemble.

Inverse design inverts the question. You know the property you want and need a structure that has it. Either search a generative model's latent space against a property predictor, or condition the generator on the property directly.

11Build this

Section 03 showed the gap at one training-set size. The interesting object is the whole curve, and it takes an afternoon on a laptop.

12What breaks

Almost none of these are modelling failures. They are evaluation failures that a modelling story got attached to.

• The random split leaked. Molecular datasets are full of near-duplicate scaffolds, so a random split puts relatives on both sides. Re-split by scaffold or by date and the headline number drops, sometimes by half.
• The potential is accurate and the simulation explodes. Force error on the held-out set looks excellent. Molecular dynamics wanders into configurations the training set never contained and the trajectory diverges.
• Symmetry was built in, then broken by preprocessing. A canonical orientation, an unsorted feature vector, or a global coordinate frame quietly reintroduces pose dependence that the architecture had eliminated.
• The PINN found a trivial solution. The residual is satisfied, the boundary term had a weight three orders of magnitude too small, and the network is happily returning something close to zero.
• The neural operator was only ever tested at training resolution. The discretisation-invariance claim is the selling point and is the thing nobody checked.
• The generated molecules cannot be made. Validity, novelty and synthesisability are three different properties. Only the first is usually in the loss.
• A confidence score is treated as a probability. pLDDT, ensemble variance and softmax outputs all rank well and none of them are calibrated out of the box.
• The result is a list with no hit rate. A model proposed 380,000 candidate crystals. The number that matters is what fraction survived contact with a chemist, and page 37 has that story in full.

13Where you meet this in the wild

Same inductive-bias argument, four very different verifiers.

14Interview questions

15Go deeper

Primary sources for each mechanism on this page, including the papers that report the limitations.

16Related pages

Other techniques for this problem

A scoped slice of the full Technique Map — every technique this page covers, grouped by what it solves.