3D, Spatial AI & Autonomous Driving

01Intuition

No single sensor tells you what's out there, so the whole field is about combining several incomplete answers into one you can trust.

A camera image is a 2D projection of the world. By the time light hits the sensor, depth is already gone. Two objects at wildly different distances can land on the exact same pixel if one sits behind the other, and a monocular camera alone cannot tell you which is which. It needs additional assumptions, or a second view to triangulate against.

The other two sensors each fix part of that, and break somewhere else:

• LiDAR fires laser pulses and times their return, giving precise, direct 3D range measurements. But a rotating unit only samples a sparse, uneven point cloud: dense up close, sparse at range, and silent about color or fine texture. It is also the most expensive sensor in the stack.
• Radar measures velocity directly and instantly via Doppler shift, which neither cameras nor LiDAR do without inferring it across multiple frames. Its spatial resolution is coarse enough that it often cannot tell you an object's exact shape, or sometimes whether two nearby objects are one object or two.

A camera turns a point in 3D into a pixel by dividing by its distance. That one division is why far things look small and why a single camera can't tell you how far away something is.

f = 800.0                                    # focal length in pixels
cx, cy = 320.0, 240.0                        # image center

def project(X, Y, Z):
    return f * X / Z + cx, f * Y / Z + cy    # x = f X / Z

for Z in (5, 10, 20):
    u, v = project(1.0, 0.5, Z)
    print(f"{Z:>2} m away -> pixel ({u:.0f}, {v:.0f})")

02Timeline

03The stack, piece by piece

A modern autonomous driving stack is a pipeline, much like the robotics pipeline in 21, but with 3D geometry doing more of the early work. Raw sensors feed geometric and learned perception. Perception output gets fused into one shared representation. That representation feeds prediction and planning. Planning's output becomes control commands.

Camera geometry: intrinsics, extrinsics, and why multi-camera needs both

Two transforms, doing two different jobs:

• The intrinsic matrix $K$ describes properties fixed by the lens and sensor: focal length, and the principal point where the optical axis hits the sensor. It maps a 3D point already expressed in the camera's own frame down to a 2D pixel.
• The extrinsic transform, a rotation $R$ and translation $t$, describes where that camera physically sits and points relative to the vehicle. It converts a point from world coordinates into that camera's frame, before the intrinsics project it to a pixel.

A single camera only ever needs its own intrinsics and extrinsics. A multi-camera rig needs every camera's extrinsics relative to one shared reference frame, typically the vehicle's own center. That shared frame is what lets you take features from six differently-mounted cameras and place them correctly relative to each other, instead of holding six disconnected pixel grids.

Epipolar geometry, stereo, and depth

Given two cameras with a known, fixed offset, the same real-world point appears in both images. Epipolar geometry says something powerful about where. Once you know the two cameras' relative pose, a point's location in the first image constrains its possible location in the second to a single line, the epipolar line, rather than the whole 2D image.

That collapses stereo matching from a 2D search into a 1D one. Which is the difference between "compute this on every frame in real time" and "don't."

Once a correspondence is found, depth falls out of the geometry directly. Farther points shift less between the two images (smaller disparity), and nearer points shift more. For a fixed baseline, disparity is inversely proportional to depth.

SLAM and visual-inertial odometry

Simultaneous localization and mapping is a genuine chicken-and-egg problem. To know where you are, it helps to already have a map. To build a map, it helps to already know where you are. SLAM solves both at once, jointly estimating the camera's trajectory and a map of tracked landmarks, refining both as new observations arrive.

Visual-inertial SLAM tightly couples camera tracking with an IMU's acceleration and angular velocity readings. ORB-SLAM3 is the well-known open-source reference implementation. The two sensors cover for each other:

• The IMU fills in fast, high-frequency motion between camera frames, and stays useful through the camera's worst moments: motion blur, brief occlusion, a tunnel.
• The camera corrects the IMU's slow drift, whenever it can see stable landmarks again.

3D object detection: camera-based and LiDAR-based

LiDAR-based detection has to solve an awkward data-structure problem first. A point cloud is sparse, unordered, and wildly uneven in density: dense close to the sensor, sparse far away. A standard CNN, built for dense regular pixel grids, cannot consume that directly.

PointPillars answers it in three moves:

• Discretize the ground plane into a grid of vertical columns, the "pillars."
• Run a small shared network over the points inside each pillar, producing one feature vector per pillar.
• Arrange those vectors into a 2D pseudo-image that an entirely ordinary 2D CNN detector can process.

That turns an irregular 3D problem into a regular 2D one, with no 3D convolutions at all. Which is exactly why it runs fast enough for a moving vehicle.

Camera-based 3D detection has the harder job: estimating depth that was never directly measured. It can do that per-camera, via monocular depth estimation, which is generally less accurate. Or jointly across camera views, the way the BEV methods below do. That accuracy gap is a large part of why multi-camera BEV approaches displaced monocular 3D detection as the camera-only default.

BEV representations: one shared top-down frame

Bird's-eye-view is a top-down 2D grid over the ground plane around the vehicle. It is the natural shared coordinate frame for driving specifically, because planning a route and reasoning about lanes is inherently a top-down problem. Nobody plans a maneuver by thinking in raw camera-pixel coordinates.

BEVFormer builds this grid directly with attention. A fixed grid of learned BEV queries, one per top-down cell, uses two attention mechanisms:

• Spatial cross-attention pulls in the relevant image features from whichever camera views see that patch of ground.
• Temporal self-attention fuses the current BEV grid with the previous timestep's grid.

That temporal half matters more than it first looks. It noticeably helps estimate a moving object's velocity, and recovers objects briefly occluded a frame ago, because the model is not restarting perception from zero every single frame.

Occupancy networks: past the bounding box

A bounding box assumes the world is made of a fixed, known set of object categories with roughly box-shaped extents. That is a bad assumption for a fallen ladder, a spilled load of cargo, or any obstacle shape a detector's training set never named.

Occupancy networks sidestep it: for every cell in a grid around the vehicle they predict whether it is occupied and, if so, by roughly what kind of thing. That turns "list the objects" into "describe free space directly", which degrades far more gracefully on anything unusual.

TPVFormer's trick for making this tractable: represent the volume as three orthogonal 2D planes, the top-down view plus the two perpendicular side views, instead of one dense 3D voxel grid. Combining the three planes recovers the full 3D structure at meaningfully lower cost than predicting every voxel directly.

NeRF, Gaussian splatting, and scene reconstruction

06 covers how NeRF and Gaussian splatting work as general 3D scene representations. The driving-specific use is reconstruction for simulation.

Take a real recorded driving log. Reconstruct the actual scene the car drove through as a NeRF or a set of Gaussians. Then re-render it from camera viewpoints the car never took, or with other actors inserted or removed. That gets much closer to testing against reality than a purely synthetic simulator, because the environment was reconstructed from a real drive and not hand-modeled.

Scene graphs and spatial reasoning

A scene graph represents the world as nodes (vehicles, pedestrians, lanes, traffic signals) connected by labeled relational edges: "in front of," "yielding to," "occupies this lane." That is richer than a flat, unstructured list of detections, and it makes higher-level spatial queries answerable directly. "Which vehicles are relevant to my current lane change" becomes a graph query over relationships, rather than a re-derivation from raw geometry every time it is asked.

Sensor fusion: camera, LiDAR, and radar together

Fusion can happen at three points in the pipeline, and where you put it is the whole design decision:

• Early. Combine near-raw sensor data before much processing. Hardest to get right, but no information is thrown away before combination.
• Late. Run separate full detectors per sensor and combine only their final outputs. Simplest to build and debug, but each detector may have already discarded a cue only visible when signals are considered jointly.
• In between. Fuse intermediate features. BEVFormer-style camera fusion, extended to also ingest LiDAR or radar features into the same shared grid, is the currently favored middle ground.

Trajectory prediction and motion planning

Trajectory prediction takes each tracked agent's recent motion history plus map context (lanes, right-of-way, traffic controls) and forecasts several plausible future paths. Several paths and not one, because "what will that pedestrian do" is inherently multimodal. They might cross or they might not, and a model that averages both possibilities into a single path is wrong in exactly the way 21's behavioral-cloning discussion describes.

MotionNet illustrates one clean way to structure this. It takes BEV maps as input and predicts, per grid cell, both a category and a short-horizon motion vector. Detection and near-term prediction live inside one shared BEV representation instead of two disconnected models.

Motion planning then takes the ego vehicle's goal plus every predicted agent trajectory, and searches for a path that is:

• Safe. Does not intersect anyone else's likely future path.
• Comfortable. Bounded acceleration and jerk.
• Legal. Obeys lane and traffic-control constraints.

Conceptually this is the same constrained trajectory-optimization idea as MPC in 21. The difference is scope: one long route rather than one short re-planned step, re-run continuously as new predictions arrive.

End-to-end autonomous driving, and world models for simulation

End-to-end driving trains one network to map sensor inputs directly to a planned trajectory, or even to control outputs. It optimizes jointly against one end-task loss instead of stage-by-stage intermediate losses.

In principle that lets it discover something the modular pipeline structurally cannot see: that a slightly different detection threshold would have produced a much better downstream plan. Section 05 covers the tradeoffs, and why the industry has not broadly converged on this for safety-critical deployment.

World models (22) increasingly get used here as fast, learned simulators. Instead of only testing planning against a fixed replay log or a hand-built physics simulator, you roll candidate plans forward inside a learned model of how the scene evolves. Cheaply, and safely, before anything touches the real world.

Closed-loop evaluation for driving

The same open-loop-versus-closed-loop distinction from 21 applies here, with higher stakes.

• Open-loop evaluation replays a fixed recorded log and checks whether the planner's outputs match what happened. It is cheap, fast and fully reproducible, and also blind to compounding error, because the ego vehicle's hypothetical actions never change what the scenario does next. A bad early decision never gets the chance to cascade.
• Closed-loop evaluation lets the planner's decisions affect the simulated scenario going forward. Increasingly that simulator is a NeRF or Gaussian-splatting reconstruction of a real log, or a learned world model, rather than something hand-authored.

Closed-loop is the only way to catch the failures that appear after the car has already made one small mistake.

04The math

Two equations carry almost all of the geometry on this page: the camera projection equation, and the epipolar constraint.

• X, Y, Z a point's coordinates in the world (or vehicle) frame.
• R, t the camera's extrinsics: a rotation and translation converting that world-frame point into the camera's own coordinate frame.
• K the camera's intrinsic matrix (focal length and principal point), projecting the camera-frame 3D point down onto the 2D image plane.
• u, v the resulting pixel coordinates, up to the scale factor $\lambda$. That scale factor is the point's depth, and the projection alone throws it away. Hence depth estimation being a separate, hard problem.

The epipolar constraint, for two calibrated cameras viewing the same point:

• x, x' the same real-world point's pixel coordinates in the first and second camera respectively.
• F the fundamental matrix, encoding the two cameras' relative geometry: their relative $R, t$ plus intrinsics.
• Given $x$ and $F$, this equation defines a single line in the second image, the epipolar line, that $x'$ is guaranteed to lie on. This is why stereo matching needs only a 1D search along a line and not a full 2D image search.

05Why BEV/occupancy, and why modular over end-to-end

Fusing raw per-camera detections after the fact is the obvious first design. It is also a worse one. Each camera's detector has already thrown information away by the time it commits to a final box. So if two cameras jointly saw a partially-occluded object, but neither alone saw enough to be confident, late fusion never gets to combine their partial evidence. It was discarded before fusion ran.

Lifting features into a shared BEV grid before detecting anything keeps that partial evidence alive long enough to combine. This is why BEVFormer-style architectures displaced "detect per camera, then merge boxes" as the default.

Occupancy networks push the same logic further. A bounding-box vocabulary is a closed set of categories decided at training time, and the real world reliably produces obstacles outside that set. Predicting occupancy directly, instead of committing to "this is definitely a car, definitely this size," degrades gracefully on exactly the long-tail shapes a fixed category list cannot name.

The modular-versus-end-to-end question does not have a settled answer. Treating it as settled is a tell that someone has not thought about the actual tradeoff.

A modular pipeline keeps perception, prediction and planning as separate stages with separate losses. Every stage is independently testable. If the car braked unnecessarily, you can inspect whether perception mis-detected something, prediction mis-forecast a trajectory, or planning made a bad call given otherwise-correct inputs. Each stage validates against its own held-out data.

An end-to-end network optimized on one final task loss can in principle find a jointly better solution across all three at once: a slightly different detection behavior that happens to produce much better downstream plans. But that joint optimum arrives with no clean way to say why the network did what it did. Which is precisely the property regulators, safety cases and post-incident investigations need.

06Complexity, failure modes, and what breaks

What shows up in production

• Sensor miscalibration. Extrinsics drift out of alignment over time, from a bump, a temperature change, a minor collision. Every downstream fusion step then silently trusts a wrong camera-to-vehicle transform, producing subtly wrong 3D positions that look entirely plausible.
• Long-tail object shapes that a bounding-box detector's training categories never covered. Occupancy networks are built to catch these, and pure bounding-box pipelines fail on them quietly instead of loudly.
• Radar/camera disagreement in edge cases. A large stationary metal object can produce a weak or ambiguous radar return. If the fusion logic over-trusts radar's usually-reliable velocity signal, a real stationary hazard can be under-weighted.
• Compounding error across stages. Prediction errors feed straight into planning as if they were ground truth. An overconfident, wrong trajectory forecast for another vehicle produces a planning decision that is optimal for the wrong predicted future.

07Build this

The projection equation in section 04 throws depth away in a single line. Recovering it from two photographs, with your own code, is what makes the rest of this page feel mechanical instead of magical.

Where this runs in production

A representative modern stack runs five stages:

• Sense. Six to eight cameras around the vehicle, sometimes paired with LiDAR.
• Fuse. A BEV transformer lifts every camera's features into one shared top-down grid, fusing across cameras and across the last few timesteps in the same step.
• Detect and track. Running on the fused BEV grid, producing tracked objects with position, velocity and a class.
• Predict. A trajectory predictor takes those tracks plus map context and forecasts several plausible short-horizon futures per agent.
• Plan and execute. A planner searches candidate ego trajectories against all those predictions plus map and traffic-rule constraints. The winning trajectory goes to a classical trajectory-tracking controller (21).

Public benchmarks like nuScenes and the Waymo Open Dataset exist so different groups' perception and prediction stacks can be compared against the same real, sensor-synchronized driving logs, rather than each group's own private data.

08Interview questions

09Go deeper

●Now write it yourself

Reading the derivation and being able to produce it are different skills. These are Deep-ML problems that exercise what this page covers — each one is checked against real test cases, not multiple choice.

Matched to this page from Deep-ML's catalogue of 1,380 problems. More at deep-ml.com, and Where to practise covers the other platforms and what each one trains.

10Related pages