Time Series & Forecasting

01The constraint that makes this a different subject

Look at the diagram before reading anything else. The rest of this page is consequences of the gap between the two halves of it.

In ordinary supervised learning the rows are exchangeable. A house in row 900 tells you nothing special about the house in row 901, so shuffling before splitting is free.

Time series rows are not exchangeable. Row $t$ and row $t+1$ are the same system one step apart, and they are usually almost equal. Random k-fold cross-validation puts one in train and the other in test.

Three separate leaks come out of that, and they compound:

• Temporal leakage. Training rows are dated after test rows. The model has seen the level, the regime and the shocks it is being asked to predict.
• Leakage through smoothness. Even with no future rows in train, a neighbouring row carries nearly the same target. Any model that averages nearby points gets the answer handed to it.
• Preprocessing leakage. A scaler, an imputer or a target encoder fitted on the whole series before splitting. The test statistics are baked into the transform.

None of this is hypothetical. Here is the gap on a series with a mild trend.

import numpy as np
rng = np.random.default_rng(0)

n = 300                                 # a drifting, cyclical series:
t = np.arange(n)                        # the level in month 300 is not
y = .05 * t + np.sin(t / 6) + rng.normal(0, .1, n)   # the month-1 level

L = 3                                   # three lags, the standard setup
X = np.stack([y[L-1-k : n-1-k] for k in range(L)], 1)
target = y[L:]

def mae(tr, te, k=3):                   # 3-NN: any model that averages
    d = ((X[te][:, None] - X[tr][None]) ** 2).sum(-1)   # nearby points
    nn = np.argsort(d, 1)[:, :k]                        # behaves this way
    return np.abs(target[tr][nn].mean(1) - target[te]).mean()

order = np.arange(len(target))
cut = int(.8 * len(target))
shuffled = rng.permutation(order)       # the ordinary k-fold split
print("X", X.shape, "target", target.shape)
a = mae(shuffled[:cut], shuffled[cut:])
b = mae(order[:cut], order[cut:])       # every test point after train
print("shuffled CV MAE %.3f" % a)
print("rolling-origin  %.3f   (%.1fx worse)" % (b, b / a))

02Stationarity, and why differencing is the first move

A model fitted on the past is only useful if the past and the future are the same kind of thing. Stationarity is the formal version of that hope.

A series is weakly stationary when three things hold: the mean does not depend on time, the variance does not depend on time, and the covariance between two points depends only on the lag between them.

Almost no interesting series is stationary as recorded. Sales grow and prices wander. The classical answer is to transform until what is left is stationary, fit there, then invert the transform.

Differencing is the main tool. Replace each value with the change since the previous value:

The first removes a trend. The second, seasonal differencing with period $m$, removes a repeating annual or weekly pattern.

The number of ordinary differences is the $d$ in ARIMA. The number of seasonal differences is the $D$ in SARIMA, and that is all those letters mean.

import numpy as np
rng = np.random.default_rng(0)

n = 400                                 # a random walk with drift, the
step = rng.normal(.3, 1.0, n)           # canonical non-stationary series
y = step.cumsum()

def halves(s, name):                    # same statistic, two windows
    a, b = s[:len(s)//2], s[len(s)//2:]
    print("%-13s mean %6.2f vs %6.2f    var %6.2f vs %6.2f"
          % (name, a.mean(), b.mean(), a.var(), b.var()))

print("y", y.shape, "   diff(y)", np.diff(y).shape, "\n")
halves(y, "raw")                        # nothing about it holds still
halves(np.diff(y), "differenced")       # both halves agree: stationary
halves(np.diff(y, 2), "twice-diffed")   # variance paid, nothing gained

Two tests are used to decide how many differences you need, and they are mirror images. The ADF test has non-stationarity as its null hypothesis. The KPSS test has stationarity as its null. Running both and agreeing is more informative than running either alone.

03Decomposition, seasonality and the baseline you skipped

Most series are a slow trend, a repeating cycle and a remainder. Separating them is both a diagnostic and a forecasting method in its own right.

The additive decomposition writes the series as a sum of three parts:

A series splits into trend-cycle, seasonal and remainder components. If the seasonal swing grows with the level, use a multiplicative form instead, or take logs first and stay additive.

Three ways to do the split, in increasing order of usefulness:

• Classical decomposition. A centred moving average for the trend, then average the detrended values by season. Simple, and it throws away the first and last few points.
• STL (Seasonal-trend decomposition using Loess) lets the seasonal shape change slowly over the years and is robust to outliers. It is the default worth reaching for.
• Prophet-style decomposable models. Fit trend, Fourier-series seasonality and holiday effects as an explicit curve-fitting problem, then forecast each piece forward.

Prophet deserves a fair hearing. It handles missing data, irregular sampling and holiday calendars without complaint, and an analyst can override the trend changepoints by hand. It is a curve fit, though, and not a stochastic process: it models the shape of the series and largely ignores the autocorrelation in the remainder.

Baselines are not a formality

Three forecasts cost nothing and beat a surprising fraction of real models:

• Naive. Tomorrow equals today. Optimal for a random walk, which is more series than you would like.
• Seasonal naive. This Tuesday equals last Tuesday. The baseline to beat on anything with a weekly or annual cycle.
• Drift. Extend the straight line from the first observation to the last.

Report your model's error as a ratio against one of these. That ratio is what MASE formalises, and it is the only error number that travels between series on different scales.

import numpy as np
rng = np.random.default_rng(0)

m, n = 7, 84                            # 12 weeks of noisy daily data
t = np.arange(n)
season = np.array([0., 1.2, 1.0, .8, 1.4, 3.5, 3.2])
y = 10 + .02 * t + season[t % m] + rng.normal(0, .8, n)
tr, te = y[:56], y[56:]                 # train 8 weeks, test 4 weeks

naive = np.full(te.shape, tr[-1])            # repeat the last value
snaive = np.array([y[56 + i - m * (i // m + 1)]
                   for i in range(len(te))]) # same weekday, last week
p = 14                                       # AR(14) by OLS on 42 rows
A = np.stack([tr[p-1-k : 55-k] for k in range(p)], 1)
w = np.linalg.lstsq(A, tr[p:], rcond=None)[0]
hist, ar = list(tr), []
for _ in te:                                 # recursive multi-step
    ar.append(float(w @ hist[-1:-p-1:-1])); hist.append(ar[-1])

print("train", tr.shape, "test", te.shape, "AR rows", A.shape)
base = np.abs(snaive - te).mean()
for nm, f in [("naive", naive), ("seasonal naive", snaive),
              ("AR(14) OLS", np.array(ar))]:
    e = np.abs(f - te).mean()
    print("%-15s MAE %.3f   vs snaive %.2fx" % (nm, e, e / base))

04The classical toolkit, in one pass

ARIMA and exponential smoothing are the two families that dominated forecasting for forty years. Both are still the right answer for a single short series.

Start with the two building blocks. An autoregressive term regresses the series on its own past. A moving average term regresses it on its own past errors:

The $\phi$ terms are AR($p$), the $\theta$ terms are MA($q$). Apply this to a $d$-times differenced series and you have ARIMA($p,d,q$).

SARIMA adds a second copy of the same structure at the seasonal lag, written ARIMA($p,d,q$)($P,D,Q$)$_m$. Seven numbers, which is why automatic order selection exists. auto.arima and its Python equivalents search the grid by AICc.

Exponential smoothing comes at the same problem from the other end. Simple exponential smoothing forecasts a weighted average of the past where the weights decay geometrically:

It has one parameter. Holt adds a trend component, Holt-Winters adds a seasonal one, and the ETS taxonomy enumerates the combinations of error, trend and seasonality.

The third row is worth dwelling on. A state space model splits the world into a hidden state that evolves and a noisy observation of it. The Kalman filter updates that state one observation at a time.

Two properties make it valuable. Missing observations are free, because you skip the update step. And the filter produces the exact likelihood, so the model can be fitted by maximum likelihood rather than by a bespoke recursion. Most modern ETS and ARIMA implementations are state space models underneath.

05Turning a series into a table

Every tree and every neural network on this page is a regression on a feature matrix. Building that matrix is the whole job, and it is where leakage lives.

The move is mechanical: for each timestamp, build a row whose target is $y_{t+h}$ and whose features use nothing dated later than $t$. Five kinds of feature do almost all the work:

• Lags of the target. $y_t, y_{t-1}, \dots$ plus the seasonal lags $y_{t-m}, y_{t-2m}$. The single most important family.
• Rolling statistics. Mean, standard deviation, min and max over the last 7, 28 or 90 periods. These encode level and volatility cheaply.
• Calendar features. Day of week, week of year, month, holiday flags, days to the next holiday. Cheap and known in advance.
• Known-future covariates. Planned price, scheduled promotion, published holiday calendar. You may use their value at $t+h$, because you will know it.
• Past-only covariates. Observed weather, competitor traffic, upstream demand. You may use their value at $t$ and no later.

The distinction in the last two bullets is the one people get wrong. It is also the heart of multivariate forecasting.

Rolling windows carry the same hazard in miniature. A 7-day rolling mean centred on $t$ includes three days after $t$. A 7-day mean ending at $t$ does not. One of those is a feature and the other is the answer.

There is also a choice about the horizon itself:

• Recursive. Train one one-step model and feed its own predictions back in. Compact, and errors compound, as the AR baseline above showed.
• Direct. Train a separate model for each horizon $h$. No compounding, more models, and no guarantee the forecasts are mutually consistent.

06Gradient-boosted trees, and why they keep winning

The honest summary of the last decade: for most tabular forecasting problems with many series, LightGBM on lag features is still the model to beat.

Once the series is a table, the usual tabular champion applies. Gradient boosting handles hundreds of mixed features, learns interactions without being told, tolerates missing values and trains in minutes on a laptop.

The evidence is the M-competitions, which are the closest thing forecasting has to a shared benchmark. In M4 (2018, 100,000 series) the pure machine-learning entries did badly. The winner was a hybrid of exponential smoothing and a recurrent network, and second place was an ensemble of classical statistical methods.

In M5 (2020, Walmart sales) the picture reversed. Gradient-boosted trees on engineered features dominated the leaderboard, and the winning accuracy entry was built on LightGBM. The difference was the data: many related series, rich covariates, real hierarchy.

Trees also need a reason to know which series a row belongs to. Series identity, category and location go in as static features, so a single global model can serve thousands of series while still specialising.

07Deep forecasting: four architectures

The contribution of deep learning here is not a better function class. It is the global model: one network trained across every series at once.

Classical methods fit each series independently, so a series with 40 observations gets 40 observations of evidence. A global model trained on 40,000 related series learns the shape of seasonality once and shares it, and that is where the gains come from.

Four designs cover most of the field:

• DeepAR (2017). An autoregressive recurrent network trained across many series that outputs the parameters of a distribution instead of a point. Forecasts are produced by sampling forward, so uncertainty comes for free.
• N-BEATS (2019). A deep stack of fully connected blocks, no recurrence at all. Each block emits a backcast and a forecast, and the backcast is subtracted before the next block sees the input. Basis functions can be constrained to be polynomial or sinusoidal, which makes the decomposition readable.
• Temporal Fusion Transformer (2019). Built for the messy case: static covariates, known-future covariates and past-only covariates, all at once. Variable selection networks score which inputs matter, gating layers let it skip unused capacity, and it emits quantiles directly.
• PatchTST (2022). Chops the series into patches, embeds each patch as a token, and treats every channel independently. Patching shortens the sequence and makes attention see local shape rather than single points.

Dilated causal convolutions are the fourth family, inherited from WaveNet. They give a long receptive field with fixed-cost inference and no recurrence. Page 13 covers that stack in its original setting.

Take the ordering seriously: baseline first, trees second, and deep models when you have many related series, rich covariates and enough history to justify them.

08Foundation models for forecasting

Since 2023 several groups have pretrained a single model on very large collections of series, then forecast new ones with no fitting at all.

The recipe borrows directly from language modelling. Take a scaled window of values, turn it into tokens or patches, and train a sequence model to predict the continuation. Four public examples:

• Chronos (Amazon, 2024). Scales and quantises values into a fixed vocabulary, then trains an off-the-shelf T5 on them. The series is literally treated as text.
• TimesFM (Google, 2023). A decoder-only model over input patches, with an output patch longer than the input patch so it can forecast far in one step.
• Moirai (Salesforce, 2024). Targets the awkward parts directly: multiple frequencies, variable numbers of covariates, and probabilistic output.
• Lag-Llama (2023). A decoder-only transformer over lagged features, and one of the first open univariate probabilistic attempts.

What they buy you is a usable forecast on a new series in seconds, with no tuning. For a cold-start series, or for the long tail of series nobody has time to model, that is a real win.

There are two cautions. First, the zero-shot claims are only as good as the separation between the pretraining corpus and the evaluation set, and that separation is hard to audit from outside. Second, most of these models take the target series and little else, so any covariate you have worked hard to build has nowhere to go.

Treat a foundation model as a strong baseline that arrives for free. Then check whether LightGBM on your own features beats it, because on a well-understood problem it often does.

09Probabilistic forecasts, quantiles and anomalies

A point forecast is a decision waiting to be made badly. Inventory, staffing and capacity all need a distribution, and a single number is not enough.

The cheap way to get one is quantile regression. Train the same model several times with a different loss for each target quantile. The pinball loss for quantile $\tau$ is asymmetric:

At $\tau=0.9$ under-predicting costs nine times as much as over-predicting, so the fitted value settles at the 90th percentile.

Whatever produces the interval, check its calibration. If your 90% interval contains the truth 62% of the time, it is not a 90% interval. Check coverage separately at each horizon, because intervals must widen as the horizon grows and often do not.

The metrics deserve the same care as the model:

• MAE targets the median, RMSE targets the mean. On skewed demand data these are different forecasts, and optimising one while reporting the other is a common own goal.
• MAPE divides by the actual value. It explodes at zero and it punishes over-forecasting more than under-forecasting. Intermittent demand breaks it completely.
• MASE scales by the in-sample naive error, so it is comparable across series and has a clear reading: below 1 beats the naive forecast.
• CRPS and weighted quantile loss score the whole predictive distribution. These are what probabilistic competitions use.

Anomaly detection is forecasting with the residual kept

The standard construction is to forecast one step ahead, then compare. A point is anomalous when it falls far outside its own prediction interval. Everything upstream on this page applies, because the quality of the anomaly flag is the quality of the forecast.

Two things cause most false alarms. Seasonality that the model does not represent, so every Monday looks like a spike. And historical anomalies left in the training data, which widen the intervals until nothing trips them.

10Backtesting that you can trust

Back to where the page started. A backtest is a simulation of deployment, and it is only worth anything if the simulation refuses to look forward.

Rolling-origin evaluation is the whole method: pick an origin, train on everything before it, forecast the next $h$ periods, score, move the origin forward, repeat. The bottom half of the diagram is the picture, and chapter 5.10 of Forecasting: Principles and Practice is the canonical write-up.

Four decisions inside that loop:

• Expanding or sliding window. Expanding keeps all history and assumes the old regime still applies. Sliding uses a fixed-length window and adapts faster after a structural break.
• Horizon. It must equal the horizon you forecast in production. A model tuned at one step ahead is not the model you want at 28.
• Gap. Leave an embargo between the end of training and the start of the test window whenever features use rolling statistics or labels arrive late.
• Refit policy. Refitting at every origin is honest and slow. Refitting monthly is what you will do in production, so backtest that instead.

Score per horizon and then aggregate, never the other way round. A model that is excellent at $h=1$ and useless at $h=28$ can show a perfectly respectable average.

The leaks, roughly in the order they bite:

• A shuffled split, from a default cv=5 nobody read.
• A scaler, imputer or target encoder fitted before the split.
• A rolling window that includes the current point, or is centred.
• A covariate used at $t+h$ that is not known at $t$.
• Feature selection or hyperparameter search run once over all the data, then "validated" on part of it.
• Restated data: a warehouse table that was corrected later, so the backtest sees numbers that did not exist at the time.
• Deduplication or outlier removal performed globally before splitting.

The last two are the hardest to catch, because nothing in the modelling code is wrong. The data itself is the wrong vintage.

11Build this

The claim on this page is that leakage is worth more than modelling, and that is measurable in an afternoon.

12What breaks

Forecasting failures are rarely a model that cannot fit. They are a number that was never measuring what you thought.

• The backtest was excellent and production is not. The default outcome of a shuffled split. Nothing in the code looks wrong, and the gap only appears after deployment.
• A covariate was not available at forecast time. Someone joined a table on timestamp without asking when the row was written. The feature importance chart looks wonderful.
• The tree forecast flatlines. A growing series, no differencing, and every long-horizon prediction pinned to the top of the training range.
• Nobody ran seasonal naive. Six weeks of modelling to land 8% worse than repeating last Tuesday, discovered by someone else.
• MAPE on a series that touches zero. The metric goes to infinity, someone clips it, and the clipped version now rewards under-forecasting.
• Intervals that never widen. The 90% band at horizon 28 is the same width as at horizon 1, so coverage collapses exactly where the decisions are risky.
• A structural break mid-history. A pricing change or a pandemic sits inside the training window, and an expanding window keeps averaging over a regime that no longer exists.
• Restated history. The warehouse corrected last quarter's numbers. Your backtest read the corrected version, and the live model never will.
• Hierarchies that do not add up. Store forecasts summed do not equal the regional forecast, and the planning system needs them to.

13Where you meet this in the wild

The constraint is the same everywhere, and the consequences of getting it wrong differ a lot.

14Interview questions

15Go deeper

●Now write it yourself

Reading the derivation and being able to produce it are different skills. These are Deep-ML problems that exercise what this page covers — each one is checked against real test cases, not multiple choice.

Matched to this page from Deep-ML's catalogue of 1,380 problems. More at deep-ml.com, and Where to practise covers the other platforms and what each one trains.

16Related pages

Other techniques for this problem

A scoped slice of the full Technique Map — every technique this page covers, grouped by what it solves.