Extrapolation Lab

Points come from a 1-D function plus Gaussian noise, but only on a limited slice of its domain. Every model is fit with no regularisation and no early stopping. We then measure the empirical risk, the true risk inside the training domain, and the true risk outside it — and sweep the training-set size, the noise, and the complexity of the truth.

True function

Family Complexity Truth seed

Training data

Training set size n Noise σ Data seed

Training domain x ∈ [0.5, 3.5]; extrapolation is scored on x ∈ [3.5, 6.5].

Models

Polynomial A (least squares) Degree

Polynomial B (least squares) Degree

If degree + 1 > n the fit is the minimum-norm interpolant.

ReLU MLP, 2 hidden layers Width Adam steps (full batch)

No weight decay. Fewer steps = early stopping; keep it high to see the unregularised answer.

Gradient-boosted trees Trees Max depth

Squared loss, learning rate 0.1, full sample every round.

Sweeps

Repeats per grid point (median, IQR band) Largest n in the n-sweep

What to look for

Inside the domain every flexible model heads to the σ² floor as n grows (the excess risk of a d-parameter model is roughly d·σ²/n). Noise raises the floor but does not change who wins.
Outside the domain more data barely helps: the error is set by the model's inductive bias, not by variance. Polynomials leave at rate x^degree, a ReLU net becomes a straight line, trees go flat.
Only a well-specified small model (Polynomial A with the true degree, quadratic family) extrapolates — and then noise is what hurts, because coefficient error is amplified by x^k.
Raise the complexity of the truth: the well-specified degree moves, the "safe" polynomial becomes misspecified, and the trig family has no correct polynomial at all.
Empirical risk lies whenever the model can interpolate (degree + 1 ≥ n, deep trees, small n): it is ~0 while the true risk is at its worst.

model	empirical risk (train MSE)	true risk, in-domain	true risk, out-of-domain

Sweeps hold everything else fixed at the sidebar values and show the median over repeats (band = interquartile range). Empirical risks below 10⁻¹² are clamped.