Bias-Variance Decomposition — Formula and Principle

The bias-variance decomposition is an essential tool for understanding the generalization performance of supervised learning models. For regression, the expected prediction error splits into squared bias, variance and irreducible noise.

Decomposition of the expected prediction error

\(\mathbb{E}[(y - \hat{f}(x))^2] = \underbrace{(\mathbb{E}[\hat{f}(x)] - f(x))^2}_{\text{Bias}^2} + \underbrace{\mathbb{E}[(\hat{f}(x) - \mathbb{E}[\hat{f}(x)])^2]}_{\text{Variance}} + \underbrace{\sigma^2}_{\text{Irreducible Error}}\)
Squared bias (Bias²)
\(\text{Bias}^2 = (\mathbb{E}[\hat{f}(x)] - f(x))^2\)
Variance
\(\text{Var} = \mathbb{E}[(\hat{f}(x) - \mathbb{E}[\hat{f}(x)])^2]\)
Irreducible error
\(\sigma^2\)
Total error
\(\text{Total Error} = \text{Bias}^2 + \text{Variance} + \sigma^2\)

Formula Parameters

xinput feature vector
ytarget variable (true value)
f(x)the true function (unknown)
f̂(x)the model prediction (depends on the training set)
E[Ā·]expectation over all possible training sets
σ²variance of the data noise; irreducible
Bias²how far the average prediction is from the truth
Variancehow sensitive the model is to changes in the training set

Interactive Simulation Lab

Adjust the complexity (degree) of the polynomial regression, the number of training samples and the noise level, then watch how bias, variance and total error change. Click ā€œRegenerateā€ for a fresh data set.

Polynomial degree (complexity)
d 3
Sample size N
N 30
Noise std. dev. σ
σ 0.20
Simulations (used to estimate bias/variance)
M 100
Training Data and Model Fit (one example run)
Bias², Variance and Total Error vs Model Complexity
Error Decomposition at the Current Settings

A Deeper Look at Bias and Variance

Standard Introduction

In supervised learning, a model generalization error can be decomposed into three parts: bias, variance and irreducible error. Bias measures the gap between the model average prediction and the true value, reflecting how well the model can fit; variance measures how sensitive the model is to a particular training set, reflecting its stability. The irreducible error is the variance of the noise inherent in the data and is independent of the model. Too little complexity (a low-degree polynomial, say) produces high bias, i.e. underfitting; too much complexity (a very high-degree polynomial) produces high variance, i.e. overfitting. Model selection therefore has to strike a balance between bias and variance.

Plain-Language Introduction

Imagine you are in a shooting competition and want to hit the bullseye. Bias describes how far your average point of aim is from the bullseye — if you always shoot to the left, your bias is large. Variance describes how spread out your shots are — if the bullet holes are scattered widely, your variance is large. An ideal shooter both aims well (low bias) and is consistent (low variance). In machine learning, however, we can only ā€œaimā€ using a finite training set, so achieving both at once is hard. A model that is too simple (fitting curved data with a straight line) is like always firing at a fixed angle: the aiming system is too crude, so the bias is large. A model that is too complex (a high-degree curve passing exactly through every training point) is like over-correcting after every shot: it becomes oversensitive to the training data, so the variance is large.

What Problem Does It Solve

The bias-variance decomposition provides a theoretical framework for understanding and diagnosing generalization. It explains why a model can perform beautifully on the training set yet poorly on the test set (overfitting): training error ignores the variance term, whereas test error includes both bias and variance. The decomposition tells us which way to improve: if bias is high, increase model complexity or use richer features; if variance is high, collect more data, apply regularization or reduce complexity. The framework also guides the design of ensemble methods — Bagging reduces variance, Boosting reduces bias.

Why It Matters

The bias-variance trade-off is one of the core ideas in machine learning. It underpins the basic rule of model selection: no single model is best in every respect, and the optimal complexity depends on the amount of data, the noise level and the smoothness of the true function. In practice we estimate the test error by cross-validation and pick the model that balances bias against variance. In both research and engineering, this concept helps us avoid blindly chasing complexity and instead make rational decisions for the problem at hand. Many advanced techniques — regularization, early stopping, dropout — are designed around the bias-variance trade-off.

Application Scenarios

  • Model selection:compare models of different complexity (polynomial degree, number of neural network layers) and pick the one with the lowest test error;
  • Regularization tuning:adjust the strength of L1/L2 regularization to balance bias against variance;
  • Ensemble learning:Bagging averages many high-variance models to cut variance, while Boosting combines weak learners to cut bias;
  • Feature engineering:decide whether to add more features — too many can raise variance, too few can raise bias;
  • Data collection strategy:judge whether more training data will actually help (it mainly helps high-variance models).

Limitations and Drawbacks

  • Idealized assumptions:the decomposition usually assumes squared loss and unlimited data; with finite data sets it is only an approximation;
  • Not directly applicable to classification:0-1 loss does not decompose cleanly, so surrogate losses or special analyses are used instead;
  • Hard to estimate precisely:bias and variance depend on knowing the true function, so in practice they can only be approximated by resampling;
  • Ignores model uncertainty:it does not account for priors over the parameters or for Bayesian inference;
  • Computationally expensive:estimating bias and variance accurately requires training the model many independent times, which may be infeasible for large models;
  • Overly simplistic:real-world error sources can be more complex — distribution shift or concept drift, for example, are not fully explained by this framework.