Error Backpropagation
Backpropagation — the core engine behind neural network learning
Deep Learning · Gradient DescentCore Formulas
The error backpropagation (BP) algorithm was popularized by Rumelhart, Hinton and Williams in 1986 and is still the most widely used algorithm for training multilayer neural networks. Using the chain rule, it propagates the output-layer error back to the hidden layers, computes the gradient of the loss function with respect to every weight, and then updates the parameters by gradient descent. The formulas below describe a typical 2-2-1 network (2 input neurons, 2 hidden neurons, 1 output neuron).
Forward propagation: $$h_j = \sigma\left(\sum_{i=1}^{2} w_{ij} x_i + b_j\right), \quad j=1,2$$ $$y = \sigma\left(\sum_{j=1}^{2} v_j h_j + c\right)$$
Loss function (mean squared error): $$E = \frac{1}{2}(y - t)^2$$
Output layer error: $$\delta_y = \frac{\partial E}{\partial y} \cdot \sigma'(z_y) = (y - t) \cdot y(1-y)$$
Hidden layer error: $$\delta_j = v_j \cdot \delta_y \cdot h_j(1-h_j), \quad j=1,2$$
Weight update (gradient descent): $$v_j \leftarrow v_j - \eta \frac{\partial E}{\partial v_j} = v_j - \eta \delta_y h_j$$ $$w_{ij} \leftarrow w_{ij} - \eta \frac{\partial E}{\partial w_{ij}} = w_{ij} - \eta \delta_j x_i$$ $$c \leftarrow c - \eta \delta_y, \quad b_j \leftarrow b_j - \eta \delta_j$$
Parameter Notes
Interactive Simulation
This is a 2-2-1 neural network. Adjust the inputs, the target value and the learning rate, set the initial weights, then click "Run one training step" to watch forward propagation, error computation and backpropagation update the weights.
Parameter Control
Initial weights and biases (training resets automatically when they change):
Network Architecture (2-2-1)
Loss Curve
Standard Overview
The error backpropagation (BP) algorithm is one of the core algorithms for training artificial neural networks; it was systematically described and popularized by Rumelhart, Hinton and Williams in their 1986 paper. Built on gradient descent, BP uses the chain rule to efficiently compute the partial derivative of the loss function with respect to every weight in the network, and uses those gradients to update the weights. Its core idea is: first compute the network output by forward propagation, then measure the error between that output and the desired target, then propagate the error layer by layer from the output back to the input, computing an error signal for every neuron along the way, and finally update the connection weights and biases accordingly.
BP made it possible to train multilayer neural networks and overcame the linear limitation of the single-layer perceptron. It laid the foundation of deep learning and is still used to train virtually every modern architecture (CNN, RNN, Transformer and others). Although modern deep learning frameworks compute gradients automatically, understanding BP remains essential for truly mastering how neural networks work.
Plain-Language Introduction
Imagine teaching a child to shoot a basketball. You tell them which way to throw; they throw and the ball lands to the left. You say "a bit more to the right." They adjust and throw again — this time it goes too far right. You say "back to the left a little." After many adjustments they get more and more accurate. Throughout this process you are really using the error (how far the ball is from the hoop) to continuously correct their motion.
BP does something similar. The neural network is the shooter: it takes a first shot (forward propagation) and sees how far it is from the target (computing the loss). Then it asks "which part of the motion caused the deviation?" If a weight (the force, the angle) has a large influence on the error it is adjusted more; if its influence is small it is adjusted less. That step of "analyzing which motion caused the deviation" is backpropagation — starting from the final error, the chain rule traces how much each parameter contributed, and gradient descent corrects them step by step. After a great deal of training the network's shots get steadily more accurate.
What Problem Does It Solve
BP solves the problem of training the hidden-layer weights of a multilayer neural network. Before BP, a perceptron had only an input layer and an output layer, so its weights could be adjusted directly from the error. In a multilayer network, however, the hidden layers have no desired output of their own, so how to adjust hidden-layer weights was an open problem.
BP solves this elegantly through error backpropagation: it distributes the output-layer error back to the hidden layers in proportion to the connection weights, giving each hidden neuron a "virtual error" so that its weights can be updated just like those of the output layer. This innovation made it possible to train feedforward networks of any depth and directly triggered the revival of neural network research.
In addition, combining BP with gradient descent provides a general parameter optimization framework that can train any differentiable model — CNNs, RNNs, Transformers and beyond. One could say that the training of almost every modern deep learning model relies on the idea behind BP.
Why It Matters
BP is the technical foundation of deep learning. Without it, modern deep learning would not exist. It turned the training of multilayer networks from a theoretical possibility into an engineering reality, and directly sparked the neural network boom of the late 1980s and early 1990s. Deep learning later went through a long winter, but the core idea of BP survived and, after the deep learning revival of 2006, became an indispensable tool.
Mathematically, BP is essentially an efficient implementation of the chain rule. Exploiting the dynamic-programming structure of the computational graph, it avoids recomputing gradients and reduces the cost from exponential to linear, which is what makes training large-scale networks feasible. The automatic differentiation built into modern frameworks such as TensorFlow and PyTorch is still BP at its core.
BP also inspired many later optimization techniques — momentum, Adam, RMSprop and others all improve the update rule on top of the gradients BP computes, in order to train faster and more stably. Understanding BP is a necessary step on the path to deep learning research.
Application Scenarios
Image Recognition Speech Recognition Natural Language Processing Recommender Systems Medical Diagnosis Financial Forecasting Autonomous Driving Game AI
BP runs through essentially every deep learning application:
Computer vision: training convolutional neural networks (CNNs) for image
classification, object detection and semantic segmentation;
Natural language processing: training recurrent neural networks (RNNs),
LSTMs and Transformers for machine translation, text generation and more;
Speech recognition: training deep networks that map acoustic features to
phonemes or characters;
Recommender systems: training deep collaborative filtering models to learn
user and item embeddings;
Reinforcement learning: training policy and value networks, such as the deep
networks behind AlphaGo.
Any task that needs to learn a complex mapping from data depends on BP. It is the bridge between data and models, and essential knowledge for every machine learning engineer.
Limitations and Drawbacks
1. Vanishing and exploding gradients: in deep networks that use
saturating activations such as sigmoid, gradients shrink (vanish) or grow (explode)
exponentially as they propagate backward, making training difficult.
2. Prone to local optima: the loss function is usually non-convex, so
gradient descent may converge to a local minimum instead of the global one. Results are often
good enough in practice, but global optimality cannot be guaranteed in theory.
3. Sensitive to weight initialization: poor initialization can slow
convergence or trap the model in a bad local minimum, so schemes such as Xavier or He
initialization are needed.
4. Heavy computational cost: training a large network means repeating
forward and backward passes, which is expensive — especially as data volume and model size
grow.
5. Risk of overfitting: models trained with BP have many parameters and
easily overfit the training data, so regularization, dropout and early stopping are needed to
mitigate this.
6. Difficulty choosing the learning rate: too large a learning rate causes
oscillation or even divergence; too small one converges slowly. Manual tuning or adaptive
learning-rate algorithms are usually required.
Despite these limitations, BP is still one of the most effective ways to train neural networks. A large body of improvements — ReLU activations, batch normalization, residual connections and more — has alleviated the problems above and made deep networks trainable in practice.