Understanding the neural network backpropagation algorithm

Modern machine learning owes much of its recent progress to a single elegant algorithm. Backpropagation, the mechanism by which neural networks learn from their mistakes, sits at the core of every deep model trained today, from image classifiers running in hospitals in Melbourne to the recommendation engines that power streaming services in Sydney. The idea is conceptually simple: compute how much each weight in the network contributed to the error and adjust accordingly. The execution, however, involves careful bookkeeping across hundreds of thousands or millions of parameters, and a solid intuition for the chain rule from calculus.

This article walks through what backpropagation actually does, why it works mathematically, and how to translate the procedure into code. We will start with the intuition behind forward propagation, then derive the backward pass using the chain rule, look at a worked example with pseudo-code, and finish with practical pitfalls that arise when training deeper networks. Whether you are a student at the University of Queensland, a researcher at CSIRO's Data61, or a hobbyist building a model on your laptop in Adelaide, the ideas transfer directly to whatever framework you use.

The role of gradient descent in training neural networks

Training a neural network means finding values for its weights and biases that minimise a loss function. Gradient descent does this iteratively: given a current set of parameters, it moves them a small step in the direction that decreases the loss most quickly. The direction is given by the negative gradient of the loss with respect to each parameter. Computed naively, this gradient would require evaluating the loss many times, perturbing one parameter at a time, which scales miserably with model size.

Backpropagation solves this scaling problem by computing all gradients in a single sweep through the network using the chain rule. It is essentially a clever application of dynamic programming to the computation graph defined by the network's architecture. The technique was popularised in the 1980s through the work of Rumelhart, Hinton, and Williams, and it remains the engine behind deep learning practice across the globe, from the research labs of CSIRO to industrial teams in Brisbane working on language models for the resources sector.

The procedure has two phases. The forward pass evaluates the network on a training example, computing and storing intermediate activations. The backward pass traverses the network in reverse, propagating error signals layer by layer until every weight has a gradient attached to it. The weights can then be updated by an optimisation rule such as stochastic gradient descent or Adam, and the cycle repeats for the next batch.

Building the computation graph during the forward pass

Every neural network can be represented as a directed acyclic graph whose nodes are elementary operations and whose edges carry tensors. A logistic regression unit computes a weighted sum followed by a sigmoid. A transformer block combines attention, normalisation, and feed-forward sublayers. During the forward pass, the network evaluates this graph from inputs to outputs, caching every intermediate tensor for later reuse.

The cache is essential. Without it, the backward pass would need to recompute expensive operations many times. With it, the algorithm can revisit each cached value exactly when needed. This design mirrors how efficient algorithms reuse earlier work: a similar philosophy appears in Tarjan's offline LCA tutorial, where a union-find structure stores ancestor information so queries are answered in logarithmic time without repeating traversals. The same principle of trading a little memory for a great deal of speed drives backpropagation's caching step.

A second subtlety is the role of vectorisation. In practice, examples are processed in mini-batches rather than one at a time, and operations are performed on matrices rather than scalars. The forward pass therefore produces tensors of shape batch by layer dimensions, and gradients computed later have the same shape. Modern frameworks hide this detail behind broadcasting rules, but understanding the shapes helps when debugging mismatches in custom layers, a chore familiar to anyone wiring together prototypes in Sydney's growing AI consultancy scene.

Why the chain rule is the mathematical foundation

The chain rule states that the derivative of a composite function equals the product of the derivatives of its outer and inner parts. Backpropagation applies this rule repeatedly, marching from the loss at the end of the computation graph back toward the input, multiplying local Jacobians at each step. Concretely, if a layer computes y = f(Wx + b) and the upstream gradient is dL/dy, then the gradient with respect to W is dL/dW = (dL/dy) times xᵀ, and the gradient with respect to the input is dL/dx = Wᵀ times (dL/dy).

This is where autograd systems earn their keep. Modern frameworks replace the manual chain rule calculations with automatic differentiation, building the computation graph dynamically during the forward pass and replaying it in reverse. To an end user in Perth running a fault-detection model on mining sensor data, the library handles every partial derivative. To an implementer, however, the visibility into what the framework actually computes can be invaluable when chasing subtle numerical bugs.

Two implementation patterns dominate. Define-by-run frameworks, including PyTorch and JAX, build the graph on the fly as operations execute, which makes debugging easier and supports dynamic control flow. Define-and-run frameworks require the graph up front. Both styles converge on the same numerical result, and both store enough intermediate state to drive the backward pass without recomputation.

Stepping through the backward pass with an example

Consider a tiny network with one hidden layer using tanh activations and a softmax output. The forward pass computes the hidden activations h, the output probabilities p, and a cross-entropy loss L. The backward pass begins at the loss, where the initial gradient is p minus y, with y being the one-hot target vector.

From there, gradients flow upstream into the output linear layer, producing derivatives for its weights and biases and for its input, which is the hidden activation h. The gradient on h is then pushed through the tanh nonlinearity, scaled by 1 minus tanh squared at h, before reaching the input linear layer, whose weights and biases receive their own gradients. Each step uses the chain rule, and each step reuses cached values from the forward pass, so the entire backward sweep is roughly as expensive as a second forward pass rather than many times costlier.

This proportionality is why training a large network is feasible at all. A model trained on a cluster in Adelaide or Canberra produces gradients at a cost comparable to its evaluation cost, simply because backpropagation uses cached state instead of re-evaluating the network from scratch. The same caching idea also shows up in algorithms that seem unrelated at first glance, such as a bucket sort tutorial, where auxiliary buckets trade memory for speed in a single pass over the input.

Translating backpropagation into working code

In NumPy, a minimal implementation starts by initialising weights and biases for each layer. The forward function multiplies inputs by weights, adds biases, and applies an activation, storing every intermediate tensor in a Python list. A separate backward function then receives the gradient of the loss with respect to the output and walks the layer list in reverse. At each layer it computes gradients using the cached tensors and propagates upstream to the layer below.

Vectorising across a batch turns the inner loops into matrix products that run fast on modern hardware. Numerical stability matters: subtracting large logits from their maximum before exponentiating avoids overflow in softmax, and adding a small epsilon to denominators prevents division by zero in batch normalisation layers. Frameworks such as PyTorch wrap all of this with autograd, so the developer sees only the forward computation while the backward graph is constructed behind the scenes.

For deeper networks, a handful of practical refinements tend to make training far more reliable.

• Initialise weights with variance-scaling schemes such as He or Xavier initialisation so signals neither vanish nor explode at the start of training.
• Normalise activations using batch norm, layer norm, or group norm to keep distributions stable across layers.
• Prefer an adaptive optimiser such as Adam or RMSProp over vanilla stochastic gradient descent when the loss landscape is steep or curved.
• Apply gradient clipping when training transformers or recurrent models, since their gradients can spike unexpectedly.
• Decay the learning rate gradually, often with a cosine schedule, to refine solutions near the end of training.

These adjustments do not change the underlying algorithm but make it tractable on real problems. Researchers at the University of Melbourne and CSIRO routinely apply several of them on production models for medical imaging and climate science.

Diagnosing vanishing gradients and other failure modes

Even a textbook-correct implementation can fail to learn. The classic symptom is a loss curve that plateaus or diverges after a few iterations, leaving practitioners staring at a flat line on a dashboard in their Brisbane office. Most such failures trace to one of a few root causes worth recognising early.

• Activations collapse to zero, a sign of dead ReLUs, learning rates that are too high, or normalisation that saturates.
• Gradients vanish in early layers, a problem deep recurrent networks suffered from before the LSTM, and which transformer-style models avoid through residual connections.
• Loss explodes mid-training, often from a learning rate that is too aggressive or from a bad batch triggering a large update that destabilises the entire network.
• Validation loss rises while training loss falls, the signature of overfitting that calls for regularisation, more data, or early stopping.

A local example makes the diagnostic value clear. A team in Brisbane building a crop-yield model from satellite imagery noticed their deep network refused to improve beyond a baseline linear regressor. Inspection revealed the input images arrived in BGR rather than RGB, and a small offset in the first batch norm layer was starving early gradients of signal. Fixing the data pipeline and switching to He initialisation took the model above the baseline within a single epoch. The lesson generalises beyond Australia and beyond this particular dataset: a backpropagation bug is rarely a math bug, more often a data bug or a shape bug that the algorithm faithfully propagates through every layer.

Backpropagation is one of those algorithms that rewards careful study. Once the chain rule, the cached forward activations, and the reverse traversal click into place, every framework and every debugging session becomes more transparent. For further reading on algorithmic techniques and the rest of the site's catalogue of tutorials, the hello ML website collects everything in one searchable place and is worth keeping open while you experiment.