The Basics of Neural Networks: Perceptrons and Backpropagation
Neural networks are computational models that learn relationships between inputs and outputs. They are used for tasks such as recognizing images, classifying documents, predicting prices, and generating recommendations. Although modern systems can contain millions or billions of parameters, their central ideas can be understood through a small building block: the artificial neuron.
Two concepts provide a useful foundation. A perceptron shows how a model can combine input features and make a simple decision. Backpropagation explains how a multilayer network adjusts its parameters after making a prediction. Together, these ideas connect linear algebra, calculus, optimization, and practical machine learning.
The goal is not to memorize a collection of formulas. It is to understand the flow of information: inputs are transformed into an output during the forward pass, an error is measured, and gradients guide the next parameter update. This process is repeated over many examples until the model captures useful patterns.
Neural Networks as Layered Functions
An artificial neuron receives numerical inputs, multiplies each input by a learned weight, adds the results, and applies an activation function. For inputs (x_1, x_2, \ldots, x_n), weights (w_1, w_2, \ldots, w_n), and bias (b), the weighted sum is:
[ z = \sum_{i=1}^{n} w_i x_i + b ]
The neuron then produces (a = f(z)), where (f) is an activation function. The bias shifts the decision boundary, while the weights determine how strongly each feature affects the result. In vector notation, the same operation is written as (z = \mathbf{w}^{T}\mathbf{x}+b).
A neural network connects many such units into layers. The input layer holds feature values, hidden layers transform those values, and the output layer produces a prediction. A network with no hidden layer can represent only a limited class of relationships, while hidden layers and nonlinear activations allow the model to approximate more complex functions.
The word “neural” refers to a loose analogy with biological neurons, not a claim that the mathematical unit works like a real brain cell. In software, a neural network is a parameterized function. Training means finding weights and biases that reduce a chosen loss function on the available data.
How a Perceptron Makes a Decision
The original perceptron is a binary classifier. It calculates a weighted sum and applies a step function:
[ \hat{y} = \begin{cases} 1 & \text{if } \mathbf{w}^{T}\mathbf{x}+b > 0 \ 0 & \text{otherwise} \end{cases} ]
This creates a linear decision boundary. With two features, the boundary is a line; with three features, it is a plane; with more features, it is a hyperplane. The perceptron can learn problems such as logical AND and OR when the classes are linearly separable.
During training, the model compares its prediction with the target label. If the example is misclassified, the parameters are updated using a learning rate (\eta):
[ \mathbf{w} \leftarrow \mathbf{w} + \eta(y-\hat{y})\mathbf{x} ]
[ b \leftarrow b + \eta(y-\hat{y}) ]
Correctly classified examples may produce no update. Repeated passes through the training set gradually move the decision boundary toward a position that separates the classes.
The perceptron is valuable because its training rule is easy to implement and inspect. It also appears in broader problem-solving practice, where breaking a process into inputs, operations, conditions, and updates helps clarify an algorithm. However, a single perceptron cannot solve the XOR problem, because XOR requires a nonlinear decision boundary.
From One Neuron to a Trainable Network
A multilayer perceptron solves the limitations of a single threshold unit by placing hidden layers between the input and output. Each layer performs an affine transformation followed by an activation:
[ \mathbf{a}^{(l)} = f(\mathbf{W}^{(l)}\mathbf{a}^{(l-1)}+\mathbf{b}^{(l)}) ]
Here, (l) identifies a layer, (\mathbf{W}^{(l)}) is its weight matrix, and (\mathbf{b}^{(l)}) is its bias vector. The activation function introduces nonlinearity, which allows several layers to build complicated mappings from simple transformations.
Common activations include sigmoid, hyperbolic tangent, and ReLU. Sigmoid maps values to the range from zero to one and is useful for some binary probability outputs. Tanh produces values between negative one and one. ReLU, defined as (\max(0,z)), is widely used in hidden layers because it is simple and usually provides stronger gradients than sigmoid for positive inputs.
The output activation depends on the task. A linear output is common for regression. A sigmoid output can represent a binary classification probability. For multiple mutually exclusive classes, softmax converts output scores into values that sum to one:
[ \text{softmax}(z_i)=\frac{e^{z_i}}{\sum_j e^{z_j}} ]
The architecture does not automatically guarantee good results. Input scaling, initialization, model size, regularization, and the quality of the training data all influence performance.
Measuring Error and Following Gradients
A network needs a loss function to quantify the difference between its prediction and the target. Mean squared error is often used for regression:
[ L = \frac{1}{n}\sum_{i=1}^{n}(y_i-\hat{y}_i)^2 ]
For classification, cross-entropy is usually a better choice because it penalizes confident incorrect predictions effectively. For a binary target (y) and predicted probability (\hat{y}), binary cross-entropy is:
[ L = -[y\log(\hat{y})+(1-y)\log(1-\hat{y})] ]
Backpropagation computes how much each weight contributed to the final loss. It applies the chain rule from calculus, beginning at the output and moving backward through the layers. If a weight has a large gradient, a small change in that weight can significantly affect the loss. If its gradient is close to zero, its immediate influence is small.
Once gradients are available, gradient descent updates the parameters:
[ \theta \leftarrow \theta-\eta\nabla_{\theta}L ]
The symbol (\theta) represents all trainable parameters. The negative sign moves the parameters in the direction that reduces the loss, while (\eta) controls the step size. A learning rate that is too large can cause unstable training; one that is too small can make learning painfully slow.
For a simple chain (x \rightarrow z \rightarrow a \rightarrow L), the derivative has the form:
[ \frac{\partial L}{\partial w}
\frac{\partial L}{\partial a} \frac{\partial a}{\partial z} \frac{\partial z}{\partial w} ]
Backpropagation is efficient because it reuses intermediate values from the forward pass instead of independently recalculating every derivative.
Comparing Core Learning Components
The following distinctions help separate the role of each part of a basic neural network system.
| Component | Main purpose | Typical expression or choice | Important limitation or concern |
|---|---|---|---|
| Perceptron | Make a binary linear decision | Step activation | Cannot represent nonlinear boundaries |
| Hidden layer | Learn intermediate feature transformations | Weighted sum plus activation | Too many units may overfit |
| ReLU | Add nonlinearity in hidden layers | (\max(0,z)) | Units can become inactive for negative inputs |
| Sigmoid | Produce a binary probability | (1/(1+e^{-z})) | Can create very small gradients at extremes |
| Cross-entropy | Measure classification error | Negative log likelihood | Requires appropriate probability outputs |
| Backpropagation | Compute parameter gradients | Chain rule | Depends on correct derivatives and stable computation |
| Gradient descent | Update weights and biases | (\theta-\eta\nabla L) | Sensitive to learning-rate selection |
A training loop usually combines these components in a fixed sequence. The model reads a batch of examples, calculates predictions, evaluates the loss, computes gradients, and updates the parameters. One complete pass through the training data is called an epoch.
Mini-batch training uses a subset of examples for each update. It is generally faster and less memory-intensive than calculating a gradient from the entire dataset. The slight randomness introduced by batches can also help optimization move through complicated loss landscapes.
A Practical Training Workflow
A small neural network can be implemented from scratch with arrays and basic mathematical operations. Before writing code, define the input shape, output format, loss function, activation functions, and parameter dimensions. Shape errors are among the most common problems in a first implementation, especially when matrix multiplication and batch dimensions are mixed.
A clear workflow makes debugging easier:
- Normalize or standardize numerical input features when their scales differ substantially.
- Initialize weights with small, carefully chosen random values and biases with zeros or small constants.
- Run a forward pass and save intermediate values needed for differentiation.
- Compute the loss, perform backpropagation, and update parameters with a chosen optimizer.
- Track training and validation metrics to detect overfitting or underfitting.
For binary classification, a useful first experiment is a network trained on a small synthetic dataset. Start with one or two input features so that predictions and decision boundaries can be visualized. Then compare a single perceptron with a network containing a hidden layer on a nonlinear dataset such as XOR.
When the implementation works, frameworks such as NumPy-based tools, PyTorch, or TensorFlow can automate gradient calculations and provide optimized tensor operations. Automation is convenient, but understanding the manual version makes framework errors easier to diagnose. Data preparation and model outputs may eventually connect to a web interface, where web development resources can help expose predictions through a simple application.
Common Problems During Training
A network that performs well on its training data may still fail on unseen examples. This is overfitting: the model has memorized patterns or noise instead of learning relationships that generalize. Validation data, smaller architectures, weight regularization, dropout, and early stopping are common responses.
Underfitting has the opposite pattern. The model is too simple, poorly trained, or supplied with weak features, so it performs badly on both training and validation data. Increasing capacity may help, but so can improving preprocessing, training for more epochs, or selecting a more suitable activation and loss function.
Vanishing gradients occur when derivatives become extremely small as they are multiplied through many layers. Saturating sigmoid or tanh units are especially vulnerable. ReLU-family activations, careful initialization, normalization, residual connections, and modern optimizers help reduce this problem. Exploding gradients, where values become excessively large, may require gradient clipping or a smaller learning rate.
A loss curve is often more informative than a single final accuracy score. A steadily decreasing training loss suggests progress, while a rising validation loss can reveal overfitting. For imbalanced classification, accuracy may be misleading; precision, recall, F1 score, and a confusion matrix can provide a more accurate view of the model’s behavior.
The perceptron offers the simplest picture of a learned decision boundary, while backpropagation extends that idea to networks with many interacting layers. Once the forward pass, loss calculation, gradient computation, and parameter update are clear, more advanced topics become easier to approach. Build a small example, inspect every intermediate value, and use the resulting model as a foundation for deeper study and experimentation.