Understanding ridge regression for regularised linear models

Ridge regression is a practical way to fit a linear model when predictors are numerous, correlated or measured at different scales. It keeps the familiar prediction rule of ordinary least squares but adds a penalty that pulls coefficients towards zero. The aim is not to make the training fit perfect; it is to reduce prediction variance on data the model has not seen.

The method appears in many working modelling tasks, from forecasting weekly apartment rents in Sydney and Melbourne to estimating demand for products sold in Australian dollars. This article develops the objective, explains how the penalty behaves, and shows how to tune and implement the method in Python. It also places ridge beside related approaches so its role is easy to judge.

Why shrinkage matters for real datasets

Linear models become fragile when columns of a dataset repeat similar information. A property-price example might include floor area, number of bedrooms, bathrooms, garage spaces, distance from a transport stop and postcode. If two predictors move almost in lockstep, least squares can produce large coefficients with opposite signs even when the underlying relationship is modest.

That instability is multicollinearity, and it matters in Australian housing data because location, property type and nearby amenities overlap. A model may look excellent on its training rows while making erratic predictions for a new listing. Shrinkage limits how strongly individual coefficients can swing, trading a little bias for more stable generalisation.

Ridge is therefore a compromise, not a magic repair for every dataset. It cannot fix a biased sample, meaningless features or a nonlinear relationship. Its clearest advantage appears when there are many related predictors and the goal is accurate numerical prediction rather than selecting a short list of variables.

From least squares to a penalised objective

Start with ordinary least squares. For observations ((x_i, y_i)), the model predicts (y_i) using an intercept (\beta_0) and slopes (\beta_1,\ldots,\beta_p). Least squares chooses these values to minimise the sum of squared residuals, (\sum_i (y_i-\hat y_i)^2).

Ridge changes the optimisation problem by adding an (L_2) penalty on the slopes:

[ \hat{\boldsymbol{\beta}}

\arg\min_{\boldsymbol{\beta}} \left[ \sum_i (y_i-\beta_0-\mathbf{x}i^\top\boldsymbol{\beta})^2 + \lambda\sum{j=1}^{p}\beta_j^2 \right]. ]

The hyperparameter (\lambda), called alpha in scikit-learn, controls the trade-off. A value of zero recovers least squares; larger values penalise coefficient size more strongly. The intercept is normally excluded from the penalty because shrinking it would move the baseline prediction towards zero.

With centred, standardised predictors and the intercept handled separately, the slopes have the closed form

[ \hat{\boldsymbol{\beta}}

(\mathbf{X}^{\top}\mathbf{X}+\lambda\mathbf{I})^{-1} \mathbf{X}^{\top}\mathbf{y}. ]

This expression reveals why ridge can work when the number of predictors exceeds the number of observations: adding a positive multiple of the identity matrix makes the system invertible. In practice, numerical solvers or a library implementation are safer than forming this matrix directly.

How the penalty changes model behaviour

The penalty has a geometric interpretation. Least squares searches for the coefficient vector that lands closest to the observed outcomes, whereas ridge searches for a nearby vector with a smaller squared length. Correlated predictors can divide a shared effect between them, and ridge keeps that division moderate instead of rewarding extreme coefficients.

The smooth (L_2) penalty generally does not send a coefficient to exactly zero. It shrinks steadily, so predictors that would otherwise receive very large weights receive smaller ones. Standardisation is crucial: if one feature is recorded in cents and another in dollars, otherwise equivalent changes would attract wildly different penalties.

Lambda should be chosen for the data rather than by habit. Too little regularisation leaves the model close to least squares; too much makes it underfit and flatten genuine relationships. A sensible baseline is no regularisation, followed by a sequence of stronger penalties evaluated on validation data.

Choosing the regularisation strength

Use cross-validation to select the regularisation strength, keeping the final test set untouched. Fit scaling only on each training fold, then predict its validation fold. Search over a logarithmic grid of lambda values, such as (10^{-4}) to (10^4), because useful penalties often span several orders of magnitude. Select the value that minimises validation error rather than the value that gives the smallest training error.

Method	Added penalty	Coefficient pattern	Main strength	Main limitation
Ordinary least squares	None	Coefficients may be large or unstable	Simple baseline for modest, independent predictors	Highly sensitive to multicollinearity and noise
Ridge regression	(\lambda\sum_j\beta_j^2)	Smooth shrinkage; correlated effects are shared	Stable predictions with many related inputs	Requires consistent scaling and careful tuning
Lasso regression	(\lambda\sum_j	\beta_j	)	Coefficients can become exactly zero

The right error metric depends on the decision attached to the prediction. Mean absolute error is easy to explain in Australian dollars when estimating a sale price; root mean squared error gives extra weight to large misses. For an operational forecast, compare ridge with a simple baseline such as the historical mean, and inspect errors by region rather than relying only on one aggregate score.

Interpreting coefficients without overclaiming

A ridge coefficient is a conditional slope: the expected change in the response for a one-unit change in that predictor, holding the other included predictors constant. If price is the target, a hypothetical coefficient of (18{,}000) means $18,000 more per unit of the predictor after adjustment. Because shrinkage changes magnitude, these coefficients are not causal effects and should not be presented as such.

Correlation among predictors also complicates interpretation. Distance from the central business district and postcode may encode overlapping information, so their separate effects can depend on how the model is specified. Report which features were included, how they were transformed and whether the slopes use original or standardised units. Local communication matters here too: an Australian team may call a simple benchmark a “fair dinkum sanity check”, making it clear that a complex model has not earned its place merely by looking sophisticated.

Residual plots remain important. Systematic curves suggest that a straight-line relationship is inadequate, while widening fans suggest non-constant variance. Adding sensible transformations or interactions may help, although each new term increases the modelling burden. Ridge regularises the chosen representation; it does not discover the correct functional form.

A practical Python workflow

Scikit-learn expresses the method through a pipeline, which keeps preprocessing consistent between training and prediction. The following example assumes a pandas DataFrame named df contains cleaned property records and predicts sale prices in Australian dollars:

import numpy as np
from sklearn.linear_model import Ridge
from sklearn.metrics import mean_absolute_error, mean_squared_error
from sklearn.model_selection import train_test_split
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler

features = ["floor_area_m2", "bedrooms", "bathrooms", "garage_spaces"]
X = df[features]
y = df["price_aud"]

X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42
)

model = Pipeline([
    ("scale", StandardScaler()),
    ("ridge", Ridge(alpha=1.0)),
])

model.fit(X_train, y_train)
predictions = model.predict(X_test)

mae = mean_absolute_error(y_test, predictions)
rmse = np.sqrt(mean_squared_error(y_test, predictions))

Here, the library's alpha parameter is the regularisation strength described as lambda above. The initial value is only a starting point. StandardScaler gives each training feature zero mean and unit variance, preventing units from deciding which coefficient receives the stronger penalty.

After fixing the pipeline, search for alpha with cross-validation and evaluate the selected model once on the test set. A grid search over np.logspace(-4, 4, 81) is a common starting pattern, although noisy data or limited samples may justify a narrower grid and repeated evaluation. If the goal is rental rather than sale prices, fit the target and metric to that market, including whether weekly or monthly rent is the intended unit.

A university or community group wanting to reproduce the exercise can find publishing and contribution details on the community contact page. Such sessions often work well when participants tune the penalty before seeing test results, then explain which coefficients changed.

Where ridge fits and where it fails

Ridge is not a universal replacement for other models. It suits continuous outcomes, substantial predictor overlap and a need for transparent, reproducible calculations. Classification targets require a different loss, while nonlinear patterns may be captured more naturally by trees or a carefully validated feature transformation. Even a well-tuned ridge model should earn its place by beating simple and nonlinear baselines on held-out data.

When hundreds of engineered variables are involved, dimensionality reduction is another option. principal component analysis creates orthogonal combinations of features rather than shrinking the original coefficients. That can improve numerical behaviour, but component loadings are harder to explain and may obscure which attributes matter to a domain expert.

Regulation and organisational context also shape adoption. An Australian bank or insurer may value a stable credit-risk or premium model because decisions affect customers and must be explainable, yet fairness checks and monitoring remain necessary. A team in Canberra should also test whether a model trained on Sydney or Melbourne behaviour transfers to Perth or Adelaide; a lower average error does not guarantee local reliability.

Ridge regression's lasting strength is its balance: it retains the interpretability of a linear prediction rule while reducing the instability that unconstrained coefficients can suffer. Used with disciplined preprocessing, cross-validation and honest reporting, (L_2) regularisation offers a dependable baseline and sometimes the best final model for the job.