A Guide to the Shapley Value for Feature Importance
Machine learning models can predict accurately while remaining difficult to explain. A random forest may combine hundreds of trees, and a neural network may transform inputs through thousands of learned parameters. When a model predicts that an Australian loan applicant has a high default risk or that a power-demand forecast will rise, people often want to know which features contributed to that result.
The Shapley value offers a principled way to assign credit among features. It comes from cooperative game theory, where players work together to produce a result. In explainable artificial intelligence, the “players” are input features and the “payout” is a model prediction. The method is powerful, though its assumptions, computational cost, and handling of correlated variables deserve careful attention.
Why Feature Attribution Matters
Feature importance is often treated as a single ranking, but there are several different questions hiding behind that phrase. Global importance asks which variables matter across a dataset. Local explanation asks why one particular prediction was high, low, or different from the average. A model can rank postcode as important globally while relying on income and account history for one individual prediction.
Shapley-based explanations are useful because they distribute a prediction’s difference from a reference value across the available features. For a classification model, the reference might be the average predicted probability. For a regression model, it could be the mean prediction. Each feature then receives an attribution showing how it moved the output away from that baseline.
This distinction matters in practical settings. A council analysing rubbish collection, a retailer in Melbourne forecasting demand, or an insurer examining claims may need both population-level patterns and case-specific reasoning. A local explanation should not be mistaken for a causal statement: a large Shapley value says that a feature helped the model produce its output, not that changing the feature would necessarily change the real-world outcome.
The Cooperative Game Behind Shapley Values
Suppose a model uses four features: age, income, credit history, and debt ratio. To calculate the contribution of income, we compare predictions made with income included against predictions made without it. However, the result depends on which other features are already present. Income may add little after credit history is known, yet add a great deal when considered before credit history.
The Shapley value resolves this order problem by averaging a feature’s marginal contribution across every possible ordering of the features. If (N) is the full feature set and (S) is a subset that does not contain feature (i), the value for (i) is:
[ \phi_i = \sum_{S \subseteq N \setminus {i}} \frac{|S|!(|N|-|S|-1)!}{|N|!} \left[v(S \cup {i}) - v(S)\right] ]
Here, (v(S)) is the model’s value when it receives the subset (S). The factorial weight ensures that every feature ordering receives equal treatment. In practice, the exact calculation requires evaluating many coalitions, which becomes expensive as the number of features grows.
The method has attractive properties. Efficiency means the feature attributions add up to the difference between the prediction and the baseline. Symmetry gives identical contributions to features that behave identically. The dummy property assigns zero to a feature that never changes the output, while additivity allows explanations from separate models to be combined. These guarantees make Shapley values more defensible than an arbitrary feature ranking.
Exact Calculation And Practical Approximations
For a model with (n) features, the number of feature subsets is (2^n). Exact Shapley computation therefore becomes impractical for many models. A brute-force implementation can also be slow because each subset may require a new model prediction. Understanding this growth is easier with asymptotic notation guide, especially when comparing exact enumeration with sampling-based methods.
Several approaches reduce the cost. KernelSHAP estimates Shapley values by sampling coalitions and fitting a weighted linear model around the prediction. TreeSHAP exploits the structure of decision trees and ensembles to calculate values much more efficiently, often with strong theoretical guarantees. DeepSHAP adapts related ideas for neural networks, while permutation-based methods estimate importance by disrupting feature values.
The word “missing” needs careful interpretation. Most tabular models cannot literally receive a row with a feature erased unless the software defines a special representation. A common solution is to replace absent features with values drawn from a background dataset. Conditional approaches instead sample missing features based on the observed features. These choices can produce different explanations, particularly when variables are strongly correlated.
| Method | Common model type | Main strength | Main limitation |
|---|---|---|---|
| Exact Shapley values | Small models or few features | Faithful to the formal definition | Exponential computational cost |
| KernelSHAP | Model-agnostic | Works with many black-box predictors | Can be slow and sensitive to background data |
| TreeSHAP | Decision trees and ensembles | Fast and structurally informed | Specific to tree-based models |
| DeepSHAP | Neural networks | Useful for layered differentiable models | Depends on approximations and reference choices |
| Permutation importance | Any predictive model | Simple global comparison | Can mislead with correlated features |
A Reliable Workflow For Model Explanations
Begin by defining the prediction output. A binary classifier may explain raw log-odds, probability, or a transformed score. These quantities are different, so a statement such as “income contributed 0.12” is incomplete without naming the output scale. For many models, explaining the raw score gives additive results, while probability explanations are easier for non-technical audiences.
Next, select a representative background dataset. It should reflect the population against which predictions are being compared rather than a convenient handful of rows. An explanation for a regional Australian energy customer can change if the baseline contains mostly Sydney households instead of a mixture of metropolitan, regional, and remote customers. Feature distributions, missing values, and seasonal patterns should be checked before calculating attributions.
Validation is part of explanation quality. A feature that appears influential in an overfit model may have little value on new data. Use a sound evaluation design and review cross-validation techniques before interpreting feature effects. For time-dependent data, random folds can leak future information, so blocked or rolling validation may be more appropriate.
A useful implementation records the model version, training data period, background sample, explainer type, output scale, and software version. Store these alongside the attribution results. If a bank changes its lending policy or an Australian retailer changes its product mix, old explanations may no longer describe current model behaviour. Reproducibility turns visual explanations into auditable evidence.
Interpreting Results Without Overclaiming
A positive Shapley value pushes a prediction above the chosen baseline, while a negative value pushes it below. The size describes contribution on the selected output scale. It does not automatically represent percentage change, statistical significance, fairness, or a causal effect. A bar chart with red and blue features can communicate direction quickly, but it cannot replace knowledge of the data-generating process.
Correlated variables create one of the hardest interpretation problems. If floor area and number of bedrooms usually increase together, the model may divide credit between them in a way that varies with the background distribution. If postcode encodes several socioeconomic and geographic signals, its attribution may reflect access, demographics, transport, or historical decisions rather than a meaningful individual characteristic.
Interactions require extra care. A feature may have a small average effect but become important when paired with another variable. For example, temperature and time of day may jointly influence electricity demand in Queensland. Interaction values or carefully designed partial dependence analyses can reveal this pattern, but they should be checked against observed data rather than presented as universal rules.
In Australia, explanation practice also sits within a setting shaped by privacy expectations, sector regulation, and uneven data coverage. A model trained on metropolitan records may perform differently in the Northern Territory or rural Western Australia. Terms such as “postcode,” “uni,” and “arvo” can be familiar locally, yet communication should still use precise definitions when explanations affect eligibility, pricing, or access to services. Background data and feature engineering deserve as much scrutiny as the final chart.
Practical Rules For Trustworthy Use
Shapley values work best as one part of a broader model review. Compare global summaries with local explanations, inspect unusual records, test stability across background samples, and ask whether the displayed contributions make sense for the model’s intended use. A polished plot can still describe a flawed model or a biased dataset.
Teams should also document who will see the explanation and what decision it may influence. A data scientist may understand log-odds and baseline distributions, while a customer or policy officer may need a plain-language account of the same result. The hello ML community reflects the value of open technical learning: explanations become more useful when their assumptions and implementation details are available for careful review.
- Define the prediction scale and baseline before calculating attributions.
- Use a background dataset that represents the relevant population and time period.
- Select an explainer suited to the model, such as TreeSHAP for tree ensembles.
- Check correlated features, missing-value handling, and possible interaction effects.
- Compare explanations across validation folds, demographic groups, and time periods.
- Treat attribution as model evidence, not proof that a feature caused the outcome.
A sound Shapley analysis therefore combines mathematics, software knowledge, and domain judgement. It can show why a particular model output differs from its baseline and can expose surprising behaviour that ordinary accuracy metrics hide. Its value depends on transparent assumptions: what counts as a missing feature, which reference population is used, what output is explained, and whether the model itself remains reliable outside the training data.