Understanding the naive bayes classifier for text classification
Text classification sits at the heart of countless products Australians use every day, from the spam filter quietly sorting junk from your Telstra inbox to the algorithm deciding whether a customer support ticket to Bunnings should land with refunds or deliveries. Among the family of algorithms that tackle this problem, the naive Bayes classifier remains one of the oldest, the simplest, and surprisingly the most reliable for many production pipelines. It learns from labelled examples and uses probability theory to assign each new document to the most likely category.
The "naive" label does not mean the model is foolish. It refers to a deliberately simplified assumption about how words relate to one another. Despite that simplification, the classifier often outperforms more sophisticated models on small or moderately sized text corpora, and it runs fast enough to score millions of documents per minute on a laptop. This makes it a popular choice for Australian startups that need to ship a text classification feature without investing in GPU infrastructure.
The probability foundation behind the model
Every naive Bayes classifier is a direct application of Bayes' theorem, a rule that updates the probability of an event when new evidence appears. In the text classification setting, the event is a class label such as "spam" or "not spam", and the evidence is the bag of words in a document. The theorem rewrites the posterior probability, what we actually want, in terms of three quantities we can estimate from training data: the prior probability of each class, the likelihood of the document given that class, and the evidence, which acts as a normalising constant.
The training step estimates those quantities by counting. The prior is just the fraction of training documents that belong to each category. The likelihood of a word given a class is the fraction of times that word appears in documents of that class, adjusted for words that may never appear. Once the counts are collected, classification becomes a matter of computing the product of likelihoods for every word in the new document, multiplying by the prior, and picking the class with the largest value.
From raw text to feature vectors
Text is messy. Before any classifier can use it, raw strings have to be converted into numeric features. The most common representation for naive Bayes is the bag-of-words model, which throws away word order and counts how often each term appears in a document. Australian English introduces a few wrinkles here: the algorithm needs to know that "arvo" and "afternoon" are interchangeable in casual writing, that "footy" can refer to AFL, NRL, or A-League depending on the city, and that place names like "the Gong" mean Wollongong. Tokenisation, lowercasing, and stopword removal are standard preprocessing steps that strip away punctuation and very common words so the classifier can focus on the discriminative vocabulary.
TF-IDF, short for term frequency–inverse document frequency, refines the bag-of-words idea by down-weighting words that appear in almost every document and up-weighting words that appear in only a few. In a collection of Australian news articles, words like "the", "a", and "of" carry little signal because they appear everywhere, while proper nouns such as "Albo", "Gaza", or "Ashes" carry strong signal about the topic. For readers who want hands-on exercises that walk through tokenisation and vectorisation, the hello ML educational blog maintains a solid set of worked examples.
Comparing the main variants
Not all naive Bayes classifiers are designed for the same kind of input. Text classification uses two variants predominantly, and a third occasionally. The summary below highlights the differences.
| Variant | Best input type | Core statistic | Typical use case |
|---|---|---|---|
| Multinomial naive Bayes | Word counts or TF-IDF values | Probability of word count given class | Document topic classification, spam filtering |
| Bernoulli naive Bayes | Binary presence or absence of each word | Probability of word being present given class | Short messages, sentiment with yes or no features |
| Complement naive Bayes | Word counts, imbalanced classes | Probability of word in complement class | News categorisation with skewed training data |
| Gaussian naive Bayes | Continuous real-valued features | Mean and variance of feature per class | Rarely used directly for raw text |
Multinomial naive Bayes is the default choice for most text classification tasks because it directly consumes word counts and TF-IDF scores. Bernoulli naive Bayes treats every word as either present or absent, which can be helpful when documents are very short, such as SMS or tweet-length messages. Complement naive Bayes is a useful adjustment when one class dominates the training set, which is common in Australian spam filters where legitimate mail vastly outnumbers junk.
Training the model and making predictions
The training procedure for multinomial naive Bayes is short enough to fit on a napkin. The algorithm iterates through every document in the training set, accumulates the word count per class, and stores the prior for each class. A common trick during training is Laplace smoothing, which adds a small constant, often one, to every word count. This prevents the classifier from assigning a probability of zero to a word it has simply not seen before, the so-called zero-frequency problem. Without smoothing, a single unseen word in a test document would multiply the entire likelihood by zero, wiping out the prediction.
At prediction time, the algorithm computes a score for each class by adding the log-probability of every word in the document, plus the log-prior of the class. Working in log space avoids numerical underflow, because multiplying many small probabilities can easily produce a number smaller than floating-point precision can represent. The class with the largest score wins. Pseudocode for this entire pipeline, alongside complexity analysis and implementation guidance in Python, is available through the problem-solving section of the hello ML site.
Strengths, pitfalls, and smoothing tricks
The classifier earns its longevity from a handful of practical advantages, but it also has limitations worth knowing before deploying it.
- Training and prediction are both linear in the number of documents and features, so it scales gracefully to millions of examples.
- It handles high-dimensional sparse data well, which is exactly the shape of bag-of-words and TF-IDF representations.
- It is interpretable: the per-class word probabilities can be opened and inspected, which helps when explaining decisions to non-technical stakeholders.
- It assumes feature independence, which is rarely true in practice, because words co-occur, idioms cluster, and grammar matters.
- It can be confidently wrong when the test distribution drifts away from the training distribution, such as when new slang enters Australian Twitter.
- Correlated features can double-count evidence and skew the posterior, so feature engineering still matters.
The independence assumption is the biggest weakness, but in many text problems the assumption is "wrong yet useful". Empirically, the classifier often produces calibrated probabilities even when the assumption is violated, because the errors roughly cancel out across thousands of words. Feature selection on the top few thousand tokens by mutual information often sharpens accuracy further, and complement naive Bayes can recover some of the performance lost to imbalance.
Where the model shows up in Australian contexts
Australian organisations have deployed naive Bayes for a wide range of text classification tasks, often without ever advertising that they are running a probabilistic model.
- Telstra and Optus use variants of naive Bayes in their SMS spam filters, which is why "click here to claim your prize" rarely reaches Australian phones.
- The ABC classifies incoming news tips and reader comments into topics so editors can route them to the right desk.
- myGov and Services Australia sort customer feedback to triage complaints and improve service delivery.
- AFL and NRL fan forums deploy the algorithm to flag abusive comments before moderators see them.
- Australian e-commerce sites such as Kogan and Catch use it to categorise product reviews and detect fake testimonials.
- University admissions teams, including those at Melbourne and Sydney, use it to triage enquiry emails for outcome-based triage.
For readers with feedback or suggestions, the site's contact page connects them directly with the team.