Naive Bayes Classification
Mahmood Amintoosi, Fall 2026
Computer Science Dept, Ferdowsi University of Mashhad
The decision rule is the one of chapter 2 of Pattern Classification by Duda, Hart, and Stork (Duda et al. (2000)); the insect example is from a set of slides by Eamonn Keogh. The Gaussian and multinomial sections follow VanderPlas (VanderPlas (2016)).
Learning objectives. By the end you should be able to
write Bayes’ theorem for a classification problem and name its four terms;
classify an object from a histogram of one feature;
apply the naive assumption and compute a posterior from several discrete features;
explain what a single zero count does to the product, and fix it with Laplace smoothing;
fit and read a Gaussian and a multinomial naive Bayes classifier with Scikit-Learn;
say when naive Bayes is a good baseline and when its independence assumption fails.
Bayes’ Theorem¶
Bayes’ Theorem is a fundamental principle in probability theory that describes how to update our beliefs about an event based on new evidence. It provides a mathematical framework for calculating conditional probabilities, specifically the probability of an event occurring given that another event has already occurred. The theorem is named after the Reverend Thomas Bayes, who first formulated this principle in the 18th century.

Thomas Bayes 1702 - 1761
The mathematical formula for Bayes’ Theorem is expressed as:
Where:
is the posterior probability: the probability of event A occurring given that B is true
is the likelihood: the probability of event B occurring given that A is true
is the prior probability: the initial probability of event A
is the marginal probability: the total probability of event B occurring
Example with Dice¶
Suppose we have two dice: one fair die (with equal probability of rolling 1-6) and one loaded die that always shows 6. We randomly select one die and roll it, getting a 6. What is the probability that we selected the loaded die?
Let’s define our events:
Event A: Selecting the loaded die
Event B: Rolling a 6
Given:
(equal chance of selecting either die)
(loaded die always shows 6)
(fair die shows 6 with probability 1/6)
Applying Bayes’ Theorem:
So given that we rolled a 6, there is a (approximately 85.7%) probability that we selected the loaded die.
Bayes Classifiers¶
Bayes’ theorem turns a model of the data into a decision rule. For a class and an observed feature vector ,
is the prior: how frequent the class is before we look at the object.
is the likelihood, or class-conditional density: how typical the object is for that class.
is the posterior.
is the evidence. It is the same for all classes, so it only scales the posteriors to sum to one and plays no role in the decision.
For two classes the evidence cancels, and the decision depends on the product of likelihood and prior:
is the minimum-error decision rule. The name naive Bayes — also idiot Bayes or simple Bayes — covers the whole family of classifiers built on this rule. The idea in one sentence: find the probability of the unseen object for each class, then pick the largest.
The slides in the next two sections write for the class and for the observed object; here we write and , as in the Bayesian Decision Theory page.
A Visual Example: Grasshoppers and Katydids¶
by Eamonn Keogh, CS, UCR.
We have two species, grasshoppers and katydids, and one feature, antenna length. With enough data we can build one histogram of antenna length per species.


Classifying an Insect¶
We catch an insect whose antennae are 3 units long. Which species is it more probably?

Formally we want, for each class, the conditional probability
and we compare the two values.
Reading the Histogram¶
Counting the insects of the histogram at antenna length 3:

The same reading at antenna length 7 and at 5:


At 3 the insect is a grasshopper, at 7 a katydid, and at 5 the two classes are equally probable: is where the decision flips. These ratios are posteriors because the two counts at a given are divided by their sum, which is the evidence of the previous section.
Example: Classifying by Name¶
Two classes, and , and one discrete feature, the first name. We meet a person called Drew and ask which posterior is larger, or . Our whole database of names and sexes is:
| Name | Sex |
|---|---|
| Drew | Male |
| Claudia | Female |
| Drew | Female |
| Drew | Female |
| Alberto | Male |
| Karin | Female |
| Nina | Female |
| Sergio | Male |

Computing the Posteriors¶
From the table, and , and inside each class
The evidence is the same for the two classes, so it is enough to compare the numerators:
The minimum-error decision for Drew is female ().


Several Features and the Naive Assumption¶
With one feature we could read the posteriors off a histogram. With features the joint density is a table over all combinations of feature values, and that table grows exponentially with . The naive Bayes assumption is that the features are conditionally independent given the class:
so the decision rule needs only one-dimensional distributions per class:
The assumption is almost never true — in a sentence the words are obviously not independent — but it keeps the number of parameters linear in , and it usually produces a useful ranking of the classes even when the probabilities themselves are estimated badly.
Worked Example: Drew with Three Features¶
| Name | Over 170 cm | Eye | Hair | Sex |
|---|---|---|---|---|
| Drew | No | Blue | Short | Male |
| Claudia | Yes | Brown | Long | Female |
| Drew | No | Blue | Long | Female |
| Drew | No | Blue | Long | Female |
| Alberto | Yes | Brown | Short | Male |
| Karin | No | Blue | Long | Female |
| Nina | Yes | Brown | Short | Female |
| Sergio | Yes | Blue | Long | Male |
Take the query — tall, brown-eyed, long hair — and count inside each class:
| Feature value | ||
|---|---|---|
| Over 170 = Yes | ||
| Eye = Brown | ||
| Hair = Long |
With the priors and the two scores are
Dividing by the sum gives the posteriors 0.42 and 0.58, and the decision is female. Note what the arithmetic did: the single feature that is much more likely for females (long hair, ) outweighed three features that all favour males.
The Zero-Count Problem¶
Now query . Among the four females nobody has short hair, so and the product for females is exactly zero, whatever the other two features say. In the log domain this is worse than a small probability: .
The standard remedy is Laplace (add-one) smoothing,
where counts the value of feature in class and is the number of
values that feature can take. With +1 in every cell no likelihood is ever zero: a value that
was never seen gets a small probability instead of a veto. The Scikit-Learn implementations do
this for you (MultinomialNB(alpha=1)); for the Gaussian model the analogous guard is
GaussianNB(var_smoothing=...).
Worked Example: Classifying Fruit¶
Adapted from this Stack Overflow answer. The training set has 1000 fruits, three binary features (long, sweet, yellow) and three classes:
| Type | Long | Not long | Sweet | Not sweet | Yellow | Not yellow | Total |
|---|---|---|---|---|---|---|---|
| Banana | 400 | 100 | 350 | 150 | 450 | 50 | 500 |
| Orange | 0 | 300 | 150 | 150 | 300 | 0 | 300 |
| Other fruit | 100 | 100 | 150 | 50 | 50 | 150 | 200 |
| Total | 500 | 500 | 650 | 350 | 800 | 200 | 1000 |
Priors and Likelihoods¶
The priors are the class totals: , and . Dividing each cell by its row total gives the likelihoods:
| Feature value | Banana | Orange | Other fruit |
|---|---|---|---|
| Long | 0.80 | 0.00 | 0.50 |
| Sweet | 0.70 | 0.50 | 0.75 |
| Yellow | 0.90 | 1.00 | 0.25 |
The column totals of the training table — 500 long, 650 sweet, 800 yellow — are the feature marginals. They are not needed for the decision, because the evidence is the same for the three classes and cancels in the comparison.
Classifying One Fruit¶
A new fruit is long, sweet and yellow. Multiplying prior and likelihoods:
| Class | |
|---|---|
| Banana | |
| Orange | |
| Other fruit |
These are unnormalised scores: they are proportional to the posteriors, with the common factor . Dividing by their sum, , gives the posteriors
and the decision is banana.
Why the Zero Matters¶
The orange column has two zeros, and they are structural rather than accidental: no orange is long, and no orange is other than yellow. As soon as a query contains one of those values the orange score is exactly zero, and no amount of evidence from the other features can revive it. Add-one smoothing from the previous section replaces 0 by and keeps the class alive. The same product is usually computed as a sum of logarithms,
which also avoids underflow when many small probabilities are multiplied.
Gaussian Naive Bayes¶
Gaussian naive Bayes takes every feature to be normally distributed inside each class, . The parameters are one mean and one variance per feature per class, and the two Gaussian parameters are all that training computes. That per-feature model is exactly where the naive assumption sits: the class conditional density has no covariance terms, so its ellipses are axis-aligned.
The figure below shows such a model fitted to two blobs: one ellipse per class, with the probability largest at the class centre.
%matplotlib inline
import numpy as np
import matplotlib.pyplot as plt
import seaborn as sns
plt.style.use('seaborn-v0_8-whitegrid') # the old 'seaborn-whitegrid' name was removed in Matplotlib 3.6;from sklearn.datasets import make_blobs
X, y = make_blobs(100, 2, centers=2, random_state=2, cluster_std=1.5)
plt.scatter(X[:, 0], X[:, 1], c=y, s=50, cmap='RdBu');

Static copy of the figure of VanderPlas (2016); the cells above reproduce it.
The ellipses here represent the Gaussian generative model for each label, with larger probability toward the center of the ellipses. With this generative model in place for each class, we have a simple recipe to compute the likelihood for any data point, and thus we can quickly compute the posterior ratio and determine which label is the most probable for a given point.
This procedure is implemented in Scikit-Learn’s sklearn.naive_bayes.GaussianNB estimator:
from sklearn.naive_bayes import GaussianNB
model = GaussianNB()
model.fit(X, y);Let’s generate some new data and predict the label:
rng = np.random.RandomState(0)
Xnew = [-6, -14] + [14, 18] * rng.rand(2000, 2)
ynew = model.predict(Xnew)Now we can plot this new data to see where the decision boundary falls:
plt.scatter(X[:, 0], X[:, 1], c=y, s=50, cmap='RdBu')
lim = plt.axis()
plt.scatter(Xnew[:, 0], Xnew[:, 1], c=ynew, s=20, cmap='RdBu', alpha=0.1)
plt.axis(lim);
We see a slightly curved boundary in the classifications: in general, the boundary produced by a Gaussian naive Bayes model is quadratic.
A nice aspect of this formalism is that it also gives a probabilistic classification, which we can read with the predict_proba method:
yprob = model.predict_proba(Xnew)
yprob[-8:].round(2)array([[0.89, 0.11],
[1. , 0. ],
[1. , 0. ],
[1. , 0. ],
[1. , 0. ],
[1. , 0. ],
[0. , 1. ],
[0.15, 0.85]])The columns give the posterior probabilities of the first and second labels, respectively. If you are looking for estimates of uncertainty in your classification, Bayesian approaches like this can be a good place to start.
Of course, the final classification will only be as good as the model assumptions that lead to it, which is why Gaussian naive Bayes often does not produce very good results. Still, in many cases—especially as the number of features becomes large—this assumption is not detrimental enough to prevent Gaussian naive Bayes from being a reliable method.
Multinomial Naive Bayes¶
The Gaussian assumption is not the only simple one. Multinomial naive Bayes takes each feature to be generated from a multinomial distribution, which describes the probability of observing counts across a number of categories. It is therefore the right choice when the features are counts or count rates — most often word counts in a document.
The recipe is unchanged: replace the best-fit Gaussian by the best-fit multinomial and plug the likelihoods into the same decision rule.
Example: Classifying Text¶
One place where multinomial naive Bayes is used is text classification, where the features are word counts or frequencies inside the documents. Here we use the sparse word count features of the 20 Newsgroups corpus, available through Scikit-Learn.
The first call downloads the corpus (about 20 MB) and caches it under ~/scikit_learn_data, so the cells below need a network connection only once:
from sklearn.datasets import fetch_20newsgroups
data = fetch_20newsgroups()
data.target_names['alt.atheism',
'comp.graphics',
'comp.os.ms-windows.misc',
'comp.sys.ibm.pc.hardware',
'comp.sys.mac.hardware',
'comp.windows.x',
'misc.forsale',
'rec.autos',
'rec.motorcycles',
'rec.sport.baseball',
'rec.sport.hockey',
'sci.crypt',
'sci.electronics',
'sci.med',
'sci.space',
'soc.religion.christian',
'talk.politics.guns',
'talk.politics.mideast',
'talk.politics.misc',
'talk.religion.misc']For simplicity here, we will select just a few of these categories and download the training and testing sets:
categories = ['talk.religion.misc', 'soc.religion.christian',
'sci.space', 'comp.graphics']
train = fetch_20newsgroups(subset='train', categories=categories)
test = fetch_20newsgroups(subset='test', categories=categories)Here is a representative entry from the data:
print(train.data[5][48:])Subject: Federal Hearing
Originator: dmcgee@uluhe
Organization: School of Ocean and Earth Science and Technology
Distribution: usa
Lines: 10
Fact or rumor....? Madalyn Murray O'Hare an atheist who eliminated the
use of the bible reading and prayer in public schools 15 years ago is now
going to appear before the FCC with a petition to stop the reading of the
Gospel on the airways of America. And she is also campaigning to remove
Christmas programs, songs, etc from the public schools. If it is true
then mail to Federal Communications Commission 1919 H Street Washington DC
20054 expressing your opposition to her request. Reference Petition number
2493.
In order to use this data for machine learning, we need to be able to convert the content of each string into a vector of numbers. For this we will use the TF-IDF vectorizer, and create a pipeline that attaches it to a multinomial naive Bayes classifier:
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.naive_bayes import MultinomialNB
from sklearn.pipeline import make_pipeline
model = make_pipeline(TfidfVectorizer(), MultinomialNB())With this pipeline, we can apply the model to the training data and predict labels for the test data:
model.fit(train.data, train.target)
labels = model.predict(test.data)Now that we have predicted the labels for the test data, we can evaluate the estimator. For example, the confusion matrix between the true and the predicted labels:
from sklearn.metrics import confusion_matrix
mat = confusion_matrix(test.target, labels)
sns.heatmap(mat.T, square=True, annot=True, fmt='d', cbar=False,
xticklabels=train.target_names, yticklabels=train.target_names,
cmap='Blues')
plt.xlabel('true label')
plt.ylabel('predicted label');
Evidently, even this very simple classifier can successfully separate space discussions from computer discussions, but it gets confused between discussions about religion and discussions about Christianity. This is perhaps to be expected!
The cool thing here is that we now have the tools to determine the category for any string, using the predict method of this pipeline.
Here’s a utility function that will return the prediction for a single string:
def predict_category(s, train=train, model=model):
pred = model.predict([s])
return train.target_names[pred[0]]Let’s try it out:
predict_category('sending a payload to the ISS')'sci.space'predict_category('discussing the existence of God')'soc.religion.christian'predict_category('determining the screen resolution')'comp.graphics'Remember that this is nothing more sophisticated than a simple probability model for the (weighted) frequency of each word in the string; nevertheless, the result is striking. Even a very naive algorithm, when used carefully and trained on a large set of high-dimensional data, can be surprisingly effective.
Naive Bayes versus k-NN, and When to Use It¶
k-NN and naive Bayes are both classifiers, but they build “similarity” differently. k-NN models nearness — Euclidean or cosine distance — and keeps the training data around; naive Bayes models probability — a generative distribution per class — and keeps only its parameters.
Because its assumptions are so strong, naive Bayes will rarely beat a more flexible model on the same data. What it offers instead:
training and prediction are a single pass over counts or means;
the output is a probability, not only a label;
the parameters are readable: they say which feature favoured which class;
there is essentially nothing to tune.
That makes it a good first baseline, as noted in Hastie et al. (2009). If it does well you have a very fast and very interpretable classifier; if it does not, you have a reference point for how much a more complex model actually buys. It tends to do particularly well
when the independence assumption more or less holds (rare);
for well-separated classes, where model complexity matters little;
in very high dimension, where flexible models need far more data than are available.
The last two points are related: in high dimension two points are rarely close, so clusters are further apart, and a simple model can separate them.
Summary¶
Naive Bayes classifies by the posterior, , and the naive assumption factorises the likelihood over the features. Everything else is a choice of the one-dimensional model:
discrete features: count the values, with Laplace smoothing;
continuous features: a Gaussian per feature per class (Gaussian NB);
word counts: a multinomial per class (multinomial NB), usually on TF-IDF features.
Two warnings. The independence assumption makes the estimated likelihood of the evidence too small, so the probabilities are usually over-confident; the ranking of the classes is much better than the numbers attached to it. And a single unseen feature value zeroes a class unless it is smoothed.
- Duda, R. O., Hart, P. E., & Stork, D. G. (2000). Pattern Classification (2nd ed.). Wiley-Interscience.
- VanderPlas, J. (2016). Python Data Science Handbook: Essential Tools for Working with Data. O’Reilly Media. https://jakevdp.github.io/PythonDataScienceHandbook/
- Hastie, T., Tibshirani, R., & Friedman, J. (2009). The elements of statistical learning: data mining, inference and prediction (2nd ed., p. 764). Springer. https://hastie.su.domains/ElemStatLearn/