Skip to article frontmatterSkip to article content
Site not loading correctly?

This may be due to an incorrect BASE_URL configuration. See the MyST Documentation for reference.

Bayesian Decision Theory

Mahmood Amintoosi, Fall 2026

Computer Science Dept, Ferdowsi University of Mashhad

This page follows chapter 2 of Pattern Classification by Duda, Hart, and Stork (Duda et al. (2000)), including its figures and the fish example. The estimates that fill in the densities come from the MLE page; the geometry of the Gaussian case is worked out on the Mahalanobis distance page, and the naive assumption that makes the densities tractable is the subject of Naive Bayes.

Learning objectives. By the end you should be able to

  1. write Bayes’ formula for a classification problem and say which factor is a prior, a likelihood, an evidence and a posterior;

  2. define a loss matrix, the conditional risk, and show that the zero-one loss gives the minimum error rate rule;

  3. derive the discriminant function for Gaussian classes in the three cases of Duda’s chapter — isotropic, equal, and arbitrary covariance matrices;

  4. say what the decision boundary looks like in each case: a hyperplane orthogonal to the line between the means, a general hyperplane, or a quadric surface;

  5. read the geometry off the eigenvalues of the covariance matrix.

English · فارسی
English · فارسی

Introduction

  • Bayesian Decision Theory is a statistical approach to pattern classification.

  • It quantifies tradeoffs between classification decisions using probabilities and costs.

  • Assumes all relevant probabilities are known.

  • Example: Classifying fish (sea bass vs. salmon) based on features like lightness.


Key Concepts

  • State of Nature (ω \omega ): Represents the true category (e.g., sea bass or salmon).

  • Prior Probability (P(ωj) P(\omega_j) ): Probability of a category before observing any data.

  • Class-Conditional Probability Density (p(x∣ωj) p(\mathbf{x} \mid \omega_j) ): Probability density of the observation x \mathbf{x} given category ωj \omega_j .

  • Posterior Probability (P(ωj∣x) P(\omega_j \mid \mathbf{x}) ): Probability of category ωj \omega_j after observing the features x \mathbf{x} .


Bayes’ Formula

  • Bayes’ Theorem:

    P(ωj∣x)=p(x∣ωj) P(ωj)p(x)P(\omega_j \mid \mathbf{x}) = \frac{p(\mathbf{x} \mid \omega_j)\,P(\omega_j)}{p(\mathbf{x})}

    that is, posterior = likelihood × prior / evidence. The evidence (p(x) p(\mathbf{x}) ) is the normalising constant, and it is often ignored in classification because it is the same for every class.

    p(x)=∑j=1cp(x∣ωj) P(ωj)p(\mathbf{x}) = \sum_{j=1}^c p(\mathbf{x} \mid \omega_j)\,P(\omega_j)

The fish example below is one-dimensional: there the observation x\mathbf{x} is a single feature xx, the lightness of the fish, and p(x∣ωj)p(x \mid \omega_j) is a curve rather than a surface.

Reading the Formula

It is the product of likelihood and prior that decides the posterior; the evidence p(x)p(\mathbf{x}) is only a scale factor, present to make the posteriors sum to one. The likelihood p(x∣ωj)p(\mathbf{x} \mid \omega_j) is so called because, other things being equal, the class for which it is large is the more likely one.

Figures 2.1 and 2.2 show the two halves of the argument for the fish: the class-conditional densities, and the posteriors they produce with priors P(ω1)=2/3P(\omega_1) = 2/3 and P(ω2)=1/3P(\omega_2) = 1/3.

Figure 2.1

Figure 2.1: hypothetical class-conditional densities for the lightness of sea bass (ω1\omega_1) and salmon (ω2\omega_2). Each curve is a density, so the area under it is 1.

Figure 2.2

Figure 2.2: the posteriors for the densities of Figure 2.1 with P(ω1)=2/3P(\omega_1) = 2/3 and P(ω2)=1/3P(\omega_2) = 1/3. At x=14x = 14 the posterior of ω2\omega_2 is about 0.08 and that of ω1\omega_1 about 0.92; at every xx the two posteriors sum to 1.

Risk and Decision Rules

Loss and Conditional Risk

The minimum-error rule is a special case of a more general criterion. Let α1,…,αa\alpha_1,\dots,\alpha_a be the actions available — for classification, “decide ωi\omega_i” — and let λij\lambda_{ij} be the loss of taking action αi\alpha_i when the true class is ωj\omega_j. The conditional risk of action αi\alpha_i for an observation x\mathbf{x} is

R(αi∣x)=∑j=1cλij P(ωj∣x),R(\alpha_i \mid \mathbf{x}) = \sum_{j=1}^{c} \lambda_{ij}\,P(\omega_j \mid \mathbf{x}),

and the Bayes decision rule selects the action of least risk:

decide αiwherei=arg⁡min⁡iR(αi∣x).\text{decide } \alpha_i \quad \text{where} \quad i = \arg\min_{i} R(\alpha_i \mid \mathbf{x}).

The zero-one loss sets λij=0\lambda_{ij} = 0 when i=ji = j and 1 otherwise, that is, it prices every error the same. Its conditional risk is the probability of being wrong,

R(αi∣x)=∑j≠iP(ωj∣x)=1−P(ωi∣x),R(\alpha_i \mid \mathbf{x}) = \sum_{j \neq i} P(\omega_j \mid \mathbf{x}) = 1 - P(\omega_i \mid \mathbf{x}),

so minimising the risk is the same as maximising the posterior. Unequal losses — a missed disease costing more than a false alarm — simply move the decision threshold; they do not change the machinery.

Minimum Error Rate Classification

  • Decide ω1\omega_1 if P(ω1∣x)>P(ω2∣x)P(\omega_1 \mid \mathbf{x}) > P(\omega_2 \mid \mathbf{x}), otherwise decide ω2\omega_2.

  • Equivalently, since the evidence is common to both classes, decide ω1\omega_1 if

    p(x∣ω1) P(ω1)>p(x∣ω2) P(ω2).p(\mathbf{x} \mid \omega_1)\,P(\omega_1) > p(\mathbf{x} \mid \omega_2)\,P(\omega_2).
  • The probability of error for a given observation is

    P(error∣x)=min⁡[P(ω1∣x),  P(ω2∣x)].P(\text{error} \mid \mathbf{x}) = \min\left[P(\omega_1 \mid \mathbf{x}),\; P(\omega_2 \mid \mathbf{x})\right].

Special Cases

Two cases show which factor drives the decision.

  1. If p(x∣ω1)=p(x∣ω2)p(\mathbf{x} \mid \omega_1) = p(\mathbf{x} \mid \omega_2), the observation is equally likely under both classes and tells us nothing; the decision rests entirely on the priors.

  2. If P(ω1)=P(ω2)P(\omega_1) = P(\omega_2), the classes are equally probable a priori and the decision rests entirely on the likelihoods.

In general both matter, and the rule gi(x)=p(x∣ωi)P(ωi)g_i(\mathbf{x}) = p(\mathbf{x} \mid \omega_i)P(\omega_i) combines them; that combination is what minimises the probability of error.

More Than Two Classes

With c>2c > 2 classes the same rule applies unchanged: assign x\mathbf{x} to the class that maximises the posterior, which — as shown above — is the action of least risk under the zero-one loss:

decide ωiwhereP(ωi∣x)>P(ωj∣x)for all j≠i.\text{decide } \omega_i \quad\text{where}\quad P(\omega_i \mid \mathbf{x}) > P(\omega_j \mid \mathbf{x}) \quad \text{for all } j \neq i.

For each class we compute one posterior, and the regions Ri\mathcal{R}_i where class ii wins are separated by the surfaces P(ωi∣x)=P(ωj∣x)P(\omega_i \mid \mathbf{x}) = P(\omega_j \mid \mathbf{x}).

Discriminant Functions

A discriminant function gi(x)g_i(\mathbf{x}) is any function whose value is largest for the class we want to choose; the classifier computes one per class and picks the maximum.

decide ωiifgi(x)>gj(x) for all j≠i.\text{decide } \omega_i \quad\text{if}\quad g_i(\mathbf{x}) > g_j(\mathbf{x}) \text{ for all } j \neq i.

For minimum error rate the obvious choice is the posterior itself,

gi(x)=P(ωi∣x),g_i(\mathbf{x}) = P(\omega_i \mid \mathbf{x}),

and because the evidence is a common positive factor we may use the numerator,

gi(x)=p(x∣ωi) P(ωi),g_i(\mathbf{x}) = p(\mathbf{x} \mid \omega_i)\,P(\omega_i),

or its logarithm, which turns the product into a sum and the exponential of a Gaussian into a quadratic form:

gi(x)=ln⁡p(x∣ωi)+ln⁡P(ωi).g_i(\mathbf{x}) = \ln p(\mathbf{x} \mid \omega_i) + \ln P(\omega_i).

Any monotone function of a discriminant is a discriminant, and any positive multiple or additive constant may be dropped: the classification is unchanged. That freedom is what makes the Gaussian case tractable below.

Figure 2.5: the structure of a general statistical classifier — dd inputs, cc discriminant functions, and a maximum selector. The arrows show the flow of information.

The Normal Density

Univariate Case

The univariate normal density is

p(x)=12πσexp⁡[−12(x−μσ)2],p(x) = \frac{1}{\sqrt{2\pi}\sigma}\exp\left[-\frac{1}{2}\left(\frac{x-\mu}{\sigma}\right)^{2}\right],

with the expected value of xx (an average over the feature space)

μ=E[x]=∫−∞∞x p(x) dx\mu = \mathbb{E}[x] = \int_{-\infty}^{\infty} x\,p(x)\,dx

and the variance

σ2=E[(x−μ)2]=∫−∞∞(x−μ)2p(x) dx.\sigma^2 = \mathbb{E}[(x-\mu)^2] = \int_{-\infty}^{\infty} (x-\mu)^2 p(x)\,dx .

The univariate normal is completely specified by μ\mu and σ2\sigma^2; we write p(x)∼N(μ,σ2)p(x) \sim N(\mu, \sigma^2). Samples cluster about the mean with a spread set by the standard deviation σ\sigma (Figure 2.7).

Figure 2.7

Figure 2.7: a univariate normal has roughly 95%95\% of its area inside ∣x−μ∣≤2σ|x-\mu| \le 2\sigma; the peak value is p(μ)=1/(2πσ)p(\mu) = 1/(\sqrt{2\pi}\sigma).

The Multivariate Normal Density

p(x)=1(2π)d/2∣Σ∣1/2exp⁡[−12(x−μ)⊤Σ−1(x−μ)],p(\mathbf{x}) = \frac{1}{(2\pi)^{d/2}|\boldsymbol{\Sigma}|^{1/2}} \exp\left[-\frac{1}{2}(\mathbf{x}-\boldsymbol{\mu})^{\top}\boldsymbol{\Sigma}^{-1}(\mathbf{x}-\boldsymbol{\mu})\right],

where the mean vector μ\boldsymbol{\mu} and the covariance matrix Σ\boldsymbol{\Sigma} replace the two scalars of the univariate case.

Formal Definitions

μ=E[x]=∫x p(x) dx,Σ=E[(x−μ)(x−μ)⊤].\boldsymbol{\mu} = \mathbb{E}[\mathbf{x}] = \int \mathbf{x}\,p(\mathbf{x})\,d\mathbf{x}, \qquad \boldsymbol{\Sigma} = \mathbb{E}\left[(\mathbf{x}-\boldsymbol{\mu})(\mathbf{x}-\boldsymbol{\mu})^{\top}\right].

The expected value of a vector or a matrix is taken componentwise: with xix_i the ii-th component of x\mathbf{x} and σij\sigma_{ij} the ijij-th component of Σ\boldsymbol{\Sigma},

μi=E[xi],σij=E[(xi−μi)(xj−μj)].\mu_i = \mathbb{E}[x_i], \qquad \sigma_{ij} = \mathbb{E}[(x_i-\mu_i)(x_j-\mu_j)].

Properties of the Covariance Matrix

  • Σ\boldsymbol{\Sigma} is symmetric and positive semidefinite; we assume it is positive definite, so that ∣Σ∣>0|\boldsymbol{\Sigma}| > 0 and the inverse exists.

  • The diagonal element σii\sigma_{ii} is the variance of xix_i, that is σi2\sigma_i^2.

  • The off-diagonal element σij\sigma_{ij} is the covariance of xix_i and xjx_j; for the length and weight of a population of fish we would expect it to be positive.

  • If xix_i and xjx_j are statistically independent then σij=0\sigma_{ij} = 0; if all off-diagonal elements vanish, p(x)p(\mathbf{x}) factorises into the product of univariate normal densities.

Parameters and Geometry

The multivariate normal is completely specified by d+d(d+1)/2d + d(d+1)/2 parameters: the dd elements of μ\boldsymbol{\mu} and the d(d+1)/2d(d+1)/2 independent elements of Σ\boldsymbol{\Sigma}.

Samples fall in a single cluster whose centre is μ\boldsymbol{\mu} and whose shape is Σ\boldsymbol{\Sigma}. The loci of constant density are the hyperellipsoids

(x−μ)⊤Σ−1(x−μ)=constant,(\mathbf{x}-\boldsymbol{\mu})^{\top}\boldsymbol{\Sigma}^{-1}(\mathbf{x}-\boldsymbol{\mu}) = \text{constant},

whose principal axes are the eigenvectors of Σ\boldsymbol{\Sigma} and whose axis lengths follow from the eigenvalues. The quadratic form itself is the squared Mahalanobis distance from x\mathbf{x} to μ\boldsymbol{\mu}: contours of constant density are contours of constant Mahalanobis distance, and the volume they enclose measures the scatter of the samples.

Figure 2.9

Figure 2.9: samples from a two-dimensional Gaussian lie in a cloud centred on μ\boldsymbol{\mu}; the red ellipses are lines of equal probability density.

Discriminant Functions for Normal Densities

Case 1: Isotropic Covariance, Σi=σ2I\boldsymbol{\Sigma}_i = \sigma^2\mathbf{I}

The simplest case: all classes share the same spherical covariance. Then ∣Σi∣=σ2d|\boldsymbol{\Sigma}_i| = \sigma^{2d} and Σi−1=I/σ2\boldsymbol{\Sigma}_i^{-1} = \mathbf{I}/\sigma^2, so the density and its logarithm are

p(x∣ωi)=1(2πσ2)d/2exp⁡(−∥x−μi∥22σ2),ln⁡p(x∣ωi)=−∥x−μi∥22σ2−d2ln⁡(2πσ2).p(\mathbf{x} \mid \omega_i) = \frac{1}{(2\pi\sigma^2)^{d/2}} \exp\left(-\frac{\|\mathbf{x}-\boldsymbol{\mu}_i\|^2}{2\sigma^2}\right), \qquad \ln p(\mathbf{x} \mid \omega_i) = -\frac{\|\mathbf{x}-\boldsymbol{\mu}_i\|^2}{2\sigma^2} - \frac{d}{2}\ln(2\pi\sigma^2).

Substituting into gi(x)=ln⁡p(x∣ωi)+ln⁡P(ωi)g_i(\mathbf{x}) = \ln p(\mathbf{x} \mid \omega_i) + \ln P(\omega_i), and dropping the term d2ln⁡(2πσ2)\frac{d}{2}\ln(2\pi\sigma^2) because it is the same for every class:

gi(x)=−∥x−μi∥22σ2+ln⁡P(ωi)\boxed{g_i(\mathbf{x}) = -\frac{\|\mathbf{x}-\boldsymbol{\mu}_i\|^2}{2\sigma^2} + \ln P(\omega_i)}

With equal priors the rule is “assign x\mathbf{x} to the nearest mean”, with the Euclidean distance scaled by 2σ22\sigma^2. With unequal priors, a point that is equally near two means is given to the a priori more likely class, because ln⁡P(ωi)\ln P(\omega_i) is the tie-breaker.

The Linear Form and the Boundary

Expanding the quadratic form,

gi(x)=−12σ2[x⊤x−2μi⊤x+μi⊤μi]+ln⁡P(ωi).g_i(\mathbf{x}) = -\frac{1}{2\sigma^2}\left[\mathbf{x}^{\top}\mathbf{x} - 2\boldsymbol{\mu}_i^{\top}\mathbf{x} + \boldsymbol{\mu}_i^{\top}\boldsymbol{\mu}_i\right] + \ln P(\omega_i).

The term x⊤x\mathbf{x}^{\top}\mathbf{x} does not depend on ii and can be dropped as an additive constant, which leaves a linear discriminant

gi(x)=wi⊤x+wi0,wi=μiσ2,wi0=−μi⊤μi2σ2+ln⁡P(ωi),g_i(\mathbf{x}) = \mathbf{w}_i^{\top}\mathbf{x} + w_{i0}, \qquad \mathbf{w}_i = \frac{\boldsymbol{\mu}_i}{\sigma^2}, \qquad w_{i0} = -\frac{\boldsymbol{\mu}_i^{\top}\boldsymbol{\mu}_i}{2\sigma^2} + \ln P(\omega_i),

where wi0w_{i0} is the threshold (or bias) of class ii. A classifier built from linear discriminants is a linear machine: its decision surfaces are pieces of hyperplanes, given by gi(x)=gj(x)g_i(\mathbf{x}) = g_j(\mathbf{x}) for the two classes whose posteriors are largest. For two classes that equation reads

w⊤(x−x0)=0,w=μi−μj,x0=12(μi+μj)−σ2∥μi−μj∥2ln⁡P(ωi)P(ωj) (μi−μj).\mathbf{w}^{\top}(\mathbf{x}-\mathbf{x}_0) = 0, \qquad \mathbf{w} = \boldsymbol{\mu}_i - \boldsymbol{\mu}_j, \qquad \mathbf{x}_0 = \frac{1}{2}(\boldsymbol{\mu}_i + \boldsymbol{\mu}_j) - \frac{\sigma^2}{\|\boldsymbol{\mu}_i-\boldsymbol{\mu}_j\|^2} \ln\frac{P(\omega_i)}{P(\omega_j)}\,(\boldsymbol{\mu}_i-\boldsymbol{\mu}_j).

So the boundary is a hyperplane through x0\mathbf{x}_0, orthogonal to w=μi−μj\mathbf{w} = \boldsymbol{\mu}_i-\boldsymbol{\mu}_j — orthogonal, that is, to the line joining the two means. With equal priors x0\mathbf{x}_0 is the midpoint and the boundary is the perpendicular bisector of the segment between the means; with unequal priors it slides away from the more likely mean, by an amount that is large when σ2\sigma^2 is large compared with ∥μi−μj∥2\|\boldsymbol{\mu}_i-\boldsymbol{\mu}_j\|^2 and negligible when the means are far apart.

Figure 2.10

Figure 2.10: with equal isotropic covariances the class regions are separated by a generalised hyperplane of d−1d-1 dimensions, perpendicular to the line joining the means.

Minimum Distance Classifier

If all priors are also equal, ln⁡P(ωi)\ln P(\omega_i) is an additive constant, and the rule reduces to measuring the Euclidean distance ∥x−μi∥\|\mathbf{x}-\boldsymbol{\mu}_i\| and assigning the nearest mean. This is the minimum distance classifier. Reading each mean as an ideal prototype of its class makes it a template-matching procedure, and it is exactly the nearest-neighbour rule of the previous chapter with one template per class instead of the whole training set.

What a Hyperplane Looks Like in Two Dimensions

The equation w⊤(x−x0)=0\mathbf{w}^{\top}(\mathbf{x}-\mathbf{x}_0) = 0 says that the vector x−x0\mathbf{x}-\mathbf{x}_0 is orthogonal to w\mathbf{w} for every point x\mathbf{x} of the surface, so:

  • w\mathbf{w} is a normal to the surface and fixes its orientation;

  • x0\mathbf{x}_0 is one fixed point of the surface;

  • in d=2d = 2 the surface is a line, and the equation becomes w1(x−x01)+w2(x−x02)=0w_1(x-x_{01}) + w_2(x-x_{02}) = 0.

For example, with w=(2,1)⊤\mathbf{w} = (2,1)^{\top} and x0=(1,1)⊤\mathbf{x}_0 = (1,1)^{\top} the line is 2x+y−3=02x + y - 3 = 0. The next cell draws it together with the normal vector.

<Figure size 640x480 with 1 Axes>

Case 2: Equal but Arbitrary Covariance Matrices, Σi=Σ\boldsymbol{\Sigma}_i = \boldsymbol{\Sigma}

When all classes share the same non-spherical covariance, the general discriminant

gi(x)=−12(x−μi)⊤Σ−1(x−μi)+ln⁡P(ωi),g_i(\mathbf{x}) = -\frac{1}{2}(\mathbf{x}-\boldsymbol{\mu}_i)^{\top}\boldsymbol{\Sigma}^{-1}(\mathbf{x}-\boldsymbol{\mu}_i) + \ln P(\omega_i),

follows from the Gaussian after dropping ln⁡∣Σ∣\ln|\boldsymbol{\Sigma}| and d2ln⁡(2π)\frac{d}{2}\ln(2\pi), which are now common to all classes. Expanding the quadratic term (using that Σ−1\boldsymbol{\Sigma}^{-1} is symmetric, so x⊤Σ−1μi=μi⊤Σ−1x\mathbf{x}^{\top}\boldsymbol{\Sigma}^{-1}\boldsymbol{\mu}_i = \boldsymbol{\mu}_i^{\top}\boldsymbol{\Sigma}^{-1}\mathbf{x}):

(x−μi)⊤Σ−1(x−μi)=x⊤Σ−1x−2μi⊤Σ−1x+μi⊤Σ−1μi.(\mathbf{x}-\boldsymbol{\mu}_i)^{\top}\boldsymbol{\Sigma}^{-1}(\mathbf{x}-\boldsymbol{\mu}_i) = \mathbf{x}^{\top}\boldsymbol{\Sigma}^{-1}\mathbf{x} - 2\boldsymbol{\mu}_i^{\top}\boldsymbol{\Sigma}^{-1}\mathbf{x} + \boldsymbol{\mu}_i^{\top}\boldsymbol{\Sigma}^{-1}\boldsymbol{\mu}_i .

The first term does not depend on ii and is dropped; what remains is again linear,

gi(x)=wi⊤x+wi0,wi=Σ−1μi,wi0=−12μi⊤Σ−1μi+ln⁡P(ωi),g_i(\mathbf{x}) = \mathbf{w}_i^{\top}\mathbf{x} + w_{i0}, \qquad \mathbf{w}_i = \boldsymbol{\Sigma}^{-1}\boldsymbol{\mu}_i, \qquad w_{i0} = -\frac{1}{2}\boldsymbol{\mu}_i^{\top}\boldsymbol{\Sigma}^{-1}\boldsymbol{\mu}_i + \ln P(\omega_i),

so the boundaries are hyperplanes. Setting gi(x)=gj(x)g_i(\mathbf{x}) = g_j(\mathbf{x}):

(μi−μj)⊤Σ−1x−12(μi⊤Σ−1μi−μj⊤Σ−1μj)+ln⁡P(ωi)P(ωj)=0.(\boldsymbol{\mu}_i-\boldsymbol{\mu}_j)^{\top}\boldsymbol{\Sigma}^{-1}\mathbf{x} - \frac{1}{2}\left(\boldsymbol{\mu}_i^{\top}\boldsymbol{\Sigma}^{-1}\boldsymbol{\mu}_i - \boldsymbol{\mu}_j^{\top}\boldsymbol{\Sigma}^{-1}\boldsymbol{\mu}_j\right) + \ln\frac{P(\omega_i)}{P(\omega_j)} = 0 .

The Geometry of Case 2

Write the boundary as w⊤x+w0=0\mathbf{w}^{\top}\mathbf{x} + w_0 = 0 with

w=Σ−1(μi−μj),w0=−12(μi⊤Σ−1μi−μj⊤Σ−1μj)+ln⁡P(ωi)P(ωj).\mathbf{w} = \boldsymbol{\Sigma}^{-1}(\boldsymbol{\mu}_i-\boldsymbol{\mu}_j), \qquad w_0 = -\frac{1}{2}\left(\boldsymbol{\mu}_i^{\top}\boldsymbol{\Sigma}^{-1}\boldsymbol{\mu}_i - \boldsymbol{\mu}_j^{\top}\boldsymbol{\Sigma}^{-1}\boldsymbol{\mu}_j\right) + \ln\frac{P(\omega_i)}{P(\omega_j)} .

The normal w\mathbf{w} points along Σ−1(μi−μj)\boldsymbol{\Sigma}^{-1}(\boldsymbol{\mu}_i-\boldsymbol{\mu}_j), which in general is not the direction of μi−μj\boldsymbol{\mu}_i-\boldsymbol{\mu}_j: the shared covariance rotates the boundary. Only when Σ\boldsymbol{\Sigma} is proportional to I\mathbf{I} do we recover Case 1. With equal priors the logarithm vanishes and the boundary passes through x0=12(μi+μj)\mathbf{x}_0 = \frac{1}{2}(\boldsymbol{\mu}_i+\boldsymbol{\mu}_j), but it is not perpendicular to the line between the means.

The surfaces of constant Mahalanobis distance from the two means are hyperellipsoids of the same shape and orientation, one per class, because Σ\boldsymbol{\Sigma} is shared. They are linear boundaries only because the quadratic terms x⊤Σ−1x\mathbf{x}^{\top}\boldsymbol{\Sigma}^{-1}\mathbf{x} cancel in gi−gjg_i - g_j.

Figure 2.12: equal covariance matrices give linear boundaries that are not necessarily perpendicular to the line joining the means; both classes are shown with the same ellipses of equal density, and the boundary moves with the priors. The next figure shows it sliding as the priors change, for one-, two- and three-dimensional spherical classes.

Figure 2.11

Figure 2.11: as the priors are changed, the decision boundary shifts; for sufficiently disparate priors the boundary will not lie between the means of these 1-, 2- and 3-dimensional spherical Gaussian distributions

Case 3: Arbitrary Covariance Matrices, Σi\boldsymbol{\Sigma}_i

The general case: each class has its own mean μi\boldsymbol{\mu}_i and its own covariance Σi\boldsymbol{\Sigma}_i. Starting from

gi(x)=ln⁡p(x∣ωi)+ln⁡P(ωi)=−12(x−μi)⊤Σi−1(x−μi)−12ln⁡∣Σi∣+ln⁡P(ωi),g_i(\mathbf{x}) = \ln p(\mathbf{x} \mid \omega_i) + \ln P(\omega_i) = -\frac{1}{2}(\mathbf{x}-\boldsymbol{\mu}_i)^{\top}\boldsymbol{\Sigma}_i^{-1}(\mathbf{x}-\boldsymbol{\mu}_i) - \frac{1}{2}\ln|\boldsymbol{\Sigma}_i| + \ln P(\omega_i),

and expanding the quadratic form exactly as in Case 2, but now with Σi\boldsymbol{\Sigma}_i:

gi(x)=−12x⊤Σi−1x+μi⊤Σi−1x−12μi⊤Σi−1μi−12ln⁡∣Σi∣+ln⁡P(ωi).g_i(\mathbf{x}) = -\frac{1}{2}\mathbf{x}^{\top}\boldsymbol{\Sigma}_i^{-1}\mathbf{x} + \boldsymbol{\mu}_i^{\top}\boldsymbol{\Sigma}_i^{-1}\mathbf{x} - \frac{1}{2}\boldsymbol{\mu}_i^{\top}\boldsymbol{\Sigma}_i^{-1}\boldsymbol{\mu}_i - \frac{1}{2}\ln|\boldsymbol{\Sigma}_i| + \ln P(\omega_i).

This is a quadratic form in x\mathbf{x},

gi(x)=x⊤Wix+wi⊤x+wi0,Wi=−12Σi−1,wi=Σi−1μi,g_i(\mathbf{x}) = \mathbf{x}^{\top}\mathbf{W}_i\mathbf{x} + \mathbf{w}_i^{\top}\mathbf{x} + w_{i0}, \qquad \mathbf{W}_i = -\frac{1}{2}\boldsymbol{\Sigma}_i^{-1}, \qquad \mathbf{w}_i = \boldsymbol{\Sigma}_i^{-1}\boldsymbol{\mu}_i,
wi0=−12μi⊤Σi−1μi−12ln⁡∣Σi∣+ln⁡P(ωi).w_{i0} = -\frac{1}{2}\boldsymbol{\mu}_i^{\top}\boldsymbol{\Sigma}_i^{-1}\boldsymbol{\mu}_i - \frac{1}{2}\ln|\boldsymbol{\Sigma}_i| + \ln P(\omega_i).

The difference from Case 2 is the term −12ln⁡∣Σi∣-\frac{1}{2}\ln|\boldsymbol{\Sigma}_i|: with unequal covariances the volume of the class matters, and a diffuse class is penalised. The discriminant is now quadratic in x\mathbf{x} and the boundary, given by gi(x)=gj(x)g_i(\mathbf{x}) = g_j(\mathbf{x}),

x⊤(Wi−Wj)x+(wi−wj)⊤x+(wi0−wj0)=0,\mathbf{x}^{\top}(\mathbf{W}_i-\mathbf{W}_j)\mathbf{x} + (\mathbf{w}_i-\mathbf{w}_j)^{\top}\mathbf{x} + (w_{i0}-w_{j0}) = 0,

does not lose its quadratic term, because Wi≠Wj\mathbf{W}_i \neq \mathbf{W}_j.

What the Quadratic Boundaries Look Like

A surface of this kind is a hyperquadric — a hyperplane, a pair of hyperplanes, a hypersphere, a hyperellipsoid, a hyperparaboloid or a hyperhyperboloid, depending on the eigenvalues of Wi−Wj\mathbf{W}_i - \mathbf{W}_j and on the linear and constant terms. In two dimensions those are a line, a pair of lines, a circle, an ellipse, a parabola and a hyperbola. Every one of these shapes can arise from two Gaussians, and conversely every hyperquadric is the Bayes boundary of some pair of Gaussians.

Figure 2.6

Figure 2.6: a two-dimensional two-category Gaussian classifier with unequal covariances. The two hyperbolas are the boundary, so the decision region R2\mathcal{R}_2 is not simply connected — a reminder that “the” boundary may have several branches.

Figure 2.14

Figure 2.14: arbitrary Gaussians give hyperquadric boundaries; the four panels show an ellipse, a hyperbola, a parabola and a pair of lines.

More Than Two Classes

Extending the result to cc classes is straightforward, but one has to keep track of which two classes are responsible for each piece of a boundary. Figure 2.16 shows the decision regions of a four-class problem with Gaussian classes: even with four classes the regions are already complicated, and the same theory still governs them.

Figure 2.16

Figure 2.16: the decision regions of four normal distributions.


Exercise: Verifying Case 2 with Scikit-Learn

Objective. Reproduce the Case 2 boundary twice — once from the formula above and once from GaussianNB — and check that they agree.

  1. Generate data. Two classes, 200 points each, both with covariance Σ=[2112]\boldsymbol{\Sigma} = \begin{bmatrix} 2 & 1 \\ 1 & 2\end{bmatrix} and means μ1=(2,3)⊤\boldsymbol{\mu}_1 = (2,3)^{\top}, μ2=(6,5)⊤\boldsymbol{\mu}_2 = (6,5)^{\top}.

  2. Theoretical boundary. With P(ω1)=P(ω2)=0.5P(\omega_1) = P(\omega_2) = 0.5, use

    gi(x)=−12(x−μi)⊤Σ−1(x−μi)+ln⁡P(ωi)g_i(\mathbf{x}) = -\frac{1}{2}(\mathbf{x}-\boldsymbol{\mu}_i)^{\top}\boldsymbol{\Sigma}^{-1}(\mathbf{x}-\boldsymbol{\mu}_i) + \ln P(\omega_i)

    and draw the curve g1(x)=g2(x)g_1(\mathbf{x}) = g_2(\mathbf{x}) on a grid with plt.contour.

  3. Estimated boundary. Fit GaussianNB on the 200 + 200 points and draw the boundary of its predictions on the same grid.

  4. Compare. Plot the two boundaries together, with the points, the line joining the two means, and its perpendicular bisector. The two boundaries should be close but not identical: GaussianNB estimates μi\boldsymbol{\mu}_i and Σi\boldsymbol{\Sigma}_i from the sample, while step 2 uses the true parameters. The angle between the boundary normal and the line joining the means is the visible consequence of the shared covariance (Case 2).

Summary

Bayesian decision theory turns probabilities into decisions.

  • The ingredients are the priors P(ωi)P(\omega_i), the class-conditional densities p(x∣ωi)p(\mathbf{x} \mid \omega_i), the posteriors P(ωi∣x)P(\omega_i \mid \mathbf{x}) and the evidence p(x)p(\mathbf{x}), which normalises the posteriors and cancels from every decision.

  • The general criterion is the conditional risk with a loss matrix λij\lambda_{ij}; the zero-one loss reduces it to the minimum error rate rule, “pick the largest posterior”.

  • Any monotone function of the posterior is a valid discriminant; using the log turns the Gaussian into a quadratic form.

  • For Gaussian classes there are three regimes. Isotropic and equal covariances give linear discriminants and hyperplane boundaries; arbitrary covariances give quadratic discriminants and hyperquadric boundaries.

  • The quadratic form inside the Gaussian is the squared Mahalanobis distance, and the three regimes are exactly LDA, LDA with a general Σ\boldsymbol{\Sigma}, and QDA.

  • Density parameters are estimated by maximum likelihood; when classes are too small to estimate a full covariance, the naive assumption (independent features) is the usual way out.

References

Duda et al. (2000)

Duda, R. O., Hart, P. E., & Stork, D. G. (2000). Pattern Classification (2nd ed.). Wiley-Interscience, chapter 2.

References
  1. Duda, R. O., Hart, P. E., & Stork, D. G. (2000). Pattern Classification (2nd ed.). Wiley-Interscience.