Case 1: Equal Covariance Matrices (${\mathbf{\Sigma }}_i={\sigma }^2\mathbf{I}$):#

• The discriminant function is linear:

How?

Let’s derive the simplified form of the log-likelihood $g_i(\mathbf{x})$ for the case where the covariance matrices are equal and isotropic (${\mathbf{\Sigma }}_i={\sigma }^2\mathbf{I}$). We’ll start with the given multivariate normal density and show how it reduces to the stated form.

Given:#

The probability density function (PDF) for class ${\omega }_i$ is:

where:

• ${\mathbf{\Sigma }}_i={\sigma }^2\mathbf{I}$ (isotropic covariance, equal across classes).

• $|{\mathbf{\Sigma }}_i|=({\sigma }^2{)}^d$ (determinant of ${\sigma }^2\mathbf{I}$).

• ${\mathbf{\Sigma }}_i^{-1}=\frac{1}{{\sigma }^2}\mathbf{I}$ (inverse of diagonal matrix).

Substitute ${\mathbf{\Sigma }}_i={\sigma }^2\mathbf{I}$ into the PDF#

Simplify the exponent:

where $\Vert \mathbf{x}-{\mathit{\mu }}_i{\Vert }^2$ is the squared Euclidean distance.

Thus, the PDF becomes:

Apply Bayes’ Rule for Posterior Probability#

The posterior probability $P({\omega }_i\mid \mathbf{x})$ is:

where $p(\mathbf{x})=\sum _{j}p(\mathbf{x}\mid {\omega }_j)P({\omega }_j)$ is the evidence (ignored in discriminant functions).

The log-posterior (discriminant function $g_i(\mathbf{x})$) is:

Take the Logarithm of the PDF#

Compute $\ln p(\mathbf{x}\mid {\omega }_i)$:

Simplify the first term:

Thus:

Combine with Prior and Ignore Constants#

The discriminant function becomes:

Since $\frac{d}{2}\ln (2\pi {\sigma }^2)$ is constant across all classes ${\omega }_i$, it does not affect the classification decision (we compare $g_i(\mathbf{x})$ across $i$, and the constant cancels out). Therefore, we drop it and write:

• Decision boundaries are hyperplanes.

• If the prior probabilities are not equal, the squared distance $\Vert \mathbf{x}-{\mathit{\mu }}_i{\Vert }^2$ is normalized by the variance ${\sigma }^2$ and offset by adding $\ln P({\omega }_i)$. This means that if $\mathbf{x}$ is equally near two different mean vectors, the optimal decision will favor the a priori more likely category.

• It is not necessary to compute distances explicitly. Expanding the quadratic form $(\mathbf{x}-{\mathit{\mu }}_i{)}^T(\mathbf{x}-{\mathit{\mu }}_i)$ yields:

  $$g_i(\mathbf{x})=-\frac{1}{2{\sigma }^2}[{\mathbf{x}}^T\mathbf{x}-2{\mathit{\mu }}_i^T\mathbf{x}+{\mathit{\mu }}_i^T{\mathit{\mu }}_i]+\ln P({\omega }_i)$$

  The quadratic term ${\mathbf{x}}^T\mathbf{x}$ is the same for all $i$, so it can be ignored as an additive constant. This simplifies the discriminant function to:

  $$\boxed{{\displaystyle g_i(\mathbf{x})={\mathbf{w}}_i^T\mathbf{x}+w_{i0}}}$$

  where:

  $${\mathbf{w}}_i=\frac{{\mathit{\mu }}_i}{{\sigma }^2}\text{(52)}$$

  and

  $$w_{i0}=-\frac{{\mathit{\mu }}_i^T{\mathit{\mu }}_i}{2{\sigma }^2}+\ln P({\omega }_i)$$

  Here, $w_{i0}$ is called the threshold or bias in the $i^{th}$ direction.

• A classifier that uses linear discriminant functions is called a linear machine. The decision surfaces for a linear machine are pieces of hyperplanes defined by the linear equations $g_i(\mathbf{x})=g_j(\mathbf{x})$ for the two categories with the highest posterior probabilities. For our case, this equation can be written as:

  $${\mathbf{w}}^T(\mathbf{x}-{\mathbf{x}}_0)=0$$

  where:

  $$\mathbf{w}={\mathit{\mu }}_i-{\mathit{\mu }}_j$$

  and

  $${\mathbf{x}}_0=\frac{1}{2}({\mathit{\mu }}_i+{\mathit{\mu }}_j)-\frac{{\sigma }^2}{\Vert {\mathit{\mu }}_i-{\mathit{\mu }}_j{\Vert }^2}\ln \frac{P({\omega }_i)}{P({\omega }_j)}({\mathit{\mu }}_i-{\mathit{\mu }}_j)\text{(56)}.$$

  This defines a hyperplane through the point ${\mathbf{x}}_0$ and orthogonal to the vector $\mathbf{w}$. Since $\mathbf{w}={\mathit{\mu }}_i-{\mathit{\mu }}_j$, the hyperplane separating ${\mathcal{R}}_i$ and ${\mathcal{R}}_j$ is orthogonal to the line linking the means. If $P({\omega }_i)=P({\omega }_j)$, the hyperplane is the perpendicular bisector of the line between the means. If the prior probabilities are unequal, the point ${\mathbf{x}}_0$ shifts away from the more likely mean.

Figure 2.10: If the covariances of two distributions are equal and proportional to the identity matrix, the distributions are spherical in $d$ dimensions, and the boundary is a generalized hyperplane of $d-1$ dimensions, perpendicular to the line separating the means.

The equation ${\mathbf{w}}^T(\mathbf{x}-{\mathbf{x}}_0)=0$ represents a hyperplane in $n$-dimensional space. Here’s how to interpret and visualize it:

Interpretation

1. $\mathbf{w}$: A normal vector to the hyperplane (defines the orientation of the hyperplane).

2. ${\mathbf{x}}_0$: A fixed point on the hyperplane.

3. $\mathbf{x}$: A variable point on the hyperplane.

The equation states that the vector $\mathbf{x}-{\mathbf{x}}_0$ is perpendicular to $\mathbf{w}$, meaning all points $\mathbf{x}$ on the hyperplane satisfy this condition.

Visualization in 2D In 2D space ($n=2$), the equation reduces to a line. Let’s break it down:

1. $\mathbf{w}=\left[\begin{array}{c}{w}_1\\ w_2\end{array}\right]$: The normal vector to the line.

2. ${\mathbf{x}}_0=\left[\begin{array}{c}{x}_0\\ y_0\end{array}\right]$: A fixed point on the line.

3. $\mathbf{x}=\left[\begin{array}{c}x\\ y\end{array}\right]$: A variable point on the line.

The equation becomes: $w_1(x-x_0)+w_2(y-y_0)=0$

This is the equation of a line in 2D.

Drawing the Hyperplane (Line in 2D) Here’s how to draw it:

1. Plot the point ${\mathbf{x}}_0$: This is a fixed point on the line.

2. Draw the normal vector $\mathbf{w}$: This vector is perpendicular to the line.

3. Draw the line: The line passes through ${\mathbf{x}}_0$ and is perpendicular to $\mathbf{w}$.

Example Let’s use the following values:

• $\mathbf{w}=\left[\begin{array}{c}2\\ 1\end{array}\right]$ (normal vector).

• ${\mathbf{x}}_0=\left[\begin{array}{c}1\\ 1\end{array}\right]$ (a point on the line).

The equation becomes: $2(x-1)+1(y-1)=0$ Simplify: $2x-2+y-1=0⟹2x+y-3=0$

import matplotlib.pyplot as plt
import numpy as np

# Define the normal vector w and point x0
w = np.array([2, 1])  # Normal vector
x0 = np.array([1, 1])  # Point on the line

# Define the line equation: 2x + y - 3 = 0 => y = -2x + 3
x_values = np.linspace(-5, 5, 100)  # Range of x values
y_values = -2 * x_values + 3  # Corresponding y values

# Plot the line
plt.plot(x_values, y_values, label="Line: $2x + y - 3 = 0$")

# Plot the point x0
plt.scatter(x0[0], x0[1], color="red", label="Point $\mathbf{x}_0 = (1, 1)$")

# Plot the normal vector w starting from x0
plt.quiver(x0[0], x0[1], w[0], w[1], angles='xy', scale_units='xy', scale=1, color="green", label="Normal vector $\mathbf{w} = (2, 1)$")

# Add labels and legend
plt.xlabel("x")
plt.ylabel("y")
plt.axhline(0, color="black", linewidth=0.5)  # x-axis
plt.axvline(0, color="black", linewidth=0.5)  # y-axis
plt.grid(True)
plt.legend()
plt.title("Line and Normal Vector Visualization")
plt.xlim(-5, 5)  # Adjust x-axis limits
plt.ylim(-5, 5)  # Adjust y-axis limits
plt.gca().set_aspect("equal", adjustable="box")  # Equal aspect ratio
plt.show()

Hyperplane and Decision Boundary

The equation:

defines a hyperplane through the point ${\mathbf{x}}_0$ and orthogonal to the vector $\mathbf{w}$. Since $\mathbf{w}={\mathit{\mu }}_i-{\mathit{\mu }}_j$, the hyperplane separating ${\mathcal{R}}_i$ and ${\mathcal{R}}_j$ is orthogonal to the line linking the means.

• If $P({\omega }_i)=P({\omega }_j)$, the second term in Eq. 56 vanishes, and the point ${\mathbf{x}}_0$ is halfway between the means. In this case, the hyperplane is the perpendicular bisector of the line between the means (see Figure 2.11).

• If the prior probabilities are unequal, the point ${\mathbf{x}}_0$ shifts away from the more likely mean.

• Note that if the variance ${\sigma }^2$ is small relative to the squared distance $\Vert {\mathit{\mu }}_i-{\mathit{\mu }}_j{\Vert }^2$, the position of the decision boundary is relatively insensitive to the exact values of the prior probabilities.

Minimum Distance Classifier

• If the prior probabilities $P({\omega }_i)$ are the same for all $c$ classes, the term $\ln P({\omega }_i)$ becomes an unimportant additive constant and can be ignored.

• In this case, the optimum decision rule simplifies to:

  • Measure the Euclidean distance $\Vert \mathbf{x}-{\mathit{\mu }}_i\Vert$ from $\mathbf{x}$ to each of the $c$ mean vectors.

  • Assign $\mathbf{x}$ to the category of the nearest mean.

• Such a classifier is called a minimum distance classifier.

• If each mean vector is considered an ideal prototype or template for patterns in its class, this is essentially a template-matching procedure (see Figure 2.10). This technique is similar to nearest-neighbor algorithm visited in the previous chapter.

Figure 2.11: As the priors are changed, the decision boundary shifts; for sufficiently disparate priors the boundary will not lie between the means of these 1-, 2- and 3-dimensional spherical Gaussian distributions

Case 2: Equal but Arbitrary Covariance Matrices (${\mathbf{\Sigma }}_i=\mathbf{\Sigma }$ for all $i$)#

When the covariance matrices for all classes are identical but arbitrary (${\mathbf{\Sigma }}_i=\mathbf{\Sigma }$), the discriminant function defined in Eq. 51 simplifies. First, the term $\ln |{\mathbf{\Sigma }}_i|$ becomes $\ln |\mathbf{\Sigma }|$, which is the same constant for all classes $i$. Since it doesn’t affect the decision, we can ignore it. Similarly, the term $-\frac{d}{2}\ln (2\pi )$ is also a constant independent of $i$ and can be dropped. Thus, we have:

Now, let’s expand the quadratic term:

Since $\mathbf{\Sigma }$ is symmetric, its inverse ${\mathbf{\Sigma }}^{-1}$ is also symmetric. The term ${\mathbf{x}}^T{\mathbf{\Sigma }}^{-1}{\mathit{\mu }}_i$ is a scalar, so it is equal to its transpose: ${\mathbf{x}}^T{\mathbf{\Sigma }}^{-1}{\mathit{\mu }}_i=({\mathbf{x}}^T{\mathbf{\Sigma }}^{-1}{\mathit{\mu }}_i{)}^T={\mathit{\mu }}_i^T({\mathbf{\Sigma }}^{-1}{)}^T({\mathbf{x}}^T{)}^T={\mathit{\mu }}_i^T{\mathbf{\Sigma }}^{-1}\mathbf{x}$. Therefore:

Substituting this back into the expression for $g_i(\mathbf{x})$:

Notice that the term $-\frac{1}{2}({\mathbf{x}}^T{\mathbf{\Sigma }}^{-1}\mathbf{x})$ is independent of the class $i$. Since we are interested in comparing $g_i(\mathbf{x})$ and $g_j(\mathbf{x})$ (e.g., finding the maximum), any term that does not depend on $i$ can be dropped without affecting the decision rule or the location of the decision boundaries. Thus, we can simplify the discriminant function to:

This discriminant function is linear in $\mathbf{x}$. We can rewrite it in the familiar linear form:

where the weight vector ${\mathbf{w}}_i$ and the bias term $w_{i0}$ are defined as:

Since the discriminant function $g_i(\mathbf{x})$ is linear, the decision boundaries between any two classes ${\omega }_i$ and ${\omega }_j$, defined by $g_i(\mathbf{x})=g_j(\mathbf{x})$, are hyperplanes. Let’s examine the equation for the decision boundary:

Substituting the expressions for ${\mathbf{w}}_i$, ${\mathbf{w}}_j$, $w_{i0}$, and $w_{j0}$:

This equation defines a hyperplane. Let $\mathbf{w}={\mathbf{\Sigma }}^{-1}({\mathit{\mu }}_i-{\mathit{\mu }}_j)$. The equation can be written as:

where

The vector $\mathbf{w}={\mathbf{\Sigma }}^{-1}({\mathit{\mu }}_i-{\mathit{\mu }}_j)$ is normal to the decision hyperplane. Note that this normal vector is generally not in the direction of the difference between the means $({\mathit{\mu }}_i-{\mathit{\mu }}_j)$ unless $\mathbf{\Sigma }$ is proportional to the identity matrix (Case 1).

The location of the hyperplane is influenced by the prior probabilities $P({\omega }_i)$ and $P({\omega }_j)$. If the priors are equal ($P({\omega }_i)=P({\omega }_j)$), then $\ln \frac{P({\omega }_i)}{P({\omega }_j)}=\ln 1=0$. The equation simplifies further, and the threshold $w_0$ is determined solely by the means and the covariance matrix. Specifically, the boundary passes through the point ${\mathbf{x}}_0=\frac{1}{2}({\mathit{\mu }}_i+{\mathit{\mu }}_j)$ if $P({\omega }_i)=P({\omega }_j)$.

Geometric Interpretation: The decision boundaries are hyperplanes. Unlike Case 1 where the hyperplanes are orthogonal to the line connecting the means, here the orientation of the hyperplanes depends on the shared covariance matrix $\mathbf{\Sigma }$. The surfaces of constant Mahalanobis distance from the means are hyperellipsoids, all having the same shape and orientation determined by $\mathbf{\Sigma }$. The decision boundaries are linear because the quadratic terms ${\mathbf{x}}^T{\mathbf{\Sigma }}^{-1}\mathbf{x}$ cancelled out.

Figure 2.12 (from DHS book) illustrates this scenario for Gaussian distributions. It shows how the equal covariance matrices lead to linear decision boundaries, but these boundaries are not necessarily perpendicular to the line connecting the class means, unlike the simpler Case 1. The shape and orientation of the ellipses (representing contours of equal probability density) are the same for both classes, reflecting the shared covariance matrix $\mathbf{\Sigma }$. The decision boundary shifts depending on the prior probabilities.

Exercise: Validating the Decision Boundary in Case 2 and Comparing with GaussianNB

Objective In this exercise, you will:

1. Generate two groups of data with Gaussian distributions.

2. Compute the decision boundary for Case 2 (equal but arbitrary covariance matrices) using the theoretical approach.

3. Compare the results with the decision boundary generated by Scikit-Learn’s GaussianNB.

4. Visualize the results in a single plot or side-by-side plots.

Steps to Follow

1. Generate Data:

   • Create two classes of data points, each following a multivariate Gaussian distribution with the same covariance matrix but different mean vectors.

   • Use the following parameters for the two classes:

     • Class 1: Mean vector ${\mathit{\mu }}_1=[2,3]$, Covariance matrix $\mathbf{\Sigma }=\left[\begin{array}{cc}2& 1\\ 1& 2\end{array}\right]$.

     • Class 2: Mean vector ${\mathit{\mu }}_2=[6,5]$, Covariance matrix $\mathbf{\Sigma }=\left[\begin{array}{cc}2& 1\\ 1& 2\end{array}\right]$.

   • Generate 200 data points for each class.

2. Compute the Decision Boundary (Theoretical Approach):

   • Use the linear discriminant function for Case 2:

     $$g_i(\mathbf{x})=-\frac{1}{2}(\mathbf{x}-{\mathit{\mu }}_i{)}^T{\mathbf{\Sigma }}^{-1}(\mathbf{x}-{\mathit{\mu }}_i)+\ln P({\omega }_i).$$

   • Assume equal prior probabilities ($P({\omega }_1)=P({\omega }_2)=0.5$).

   • The decision boundary is the set of points where $g_1(\mathbf{x})=g_2(\mathbf{x})$.

3. Compute the Decision Boundary (GaussianNB):

   • Use Scikit-Learn’s GaussianNB to fit the data and predict the labels for test points.

4. Generate Test Points:

   • Create a grid of test points covering the entire plot area (e.g., using np.meshgrid).

   • Predict the class labels for these test points using both the theoretical decision boundary and GaussianNB.

5. Visualize the Results:

   • Plot the original data points for both classes.

   • Plot the test points with their predicted labels (use transparency to avoid obscuring the original data).

   • Draw the decision boundary for both the theoretical approach and GaussianNB.

   • Draw the line connecting the two class centers and the perpendicular bisector.

Expected Output

• A figure with two subplots:

  1. Theoretical Decision Boundary:

     • The original data points for both classes.

     • The test points with their predicted labels (shaded regions).

     • The theoretical decision boundary (black line).

     • The line connecting the two class centers (dashed black line).

     • The perpendicular bisector (dotted green line).

  2. GaussianNB Decision Boundary:

     • The original data points for both classes.

     • The test points with their predicted labels (shaded regions).

     • The decision boundary generated by GaussianNB (black line).

     • The line connecting the two class centers (dashed black line).

     • The perpendicular bisector (dotted green line).

Case 3: Arbitrary Covariance Matrices (${\mathbf{\Sigma }}_i$):#

This is the most general case for Gaussian distributions. Here, each class ${\omega }_i$ is modeled by a multivariate normal distribution $N({\mathit{\mu }}_i,{\mathbf{\Sigma }}_i)$, where both the mean vector ${\mathit{\mu }}_i$ and the covariance matrix ${\mathbf{\Sigma }}_i$ can be different for each class.

We start again with the general discriminant function, dropping the constant term $-\frac{d}{2}\ln (2\pi )$ as it doesn’t affect the decision:

Now, we expand the quadratic term $(\mathbf{x}-{\mathit{\mu }}_i{)}^T{\mathbf{\Sigma }}_i^{-1}(\mathbf{x}-{\mathit{\mu }}_i)$:

Since ${\mathbf{\Sigma }}_i^{-1}$ is symmetric, the two middle terms are equal scalar values: ${\mathbf{x}}^T{\mathbf{\Sigma }}_i^{-1}{\mathit{\mu }}_i=({\mathit{\mu }}_i^T{\mathbf{\Sigma }}_i^{-1}\mathbf{x}{)}^T={\mathit{\mu }}_i^T{\mathbf{\Sigma }}_i^{-1}\mathbf{x}$. So the expansion becomes:

Substituting this back into the expression for $g_i(\mathbf{x})$:

Rearranging the terms to group by powers of $\mathbf{x}$:

We can see this is a quadratic function of $\mathbf{x}$. It can be written in the general quadratic form:

where the components are identified by comparing the terms:

• Matrix ${\mathbf{W}}_i$: Determines the quadratic term.

  $${\mathbf{W}}_i=-\frac{1}{2}{\mathbf{\Sigma }}_i^{-1}$$

• Vector ${\mathbf{w}}_i$: Determines the linear term. Note that ${\mathit{\mu }}_i^T{\mathbf{\Sigma }}_i^{-1}\mathbf{x}=({\mathbf{\Sigma }}_i^{-1}{\mathit{\mu }}_i{)}^T\mathbf{x}$.

  $${\mathbf{w}}_i={\mathbf{\Sigma }}_i^{-1}{\mathit{\mu }}_i$$

• Scalar $w_{i0}$: Represents the constant or threshold term.

  $$w_{i0}=-\frac{1}{2}{\mathit{\mu }}_i^T{\mathbf{\Sigma }}_i^{-1}{\mathit{\mu }}_i-\frac{1}{2}\ln |{\mathbf{\Sigma }}_i|+\ln P({\omega }_i)$$

Since the discriminant function $g_i(\mathbf{x})$ contains the quadratic term ${\mathbf{x}}^T{\mathbf{W}}_i\mathbf{x}$, where ${\mathbf{W}}_i=-\frac{1}{2}{\mathbf{\Sigma }}_i^{-1}$ depends on the class $i$, the resulting function is quadratic in $\mathbf{x}$.

The decision boundary between two classes ${\omega }_i$ and ${\omega }_j$ is defined by $g_i(\mathbf{x})=g_j(\mathbf{x})$, which leads to:

Because ${\mathbf{W}}_i\ne {\mathbf{W}}_j$ in general (since ${\mathbf{\Sigma }}_i\ne {\mathbf{\Sigma }}_j$), the quadratic term ${\mathbf{x}}^T({\mathbf{W}}_i-{\mathbf{W}}_j)\mathbf{x}$ does not cancel out. This equation describes a hyperquadric surface in the feature space ${\mathbb{R}}^d$.

These hyperquadric decision boundaries can take various forms, including:

• Hyperplanes (only if ${\mathbf{\Sigma }}_i={\mathbf{\Sigma }}_j$, which reduces to Case 2)

• Pairs of hyperplanes

• Hyperspheres

• Hyperellipsoids

• Hyperparaboloids

• Hyperhyperboloids

In 2D space, these correspond to familiar shapes like lines, pairs of lines, circles, ellipses, parabolas, and hyperbolas. The specific shape depends on the eigenvalues of the matrix $({\mathbf{W}}_i-{\mathbf{W}}_j)$ and the other terms in the equation.

This general case allows for much more complex decision boundaries compared to the linear boundaries of Cases 1 and 2, reflecting the greater flexibility afforded by allowing each class to have its own unique covariance matrix.

The figures referenced illustrate this complexity:

• Figure 2.14: Shows various quadratic decision boundaries (ellipses, hyperbolas, parabolas, lines) that can arise in 2D when covariance matrices are arbitrary.

Figure 2.14: Arbitrary Gaussian distributions lead to Bayes decision boundaries that are general hyperquadrics. Conversely, given any hyperquadratic, one can find two Gaussian distributions whose Bayes decision boundary is that hyperquadric.

This quadratic classifier is the most general form for Gaussian distributions under the Bayesian framework.

The extension of these results to more than two categories is straightforward though we need to keep clear which two of the total c categories are responsible for any boundary segment. Figure 2.16 shows the decision surfaces for a four-category case made up of Gaussian distributions. Of course, if the distributions are more complicated, the decision regions can be even more complex, though the same underlying theory holds there too.

Figure 2.16: The decision regions for four normal distributions. Even with such a low number of categories, the shapes of the boundary regions can be rather complex.