Mahalanobis Distance

Understanding Covariance, Geometric Interpretation, and Bayesian Decision Theory

In this tutorial, we will explore the covariance matrix, its geometric interpretation, and its critical role in Bayesian Decision Theory, particularly in the context of the Mahalanobis Distance. We will derive the mathematical properties of these concepts and illustrate them with Python code and visualizations.

Resources: Adapted from Vision Dummy, SAS Blogs, Wikipedia, and Chapter 2 of Pattern Classification by Duda, Hart, and Stork Duda et al. (2000).

1. Why Do We Need Mahalanobis Distance?¶

In many real-world applications, we deal with multivariate data where features are often correlated and have different scales. While the Euclidean distance ($\|\mathbf{x} - \mathbf{y}\|_2$) is a common choice for measuring distances, it has a significant limitation: it assumes that all features are uncorrelated and have the same variance. This assumption rarely holds in practice.

Consider a dataset where one feature is measured in meters and another in kilometers. Euclidean distance would disproportionately weigh the feature with larger values. Similarly, if features are correlated, the “shape” of the data distribution is an ellipsoid rather than a sphere. Euclidean distance fails to account for this underlying structure, leading to misleading results in classification and clustering.

The Mahalanobis distance solves this by measuring the distance between a point and a distribution, normalized by the variance of each feature and the correlations between them.

Illustrative Example: Euclidean vs. Mahalanobis Distance¶

Let’s visualize this using a 2D Gaussian distribution with high correlation.

Observation: Point 1 and Point 2 have the same Euclidean distance from the center, yet they lie in different “probability regions.” Point 2 is in a direction of lower variance (orthogonal to the main correlation), making it statistically less likely (further in Mahalanobis distance) than Point 1.

The effect of feature scaling¶

Case Study: Iranian Missile Similarity¶

In this example, we will use a dataset of Iranian Ballistic Missiles to solve a similarity problem. We want to determine which missile is more similar to the Khaybar Shikan (Missile A) based on Range and Speed.

1. The Dataset¶

We have the following data for three missiles:

Missile	Range (km)	Speed (Mach)
A. Khaybar Shikan	1,450	5
B. Sejjil	2,000	10
C. Fatāh	1,400	14

2. The Naive Approach (Raw Distance)¶

Using standard Euclidean distance, we calculate the distance between the Target (Khaybar) and the other missiles.

• Khaybar vs. Sejjil (B):

  • $\Delta \text{Range} = |2000 - 1450| = 550 \text{ km}$

  • $\Delta \text{Speed} = |10 - 5| = 5 \text{ Mach}$

  • $D_{raw} = \sqrt{550^2 + 5^2} = \sqrt{302,500 + 25} \approx \mathbf{550.02}$

• Khaybar vs. Fatāh (C):

  • $\Delta \text{Range} = |1400 - 1450| = 50 \text{ km}$

  • $\Delta \text{Speed} = |14 - 5| = 9 \text{ Mach}$

  • $D_{raw} = \sqrt{50^2 + 9^2} = \sqrt{2,500 + 81} \approx \mathbf{50.80}$

❌ Problem: Fatāh looks much closer (50.8) than Sejjil (550.0). Reality Check: 50 km is negligible in missile range, but 9 Mach is a massive difference in speed class (Hypersonic vs. Medium). The raw calculation is dominated by the Range feature.

3. The Corrected Approach (Standard Scaling)¶

To fix this, we normalize the data using Z-scores ($Z = \frac{x - \mu}{\sigma}$). This puts both features on the same scale.

Dataset Statistics (from Code Output):

• Range: Mean = 1,616.67 km | Std Dev (Sample) $\approx$ 332.92 km

• Speed: Mean = 9.67 Mach | Std Dev (Sample) $\approx$ 4.51 Mach

Note: The StandardScaler algorithm typically uses Population Std Dev (approx 272 km / 3.7 Mach) for internal transformation to ensure consistency, which results in the final distances below.

A. Khaybar vs. Sejjil (B):

• $D_{scaled} = \sqrt{1.90^2 + 1.40^2} \approx \mathbf{2.44}$

B. Khaybar vs. Fatāh (C):

• $D_{scaled} = \sqrt{0.20^2 + 2.40^2} \approx \mathbf{2.45}$

4. The Result¶

✅ Result: Sejjil (2.44) is now correctly identified as the closest match to Khaybar, not Fatāh.

5. Why the Change?¶

1. Raw Data: The Range feature (km) was numerically huge (1400-2000) compared to Speed (5-14). The algorithm ignored Speed because a change of 50 km is a tiny percentage of the total range, but a change of 9 Mach is a 180% change in Speed.

2. Scaled Data: By dividing by the Standard Deviation, we turned Speed into the dominant factor.

3. Conclusion: The Scaled Distance penalizes the massive speed difference in Fatāh, correctly identifying Sejjil as the more similar missile platform.

Code Logic Summary¶

The Python script used StandardScaler to handle this automatically. It:

1. Calculated the Mean and Standard Deviation for each column.

2. Transformed the values into Z-scores.

3. Calculated Euclidean distance on the transformed (standardized) data.

This ensures features with different units (km vs Mach) contribute equally to the similarity score.

Key Takeaway¶

When building ML models, Unit Mismatch is a silent killer.

Always scale your features (Standardization/Normalization) before calculating distances, training Neural Networks, or using K-Means Clustering.

2. Introduction to Covariance Matrix¶

The covariance matrix is a square matrix that summarizes the variances and covariances of a set of random variables. For a dataset with $d$ features, the covariance matrix $\boldsymbol{\Sigma}$ is a $d \times d$ matrix defined as:

Where:

• $\sigma_{ii} = \operatorname{Var}(x_i)$: The variance of the $i$-th feature (diagonal elements).

• $\sigma_{ij} = \operatorname{Cov}(x_i, x_j)$: The covariance between the $i$-th and $j$-th features (off-diagonal elements).

Properties:

1. Symmetric: $\sigma_{ij} = \sigma_{ji}$, so $\boldsymbol{\Sigma} = \boldsymbol{\Sigma}^{\top}$.

2. Positive Semi-Definite: All eigenvalues are non-negative ($\lambda_i \geq 0$).

3. Geometric Interpretation of Covariance Matrix¶

The covariance matrix defines the shape and orientation of the data distribution in the feature space.

1. Eigenvectors (Directions): The eigenvectors of $\boldsymbol{\Sigma}$ represent the principal axes of the data distribution. They indicate the directions of maximum variance.

2. Eigenvalues (Magnitudes): The corresponding eigenvalues represent the magnitude of variance along these eigenvector directions.

3. Ellipsoid: In 2D, the covariance matrix defines an ellipse. In $d$-dimensions, it defines a hyper-ellipsoid. The axes of this ellipsoid are aligned with the eigenvectors, and the lengths of the axes are proportional to the square roots of the eigenvalues.

Whitening Transformation

Whitening is a process that transforms data so that its covariance matrix becomes the identity matrix. This transformation removes correlations between features and scales the features to have unit variance. Whitening is particularly useful in machine learning and signal processing because it simplifies the structure of the data, making it easier to apply algorithms that assume uncorrelated features.

Figure 2.8 Duda et al. (2000): The action of a linear transformation on the feature space will convert an arbitrary normal distribution into another normal distribution.

In the following we will see that a linear transformation $W^{\!\top}$ applied to a zero-mean vector $\mathbf{x}-\boldsymbol{\mu}$ produces a new vector whose covariance is obtained by “sandwiching” the old covariance between $W^{\!\top}$ and $W$. This property is fundamental to whitening, where we choose $W$ specifically so that the new covariance becomes the Identity matrix $\mathbf{I}$.

Example: Whitening in Python

The following Python code demonstrates how to whiten a dataset:

4. Linear Transformations and Covariance¶

To fully understand Mahalanobis distance and whitening, we must understand how covariance changes under linear transformations. The following theorem provides the mathematical foundation for transforming data to a “whitened” space.

Theorem 1 (Covariance Transformation Property)¶

Let $\mathbf{x} \in \mathbb{R}^d$ be a random vector with mean $\boldsymbol{\mu} = \mathbb{E}[\mathbf{x}]$ and covariance matrix $\boldsymbol{\Sigma} = \operatorname{Cov}(\mathbf{x})$. Let $W \in \mathbb{R}^{d \times d}$ be a constant transformation matrix. If the transformed random vector $\mathbf{y}$ defines as $\mathbf{y} = W^{\!\top}\bigl(\mathbf{x} - \boldsymbol{\mu}\bigr)$; then, the covariance matrix of the transformed vector $\mathbf{y}$ be as follows:

Proof: We proceed by applying the definition of covariance and the linearity of expectation.

1. Mean of $\mathbf{y}$: First, observe that the mean of the centered vector is zero:

   $$\begin{aligned} \mathbb{E}[\mathbf{y}] &= \mathbb{E}\bigl[W^{\!\top}(\mathbf{x} - \boldsymbol{\mu})\bigr] \\ &= W^{\!\top}\bigl(\mathbb{E}[\mathbf{x}] - \boldsymbol{\mu}\bigr) \\ &= W^{\!\top}(\boldsymbol{\mu} - \boldsymbol{\mu}) \\ &= \mathbf{0}. \end{aligned}$$

   (3)

2. Covariance of $\mathbf{y}$:

   $$\text{Cov}(\mathbf{y}) = \mathbb{E}[(\mathbf{y} - \mathbb{E}[\mathbf{y}])(\mathbf{y} - \mathbb{E}[\mathbf{y}])^T]$$

   (4)

   Since $\mathbb{E}[\mathbf{y}] = \mathbf{0}$, the covariance definition simplifies to $\operatorname{Cov}(\mathbf{y}) = \mathbb{E}[\mathbf{y}\mathbf{y}^{\!\top}]$.

   $$\begin{aligned} \operatorname{Cov}(\mathbf{y}) &= \mathbb{E}\bigl[\mathbf{y}\,\mathbf{y}^{\!\top}\bigr] \\ &= \mathbb{E}\Big[W^{\!\top}(\mathbf{x}-\boldsymbol{\mu})\,(\mathbf{x}-\boldsymbol{\mu})^{\!\top}W\Big]. \end{aligned}$$

   (5)

   By the linearity of the expectation operator, constant matrices $W^{\!\top}$ and $W$ can be pulled out of the expectation:

   $$\begin{aligned} \operatorname{Cov}(\mathbf{y}) &= W^{\!\top}\;\mathbb{E}\Big[(\mathbf{x}-\boldsymbol{\mu})(\mathbf{x}-\boldsymbol{\mu})^{\!\top}\Big]\;W. \end{aligned}$$

   (6)

   Recognizing that the term inside the expectation is exactly the definition of $\operatorname{Cov}(\mathbf{x}) = \boldsymbol{\Sigma}$:

   $$\operatorname{Cov}(\mathbf{y}) = W^{\!\top}\boldsymbol{\Sigma}W. \quad \square$$

   (7)

Remark (Whitening Application): This property is fundamental to whitening. By choosing $W$ such that $W^{\!\top}\boldsymbol{\Sigma}W = \mathbf{I}$, we transform the data into a space where features are uncorrelated and have unit variance. As shown in Section 5, this $W$ is constructed using the eigendecomposition of $\boldsymbol{\Sigma}$.

5. Whitening Transformation¶

Whitening is a preprocessing step that transforms data so that its covariance matrix becomes the Identity matrix ($\mathbf{I}$). This implies the transformed features are uncorrelated and have unit variance.

Mathematical Formulation¶

We seek a transformation matrix $W$ such that if $\mathbf{y} = W^{\top}(\mathbf{x} - \boldsymbol{\mu})$, then $\operatorname{Cov}(\mathbf{y}) = \mathbf{I}$.

From the proof in Section 4:

We can solve for $W$ using the Eigen Decomposition of $\boldsymbol{\Sigma}$. Let $\boldsymbol{\Sigma} = \Phi \Lambda \Phi^{\top}$, where:

• $\Phi$ is the orthogonal matrix of eigenvectors.

• $\Lambda$ is the diagonal matrix of eigenvalues.

We choose $W = \Phi \Lambda^{-1/2}$. Let’s verify:

Geometric Interpretation¶

1. Rotation ($\Phi^{\top}$): Aligns the data with the principal axes (removes correlations).

2. Scaling ($\Lambda^{-1/2}$): Stretches or compresses the data along these axes so that all axes have length 1 (unit variance).

Result: The data cloud becomes a perfect sphere (isotropic). In this whitened space, the Euclidean distance is equivalent to the Mahalanobis distance in the original space.

6. Mahalanobis Distance¶

The Mahalanobis distance is a measure of the distance between a point $\mathbf{x}$ and a distribution with mean $\boldsymbol{\mu}$ and covariance matrix $\mathbf{\Sigma}$. It is defined as:

This distance is central to many multivariate statistical methods, including Bayesian decision theory for classification, because it accounts for the underlying structure (variances and correlations) of the data.

The Key Insight¶

The covariance transformation formula from Section 4 is the fundamental mathematical foundation for both whitening transformation and Mahalanobis distance.

Recall the Mahalanobis distance:

Now consider the whitened space: If we define $\mathbf{y} = W^{\!\top}(\mathbf{x}-\boldsymbol{\mu})$ where $W$ makes $\operatorname{Cov}(\mathbf{y}) = \mathbf{I}$, then:

For this to equal the Mahalanobis distance, we need to show that $W W^{\!\top} = \boldsymbol{\Sigma}^{-1}$.

Proof:

We are given: $\boldsymbol{\Sigma} = \Phi \Lambda \Phi^{\top}$ and $W = \Phi \Lambda^{-1/2}$

We aim to show that $W W^{\!\top} = \boldsymbol{\Sigma}^{-1}$

Step 1: Compute $W W^{\!\top}$

Substitute $W = \Phi \Lambda^{-1/2}$:

Now, transpose the second factor:

Since $\Lambda$ is a diagonal matrix with positive eigenvalues (implying invertibility), $\Lambda^{-1/2}$ is also diagonal and symmetric, so:

Thus,

Step 2: Compute $\boldsymbol{\Sigma}^{-1}$ using the inverse of a diagonalized matrix

We are given:

Since $\Phi$ is an orthogonal matrix (i.e., $\Phi^{\!\top} \Phi = I$), we know that:

Because $\Phi$ is orthogonal:

• $\Phi^{-1} = \Phi^{\!\top}$

• So $(\Phi^{\!\top})^{-1} = \Phi$

Therefore,

We have shown that:

Key insight: The orthogonality of $\Phi$ allows us to simplify the inverse of $\Phi \Lambda \Phi^{\!\top}$ elegantly, which is essential for establishing the result.

Key Relationship¶

7. Python Implementation: Step-by-Step Demonstration¶

8. Covariance Matrix in Bayesian Decision Theory¶

• $\boldsymbol{\mu}_i$: Mean vector for class $\omega_i$.

• $\boldsymbol{\Sigma}_i$: Covariance matrix for class $\omega_i$.

The term in the exponent is exactly half the squared Mahalanobis distance. The covariance matrix determines the “tightness” and orientation of the decision boundaries.

9. Mahalanobis Distance and Its Role¶

The Mahalanobis distance measures the distance between a point $\mathbf{x}$ and a distribution (characterized by mean $\boldsymbol{\mu}$ and covariance $\boldsymbol{\Sigma}$).

Connection to Standardization (Z-Score)¶

To build intuition, compare this to the 1D Z-score:

• 1D: We normalize by $\sigma^2$ (variance).

• d-Dimensions: We normalize by $\boldsymbol{\Sigma}$ (covariance matrix).

The inverse covariance matrix $\boldsymbol{\Sigma}^{-1}$ effectively:

1. Scales features by their variance (normalizing units).

2. Rotates the space to decorrelate features.

Mahalanobis Distance Intuition¶

If the data distribution is a cloud of points shaped like an ellipsoid, Mahalanobis distance measures how many “standard deviations away” a point is, taking into account the stretching and rotation of that ellipsoid.

The data cloud becomes a perfect sphere (isotropic). In this whitened space, the Euclidean distance is equivalent to the Mahalanobis distance in the original space.

Is Whitening Equivalent to Per-Feature Normalization?¶

No, they are NOT equivalent. This is a critical distinction.

Per-Feature Normalization (Standardization)¶

• Scales each feature independently to unit variance

• Preserves correlations between features

Whitening¶

• Rotates AND scales the entire space

• Removes correlations and normalizes variance

• Covariance becomes identity matrix $\mathbf{I}$

• Data cloud becomes a perfect sphere

Key Difference¶

Why It Matters¶

For Mahalanobis distance and Bayesian decision theory, only whitening ensures that Euclidean distance in the transformed space equals Mahalanobis distance in the original space. Per-feature normalization alone is insufficient.

10. Comparison: PCA vs. Whitening¶

Principal Component Analysis (PCA) and Whitening are closely related linear transformations based on eigendecomposition of the covariance matrix. However, they have distinct goals and effects on the data distribution.

Mathematical Relationship¶

Both transformations start with the eigen decomposition of the covariance matrix $\boldsymbol{\Sigma} = \Phi \Lambda \Phi^{\!\top}$.

1. PCA (Rotation Only): Projects data onto the principal axes (eigenvectors). It removes correlations but preserves the variance scale.

   $$\mathbf{z}_{\text{PCA}} = \Phi^{\!\top}(\mathbf{x} - \boldsymbol{\mu})$$

   (25)

   Resulting Covariance:

   $$\operatorname{Cov}(\mathbf{z}_{\text{PCA}}) = \Lambda = \operatorname{diag}(\lambda_1, \lambda_2, \dots, \lambda_d)$$

   (26)

2. Whitening (Rotation + Scaling): Applies PCA rotation followed by a scaling step to normalize the variance of each component to 1.

   $$\mathbf{y}_{\text{White}} = \Lambda^{-1/2} \Phi^{\!\top}(\mathbf{x} - \boldsymbol{\mu}) = \Lambda^{-1/2} \mathbf{z}_{\text{PCA}}$$

   (27)

   Resulting Covariance:

   $$\operatorname{Cov}(\mathbf{y}_{\text{White}}) = \mathbf{I}$$

   (28)

Key Differences¶

Visual Demonstration: PCA vs. Whitening¶

Why This Matters for Mahalanobis Distance¶

1. PCA aligns the data with the principal axes but preserves the different variances along those axes. If you measure Euclidean distance in PCA space, a point 2 units away along a high-variance axis is treated the same as a point 2 units away along a low-variance axis. This distorts the statistical probability.

2. Whitening normalizes the variances. In this space, 1 unit equals 1 standard deviation in every direction. Therefore, Euclidean distance in whitened space correctly represents the statistical “surprise” (Mahalanobis distance) of a point.

Conclusion¶

The Mahalanobis distance is the natural generalization of the Z-score to multivariate dimensions. By incorporating the covariance matrix, it provides a statistically valid measure of distance that accounts for feature scales and correlations. This makes it indispensable for:

• Anomaly Detection: Identifying points that are statistically unlikely.

• Classification: Bayesian decision boundaries rely on this distance metric.

• Preprocessing: Whitening transforms data to simplify downstream algorithms (e.g., PCA, Neural Networks).

Key Takeaways¶

• Euclidean Distance assumes spherical data; Mahalanobis Distance assumes elliptical data.

• The Covariance Matrix $\boldsymbol{\Sigma}$ captures the shape, scale, and orientation of the data distribution.

• Whitening transforms $\boldsymbol{\Sigma}$ to $\mathbf{I}$, effectively “spherifying” the data cloud.

• Linear Transformation Property: If $\mathbf{y} = W^{\top}(\mathbf{x}-\boldsymbol{\mu})$, then $\operatorname{Cov}(\mathbf{y}) = W^{\top}\boldsymbol{\Sigma}W$.

• In the Whitened Space, Euclidean distance equals Mahalanobis distance.

References¶

Duda et al. (2000)

Duda, R. O., Hart, P. E., & Stork, D. G. (2000). Pattern Classification (2nd ed.). Wiley-Interscience.