Skip to article frontmatterSkip to article content
Site not loading correctly?

This may be due to an incorrect BASE_URL configuration. See the MyST Documentation for reference.

Mahalanobis Distance

Mahmood Amintoosi, Fall 2026

Computer Science Dept, Ferdowsi University of Mashhad

The covariance material follows chapter 2 of Pattern Classification by Duda, Hart, and Stork (Duda et al. (2000)), whose notation we use; the geometric explanations are adapted from Vision Dummy and from the SAS blog on the Mahalanobis distance.

Learning objectives. By the end you should be able to

  1. say why Euclidean distance fails when features have different units or are correlated;

  2. read a covariance matrix, and its eigenvalues and eigenvectors, as shape and orientation;

  3. prove that Cov⁡(W⊤(x−μ))=W⊤ΣW\operatorname{Cov}(W^{\top}(\mathbf{x}-\boldsymbol{\mu})) = W^{\top}\boldsymbol{\Sigma}W;

  4. build the whitening matrix W=ΦΛ−1/2W = \Phi\Lambda^{-1/2} and show that whitening turns the Mahalanobis distance into a Euclidean one;

  5. compute Mahalanobis distances and whitened coordinates with NumPy;

  6. tell apart standardization, PCA and whitening;

  7. place the Mahalanobis distance in the Gaussian discriminant of Bayesian decision theory.

English · فارسی
English · فارسی

1. Why Do We Need the Mahalanobis Distance?

In most applications the data are multivariate, the features are correlated and they are measured on different scales. The Euclidean distance (∥x−y∥2\|\mathbf{x}-\mathbf{y}\|_2) assumes the opposite: uncorrelated features with the same variance. If one feature is in metres and another in kilometres, the second one dominates the sum by its numerical size alone. If the features are correlated, the cloud is an ellipsoid rather than a sphere, and Euclidean distance ignores the orientation of that ellipsoid — misleading results in classification and in clustering follow.

The Mahalanobis distance measures the distance between a point and a distribution, normalised by the variance of each feature and by the correlations between them.

Illustrative Example: Euclidean versus Mahalanobis

A two-dimensional Gaussian with high correlation, and two points at the same Euclidean distance from its centre, one along the direction of large variance and one across it:

Source
<Figure size 1000x800 with 1 Axes>

Observation: Point 1 and Point 2 have the same Euclidean distance from the center, yet they lie in different “probability regions.” Point 2 is in a direction of lower variance (orthogonal to the main correlation), making it statistically less likely (further in Mahalanobis distance) than Point 1.


2. Feature Scaling: A Case Study

The Data

Three ballistic missiles, described by two features: range and speed.

MissileRange (km)Speed (Mach)
A. Khaybar Shikan (target)1,4505
B. Sejjil2,00010
C. Fatāh1,40014

Which of B and C is more similar to A?

Raw Euclidean Distance

  • Khaybar vs. Sejjil (B): Δrange=550\Delta\text{range} = 550 km and Δspeed=5\Delta\text{speed} = 5 Mach, so Draw=5502+52=550.02D_{\text{raw}} = \sqrt{550^2 + 5^2} = 550.02.

  • Khaybar vs. Fatāh (C): Δrange=50\Delta\text{range} = 50 km and Δspeed=9\Delta\text{speed} = 9 Mach, so Draw=502+92=50.80D_{\text{raw}} = \sqrt{50^2 + 9^2} = 50.80.

Fatāh looks ten times closer. The reason is arithmetic rather than similarity: the range column takes values in the thousands, so 50 km is a small number next to 5 or 9 in the speed column. The feature with the larger numerical range decides the answer.

Standardized Distance

Divide each feature by its standard deviation, z=(x−μ)/σz = (x-\mu)/\sigma, so that both are measured in standard deviations. For the three missiles

  • range: μ=1616.7\mu = 1616.7 km, population σ≈271.8\sigma \approx 271.8 km (sample s≈332.9s \approx 332.9 km);

  • speed: μ=9.67\mu = 9.67 Mach, population σ≈3.68\sigma \approx 3.68 Mach (sample s≈4.51s \approx 4.51 Mach).

StandardScaler divides by the population standard deviation, by NN and not by N−1N-1, which is why the distances below use 271.8 and 3.68 while pandas prints 332.9 and 4.51. In z-units A is (−0.61,−1.27)(-0.61, -1.27), B is (1.41,0.09)(1.41, 0.09) and C is (−0.80,1.18)(-0.80, 1.18), so

  • Khaybar vs. Sejjil: Dscaled=2.022+1.362=2.44D_{\text{scaled}} = \sqrt{2.02^2 + 1.36^2} = 2.44;

  • Khaybar vs. Fatāh: Dscaled=0.182+2.442=2.45D_{\text{scaled}} = \sqrt{0.18^2 + 2.44^2} = 2.45.

ComparisonRaw distanceStandardized distance
Khaybar vs. Sejjil550.022.44
Khaybar vs. Fatāh50.802.45

The answer flips, by a hair: after standardization Sejjil is the nearer neighbour.

Which Answer Is Right?

Neither. Standardizing says the two features deserve equal weight; the raw distance says one kilometre counts as much as one Mach. Both are assumptions about the problem, not facts about missiles, and the example is built so that they disagree. That is the lesson to keep: for any method based on Euclidean distance — k-NN, k-means, the input layer of a network — the scaling is part of the model, and it should be chosen rather than inherited from the units in which the data happen to be stored.

The Code

The next cell does the arithmetic: raw distances, then StandardScaler, then the same Euclidean distance on the standardized columns.

Source
=== RAW EUCLIDEAN DISTANCE (Unscaled) ===
Distance to Sejjil: 550.02
Distance to Fatāh: 50.80

=== SCALED EUCLIDEAN DISTANCE (Normalized) ===
Distance to Sejjil: 2.44
Distance to Fatāh: 2.45

--- ANALYSIS ---
Winner with raw distances: Fatāh
Winner after standardization: Sejjil

--- DATASET STATISTICS ---
Range Std Dev: 332.92 km
Speed Std Dev: 4.51 Mach

The two printed statistics use different denominators: df.std() is the sample standard deviation (N−1N-1) while StandardScaler uses the population one (NN). Both are reasonable; the distances above come from the population version.

3. The Covariance Matrix

The covariance matrix is a square matrix that summarizes the variances and covariances of a set of random variables. For a dataset with dd features, the covariance matrix Σ\boldsymbol{\Sigma} is a d×dd \times d matrix defined as:

Σ=[σ11σ12⋯σ1dσ21σ22⋯σ2d⋮⋮⋱⋮σd1σd2⋯σdd]\boldsymbol{\Sigma} = \begin{bmatrix} \sigma_{11} & \sigma_{12} & \cdots & \sigma_{1d} \\ \sigma_{21} & \sigma_{22} & \cdots & \sigma_{2d} \\ \vdots & \vdots & \ddots & \vdots \\ \sigma_{d1} & \sigma_{d2} & \cdots & \sigma_{dd} \end{bmatrix}

Where:

  • σii=Var⁡(xi)\sigma_{ii} = \operatorname{Var}(x_i): The variance of the ii-th feature (diagonal elements).

  • σij=Cov⁡(xi,xj)\sigma_{ij} = \operatorname{Cov}(x_i, x_j): The covariance between the ii-th and jj-th features (off-diagonal elements).

Properties:

  1. Symmetric: σij=σji\sigma_{ij} = \sigma_{ji}, so Σ=Σ⊤\boldsymbol{\Sigma} = \boldsymbol{\Sigma}^{\top}.

  2. Positive Semi-Definite: All eigenvalues are non-negative (λi≥0\lambda_i \geq 0).


4. Geometric Interpretation of the Covariance Matrix

The covariance matrix defines the shape and orientation of the data distribution in the feature space.

  1. Eigenvectors (Directions): The eigenvectors of Σ\boldsymbol{\Sigma} represent the principal axes of the data distribution. They indicate the directions of maximum variance.

  2. Eigenvalues (Magnitudes): The corresponding eigenvalues represent the magnitude of variance along these eigenvector directions.

  3. Ellipsoid: In 2D, the covariance matrix defines an ellipse. In dd-dimensions, it defines a hyper-ellipsoid. The axes of this ellipsoid are aligned with the eigenvectors, and the lengths of the axes are proportional to the square roots of the eigenvalues.


5. Linear Transformations and Covariance

To fully understand Mahalanobis distance and whitening, we must understand how covariance changes under linear transformations. The following theorem provides the mathematical foundation for transforming data to a “whitened” space.

Theorem 1 (Covariance Transformation Property)

Let x∈Rd\mathbf{x} \in \mathbb{R}^d be a random vector with mean μ=E[x]\boldsymbol{\mu} = \mathbb{E}[\mathbf{x}] and covariance matrix Σ=Cov⁡(x)\boldsymbol{\Sigma} = \operatorname{Cov}(\mathbf{x}). Let W∈Rd×dW \in \mathbb{R}^{d \times d} be a constant transformation matrix. For a transformed random vector y=W ⁣⊤(x−μ)\mathbf{y} = W^{\!\top}(\mathbf{x} - \boldsymbol{\mu}), the covariance matrix of y\mathbf{y} is

Cov⁡(y)=W ⁣⊤ΣW\boxed{\operatorname{Cov}(\mathbf{y}) = W^{\!\top}\boldsymbol{\Sigma}W}

Proof: We proceed by applying the definition of covariance and the linearity of expectation.

  1. Mean of y\mathbf{y}: First, observe that the mean of the centered vector is zero:

    E[y]=E[W ⁣⊤(x−μ)]=W ⁣⊤(E[x]−μ)=W ⁣⊤(μ−μ)=0.\begin{aligned} \mathbb{E}[\mathbf{y}] &= \mathbb{E}\bigl[W^{\!\top}(\mathbf{x} - \boldsymbol{\mu})\bigr] \\ &= W^{\!\top}\bigl(\mathbb{E}[\mathbf{x}] - \boldsymbol{\mu}\bigr) \\ &= W^{\!\top}(\boldsymbol{\mu} - \boldsymbol{\mu}) \\ &= \mathbf{0}. \end{aligned}
  2. Covariance of y\mathbf{y}:

    Cov(y)=E[(y−E[y])(y−E[y])T]\text{Cov}(\mathbf{y}) = \mathbb{E}[(\mathbf{y} - \mathbb{E}[\mathbf{y}])(\mathbf{y} - \mathbb{E}[\mathbf{y}])^T]

    Since E[y]=0\mathbb{E}[\mathbf{y}] = \mathbf{0}, the covariance definition simplifies to Cov⁡(y)=E[yy ⁣⊤]\operatorname{Cov}(\mathbf{y}) = \mathbb{E}[\mathbf{y}\mathbf{y}^{\!\top}].

    Cov⁡(y)=E[y y ⁣⊤]=E[W ⁣⊤(x−μ) (x−μ) ⁣⊤W].\begin{aligned} \operatorname{Cov}(\mathbf{y}) &= \mathbb{E}\bigl[\mathbf{y}\,\mathbf{y}^{\!\top}\bigr] \\ &= \mathbb{E}\Big[W^{\!\top}(\mathbf{x}-\boldsymbol{\mu})\,(\mathbf{x}-\boldsymbol{\mu})^{\!\top}W\Big]. \end{aligned}

    By the linearity of the expectation operator, constant matrices W ⁣⊤W^{\!\top} and WW can be pulled out of the expectation:

    Cov⁡(y)=W ⁣⊤  E[(x−μ)(x−μ) ⁣⊤]  W.\begin{aligned} \operatorname{Cov}(\mathbf{y}) &= W^{\!\top}\;\mathbb{E}\Big[(\mathbf{x}-\boldsymbol{\mu})(\mathbf{x}-\boldsymbol{\mu})^{\!\top}\Big]\;W. \end{aligned}

    Recognizing that the term inside the expectation is exactly the definition of Cov⁡(x)=Σ\operatorname{Cov}(\mathbf{x}) = \boldsymbol{\Sigma}:

    Cov⁡(y)=W ⁣⊤ΣW.□\operatorname{Cov}(\mathbf{y}) = W^{\!\top}\boldsymbol{\Sigma}W. \quad \square

Remark (Whitening Application): This property is fundamental to whitening. By choosing WW such that W ⁣⊤ΣW=IW^{\!\top}\boldsymbol{\Sigma}W = \mathbf{I}, we transform the data into a space where features are uncorrelated and have unit variance. Section 6 constructs this WW from the eigendecomposition of Σ\boldsymbol{\Sigma}.

6. Whitening Transformation

Whitening is a preprocessing step that turns the covariance matrix into the identity. The transformed features are then uncorrelated and have unit variance.

Figure 2.8 of Duda

Figure 2.8 Duda et al. (2000): a linear transformation of the feature space maps an arbitrary normal distribution to another normal distribution.

We want a matrix WW such that y=W⊤(x−μ)\mathbf{y} = W^{\top}(\mathbf{x} - \boldsymbol{\mu}) has covariance I\mathbf{I}. Section 5 gives Cov⁡(y)=W⊤ΣW\operatorname{Cov}(\mathbf{y}) = W^{\top}\boldsymbol{\Sigma}W, and the eigendecomposition of Section 4, Σ=ΦΛΦ⊤\boldsymbol{\Sigma} = \Phi\Lambda\Phi^{\top}, provides W=ΦΛ−1/2W = \Phi\Lambda^{-1/2}. Checking it:

Cov⁡(y)=(ΦΛ−1/2)⊤Σ(ΦΛ−1/2)=Λ−1/2Φ⊤(ΦΛΦ⊤)ΦΛ−1/2=Λ−1/2ΛΛ−1/2=I.\begin{aligned} \operatorname{Cov}(\mathbf{y}) &= (\Phi \Lambda^{-1/2})^{\top} \boldsymbol{\Sigma} (\Phi \Lambda^{-1/2}) \\ &= \Lambda^{-1/2} \Phi^{\top} (\Phi \Lambda \Phi^{\top}) \Phi \Lambda^{-1/2} \\ &= \Lambda^{-1/2} \Lambda \Lambda^{-1/2} = \mathbf{I}. \end{aligned}

Geometric Interpretation

  1. Rotation (Φ⊤\Phi^{\top}) aligns the data with the principal axes and removes the correlations.

  2. Scaling (Λ−1/2\Lambda^{-1/2}) stretches or compresses each axis until its variance is 1.

The cloud becomes a sphere. In this space the Euclidean distance is the Mahalanobis distance of the original space — Section 7 proves the identity, and the code below shows it numerically.

The Same Two Points, Whitened

The code takes the correlated cloud and the two starred points of Section 1, applies W=ΦΛ−1/2W = \Phi\Lambda^{-1/2}, and plots the data before and after. On the left the two points are at the same Euclidean distance from the centre; on the right their Euclidean distances are their Mahalanobis distances in the original space.

Source
<Figure size 1200x600 with 2 Axes>

Both points are 2.12 from the centre in the original Euclidean sense. Their Mahalanobis distances are 1.58 for the point along the principal axis and 4.74 for the point across it; after whitening the Euclidean distances are exactly those two numbers, in the same order. The circle of constant radius in the right panel is an ellipse of constant Mahalanobis distance in the left one.

7. The Mahalanobis Distance

In Section 6 we fixed the transformation and asked which covariance it produces. Here we ask the inverse question: which norm on the transformed coordinates reproduces the Mahalanobis distance of the original space?

The Mahalanobis distance is a measure of the distance between a point x\mathbf{x} and a distribution with mean μ\boldsymbol{\mu} and covariance matrix Σ\mathbf{\Sigma}. It is defined as:

DM(x,μ)=(x−μ)TΣ−1(x−μ)D_M(\mathbf{x}, \boldsymbol{\mu}) = \sqrt{(\mathbf{x} - \boldsymbol{\mu})^T \mathbf{\Sigma}^{-1} (\mathbf{x} - \boldsymbol{\mu})}

This distance is central to many multivariate statistical methods, including Bayesian decision theory for classification, because it accounts for the underlying structure (variances and correlations) of the data.

The Key Insight

The covariance transformation formula from Section 5 is the fundamental mathematical foundation for both whitening transformation and Mahalanobis distance.

Recall the Mahalanobis distance:

DM(x)=(x−μ)TΣ−1(x−μ)D_M(\mathbf{x}) = \sqrt{(\mathbf{x} - \boldsymbol{\mu})^T \boldsymbol{\Sigma}^{-1} (\mathbf{x} - \boldsymbol{\mu})}

Now consider the whitened space: If we define y=W ⁣⊤(x−μ)\mathbf{y} = W^{\!\top}(\mathbf{x}-\boldsymbol{\mu}) where WW makes Cov⁡(y)=I\operatorname{Cov}(\mathbf{y}) = \mathbf{I}, then:

∥y∥2=yTy=(x−μ)TWW ⁣⊤(x−μ)\|\mathbf{y}\|_2 = \sqrt{\mathbf{y}^T \mathbf{y}} = \sqrt{(\mathbf{x} - \boldsymbol{\mu})^T W W^{\!\top} (\mathbf{x} - \boldsymbol{\mu})}

For this to equal the Mahalanobis distance, we need to show that WW ⁣⊤=Σ−1 W W^{\!\top} = \boldsymbol{\Sigma}^{-1} .

Proof:

We are given: Σ=ΦΛΦ⊤ \boldsymbol{\Sigma} = \Phi \Lambda \Phi^{\top} and W=ΦΛ−1/2 W = \Phi \Lambda^{-1/2}

We aim to show that WW ⁣⊤=Σ−1W W^{\!\top} = \boldsymbol{\Sigma}^{-1}

Step 1: Compute WW ⁣⊤ W W^{\!\top}

Substitute W=ΦΛ−1/2 W = \Phi \Lambda^{-1/2} :

WW ⁣⊤=(ΦΛ−1/2)(ΦΛ−1/2) ⁣⊤W W^{\!\top} = (\Phi \Lambda^{-1/2}) (\Phi \Lambda^{-1/2})^{\!\top}

Now, transpose the second factor:

(ΦΛ−1/2) ⁣⊤=(Λ−1/2) ⁣⊤Φ ⁣⊤(\Phi \Lambda^{-1/2})^{\!\top} = (\Lambda^{-1/2})^{\!\top} \Phi^{\!\top}

Since Λ \Lambda is a diagonal matrix with positive eigenvalues (implying invertibility), Λ−1/2 \Lambda^{-1/2} is also diagonal and symmetric, so:

(Λ−1/2) ⁣⊤=Λ−1/2(\Lambda^{-1/2})^{\!\top} = \Lambda^{-1/2}

Thus,

WW ⁣⊤=ΦΛ−1/2(Λ−1/2)Φ ⁣⊤=Φ(Λ−1/2⋅Λ−1/2)Φ ⁣⊤=ΦΛ−1Φ ⁣⊤W W^{\!\top} = \Phi \Lambda^{-1/2} (\Lambda^{-1/2}) \Phi^{\!\top} = \Phi (\Lambda^{-1/2} \cdot \Lambda^{-1/2}) \Phi^{\!\top} = \Phi \Lambda^{-1} \Phi^{\!\top}

Step 2: Compute Σ−1 \boldsymbol{\Sigma}^{-1} using the inverse of a diagonalized matrix

We are given:

Σ=ΦΛΦ ⁣⊤\boldsymbol{\Sigma} = \Phi \Lambda \Phi^{\!\top}

Since Φ \Phi is an orthogonal matrix (i.e., Φ ⁣⊤Φ=I \Phi^{\!\top} \Phi = I ), we know that:

(ΦΛΦ ⁣⊤)−1=(Φ ⁣⊤)−1Λ−1Φ−1(\Phi \Lambda \Phi^{\!\top})^{-1} = (\Phi^{\!\top})^{-1} \Lambda^{-1} \Phi^{-1}

Because Φ \Phi is orthogonal:

  • Φ−1=Φ ⁣⊤ \Phi^{-1} = \Phi^{\!\top}

  • So (Φ ⁣⊤)−1=Φ (\Phi^{\!\top})^{-1} = \Phi

Therefore,

(ΦΛΦ ⁣⊤)−1=ΦΛ−1Φ ⁣⊤(\Phi \Lambda \Phi^{\!\top})^{-1} = \Phi \Lambda^{-1} \Phi^{\!\top}

We have shown that:

WW ⁣⊤=Σ−1\boxed{W W^{\!\top} = \boldsymbol{\Sigma}^{-1}}

Key insight: The orthogonality of Φ \Phi allows us to simplify the inverse of ΦΛΦ ⁣⊤ \Phi \Lambda \Phi^{\!\top} elegantly, which is essential for establishing the result.

Key Relationship

ConceptMathematical ExpressionPurpose
Covariance TransformationCov⁡(y)=W ⁣⊤ΣW\operatorname{Cov}(\mathbf{y}) = W^{\!\top}\boldsymbol{\Sigma}WDescribes how covariance changes under linear transformation
Whitening MatrixW=ΦΛ−1/2W = \Phi \Lambda^{-1/2}Transforms data to unit covariance
Mahalanobis DistanceDM=(x−μ)TΣ−1(x−μ)D_M = \sqrt{(\mathbf{x}-\boldsymbol{\mu})^T \boldsymbol{\Sigma}^{-1}(\mathbf{x}-\boldsymbol{\mu})}Distance accounting for covariance structure
Euclidean Distance (Whitened)∣y∣2=yTy|\mathbf{y}|_2 = \sqrt{\mathbf{y}^T\mathbf{y}}Distance after covariance normalization
Key EquivalenceDM(x)=∣y∣2D_M(\mathbf{x}) = |\mathbf{y}|_2Mahalanobis = Euclidean in whitened space

8. Python Implementation, Step by Step

======================================================================
STEP 1: Generate Correlated Data (Original Space)
======================================================================
Original Covariance:
[[4 3]
 [3 4]]

======================================================================
STEP 2: Eigen Decomposition (Rotation + Scaling Factors)
======================================================================

Eigenvalues (λ): [7. 1.]

Eigenvectors (Φ):
[[ 0.70710678 -0.70710678]
 [ 0.70710678  0.70710678]]

======================================================================
STEP 3: Rotation Only
======================================================================

Covariance after Rotation (diagonal, up to sampling error):
[[6.54609064 0.06692453]
 [0.06692453 0.98399756]]
Off-diagonal before rotation: 3.0000
Off-diagonal after rotation: 0.0669

======================================================================
STEP 4: Scaling Only (Λ^(-1/2) normalizes variance)
======================================================================

Covariance after Scaling (identity, up to sampling error):
[[0.93515581 0.02529509]
 [0.02529509 0.98399756]]

======================================================================
STEP 5:  Whitening (W = ΦΛ^(-1/2))
======================================================================

Whitening Matrix (W = ΦΛ^(-1/2)):
[[ 0.26726124 -0.70710678]
 [ 0.26726124  0.70710678]]

Covariance after Whitening (W^T Σ W):
[[0.93515581 0.02529509]
 [0.02529509 0.98399756]]

The printed covariances are those of a sample of 500 points, not of the true Σ=[4334]\boldsymbol{\Sigma} = \begin{bmatrix} 4 & 3 \\ 3 & 4\end{bmatrix}, and np.cov divides by n−1n-1. That is why the diagonal after rotation reads 6.55 and 0.98 instead of exactly 7 and 1, and why after whitening it reads 0.94 and 0.98 instead of exactly 1: the transformation is exact, the estimate of the covariance is not. Raise n_samples and the entries converge to the identity.


======================================================================
STEP 6: Visualization of Transformation
======================================================================
<Figure size 1400x1200 with 4 Axes>

======================================================================
STEP 7: Mahalanobis Distance = Euclidean in Whitened Space
======================================================================

Test Point: [4 4]
Mahalanobis Distance (Original Space): 1.5119
Euclidean Distance (Whitened Space): 1.5119

Mahalanobis distance and whitened Euclidean distance agree.

9. The Covariance Matrix in Bayesian Decision Theory

The Gaussian class-conditional density of the Bayesian decision theory page is

p(x∣ωi)=1(2π)d/2∣Σi∣1/2exp⁡(−12(x−μi)⊤Σi−1(x−μi)),p(\mathbf{x} \mid \omega_i) = \frac{1}{(2\pi)^{d/2}|\boldsymbol{\Sigma}_i|^{1/2}} \exp\left(-\frac{1}{2}(\mathbf{x} - \boldsymbol{\mu}_i)^{\top} \boldsymbol{\Sigma}_i^{-1} (\mathbf{x} - \boldsymbol{\mu}_i)\right),

where μi\boldsymbol{\mu}_i and Σi\boldsymbol{\Sigma}_i are estimated per class by maximum likelihood (MLE page).

  • The exponent is exactly half the squared Mahalanobis distance from x\mathbf{x} to μi\boldsymbol{\mu}_i.

  • A common covariance, Σi=Σ\boldsymbol{\Sigma}_i = \boldsymbol{\Sigma} for all ii, makes the log-discriminant linear, and the boundaries are hyperplanes: that is LDA.

  • A separate covariance per class leaves a quadratic term and gives QDA.

  • If Σ^i\hat{\boldsymbol{\Sigma}}_i is singular — too few samples per class compared with dd — the density, and with it the Mahalanobis distance, is undefined until the estimate is regularised.

So the Mahalanobis distance is not a distance formula added to classification: it is the geometry of the Gaussian classifier.

10. Mahalanobis Distance and Its Role

For one feature, the z-score measures how many standard deviations a value is from the mean. The Mahalanobis distance is the same idea for dd features:

z2=(x−μ)2σ2⟶DM2=(x−μ)⊤Σ−1(x−μ).z^2 = \frac{(x-\mu)^2}{\sigma^2} \qquad\longrightarrow\qquad D_M^2 = (\mathbf{x}-\boldsymbol{\mu})^{\top}\boldsymbol{\Sigma}^{-1}(\mathbf{x}-\boldsymbol{\mu}).

The role of σ2\sigma^2 is taken by the covariance matrix, so Σ−1\boldsymbol{\Sigma}^{-1} does two things at once: it scales each feature by its variance and it rotates the space to remove the correlations. Geometrically, DMD_M counts how many “standard deviations” a point is from the centre of an ellipsoidal cloud, measuring the distance along the stretched and rotated axes of that ellipsoid.

From a Distance to a Test

For data drawn from N(μ,Σ)\mathcal{N}(\boldsymbol{\mu}, \boldsymbol{\Sigma}) the squared Mahalanobis distance is a χ2\chi^2 variable with dd degrees of freedom, DM2∼χd2D_M^2 \sim \chi^2_d. The distance therefore converts into a probability statement, which is what an anomaly detector needs: the 95% and 99% quantiles of χ22\chi^2_2 are 5.99 and 9.21, so a point with DM2D_M^2 beyond them lies outside the corresponding probability contour of the ellipse. This is the multivariate version of the “more than two standard deviations away” rule, and it is the most common practical use of the distance.

11. Standardization, PCA and Whitening

All three transformations start from the same statistics and are easy to confuse. In terms of the eigendecomposition Σ=ΦΛΦ⊤\boldsymbol{\Sigma} = \Phi\Lambda\Phi^{\top}:

  • Standardization divides each feature by its own standard deviation. It changes the diagonal of the covariance to one but leaves the off-diagonal correlations in place.

  • PCA rotates to the principal axes, z=Φ⊤(x−μ)\mathbf{z} = \Phi^{\top}(\mathbf{x} - \boldsymbol{\mu}), and has covariance Λ\Lambda: the correlations are gone, the variances λi\lambda_i remain.

  • Whitening rotates and then scales, y=Λ−1/2Φ⊤(x−μ)=Λ−1/2z\mathbf{y} = \Lambda^{-1/2}\Phi^{\top}(\mathbf{x} - \boldsymbol{\mu}) = \Lambda^{-1/2}\mathbf{z}, and has covariance I\mathbf{I}.

PropertyStandardizationPCAWhitening
Goalcomparable unitsreduce dimension, keep varianceunit variance and no correlation
Transformationxi′=(xi−μi)/σix_i' = (x_i-\mu_i)/\sigma_iΦ⊤(x−μ)\Phi^{\top}(\mathbf{x}-\boldsymbol{\mu})Λ−1/2Φ⊤(x−μ)\Lambda^{-1/2}\Phi^{\top}(\mathbf{x}-\boldsymbol{\mu})
Covariance afterwardscorrelations keptdiagonal, Λ\Lambdaidentity, I\mathbf{I}
Shape of the cloudsame ellipseaxis-aligned ellipsesphere
Euclidean == Mahalanobis?nonoyes
Loss of informationnoneif k<dk<d components are keptnone at full rank

The one that makes Euclidean distance meaningful is whitening: after standardization the features still have different variances along the correlated directions, and after PCA the axes still have the variances λi\lambda_i. Only when every direction has variance 1 does a Euclidean ball become a contour of constant probability density.

Demonstration: PCA versus Whitening

The next cell applies both transformations to the same cloud and prints the three distances of one test point.

Source

======================================================================
COMPARISON: Original → PCA → Whitening
======================================================================
<Figure size 1800x600 with 3 Axes>

======================================================================
Distance Metrics for a Test Point
======================================================================

Test Point: [2. 1.]
Mahalanobis Distance (Original): 1.0690
Euclidean Distance (PCA Space):  2.2361
Euclidean Distance (White Space): 1.0690

Conclusion: only whitening keeps the Euclidean distance equal to the Mahalanobis one.

Why This Matters for Mahalanobis Distance

  1. PCA aligns the data with the principal axes but preserves the different variances along those axes. If you measure Euclidean distance in PCA space, a point 2 units away along a high-variance axis is treated the same as a point 2 units away along a low-variance axis. This distorts the statistical probability.

  2. Whitening normalizes the variances. In this space, 1 unit equals 1 standard deviation in every direction. Therefore, Euclidean distance in whitened space correctly represents the statistical “surprise” (Mahalanobis distance) of a point.

Conclusion

The Mahalanobis distance is the natural generalization of the z-score to several dimensions. Through the covariance matrix it accounts for the scale of each feature and for the correlation between features, which Euclidean distance cannot do. It is used for

  • anomaly detection: flagging points that are statistically unlikely, against a χ2\chi^2 quantile;

  • classification: it is the exponent of the Gaussian discriminant of Bayesian decision theory, whose parameters come from the MLE page;

  • preprocessing: whitening, after which Euclidean tools become valid.

Key Takeaways

  • Euclidean distance assumes spherical data; Mahalanobis distance assumes elliptical data.

  • The covariance matrix Σ\boldsymbol{\Sigma} carries the shape, the scale and the orientation of the distribution.

  • Whitening maps Σ\boldsymbol{\Sigma} to I\mathbf{I} and turns the cloud into a sphere.

  • If y=W⊤(x−μ)\mathbf{y} = W^{\top}(\mathbf{x}-\boldsymbol{\mu}) then Cov⁡(y)=W⊤ΣW\operatorname{Cov}(\mathbf{y}) = W^{\top}\boldsymbol{\Sigma}W; for W=ΦΛ−1/2W = \Phi\Lambda^{-1/2} this is I\mathbf{I}.

  • In the whitened space Euclidean distance equals Mahalanobis distance; per-feature standardization does not achieve this, because it keeps the correlations.

References

Duda et al. (2000)

Duda, R. O., Hart, P. E., & Stork, D. G. (2000). Pattern Classification (2nd ed.). Wiley-Interscience, chapter 2.

References
  1. Duda, R. O., Hart, P. E., & Stork, D. G. (2000). Pattern Classification (2nd ed.). Wiley-Interscience.