Skip to article frontmatterSkip to article content
Site not loading correctly?

This may be due to an incorrect BASE_URL configuration. See the MyST Documentation for reference.

Special Kinds of Matrices and Vectors

import numpy as np
import matplotlib.pyplot as plt
import seaborn as sns
Source
# Plot parameters
sns.set()
%pylab inline
pylab.rcParams['figure.figsize'] = (4, 4)
plt.rcParams['xtick.major.size'] = 0
plt.rcParams['ytick.major.size'] = 0
# Avoid inaccurate floating values (for inverse matrices in dot product for instance)
# See https://stackoverflow.com/questions/24537791/numpy-matrix-inversion-rounding-errors
np.set_printoptions(suppress=True)
Output
Populating the interactive namespace from numpy and matplotlib

Special Kinds of Matrices and Vectors

We have seen in previous sections some interesting kind of matrices. We will see other type of vectors and matrices in this chapter. It is not a big chapter but it is important to understand the next ones.

Diagonal and symmetric matrices

Example of diagonal and symmetric matrices

Diagonal matrices

Example of a diagonal matrix

Example of a diagonal matrix

A matrix Ai,j\boldsymbol{A}_{i,j} is diagonal if its entries are all zeros except on the diagonal (when i=ji=j).

Example 1.

D=[2000040000300001]\boldsymbol{D}= \begin{bmatrix} 2 & 0 & 0 & 0\\\\ 0 & 4 & 0 & 0\\\\ 0 & 0 & 3 & 0\\\\ 0 & 0 & 0 & 1 \end{bmatrix}
(1)

In this case the matrix is also square but there can be non square diagonal matrices.

Example 2.

D=[200040003000]\boldsymbol{D}= \begin{bmatrix} 2 & 0 & 0\\\\ 0 & 4 & 0\\\\ 0 & 0 & 3\\\\ 0 & 0 & 0 \end{bmatrix}
(2)

Or

D=[200004000030]\boldsymbol{D}= \begin{bmatrix} 2 & 0 & 0 & 0\\\\ 0 & 4 & 0 & 0\\\\ 0 & 0 & 3 & 0 \end{bmatrix}
(3)

The diagonal matrix can be denoted diag(v)diag(\boldsymbol{v}) where v\boldsymbol{v} is the vector containing the diagonal values.

Example 3.

D=[2000040000300001]\boldsymbol{D}= \begin{bmatrix} 2 & 0 & 0 & 0\\\\ 0 & 4 & 0 & 0\\\\ 0 & 0 & 3 & 0\\\\ 0 & 0 & 0 & 1 \end{bmatrix}
(4)

In this matrix, v\boldsymbol{v} is the following vector:

v=[2431]\boldsymbol{v}= \begin{bmatrix} 2\\\\ 4\\\\ 3\\\\ 1 \end{bmatrix}
(5)

The Numpy function diag() can be used to create square diagonal matrices:

v = np.array([2, 4, 3, 1])
np.diag(v)
array([[2, 0, 0, 0], [0, 4, 0, 0], [0, 0, 3, 0], [0, 0, 0, 1]])

The mutliplication between a diagonal matrix and a vector is thus just a ponderation of each element of the vector by vv:

Example 4.

D=[2000040000300001]\boldsymbol{D}= \begin{bmatrix} 2 & 0 & 0 & 0\\\\ 0 & 4 & 0 & 0\\\\ 0 & 0 & 3 & 0\\\\ 0 & 0 & 0 & 1 \end{bmatrix}
(6)

and

x=[3227]\boldsymbol{x}= \begin{bmatrix} 3\\\\ 2\\\\ 2\\\\ 7 \end{bmatrix}
(7)
Dx=[2000040000300001]×[3227]=[2×3+0×2+0×2+0×70×3+4×2+0×2+0×70×3+0×2+3×2+0×70×3+0×2+0×2+1×7]=[2×34×23×21×7]\begin{align*} &\boldsymbol{Dx}= \begin{bmatrix} 2 & 0 & 0 & 0\\\\ 0 & 4 & 0 & 0\\\\ 0 & 0 & 3 & 0\\\\ 0 & 0 & 0 & 1 \end{bmatrix} \times \begin{bmatrix} 3\\\\ 2\\\\ 2\\\\ 7 \end{bmatrix}\\\\ &=\begin{bmatrix} 2\times3 + 0\times2 + 0\times2 + 0\times7\\\\ 0\times3 + 4\times2 + 0\times2 + 0\times7\\\\ 0\times3 + 0\times2 + 3\times2 + 0\times7\\\\ 0\times3 + 0\times2 + 0\times2 + 1\times7 \end{bmatrix}\\\\ &= \begin{bmatrix} 2\times3\\\\ 4\times2\\\\ 3\times2\\\\ 1\times7 \end{bmatrix} \end{align*}
(8)

Non square matrices have the same properties:

Example 5.

D=[200040003000]\boldsymbol{D}= \begin{bmatrix} 2 & 0 & 0\\\\ 0 & 4 & 0\\\\ 0 & 0 & 3\\\\ 0 & 0 & 0 \end{bmatrix}
(9)

and

x=[322]\boldsymbol{x}= \begin{bmatrix} 3\\\\ 2\\\\ 2 \end{bmatrix}
(10)
Dx=[200040003000]×[322]=[2×34×23×20]\boldsymbol{Dx}= \begin{bmatrix} 2 & 0 & 0\\\\ 0 & 4 & 0\\\\ 0 & 0 & 3\\\\ 0 & 0 & 0 \end{bmatrix} \times \begin{bmatrix} 3\\\\ 2\\\\ 2 \end{bmatrix} = \begin{bmatrix} 2\times3\\\\ 4\times2\\\\ 3\times2\\\\ 0 \end{bmatrix}
(11)

The invert of a square diagonal matrix exists if all entries of the diagonal are non-zeros. If it is the case, the invert is easy to find. Also, the inverse doen’t exist if the matrix is non-square.

D=[2000040000300001]\boldsymbol{D}= \begin{bmatrix} 2 & 0 & 0 & 0\\\\ 0 & 4 & 0 & 0\\\\ 0 & 0 & 3 & 0\\\\ 0 & 0 & 0 & 1 \end{bmatrix}
(12)
D−1=[12000014000013000011]\boldsymbol{D}^{-1}= \begin{bmatrix} \frac{1}{2} & 0 & 0 & 0\\\\ 0 & \frac{1}{4} & 0 & 0\\\\ 0 & 0 & \frac{1}{3} & 0\\\\ 0 & 0 & 0 & \frac{1}{1} \end{bmatrix}
(13)
D=[2000040000300001][12000014000013000011]=[1000010000100001]\boldsymbol{D}= \begin{bmatrix} 2 & 0 & 0 & 0\\\\ 0 & 4 & 0 & 0\\\\ 0 & 0 & 3 & 0\\\\ 0 & 0 & 0 & 1 \end{bmatrix} \begin{bmatrix} \frac{1}{2} & 0 & 0 & 0\\\\ 0 & \frac{1}{4} & 0 & 0\\\\ 0 & 0 & \frac{1}{3} & 0\\\\ 0 & 0 & 0 & \frac{1}{1} \end{bmatrix}= \begin{bmatrix} 1 & 0 & 0 & 0\\\\ 0 & 1 & 0 & 0\\\\ 0 & 0 & 1 & 0\\\\ 0 & 0 & 0 & 1 \end{bmatrix}
(14)

Let’s check with Numpy that the multiplication of the matrix with its invert gives us the identity matrix:

A = np.array([[2, 0, 0, 0], [0, 4, 0, 0], [0, 0, 3, 0], [0, 0, 0, 1]])
A
array([[2, 0, 0, 0], [0, 4, 0, 0], [0, 0, 3, 0], [0, 0, 0, 1]])
A_inv = np.array([[1/2., 0, 0, 0], [0, 1/4., 0, 0], [0, 0, 1/3., 0], [0, 0, 0, 1/1.]])
A_inv
array([[0.5 , 0. , 0. , 0. ], [0. , 0.25 , 0. , 0. ], [0. , 0. , 0.33333333, 0. ], [0. , 0. , 0. , 1. ]])
A.dot(A_inv)
array([[1., 0., 0., 0.], [0., 1., 0., 0.], [0., 0., 1., 0.], [0., 0., 0., 1.]])

Great! This gives the identity matrix

Symmetric matrices

Illustration of a symmetric matrix

Illustration of a symmetric matrix

The matrix AA is symmetric if it is equal to its transpose:

A=AT\boldsymbol{A} = \boldsymbol{A}^\text{T}
(15)

This concerns only square matrices.

Example 6.

A=[24−14−80−103]\boldsymbol{A}= \begin{bmatrix} 2 & 4 & -1\\\\ 4 & -8 & 0\\\\ -1 & 0 & 3 \end{bmatrix}
(16)
A = np.array([[2, 4, -1], [4, -8, 0], [-1, 0, 3]])
A
array([[ 2, 4, -1], [ 4, -8, 0], [-1, 0, 3]])
A.T
array([[ 2, 4, -1], [ 4, -8, 0], [-1, 0, 3]])

Unit vectors

A unit vector is a vector of length equal to 1. It can be denoted by a letter with a hat: u^\hat{u}

Orthogonal vectors

Two orthogonal vectors are separated by a 90° angle. The dot product of two orthogonal vectors gives 0.

Example 7.

x = [0,0,2,2]
y = [0,0,2,-2]

plt.quiver([x[0], y[0]],
           [x[1], y[1]],
           [x[2], y[2]],
           [x[3], y[3]],
           angles='xy', scale_units='xy', scale=1)

plt.xlim(-2, 4)
plt.ylim(-3, 3)
plt.axvline(x=0, color='grey')
plt.axhline(y=0, color='grey')

plt.text(1, 1.5, r'$\vec{u}$', size=18)
plt.text(1.5, -1, r'$\vec{v}$', size=18)

plt.show()
plt.close()
<Figure size 288x288 with 1 Axes>
x=[22]\boldsymbol{x}= \begin{bmatrix} 2\\\\ 2 \end{bmatrix}
(17)

and

y=[2−2]\boldsymbol{y}= \begin{bmatrix} 2\\\\ -2 \end{bmatrix}
(18)
xTy=[22][2−2]=[2×2+2×−2]=0\boldsymbol{x^\text{T}y}= \begin{bmatrix} 2 & 2 \end{bmatrix} \begin{bmatrix} 2\\\\ -2 \end{bmatrix}= \begin{bmatrix} 2\times2 + 2\times-2 \end{bmatrix}=0
(19)

In addition, when the norm of orthogonal vectors is the unit norm they are called orthonormal.

It is impossible to have more than nn vectors mutually orthogonal in Rn\mathbb{R}^n. For instance try to draw 3 vectors in a 2-dimensional space (R2\mathbb{R}^2) that are mutually orthogonal...

Orthogonal matrices

Orthogonal matrices are important because they have interesting properties. A matrix is orthogonal if columns are mutually orthogonal and have a unit norm (orthonormal) and rows are mutually orthonormal and have unit norm.

Under the hood of an orthogonal matrix

Under the hood of an orthogonal matrix

A=[A1,1A1,2A2,1A2,2]\boldsymbol{A}= \begin{bmatrix} A_{1,1} & A_{1,2}\\\\ A_{2,1} & A_{2,2} \end{bmatrix}
(20)

This means that

[A1,1A2,1]\begin{bmatrix} A_{1,1}\\\\ A_{2,1} \end{bmatrix}
(21)

and

[A1,2A2,2]\begin{bmatrix} A_{1,2}\\\\ A_{2,2} \end{bmatrix}
(22)

are orthogonal vectors and also that the rows

[A1,1A1,2]\begin{bmatrix} A_{1,1} & A_{1,2} \end{bmatrix}
(23)

and

[A2,1A2,2]\begin{bmatrix} A_{2,1} & A_{2,2} \end{bmatrix}
(24)

are orthogonal vectors (cf. above for definition of orthogonal vectors).

Property 1: ATA=I\boldsymbol{A^\text{T}A}=\boldsymbol{I}

A orthogonal matrix has this property:

ATA=AAT=I\boldsymbol{A^\text{T}A}=\boldsymbol{AA^\text{T}}=\boldsymbol{I}
(25)

We can see that this statement is true with the following reasoning:

Let’s have the following matrix:

A=[abcd]\boldsymbol{A}=\begin{bmatrix} a & b\\\\ c & d \end{bmatrix}
(26)

and thus

AT=[acbd]\boldsymbol{A}^\text{T}=\begin{bmatrix} a & c\\\\ b & d \end{bmatrix}
(27)

Let’s do the product:

ATA=[acbd][abcd]=[aa+ccab+cdab+cdbb+dd]=[a2+c2ab+cdab+cdb2+d2]\begin{align*} &\boldsymbol{A^\text{T}A}=\begin{bmatrix} a & c\\\\ b & d \end{bmatrix} \begin{bmatrix} a & b\\\\ c & d \end{bmatrix} = \begin{bmatrix} aa + cc & ab + cd\\\\ ab + cd & bb + dd \end{bmatrix}\\\\ &= \begin{bmatrix} a^2 + c^2 & ab + cd\\\\ ab + cd & b^2 + d^2 \end{bmatrix} \end{align*}
(28)

We saw that the norm of the vector [ac]\begin{bmatrix} a & c \end{bmatrix} is equal to a2+c2a^2+c^2 (L2L^2 or squared L2L^2). In addtion, we saw that the rows of A\boldsymbol{A} have a unit norm because A\boldsymbol{A} is orthogonal. This means that a2+c2=1a^2+c^2=1 and b2+d2=1b^2+d^2=1. So we now have:

ATA=[1ab+cdab+cd1]\boldsymbol{A^\text{T}A}= \begin{bmatrix} 1 & ab + cd\\\\ ab + cd & 1 \end{bmatrix}
(29)

Also, ab+cdab+cd corresponds to the product of [ac]and[bd]\begin{bmatrix} a & c \end{bmatrix} and \begin{bmatrix} b & d \end{bmatrix}:

[ac][bd]=ab+cd\begin{bmatrix} a & c \end{bmatrix} \begin{bmatrix} b\\\\ d \end{bmatrix} = ab+cd
(30)

And we know that the columns are orthogonal which means that:

[ac][bd]=0\begin{bmatrix} a & c \end{bmatrix} \begin{bmatrix} b\\\\ d \end{bmatrix}=0
(31)

We thus have the identity matrix:

ATA=[1001]\boldsymbol{A^\text{T}A}=\begin{bmatrix} 1 & 0\\\\ 0 & 1 \end{bmatrix}
(32)

Property 2: AT=A−1\boldsymbol{A}^\text{T}=\boldsymbol{A}^{-1}

We can show that if ATA=I\boldsymbol{A^\text{T}A}=\boldsymbol{I} then AT=A−1 \boldsymbol{A}^\text{T}=\boldsymbol{A}^{-1}.

If we multiply each side of the equation ATA=I\boldsymbol{A^\text{T}A}=\boldsymbol{I} by A−1\boldsymbol{A}^{-1} we have:

(ATA)A−1=IA−1(\boldsymbol{A^\text{T}A})\boldsymbol{A}^{-1}=\boldsymbol{I}\boldsymbol{A}^{-1}
(33)

Recall that a matrix or vector doesn’t change when it is multiplied by the identity matrix. So we have:

(ATA)A−1=A−1(\boldsymbol{A^\text{T}A})\boldsymbol{A}^{-1}=\boldsymbol{A}^{-1}
(34)

We also saw that matrix multiplication is associative so we can remove the parenthesis:

ATAA−1=A−1\boldsymbol{A^\text{T}A}\boldsymbol{A}^{-1}=\boldsymbol{A}^{-1}
(35)

We also know that AA−1=I\boldsymbol{A}\boldsymbol{A}^{-1}=\boldsymbol{I} so we can replace:

ATI=A−1\boldsymbol{A^\text{T}}\boldsymbol{I}=\boldsymbol{A}^{-1}
(36)

This shows that

AT=A−1\boldsymbol{A}^\text{T}=\boldsymbol{A}^{-1}
(37)

You can refer to this question.

Example 8.

Sine and cosine are convenient to create orthogonal matrices. Let’s take the following matrix:

A=[cos(50)−sin(50)sin(50)cos(50)]\boldsymbol{A}= \begin{bmatrix} cos(50) & -sin(50)\\\\ sin(50) & cos(50) \end{bmatrix}
(38)
A = np.array([[np.cos(50), -np.sin(50)], [np.sin(50), np.cos(50)]])
A
array([[ 0.96496603, 0.26237485], [-0.26237485, 0.96496603]])
col0 = A[:, [0]]
col1 = A[:, [1]]
row0 = A[0].reshape(A.shape[1], 1)
row1 = A[1].reshape(A.shape[1], 1)

Let’s check that rows and columns are orthogonal:

col0.T.dot(col1)
array([[0.]])
row0.T.dot(row1)
array([[0.]])

Let’s check that

ATA=AAT=I\boldsymbol{A^\text{T}A}=\boldsymbol{AA^\text{T}}=\boldsymbol{I}
(39)

and thus

AT=A−1\boldsymbol{A}^\text{T}=\boldsymbol{A}^{-1}
(40)
A.T.dot(A)
array([[1., 0.], [0., 1.]])
A.T
array([[ 0.96496603, -0.26237485], [ 0.26237485, 0.96496603]])
numpy.linalg.inv(A)
array([[ 0.96496603, -0.26237485], [ 0.26237485, 0.96496603]])

Everything is correct!

Conclusion

In this chapter we saw different interesting type of matrices with specific properties. It is generally useful to recall them while we deal with this kind of matrices.

In the next chapter we will saw a central idea in linear algebra: the eigendecomposition. Keep reading!

Linear Algebra and SVD
Subspaces and Maps to Other Spaces
Linear Algebra and SVD
Projection with dot product