Multiplying Matrices and Vectors - Mathematical Foundations of Data Science

We will see some very important concepts in this chapter. The dot product is used in every equation explaining data science algorithms so it’s worth the effort to understand it. Then we will see some properties of this operation. Finally, we will to get some intuition on the link between matrices and systems of linear equations.

import numpy as np

# Avoid inaccurate floating values (for inverse matrices in dot product for instance)
# See https://stackoverflow.com/questions/24537791/numpy-matrix-inversion-rounding-errors
np.set_printoptions(suppress=True)

The standard way to multiply matrices is not to multiply each element of one with each elements of the other (this is the element-wise product) but to calculate the sum of the products between rows and columns. The matrix product, also called dot product, is calculated as following:

The dot product between a matrix and a vector

The number of columns of the first matrix must be equal to the number of rows of the second matrix. Thus, if the dimensions, or the shape of the first matrix, is ( $m \times n$ ) the second matrix need to be of shape ( $n \times x$ ). The resulting matrix will have the shape ( $m \times x$ ).

Example 1.¶

As a starter we will see the multiplication of a matrix and a vector.

\boldsymbol{A} \times \boldsymbol{b} = \boldsymbol{C}

(1)

with $ \boldsymbol{A}=

\begin{bmatrix} 1 & 2\\\\ 3 & 4\\\\ 5 & 6 \end{bmatrix}

(2)

$and$ \boldsymbol{b}=\begin{bmatrix} 2\\ 4 \end{bmatrix}$. We saw that the formula is the following:

\begin{align*} &\begin{bmatrix} A_{1,1} & A_{1,2} \\\\ A_{2,1} & A_{2,2} \\\\ A_{3,1} & A_{3,2} \end{bmatrix}\times \begin{bmatrix} B_{1,1} \\\\ B_{2,1} \end{bmatrix}=\\\\ &\begin{bmatrix} A_{1,1}B_{1,1} + A_{1,2}B_{2,1} \\\\ A_{2,1}B_{1,1} + A_{2,2}B_{2,1} \\\\ A_{3,1}B_{1,1} + A_{3,2}B_{2,1} \end{bmatrix} \end{align*}

(3)

So we will have:

\begin{align*} &\begin{bmatrix} 1 & 2 \\\\ 3 & 4 \\\\ 5 & 6 \end{bmatrix}\times \begin{bmatrix} 2 \\\\ 4 \end{bmatrix}=\\\\ &\begin{bmatrix} 1 \times 2 + 2 \times 4 \\\\ 3 \times 2 + 4 \times 4 \\\\ 5 \times 2 + 6 \times 4 \end{bmatrix}= \begin{bmatrix} 10 \\\\ 22 \\\\ 34 \end{bmatrix} \end{align*}

(4)

It is a good habit to check the dimensions of the matrix so see what is going on. We can see in this example that the shape of $\boldsymbol{A}$ is ( $3 \times 2$ ) and the shape of $\boldsymbol{b}$ is ( $2 \times 1$ ). So the dimensions of $\boldsymbol{C}$ are ( $3 \times 1$ ).

With Numpy¶

The Numpy function dot() can be used to compute the matrix product (or dot product). Let’s try to reproduce the last exemple:

A = np.array([[1, 2], [3, 4], [5, 6]])
A

array([[1, 2],
       [3, 4],
       [5, 6]])

B = np.array([[2], [4]])
B

array([[2],
       [4]])

C = np.dot(A, B)
C

array([[10],
       [22],
       [34]])

It is equivalent to use the method dot() of Numpy arrays:

C = A.dot(B)
C

array([[10],
       [22],
       [34]])

And also with @ operator:

C = A @ B
C

array([[10],
       [22],
       [34]])

Example 2.¶

Multiplication of two matrices.

\boldsymbol{A} \times \boldsymbol{B} = \boldsymbol{C}

(5)

with:

\boldsymbol{A}=\begin{bmatrix} 1 & 2 & 3 \\\\ 4 & 5 & 6 \\\\ 7 & 8 & 9 \\\\ 10 & 11 & 12 \end{bmatrix}

(6)

and:

\boldsymbol{B}=\begin{bmatrix} 2 & 7 \\\\ 1 & 2 \\\\ 3 & 6 \end{bmatrix}

(7)

So we have:

\begin{align*} &\begin{bmatrix} 1 & 2 & 3 \\\\ 4 & 5 & 6 \\\\ 7 & 8 & 9 \\\\ 10 & 11 & 12 \end{bmatrix}\times \begin{bmatrix} 2 & 7 \\\\ 1 & 2 \\\\ 3 & 6 \end{bmatrix}=\\\\ &\begin{bmatrix} 2 \times 1 + 1 \times 2 + 3 \times 3 & 7 \times 1 + 2 \times 2 + 6 \times 3 \\\\ 2 \times 4 + 1 \times 5 + 3 \times 6 & 7 \times 4 + 2 \times 5 + 6 \times 6 \\\\ 2 \times 7 + 1 \times 8 + 3 \times 9 & 7 \times 7 + 2 \times 8 + 6 \times 9 \\\\ 2 \times 10 + 1 \times 11 + 3 \times 12 & 7 \times 10 + 2 \times 11 + 6 \times 12 \\\\ \end{bmatrix}\\\\ &= \begin{bmatrix} 13 & 29 \\\\ 31 & 74 \\\\ 49 & 119 \\\\ 67 & 164 \end{bmatrix} \end{align*}

(8)

Let’s check the result with Numpy:

A = np.array([[1, 2, 3], [4, 5, 6], [7, 8, 9], [10, 11, 12]])
A

array([[ 1,  2,  3],
       [ 4,  5,  6],
       [ 7,  8,  9],
       [10, 11, 12]])

B = np.array([[2, 7], [1, 2], [3, 6]])
B

array([[2, 7],
       [1, 2],
       [3, 6]])

C = A.dot(B)
C

array([[ 13,  29],
       [ 31,  74],
       [ 49, 119],
       [ 67, 164]])

It works!

Formalization of the dot product¶

C_{i,j} = A_{i,k}B_{k,j} = \sum_{k}A_{i,k}B_{k,j}

(9)

You can find more examples about the dot product here.

Properties of the dot product¶

We will now see some interesting properties of the matrix multiplication. It will become useful as we move forward in the chapters. Using simple examples for each property will provide a way to check them while we get used to the Numpy functions.

Matrices mutliplication is distributive¶

\boldsymbol{A}(\boldsymbol{B}+\boldsymbol{C}) = \boldsymbol{AB}+\boldsymbol{AC}

(10)

Example 3.¶

\boldsymbol{A}=\begin{bmatrix} 2 & 3 \\\\ 1 & 4 \\\\ 7 & 6 \end{bmatrix}, \boldsymbol{B}=\begin{bmatrix} 5 \\\\ 2 \end{bmatrix}, \boldsymbol{C}=\begin{bmatrix} 4 \\\\ 3 \end{bmatrix}

(11)

\begin{align*} \boldsymbol{A}(\boldsymbol{B}+\boldsymbol{C})&=\begin{bmatrix} 2 & 3 \\\\ 1 & 4 \\\\ 7 & 6 \end{bmatrix}\times \left(\begin{bmatrix} 5 \\\\ 2 \end{bmatrix}+ \begin{bmatrix} 4 \\\\ 3 \end{bmatrix}\right)= \begin{bmatrix} 2 & 3 \\\\ 1 & 4 \\\\ 7 & 6 \end{bmatrix}\times \begin{bmatrix} 9 \\\\ 5 \end{bmatrix}\\\\ &= \begin{bmatrix} 2 \times 9 + 3 \times 5 \\\\ 1 \times 9 + 4 \times 5 \\\\ 7 \times 9 + 6 \times 5 \end{bmatrix}= \begin{bmatrix} 33 \\\\ 29 \\\\ 93 \end{bmatrix} \end{align*}

(12)

is equivalent to

\begin{align*} \boldsymbol{A}\boldsymbol{B}+\boldsymbol{A}\boldsymbol{C} &= \begin{bmatrix} 2 & 3 \\\\ 1 & 4 \\\\ 7 & 6 \end{bmatrix}\times \begin{bmatrix} 5 \\\\ 2 \end{bmatrix}+ \begin{bmatrix} 2 & 3 \\\\ 1 & 4 \\\\ 7 & 6 \end{bmatrix}\times \begin{bmatrix} 4 \\\\ 3 \end{bmatrix}\\\\ &= \begin{bmatrix} 2 \times 5 + 3 \times 2 \\\\ 1 \times 5 + 4 \times 2 \\\\ 7 \times 5 + 6 \times 2 \end{bmatrix}+ \begin{bmatrix} 2 \times 4 + 3 \times 3 \\\\ 1 \times 4 + 4 \times 3 \\\\ 7 \times 4 + 6 \times 3 \end{bmatrix}\\\\ &= \begin{bmatrix} 16 \\\\ 13 \\\\ 47 \end{bmatrix}+ \begin{bmatrix} 17 \\\\ 16 \\\\ 46 \end{bmatrix}= \begin{bmatrix} 33 \\\\ 29 \\\\ 93 \end{bmatrix} \end{align*}

(13)

A = np.array([[2, 3], [1, 4], [7, 6]])
A

array([[2, 3],
       [1, 4],
       [7, 6]])

B = np.array([[5], [2]])
B

array([[5],
       [2]])

C = np.array([[4], [3]])
C

array([[4],
       [3]])

$\boldsymbol{A}(\boldsymbol{B}+\boldsymbol{C})$ :

D = A.dot(B+C)
D

array([[33],
       [29],
       [93]])

is equivalent to $\boldsymbol{AB}+\boldsymbol{AC}$ :

D = A.dot(B) + A.dot(C)
D

array([[33],
       [29],
       [93]])

Matrices mutliplication is associative¶

\boldsymbol{A}(\boldsymbol{BC}) = (\boldsymbol{AB})\boldsymbol{C}

(14)

A = np.array([[2, 3], [1, 4], [7, 6]])
A

array([[2, 3],
       [1, 4],
       [7, 6]])

B = np.array([[5, 3], [2, 2]])
B

array([[5, 3],
       [2, 2]])

$\boldsymbol{A}(\boldsymbol{BC})$ :

D = A.dot(B.dot(C))
D

array([[100],
       [ 85],
       [287]])

is equivalent to $(\boldsymbol{AB})\boldsymbol{C}$ :

D = (A.dot(B)).dot(C)
D

array([[100],
       [ 85],
       [287]])

Matrix multiplication is not commutative¶

\boldsymbol{AB} \neq \boldsymbol{BA}

(15)

A = np.array([[2, 3], [6, 5]])
A

array([[2, 3],
       [6, 5]])

B = np.array([[5, 3], [2, 2]])
B

array([[5, 3],
       [2, 2]])

$\boldsymbol{AB}$ :

AB = np.dot(A, B)
AB

array([[16, 12],
       [40, 28]])

is different from $\boldsymbol{BA}$ :

BA = np.dot(B, A)
BA

array([[28, 30],
       [16, 16]])

However vector multiplication is commutative¶

\boldsymbol{x^{ \text{T}}y} = \boldsymbol{y^{\text{T}}x}

(16)

x = np.array([[2], [6]])
x

array([[2],
       [6]])

y = np.array([[5], [2]])
y

array([[5],
       [2]])

$\boldsymbol{x^\text{T}y}$ :

x_ty = x.T.dot(y)
x_ty

array([[22]])

is equivalent to $\boldsymbol{y^\text{T}x}$ :

y_tx = y.T.dot(x)
y_tx

array([[22]])

Simplification of the matrix product¶

(\boldsymbol{AB})^{\text{T}} = \boldsymbol{B}^\text{T}\boldsymbol{A}^\text{T}

(17)

A = np.array([[2, 3], [1, 4], [7, 6]])
A

array([[2, 3],
       [1, 4],
       [7, 6]])

B = np.array([[5, 3], [2, 2]])
B

array([[5, 3],
       [2, 2]])

$(\boldsymbol{AB})^{\text{T}}$ :

AB_t = A.dot(B).T
AB_t

array([[16, 13, 47],
       [12, 11, 33]])

is equivalent to $\boldsymbol{B}^\text{T}\boldsymbol{A}^\text{T}$ :

B_tA = B.T.dot(A.T)
B_tA

array([[16, 13, 47],
       [12, 11, 33]])

System of linear equations¶

This is an important part of why linear algebra can be very useful to solve variety of problems. Here we will see that it can be use to represent system of equations.

A system of equations is a set of multiple equations (at least 1). For instance we could have:

\begin{cases} y = 2x + 1 \\\\ y = \frac{7}{2}x +3 \end{cases}

(18)

A system of equations is defined by its number of equations and its number of unknowns. In our example above, the system has 2 equations and 2 unknowns ( $x$ and $y$ ). In addition we call this a system of linear equations because each equations is linear. It is easy to see that in 2 dimensions: we will have one straight line per equation and the dimensions are the unknowns. Here is the plot of the first one:

Representation of a line from an equation

Representation of a linear equation

In our system of equations, the unknowns are the dimensions and the number of equations is the number of lines (in 2D) or $n$-dimensional planes.

Using matrices to describe the system¶

Matrices can be used to describe a system of linear equations of the form $\boldsymbol{Ax}=\boldsymbol{b}$ . Here is such a system:

A_{1,1}x_1 + A_{1,2}x_2 + A_{1,n}x_n = b_1 \\\\ A_{2,1}x_1 + A_{2,2}x_2 + A_{2,n}x_n = b_2 \\\\ \cdots \\\\ A_{m,1}x_1 + A_{m,2}x_2 + A_{m,n}x_n = b_n

(19)

The unknowns (what we want to find to solve the system) are the variables $x_1$ and $x_2$ corresponding to the previous variables $x$ and $y$ . It is exactly the same form as with the last example but with all the variables on the same side. $y = 2x + 1$ becomes $-2x + y = 1$ with $x$ corresponding to $x_1$ and $y$ corresponding to $x_2$ . We will have $n$ unknowns and $m$ equations.

The variables are named $x_1, x_2, \cdots, x_n$ by convention because we will see that it can be summarised in the vector $\boldsymbol{x}$ .

Left hand side¶

The left hand term can considered as the product of a matrix $\boldsymbol{A}$ containing weights for each variable ( $n$ columns) and each equation ( $m$ rows):

\boldsymbol{A}= \begin{bmatrix} A_{1,1} & A_{1,2} & \cdots & A_{1,n} \\\\ A_{2,1} & A_{2,2} & \cdots & A_{2,n} \\\\ \cdots & \cdots & \cdots & \cdots \\\\ A_{m,1} & A_{m,2} & \cdots & A_{m,n} \end{bmatrix}

(20)

with a vector $\boldsymbol{x}$ containing the $n$ unknowns

\boldsymbol{x}= \begin{bmatrix} x_1 \\\\ x_2 \\\\ \cdots \\\\ x_n \end{bmatrix}

(21)

The dot product of $\boldsymbol{A}$ and $\boldsymbol{x}$ gives a set of equations. Here is a simple example:

Matrix form of a system of linear equations

We have a set of two equations with two unknowns. So the number of rows of $\boldsymbol{A}$ gives the number of equations and the number of columns gives the number of unknowns.

Both sides¶

The equation system can be wrote like that:

\begin{bmatrix} A_{1,1} & A_{1,2} & \cdots & A_{1,n} \\\\ A_{2,1} & A_{2,2} & \cdots & A_{2,n} \\\\ \cdots & \cdots & \cdots & \cdots \\\\ A_{m,1} & A_{m,2} & \cdots & A_{m,n} \end{bmatrix} \times \begin{bmatrix} x_1 \\\\ x_2 \\\\ \cdots \\\\ x_n \end{bmatrix} = \begin{bmatrix} b_1 \\\\ b_2 \\\\ \cdots \\\\ b_m \end{bmatrix}

(22)

Or simply:

\boldsymbol{Ax}=\boldsymbol{b}

(23)

Example 4.¶

We will try to convert the common form of a linear equation $y=ax+b$ to the matrix form. If we want to keep the previous notation we will have instead:

x_2=ax_1+b

(24)

Don’t confuse the variable $x_1$ and $x_2$ with the vector $\boldsymbol{x}$ . This vector contains actually all the variables of our equations. Here we have:

\boldsymbol{x} = \begin{bmatrix} x_1 \\ x_2 \end{bmatrix}

(25)

In this example we will use the following equation:

\begin{align*} &x_2=2x_1+1\\\\ \Leftrightarrow& 2x_1-x_2=-1 \end{align*}

(26)

In order to end up with this system when we multiply $\boldsymbol{A}$ and $\boldsymbol{x}$ we need $\boldsymbol{A}$ to be a matrix containing the weights of each variable. The weight of $x_1$ is 2 and the weights of $x_2$ is -1:

\boldsymbol{A}= \begin{bmatrix} 2 & -1 \end{bmatrix}

(27)

So we have

\begin{bmatrix} 2 & -1 \end{bmatrix} \begin{bmatrix} x_1 \\ x_2 \end{bmatrix} = \begin{bmatrix} 2x_1-1x_2 \end{bmatrix}

(28)

To complete the equation we have

\boldsymbol{b}= \begin{bmatrix} -1 \end{bmatrix}

(29)

which gives

\begin{bmatrix} 2 & -1 \end{bmatrix} \begin{bmatrix} x_1 \\ x_2 \end{bmatrix} = \begin{bmatrix} -1 \end{bmatrix}

(30)

This system of equations is thus very simple and contains only 1 equation ( $\boldsymbol{A}$ has 1 row) and 2 variables ( $\boldsymbol{A}$ has 2 columns).

To summarise, $\boldsymbol{A}$ will be the matrix of dimensions $m\times n$ containing scalars multiplying these variables (here $x_1$ is multiplied by 2 and $x_2$ by -1). The vector $\boldsymbol{x}$ contains the variables $x_1$ and $x_2$ . And the right hand term is the constant $\boldsymbol{b}$ :

\boldsymbol{A}= \begin{bmatrix} 2 & -1 \end{bmatrix}

(31)

\boldsymbol{x}= \begin{bmatrix} x_1\\\\ x_2 \end{bmatrix}

(32)

\boldsymbol{b}= \begin{bmatrix} -1 \end{bmatrix}

(33)

We can write this system

\boldsymbol{Ax}=\boldsymbol{b}

(34)

We will see at the end of the next chapter that this compact way of writing sets of linear equations can be very usefull. It provides actually a way to solve the equations.

References¶

Math is fun - Multiplying matrices