Basics of Linear Algebra

Last updated: March 27th, 2019

Basics of Linear Algebra with NumPy¶

Numpy arrays, when created with the correct set of dimensions, will resemble algebraic structures like vectors or matrices. As expected, NumPy will be able to perform the expected primitive operations associated with them. It has a very comprehensive linear algebra sub-system.

Hands on!¶

In [ ]:
import sys
import numpy as np

$$\large A = \begin{bmatrix} 1 & 2 & 3 \\ 4 & 5 & 6 \end{bmatrix}$$
In [ ]:
A = np.array([[1, 2, 3], [4, 5, 6]])
A

$$\large B = \begin{bmatrix} 5 & 5 & 5 \\ 5 & 5 & 5 \end{bmatrix}$$
In [ ]:
B = np.full((2, 3), 5)
B

$$\normalsize v = \begin{bmatrix} 6 \\ 7 \\ 8 \end{bmatrix}$$
In [ ]:
v = np.array([6, 7, 8])


Operations with Matrices and Vectors¶

Sum of matrices:

\begin{equation*} \large A + \large B = \begin{bmatrix} 1 & 2 & 3\\ 4 & 5 & 6 \end{bmatrix} + \begin{bmatrix} 5 & 5 & 5 \\ 5 & 5 & 5 \end{bmatrix} = \begin{bmatrix} 6 & 7 & 8 \\ 9 & 10 & 11 \end{bmatrix} \end{equation*}
In [ ]:
A + B


Subtraction of matrices:

\begin{equation*} \large A - \large B = \begin{bmatrix} 1 & 2 & 3\\ 4 & 5 & 6 \end{bmatrix} - \begin{bmatrix} 5 & 5 & 5 \\ 5 & 5 & 5 \end{bmatrix} = \begin{bmatrix} -4 & -3 & -2 \\ -1 & 0 & 1 \end{bmatrix} \end{equation*}
In [ ]:
A - B


(Not to be confused with matrix multiplication or dot product):

\begin{equation*} \large A \circ \large B = \begin{bmatrix} 1 & 2 & 3\\ 4 & 5 & 6 \end{bmatrix} \circ \begin{bmatrix} 5 & 5 & 5 \\ 5 & 5 & 5 \end{bmatrix} = \begin{bmatrix} 5 & 10 & 15 \\ 20 & 25 & 30 \end{bmatrix} \end{equation*}
In [ ]:
A * B


Division also works "per-entry":

In [ ]:
A / B


Matrix multiplication:

\begin{equation*} \large A \cdot \normalsize v = \begin{bmatrix} 1 & 2 & 3\\ 4 & 5 & 6 \end{bmatrix} \cdot \begin{bmatrix} 6 \\ 7 \\ 8 \\ \end{bmatrix} = \begin{bmatrix} 44 \\ 107 \end{bmatrix} \end{equation*}
In [ ]:
np.dot(A, v)


In Python 3.5, the @ operator was introduced for matrix multiplication:

In [ ]:
A @ v


Matrix transpose:

In [ ]:
A.T


The identity matrix¶

An identity matrix is a square matrix, of size n, with ones on the main diagonal and zeros elsewhere:

\begin{equation*} \normalsize I_{n} = \begin{bmatrix} 1&0&0&\cdots &0\\ 0&1&0&\cdots &0\\ 0&0&1&\cdots &0\\ \vdots &\vdots &\vdots &\ddots &\vdots \\ 0&0&0&\cdots &1 \end{bmatrix} \end{equation*}

So, for example, an identity matrix of size 3:

\begin{equation*} \normalsize I_{3} = \begin{bmatrix} 1 & 0 & 0 \\ 0 & 1 & 0 \\ 0 & 0 & 1 \end{bmatrix} \end{equation*}
In [ ]:
I = np.identity(3)
I

\begin{equation*} \normalsize C = \begin{bmatrix} 1 & 2 & 3 \\ 4 & 5 & 6 \\ 7 & 8 & 9 \end{bmatrix} \end{equation*}
In [ ]:
np.arange(1, 10)

In [ ]:
C = np.arange(1, 10).reshape((3, 3))
C


Properties of the identity matrix:

\begin{equation*} \large A_{n} \cdot I_{n} = A_{n} \end{equation*}\begin{equation*} \large I_{n} \cdot A_{n} = A_{n} \end{equation*}
In [ ]:
I @ C

In [ ]:
C @ I


Inverse of a matrix:¶

The formula to find the inverse of a 2x2 matrix is:

\begin{equation*} \normalsize A ^ {-1} = \begin{bmatrix} a_{11} & a_{12} \\ a_{21} & a_{22} \end{bmatrix}^{-1} = \frac{1}{\lvert A\rvert} \begin{bmatrix} a_{22} & -a_{12} \\ -a_{21} & a_{11}\end{bmatrix} \end{equation*}

|A| is the determinant of a matrix, which can be computed with np.linalg.det. So, as an example, given the following matrix:

\begin{equation*} \normalsize D = \begin{bmatrix} 1 & 2 \\ 3 & 4 \\ \end{bmatrix} \end{equation*}
In [ ]:
D = np.array([
[1, 2],
[3, 4]
])

D


The determinant can be calculated using:

In [ ]:
np.linalg.det(D)


And the inverse with np.linalg.inv:

In [ ]:
np.linalg.inv(D)


A square matrix that is not invertible is called singular or degenerate. A square matrix is singular if and only if its determinant is 0. Singular matrices are rare in the sense that a square matrix randomly selected from a continuous uniform distribution on its entries will almost never be singular.

In [ ]:
singular = np.array([[1, 2, 3],
[4, 5, 6],
[7, 8, 9]])

np.linalg.det(singular)


Calculating its inverse matrix will raise an error:

In [ ]:
np.linalg.inv(singular)


The main characteristic of the inverse matrix is that, it acts as a reciprocal of the matrix:

\begin{equation*} \large A \cdot A^{-1} = I \end{equation*}\begin{equation*} \large A^{-1} \cdot A = I \end{equation*}
In [ ]:
D @ np.linalg.inv(D)

In [ ]:
np.round(D @ np.linalg.inv(D))


Useful Numpy functions¶

We'll now take the chance to see in more detail the functions we've been using, the most common and useful ones in the numpy world:

arange

Return evenly spaced values within a given interval.

In [ ]:
range(5)

In [ ]:
np.arange(10)

In [ ]:
np.arange(5, 10)

In [ ]:
np.arange(0, 1, .1)


linspace

Return evenly spaced numbers over a specified interval.

In [ ]:
np.linspace(0, 1, 5)

In [ ]:
np.linspace(0, 1, 20)

In [ ]:
np.linspace(0, 1, 20, False)


zeros, ones, empty

In [ ]:
np.zeros(5)

In [ ]:
np.zeros((3, 3))

In [ ]:
np.zeros((3, 3), dtype=np.int)

In [ ]:
np.ones(5)

In [ ]:
np.ones((3, 3))

In [ ]:
np.empty(5)

In [ ]:
np.empty((2, 2))


Full filled

In [ ]:
np.full((2,2), 7)

In [ ]:
np.full((3, 3), 1)


Random filled

In [ ]:
np.random.random((2,2))


Identity and eye

In [ ]:
np.identity(3)

In [ ]:
np.eye(3, 3)

In [ ]:
np.eye(8, 4)

In [ ]:
np.eye(8, 4, k=1)

In [ ]:
np.eye(8, 4, k=-3)

In [ ]:
np.eye(8, 4, k=-3, dtype=np.int)