Matrix Algebra Review

Matrix Algebra Review

Matrix Algebra: A Review

The Prerequisites Checklist page on the Department of Statistics website lists a number of courses that require a working knowledge of Matrix algebra as a prerequisite. Students who do not have this foundation or have not reviewed this material within the past couple of years will struggle with the concepts and methods that build on this foundation. The courses that require this foundation include:

  • STAT 414 - Introduction to Probability Theory
  • STAT 501 - Regression Methods
  • STAT 504 - Analysis of Discrete Data
  • STAT 505 - Applied Multivariate Statistical Analysis

Review Materials

Many of our returning, working professional students report that they had taken courses that included matrix algebra topics but often these courses were taken a number of years ago. As a means of helping students assess whether or not what they currently know and can do will meet the expectations of instructors in the courses above, the online program has put together a brief review of these concepts and methods. This is then followed by a short self-assessment exam that will help give you an idea if you still have the necessary background.

Self-Assessment Procedure

  1. Review the concepts and methods on the pages in this section of this website. Note the courses that certain sections are aligned with as prerequisites:
      STAT 414 STAT 501 STAT 504 STAT 505
    M.1 - Matrix Definitions  Required  Required  Required Required
    M.2 - Matrix Arithmetic  Required  Required  Required Required
    M.3 - Matrix Properties  Required  Required  Required Required
    M.4 - Matrix Inverse  Required  Required  Required Required
    M.5 - Advanced Topics  Recommended  Recommended  Recommended 5.1, 5.4, Required
    5.2, 5.3, Recommended
  2. Download and complete the Self-Assessment Exam
  3. Review the Self-Assessment Exam Solutions and determine your score.

Students who score below 70% (fewer than 21 questions correct) should consider further review of these materials and are strongly encouraged to take a course like MATH 220 or an equivalent course at a local college or community college.

If you have struggled with the concepts and methods that are presented here, you will indeed struggle in the courses above that expect this foundation.


M.1 Matrix Definitions

M.1 Matrix Definitions
Matrix

A matrix is a rectangular collection of numbers. Generally, matrices are denoted as bold capital letters. For example:

\[A = \begin{pmatrix} 1 & -5 & 4\\
2 & 5 & 3 \end{pmatrix}\]

A is a matrix with two rows and three columns. For that reason, it is called a 2 by 3 matrix. This is called the dimension of a matrix.

Dimension

The dimension of a matrix is expressed as number of rows × number of columns. So,

\[B = \begin{pmatrix} 1 & -5 & 4 \\ 5 & 3 & -8 \\ 1 & 5 & 4 \\ 2 & 5 & 3 \end{pmatrix}\]

B is a 4 × 3 matrix. It is common to refer to elements in a matrix by subscripts, like so.

\[B = \begin{pmatrix} b_{1,1} & b_{1,2} & b_{1,3}\\ b_{2,1} & b_{2,2} & b_{2,3}\\ b_{3,1} & b_{3,2} & b_{3,3}\\ b_{4,1} & b_{4,2} & b_{4,3} \end{pmatrix}\]

With the row first and the column second. So in this case, \(b_{2,1} = 5\) and \(b_{1,3} =4\).

Vector

A vector is a matrix with only one row (called a row vector) or only one column (called a column vector). For example:

\[C = \begin{pmatrix} 2 & 7 & -3 & 5 \end{pmatrix}\]

C is a 4 dimensional row vector.

\[D = \begin{pmatrix} 2 \\9 \\-3 \\3\\    6\\  \end{pmatrix}\]

D is a 5 dimensional column vector. An "ordinary" number can be thought of as a 1 × 1 matrix, also known as a scalar. Some examples of scalars are shown below:

\[ E = \pi \]

\[ F = 6 \]


M.2 Matrix Arithmetic

M.2 Matrix Arithmetic

Transpose a Matrix

To take the transpose of a matrix, simply switch the rows and column of a matrix. The transpose of \(A\) can be denoted as \(A'\) or \(A^T\).

For example

\[A = \begin{pmatrix} 1 & -5 & 4 \\ 2 & 5 & 3 \end{pmatrix}\]

\[A' = A^T = \begin{pmatrix} 1 & 2\\ -5 & 5\\ 4 & 3 \end{pmatrix}\]

If a matrix is its own transpose, then that matrix is said to be symmetric. Symmetric matrices must be square matrices, with the same number of rows and columns.

One example of a symmetric matrix is shown below:

\[ A = \begin{pmatrix} 1 & -5 & 4 \\ -5 & 7 & 3\\ 4 & 3 & 3 \end{pmatrix} = A' = A^T \]

Matrix Addition

To perform matrix addition, two matrices must have the same dimensions. This means they must have the same number of rows and columns. In that case simply add each individual components, like below.

For example

\[A + B = \begin{pmatrix} 1 & -5 & 4 \\ 2 & 5 & 3 \end{pmatrix} + \begin{pmatrix} 8 & -3 & -4 \\ 4 & -2 & 9 \end{pmatrix} = \begin{pmatrix} 1 + 8 & -5 - 3 & 4 - 4 \\ 2 + 4 & 5 -2 & 3 + 9 \end{pmatrix} = \begin{pmatrix} 9 & -8 & 0\\ 6 & 3 & 12 \end{pmatrix}\]

Matrix addition does have many of the same properties as "normal" addition.

\[A + B = B + A\]

\[A + (B + C) = (A + B) + C\]

In addition, if one wishes to take the transpose of the sum of two matrices, then

\[A^T + B^T = (A+B)^T \]

Matrix Scalar Multiplication

To multiply a matrix by a scalar, also known as scalar multiplication, multiply every element in the matrix by the scalar.

For example...

\[ 6*A = 6 * \begin{pmatrix} 1 & -5 & 4\\ 2 & 5 & 3 \end{pmatrix} = \begin{pmatrix} 6 * 1 & 6 *-5 & 6 * 4\\ 6 * 2 & 6 *5 & 6 * 3 \end{pmatrix} = \begin{pmatrix} 6 & -30 & 24 \\ 12 & 30 & 18 \end{pmatrix}\]

To multiply two vectors with the same length together is to take the dot product, also called inner product. This is done by multiplying every entry in the two vectors together and then adding all the products up.

For example, for vectors x and y, the dot product is calculated below

\[ x \cdot y = \begin{pmatrix} 1 & -5 & 4 \end{pmatrix} * \begin{pmatrix} 4 & -2 & 5 \end{pmatrix} = 1*4 + (-5)*(-2) + 4*5 = 4+10+20 = 34\]

 

Matrix Multiplication

To perform matrix multiplication, the first matrix must have the same number of columns as the second matrix has rows. The number of rows of the resulting matrix equals the number of rows of the first matrix, and the number of columns of the resulting matrix equals the number of columns of the second matrix. So a 3 × 5 matrix could be multiplied by a 5 × 7 matrix, forming a 3 × 7 matrix, but one cannot multiply a 2 × 8 matrix with a 4 × 2 matrix. To find the entries in the resulting matrix, simply take the dot product of the corresponding row of the first matrix and the corresponding column of the second matrix.

For example

\[ C*D = \begin{pmatrix} 3 & -9 & -8\\ 2 & 4 & 3 \end{pmatrix} * \begin{pmatrix} 7 & -3\\ -2 & 3\\ 6 & 2 \end{pmatrix} \]

\[ C*D = \begin{pmatrix} 3*7 + (-9)*(-2) + (-8)*6 & 3*(-3) + (-9)*3 + (-8)*2 \\ 2*7 + 4*(-2) + 3*6 & 2*(-3) + 4*3 + 3*2 \end{pmatrix}\]

\[ C*D = \begin{pmatrix} 21 + 18 - 48 & - 9 - 27 - 16 \\14 - 8 + 18 & - 6 + 12 + 6 \end{pmatrix} = \begin{pmatrix} -9 & - 52\\ 24 & 12 \end{pmatrix} \]

Matrix multiplication has some of the same properties as "normal" multiplication , such as

\[ A(BC) = (AB)C\]

\[A(B + C) = AB + AC\]

\[(A + B)C = AC + BC\]

However matrix multiplication is not communicative. That is to say A*B does not necessarily equal B*A. In fact, B*A often has no meaning since the dimensions rarely match up. However, you can take the transpose of matrix multiplication. In that case \((AB)^T = B^T A^T\).


M.3 Matrix Properties

M.3 Matrix Properties

Identity Matrices

An identity matrix is a square matrix where every diagonal entry is 1 and all the other entries are 0. The following two matrices are both identity matrices and diagonal matrices.

\[ I_3 = \begin{pmatrix} 1 & 0 & 0 \\0 & 1 & 0\\ 0 & 0 & 1 \end{pmatrix} \]

\[ I_4 = \begin{pmatrix} 1 & 0 & 0 & 0\\ 0 & 1 & 0 & 0\\ 0 & 0 & 1 & 0\\ 0 & 0 & 0 & 1 \end{pmatrix} \]

They are called identity matrices, because any matrix multiplied with an identify matrix equals itself. The diagonal entries of a matrix are the entries where the column and row number are the same. \(a_{2,2}\) is a diagonal entry but \(a_{3,5}\) is not. The trace of a n × n matrix is the sum of all the diagonal entries. In other words, for n × n matrix A, \(trace(A) = tr(A) = \sum_{i=1}^{n} a_{i,i}\) For example:

\[ trace(F) = tr(F) = tr \begin{pmatrix} 1 & 3 & 3\\ 0 & 6 & 7\\ -5 & 0 & 1 \end{pmatrix} = 1 + 6 + 1 = 8 \]

The trace has some useful properties, namely that for same size square matrices A and B and scalar c,

\[ tr(A) = tr(A^T)\] \[ tr(A + B) = tr(B + A) = tr(A) + tr(B)\] \[ tr(AB) = tr(BA)\] \[ tr(cA) = c*tr(A)\]

Determintants

The determinate of a square, 2 × 2 matrix A is

\[ det(A) = |A| = \begin{vmatrix} a_{1,1} & a_{1,2} \\ a_{2,1} & a_{2,2} \end{vmatrix} = a_{1,1} * a_{2,2} - a_{1,2}*a_{2,1}\]

For example

\[ det(A) = |A| = \begin{vmatrix} 5 & 2 \\ 7 & 2 \end{vmatrix} = 5*2 - 2*7 = -4\]

For a 3 × 3 matrix B, the determinate is

\[det(B) = |B| = \begin{vmatrix} b_{1,1} & b_{1,2} & b_{1,3}\\ b_{2,1} & b_{2,2} & b_{2,3}\\ b_{3,1} & b_{3,2} & b_{3,3} \end{vmatrix} = b_{1,3} \begin{vmatrix} b_{2,1} & b_{2,2}\\ b_{3,1} & b_{3,2} \end{vmatrix} - b_{2,3} \begin{vmatrix} b_{1,1} & b_{1,2}\\ b_{3,1} & b_{3,2} \end{vmatrix} + b_{3,3} \begin{vmatrix} b_{1,1} & b_{1,2} \\b_{2,1} & b_{2,2} \end{vmatrix}\]

\[ det(B) = b_{1,3}(b_{2,1} b_{3,2} - b_{2,2} b_{3,1} ) - b_{2,3}(b_{1,1} b_{3,2} - b_{1,2} b_{3,1} ) + b_{3,3}(b_{1,1} b_{2,2} - b_{1,2} b_{2,1} ) \]

For example:

\[det(B) = |B| = \begin{vmatrix} 4 & 0 & -1\\2 & -2 & 3 \\ 7 & 5 & 0 \end{vmatrix} = -1 \begin{vmatrix} 2 & -2\\ 7 & 5 \end{vmatrix} - 3 \begin{vmatrix} 4 & 0 \\7 & 5 \end{vmatrix} + 0 \begin{vmatrix} 4 & 0 \\ 2 & -2 \end{vmatrix}\]

\[ det(B) = -1 (2*5 - (-2)*7) + 3(4*5 - 0 *7) - 0 (4*(-2) - 0 * 2) = -1*24 - 3*20 + 0 *(-8) = -24 - 60 = -84 \]

In general, to find the determinate of a n \(\times\) n matrix, choose a row or column like column 1, and take the determinates of the "minor" matrices inside the original matrix, like so:

\[det(C) = |C| = \begin{vmatrix} c_{1,1} & c_{1,2} & \ldots & c_{1,n}\\ c_{2,1} & c_{2,2} & \ldots & c_{2,n}\\ \vdots & \vdots & \ddots & \vdots \\c_{n,1} & c_{.,2} & \ldots & c_{n,n} \end{vmatrix}\]

\[det(C) = (-1)^{1+1} c_{1,1} \begin{vmatrix} c_{2,2} & \ldots & c_{2,n}\\ \vdots & \ddots & \vdots \\ c_{n,2} & \ldots & c_{n,n} \end{vmatrix} + (-1)^{2+1} c_{2,1} \begin{vmatrix} c_{1,2} & \ldots & c_{1,n}\\ c_{3,2} & \ldots & c_{3,n}\\ \vdots & \ddots & \vdots\\ c_{n,2} & \ldots & c_{n,n} \end{vmatrix} + \ldots\]

\[ \ldots + (-1)^{n+1} c_{n,1} \begin{vmatrix} c_{1,2} & \ldots & c_{1,n}\\ \vdots & \ddots & \vdots \\ c_{n-1,2} & \ldots & c_{n-1,n} \end{vmatrix} \]

This is known as Laplace's formula,

Laplace's Formula

 

\[ det(A) = \sum_{j=1}^{n} (-1)^{i+j} a_{i,j} det(A_{-i,-j}) = \sum_{i=1}^{n} (-1)^{i+j} a_{i,j} det(A_{-i,-j}) \]

For any ij, where \(A_{-i,-j}\) is matrix A with row i and column j removed. This formula works whether one goes by rows, using the first formulation, or by columns, using the second formulation. It is easiest to use Laplace's formula when one chooses the row or column with the most zeroes.

 

Matrix Determinant Properties

The matrix determinate has some interesting properties.

\[det(I) = 1\]

where I is the identity matrix.

\[det(A) = det(A^T)\]

If A and B are square matrices with the same dimensions, then

\[ det(AB) = det(A)*det(B)\] and if A is a n × n square matrix and c is a scalar, then

\[ det(cA) = c^n det(A)\]


M.4 Matrix Inverse

M.4 Matrix Inverse
Inverse of a Matrix

The matrix B is the inverse of matrix A if \(AB = BA = I\). This is often denoted as \(B = A^{-1}\) or \(A = B^{-1}\). When taking the inverse of the product of two matrices A and B,

\[(AB)^{-1} = B^{-1} A^{-1}\]

 

When taking the determinate of the inverse of the matrix A,

\[ det(A^{-1}) = \frac{1}{det(A)} = det(A)^{-1}\]

Note that not all matrices have inverses. For a matrix A to have an inverse, that is to say for A to be invertible, A must be a square matrix and \(det(A) \neq 0\). For that reason, invertible matrices are also called nonsingular matrices.

Two examples are shown below

\[ det(A) = \begin{vmatrix} 4 & 5 \\ -2 & 1 \end{vmatrix} = 4*1-5*-2 = 14 \neq 0 \]

\[ det(C) = \begin{vmatrix} 1 & 2 & -1\\ 5 & 3 & 2 \\ 6 & 0 & 6 \end{vmatrix} = -2 \begin{vmatrix} 5 & 2 \\ 6 & 6 \end{vmatrix} + 3 \begin{vmatrix} 1 & -1\\ 6 & 6 \end{vmatrix} + 0 \begin{vmatrix} 1 & -1\\ 5 & 2 \end{vmatrix}\]

\[ det(C)= - 2(5*6-2*6) + 3(1*6-(-1)*6) - 0(1*2-(-1)*5) = 0 \]

So C is not invertible, because its determinate is zero. However, A is an invertible matrix, because its determinate is nonzero. To calculate that matrix inverse of a 2 × 2 matrix, use the below formula.

\[ A^{-1} = \begin{pmatrix} a_{1,1} & a_{1,2}\\ a_{2,1} & a_{2,2} \end{pmatrix} ^{-1} = \frac{1}{det(A)} \begin{pmatrix} a_{2,2} & -a_{1,2} \\  -a_{2,1} & a_{1,1} \end{pmatrix} = \frac{1}{a_{1,1} * a_{2,2} - a_{1,2}*a_{2,1}} \begin{pmatrix} a_{2,2} & -a_{1,2} \\  -a_{2,1} & a_{1,1} \end{pmatrix}\]

For example

\[ A^{-1} = \begin{pmatrix} 4 & 5 \\ -2 & 1 \end{pmatrix} ^{-1} = \frac{1}{det(A)} \begin{pmatrix} 1 & -5 \\ 2 & 4 \end{pmatrix} = \frac{1}{4*1 - 5*(-2)} \begin{pmatrix} 1 & -5 \\ 2 & 4 \end{pmatrix} = \begin{pmatrix} \frac{1}{14} & \frac{-5}{14} \\ \frac{2}{14} & \frac{4}{14} \end{pmatrix}\]

 

For finding the matrix inverse in general, you can use Gauss-Jordan Algorithm. However, this is a rather complicated algorithm, so usually one relies upon the computer or calculator to find the matrix inverse.


M.5 Advanced Matrix Properties

M.5 Advanced Matrix Properties
Orthogonal Vectors

Two vectors, x and y, are orthogonal if their dot product is zero.

For example

\[ e \cdot f = \begin{pmatrix} 2 & 5 & 4 \end{pmatrix} * \begin{pmatrix} 4 \\ -2 \\ 5 \end{pmatrix} = 2*4 + (5)*(-2) + 4*5 = 8-10+20 = 22\]

Vectors e and f are not orthogonal.

\[ g \cdot h = \begin{pmatrix} 2 & 3 & -2 \end{pmatrix} * \begin{pmatrix} 4 \\ -2 \\ 1 \end{pmatrix} = 2*4 + (3)*(-2) + (-2)*1 = 8-6-2 = 0\]

However, vectors g and h are orthogonal. Orthogonal can be thought of as an expansion of perpendicular for higher dimensions. Let \(x_1, x_2, \ldots , x_n\) be m-dimensional vectors. Then a linear combination of \(x_1, x_2, \ldots , x_n\) is any m-dimensional vector that can be expressed as

\[ c_1 x_1 + c_2 x_2 + \ldots + c_n x_n \]

where \(c_1, \ldots, c_n\) are all scalars. For example:

\[x_1 =\begin{pmatrix}
  3  \\
  8 \\
  -2
\end{pmatrix},
x_2 =\begin{pmatrix}
  4  \\
  -2 \\
  3
\end{pmatrix}\]

\[y =\begin{pmatrix}
  -5 \\
  12 \\
  -8
\end{pmatrix} = 1*\begin{pmatrix}
  3  \\
  8 \\
  -2
\end{pmatrix} + (-2)* \begin{pmatrix}
  4  \\
  -2 \\
  3
\end{pmatrix} = 1*x_1 + (-2)*x_2\]

So y is a linear combination of \(x_1\) and \(x_2\). The set of all linear combinations of \(x_1, x_2, \ldots , x_n\) is called the span of \(x_1, x_2, \ldots , x_n\). In other words

\[ span(\{x_1, x_2, \ldots , x_n \} ) = \{ v| v= \sum_{i = 1}^{n} c_i x_i , c_i \in \mathbb{R} \} \]

A set of vectors \(x_1, x_2, \ldots , x_n\) is linearly independent if none of the vectors in the set can be expressed as a linear combination of the other vectors. Another way to think of this is a set of vectors \(x_1, x_2, \ldots , x_n\) are linearly independent if the only solution to the below equation is to have \(c_1 = c_2 = \ldots = c_n = 0\), where \(c_1 , c_2 , \ldots , c_n \) are scalars, and \(0\) is the zero vector (the vector where every entry is 0).

\[ c_1 x_1 + c_2 x_2 + \ldots + c_n x_n = 0 \]

If a set of vectors is not linearly independent, then they are called linearly dependent.

Example M.5.1

\[ x_1 =\begin{pmatrix} 3 \\ 4 \\ -2 \end{pmatrix}, x_2 =\begin{pmatrix} 4 \\ -2 \\ 2 \end{pmatrix}, x_3 =\begin{pmatrix} 6 \\ 8 \\ -2 \end{pmatrix} \]

Does there exist a vector c, such that,

\[ c_1 x_1 + c_2 x_2 + c_3 x_3 = 0 \]

To answer the question above, let:

\begin{align} 3c_1 + 4c_2 +6c_3 &= 0,\\ 4c_1 -2c_2 + 8c_3 &= 0,\\ -2c_1 + 2c_2 -2c_3 &= 0 \end{align}

Solving the above system of equations shows that the only possible solution is \(c_1 = c_2 = c_3 = 0\). Thus \(\{ x_1 , x_2 , x_3 \}\) is linearly independent. One way to solve the system of equations is shown below. First, subtract (4/3) times the 1st equation from the 2nd equation.

\[-\frac{4}{3}(3c_1 + 4c_2 +6c_3) + (4c_1 -2c_2 + 8c_3) = -\frac{22}{3}c_2 = -\frac{4}{3}0 + 0 = 0 \Rightarrow c_2 = 0 \]

Then add the 1st and 3 times the 3rd equations together, and substitute in \(c_2 = 0\).

\[ (3c_1 + 4c_2 +6c_3) + 3*(-2c_1 + 2c_2 -2c_3) = -3c_1 + 10 c_2 = -3c_1 + 10*0 = 0 + 3*0 = 0 \Rightarrow c_1 = 0 \]

Now, substituting both \(c_1 = 0\) and \(c_2 = 0\) into equation 2 gives.

\[ 4c_1 -2c_2 + 8c_3 = 4*0 -2*0 + 8c_3 = 0 \Rightarrow c_3 = 0 \]

So \(c_1 = c_2 = c_3 = 0\), and \(\{ x_1 , x_2 , x_3 \}\) are linearly independent.

Example M.5.2

\[ x_1 =\begin{pmatrix} 1 \\ -8 \\ 8 \end{pmatrix}, x_2 =\begin{pmatrix} 4 \\ -2 \\ 2 \end{pmatrix}, x_3 =\begin{pmatrix} 1 \\ 3 \\ -2 \end{pmatrix} \]

In this case \(\{ x_1 , x_2 , x_3 \}\)are  linearly dependent, because if \(c = (-1, 1, -2)\), then

\[c^T X = \begin{pmatrix}
  -1  \\
  1\\
  -2
\end{pmatrix}  \begin{pmatrix}
x_1 & x_2 & x_3
\end{pmatrix} = -1 \begin{pmatrix}
  1  \\
  -8\\
  8
\end{pmatrix}+ 1
\begin{pmatrix}
  4  \\
  -2\\
  2
\end{pmatrix} - 2 \begin{pmatrix}
  1  \\
  3 \\
  -2
\end{pmatrix} =
 \begin{pmatrix}
  -1*1 +1*4-2*1  \\
  -1*-8+1*-2-2*3 \\
  -1*8+1*2-2*-2
\end{pmatrix}=
 \begin{pmatrix}
  0  \\
  0 \\
 0
\end{pmatrix}
\]

Norm of a vector or matrix

 The norm of a vector or matrix is a measure of the "length" of said vector or matrix. For a vector x, the most common norm is the \(\mathbf{L_2}\) norm, or Euclidean norm. It is defined as

\[ \| x \| = \| x \|_2 = \sqrt{ \sum_{i=1}^{n} x_i^2 } \]

Other common vector norms include the \(\mathbf{L_1}\) norm, also called the Manhattan norm and Taxicab norm.

\[ \| x \|_1 = \sum_{i=1}^{n} |x_i| \]

Other common vector norms include the Maximum norm, also called the Infinity norm.

\[ \| x \|_\infty = max( |x_1| ,|x_2|, \ldots ,|x_n|) \]

The most commonly used matrix norm is the Frobenius norm. For a m × n matrix A, the Frobenius norm is defined as:

\[ \| A \| = \| A \|_F = \sqrt{ \sum_{i=1}^{m} \sum_{j=1}^{n} x_{i,j}^2 } \]


Quadratic Form of a Vector

The quadratic form of the vector x associated with matrix A is

\[ x^T A x = \sum_{i = 1}^{m} \sum_{j=1}^{n} a_{i,j} x_i x_j \]

A matrix A is Positive Definite if for any non-zero vector x, the quadratic form of x and A is strictly positive. In other words, \(x^T A x > 0\) for all nonzero x.

A matrix A is Positive Semi-Definite or Non-negative Definite if for any non-zero vector x, the quadratic form of x and A is non-negative . In other words, \(x^T A x \geq 0\) for all non-zero x. Similarly,

A matrix A is Negative Definite if for any non-zero vector x, \(x^T A x < 0\). A matrix A is Negative Semi-Definite or Non-positive Definite if for any non-zero vector x, \(x^T A x \leq 0\).


M.6 Range, Nullspace and Projections

M.6 Range, Nullspace and Projections
Range of a matrix

The range of m × n matrix A, is the span of the n columns of A. In other words, for

\[ A = [ a_1 a_2 a_3 \ldots a_n ] \]

where \(a_1 , a_2 , a_3 , \ldots ,a_n\) are m-dimensional vectors,

\[ range(A) = R(A) = span(\{a_1, a_2, \ldots , a_n \} ) = \{ v| v= \sum_{i = 1}^{n} c_i a_i , c_i \in \mathbb{R} \} \]

The dimension (number of linear independent columns) of the range of A is called the rank of A. So if 6 × 3 dimensional matrix B has a 2 dimensional range, then \(rank(A) = 2\).

For example

\[C =\begin{pmatrix}
  1 & 4 & 1\\
  -8 & -2 & 3\\
  8 & 2  & -2
\end{pmatrix} =  \begin{pmatrix}
x_1 & x_2 & x_3
\end{pmatrix}= \begin{pmatrix}
  y_1  \\
  y_2\\
  y_3
\end{pmatrix}\]

C has a rank of 3, because \(x_1\), \(x_2\) and \(x_3\) are linearly independent.

Nullspace
p>The nullspace of a m \(\times\) n matrix is the set of all n-dimensional vectors that equal the n-dimensional zero vector (the vector where every entry is 0) when multiplied by A.  This is often denoted as

 

\[N(A) = \{ v | Av = 0 \}\]

The dimension of the nullspace of A is called the nullity of A.  So if 6 \(\times\) 3 dimensional matrix B has a 1 dimensional range, then \(nullity(A) = 1\).

 

The range and nullspace of a matrix are closely related.  In particular, for m \(\times\) n matrix A,

\[\{w | w = u + v, u \in R(A^T), v \in N(A) \} = \mathbb{R}^{n}\]

\[R(A^T) \cap N(A) = \phi\]

This leads to the rank--nullity theorem, which says that the rank and the nullity of a matrix sum together to the number of columns of the matrix. To put it into symbols:

Rank--nullity Theorem

 

\[A \in \mathbb{R}^{m \times n} \Rightarrow rank(A) + nullity(A) = n\]

 

For example, if B is a 4 \(\times\) 3 matrix and \(rank(B) = 2\), then from the rank--nullity theorem, on can deduce that

\[rank(B) + nullity(B) = 2 + nullity(B) = 3 \Rightarrow nullity(B) = 1\]

Projection

The projection of a vector x onto the vector space J, denoted by Proj(X, J), is the vector \(v \in J\) that minimizes \(\vert x - v \vert\).  Often, the vector space J one is interested in is the range of the matrix A, and norm used is the Euclidian norm.  In that case

\[Proj(x,R(A)) = \{ v \in R(A) |  \vert x - v \vert_2 \leq \vert x - w \vert_2 \forall w \in R(A) \}\]

In other words

\[Proj(x,R(A)) = argmin_{v \in R(A)} \vert x - v \vert_2\]


M.7 Gauss-Jordan Elmination

M.7 Gauss-Jordan Elmination

Gauss-Jordan Elimination is an algorithm that can be used to solve systems of linear equations and to find the inverse of any invertible matrix. It relies upon three elementary row operations one can use on a matrix:

  1. Swap the positions of two of the rows
  2. Multiply one of the rows by a nonzero scalar.
  3. Add or subtract the scalar multiple of one row to another row.
 

For an example of the first elementary row operation, swap the positions of the 1st and 3rd row.

\[ \begin{pmatrix} 4 & 0 & -1 \\ 2 & -2 & 3 \\ 7 & 5 & 0 \end{pmatrix}\Rightarrow  \begin{pmatrix} 7 & 5 & 0 \\ 2 & -2 & 3 \\ 4 & 0 & -1 \end{pmatrix} \]

For an example of the second elementary row operation, multiply the second row by 3.

\[ \begin{pmatrix} 4 & 0 & -1 \\ 2 & -2 & 3 \\ 7 & 5 & 0 \end{pmatrix} \Rightarrow \begin{pmatrix} 4 & 0 & -1 \\ 6 & -6 & 9 \\ 7 & 5 & 0 \end{pmatrix} \]

For an example of the third elementary row operation, add twice the 1st row to the 2nd row.

\[ \begin{pmatrix} 4 & 0 & -1 \\ 2 & -2 & 3 \\ 7 & 5 & 0 \end{pmatrix}\Rightarrow  \begin{pmatrix} 4 & 0 & -1 \\ 10 & -2 & 1 \\ 7 & 5 & 0 \end{pmatrix} \]


Reduced-row echelon form

The purpose of Gauss-Jordan Elimination is to use the three elementary row operations to convert a matrix into reduced-row echelon form. A matrix is in reduced-row echelon form, also known as row canonical form, if the following conditions are satisfied:

  1. All rows with only zero entries are at the bottom of the matrix
  2. The first nonzero entry in a row, called the leading entry or the pivot, of each nonzero row is to the right of the leading entry of the row above it.
  3. The leading entry, also known as the pivot, in any nonzero row is 1.
  4. All other entries in the column containing a leading 1 are zeroes.

For example

\[A = \begin{pmatrix} 1 & 0 & 0 \\ 0 & 1 & 3 \\ 0 & 0 & 0 \end{pmatrix}, B = \begin{pmatrix} 1 & 0 & 0 \\ 0 & 1 & 0 \\ 0 & 0 & 1 \end{pmatrix}, C = \begin{pmatrix} 0 & 7 & 3 \\ 1 & 0 & 0 \\ 0 & 0 & 0 \end{pmatrix}, D = \begin{pmatrix} 1 & 7 & 3 \\ 0 & 1 & 0 \\ 0 & 0 & 1 \end{pmatrix} \]

Matrices A and B are in reduced-row echelon form, but matrices C and D are not. C is not in reduced-row echelon form because it violates conditions two and three. D is not in reduced-row echelon form because it violates condition four. In addition, the elementary row operations can be used to reduce matrix D into matrix B.


Steps for Gauss-Jordan Elimination

To perform Gauss-Jordan Elimination:

  1. Swap the rows so that all rows with all zero entries are on the bottom
  2. Swap the rows so that the row with the largest, leftmost nonzero entry is on top.
  3. Multiply the top row by a scalar so that top row's leading entry becomes 1.
  4. Add/subtract multiples of the top row to the other rows so that all other entries in the column containing the top row's leading entry are all zero.
  5. Repeat steps 2-4 for the next leftmost nonzero entry until all the leading entries are 1.
  6. Swap the rows so that the leading entry of each nonzero row is to the right of the leading entry of the row above it.

Selected video examples are shown below:

To obtain the inverse of a n × n matrix A :

  1. Create the partitioned matrix \(( A | I )\) , where I is the identity matrix.
  2. Perform Gauss-Jordan Elimination on the partitioned matrix with the objective of converting the first part of the matrix to reduced-row echelon form.
  3. If done correctly, the resulting partitioned matrix will take the form \(( I | A^{-1} )\)
  4. Double-check your work by making sure that \(AA^{-1} = I\).

M.8 Eigendecomposition

M.8 Eigendecomposition
Eigenvector of a matrix

An eigenvector of a matrix A is a vector whose product when multiplied by the matrix is a scalar multiple of itself. The corresponding multiplier is often denoted as \(lambda\) and referred to as an eigenvalue. In other words, if A is a matrix, v is a eigenvector of A, and \(\lambda\) is the corresponding eigenvalue, then \(Av = \lambda v\).

For Example

\[ A= \begin{pmatrix} 4 & 0 & -1 \\ 2 & -2 & 3 \\ 7 & 5 & 0 \end{pmatrix} \]

\[ v = \begin{pmatrix} 1 \\ 1 \\ 2 \end{pmatrix} \]

\[ Av = \begin{pmatrix} 4 & 0 & 1 \\ 2 & -2 & 3 \\ 5 & 7 & 0 \end{pmatrix} \begin{pmatrix} 1 \\ 1 \\ 2 \end{pmatrix} = \begin{pmatrix} 4*1 + 0*1 + 1*2  \\ 2*1+ -2*1+ 3*2  \\ 5*1+ 7*1+ 0*2 \end{pmatrix} = \begin{pmatrix} 6 \\ 6 \\ 12 \end{pmatrix} = 6v \]

 

In the above example, v is an eigenvector of A, and the corresponding eigenvalue is 6. To find the eigenvalues/vectors of a n × n square matrix, solve the characteristic equation of a matrix for the eigenvalues. This equation is

\[ det(A - \lambda I ) = 0\]

Where A is the matrix, \(\lambda\) is the eigenvalue, and I is an n × n identity matrix. For example, take

\[ A= \begin{pmatrix} 4 & 3 \\ 2 & -1 \end{pmatrix}\]

The characteristic equation of A is listed below.

\[ det(A - \lambda I ) = det( \begin{pmatrix} 4 & 3 \\ 2 & -1 \end{pmatrix} - \lambda \begin{pmatrix} 1 & 0 \\ 0 & 1 \end{pmatrix} ) = det \begin{pmatrix} 4 - \lambda & 3 \\ 2 & -1 - \lambda \end{pmatrix} = 0 \]

\[ det(A - \lambda I ) = (4 - \lambda)(-1 - \lambda) - 3*2 = \lambda^2 - 3 \lambda - 10 = (\lambda + 2)(\lambda - 5) = 0 \]

Therefore, one finds that the eigenvalues of A must be -2 and 5. Once the eigenvalues are found, one can then find the corresponding eigenvectors from the definition of an eigenvector. For \(\lambda = 5\), simply set up the equation as below, where the unknown eigenvector is \(v = (v_1, v_2)'\).

\[\begin{pmatrix} 4 & 3 \\ 2 & -1 \end{pmatrix} * \begin{pmatrix} v_1 \\  v_2 \end{pmatrix} = 5 \begin{pmatrix} v_1 \\  v_2 \end{pmatrix} \]

\[\begin{pmatrix} 4 v_1 + 3 v_2 \\ 2 v_1 - 1 v_2 \end{pmatrix} = \begin{pmatrix} 5 v_1 \\ 5 v_2 \end{pmatrix} \]

And then solve the resulting system of linear equations to get

\[ v = \begin{pmatrix} 3 \\ 1 \end{pmatrix} \]

For \(\lambda = -2\), simply set up the equation as below, where the unknown eigenvector is \(w = (w_1, w_2)\).

\[\begin{pmatrix} 4 & 3 \\ 2 & -1 \end{pmatrix} * \begin{pmatrix} w_1 \\ w_2 \end{pmatrix} = -2 \begin{pmatrix} w_1 \\ w_2 \end{pmatrix} \]

\[\begin{pmatrix} 4 w_1 + 3 w_2 \\ 2 w_1 - 1 w_2 \end{pmatrix} = \begin{pmatrix} -2 w_1 \\ -2 w_2 \end{pmatrix} \]

And then solve the resulting system of linear equations to get

\[ w = \begin{pmatrix} -1 \\ 2 \end{pmatrix} \]


M.9 Self-Assess

M.9 Self-Assess

Here's your chance to assess what you remember from the matrix review.

  1. Review the concepts and methods on the pages in this section. Note the courses that certain sections are aligned with as prerequisites:
      STAT 414 STAT 501 STAT 504 STAT 505
    M.1 - Matrix Definitions  Required  Required  Required Required
    M.2 - Matrix Arithmetic  Required  Required  Required Required
    M.3 - Matrix Properties  Required  Required  Required Required
    M.4 - Matrix Inverse  Required  Required  Required Required
    M.5 - Advanced Topics  Recommended  Recommended  Recommended 5.1, 5.4, Required
    5.2, 5.3, Recommended
  2. Download and Complete the Self-Assessment Exam.
  3. Determine your Score by Reviewing the Self-Assessment Exam Solutions.

 

Students that score below 70% (fewer than 21 questions correct) should consider further review of these materials and are strongly encouraged to take MATH 220 (2 credits) or an equivalent course.

If you have struggled with the concepts and methods that are presented here, you will indeed struggle in the courses above that expect this foundation.

Please Note: These materials are NOT intended to be a complete treatment of the ideas and methods used in Matrix algebra. These materials and the accompanying self-assessment are simply intended as simply an 'early warning signal' for students. Also, please note that completing the self-assessment successfully does not automatically ensure success in any of the courses that use this foundation. 

Additional Linear Algebra Online Materials


Legend
[1]Link
Has Tooltip/Popover
 Toggleable Visibility