Linear Algebra Review : Machine Learning

Linear Algebra Review

Khan Academy has excellent Linear Algebra Tutorials.

This online Linear Algebra text is also an excellent resource, particularly for a proof of the normal equation.

Matrices and Vectors

Matrices are 2-dimensional arrays:

A=⎡⎣⎢⎢⎢adgjbehkcfil⎤⎦⎥⎥⎥

The above matrix has four rows and three columns, so it is a 4 x 3 matrix.

A vector is a matrix with one column and many rows:

⎡⎣⎢⎢wxyz⎤⎦⎥⎥

So vectors are a subset of matrices. The above vector is a 4 x 1 matrix.

Notation and terms:

* Aij refers to the element in the ith row and jth column of matrix A.

* A vector with 'n' rows is referred to as an 'n'-dimensional vector

* vi refers to the element in the ith row of the vector.

* In general, all our vectors and matrices will be 1-indexed.

* Matrices are usually denoted by uppercase names while vectors are lowercase.

* "Scalar" means that an object is a single value, not a vector or matrix.

* R refers to the set of scalar real numbers

* Rn refers to the set of n-dimensional vectors of real numbers

Addition and Scalar Multiplication

Addition and subtraction are element-wise, so you simply add or subtract each corresponding element:

[acbd]+[wyxz]=[a+wc+yb+xd+z]

To add or subtract two matrices, their dimensions must be the same.

In scalar multiplication, we simply multiply every element by the scalar value:

[acbd]∗x=[a∗xc∗xb∗xd∗x]

Matrix-Vector Multiplication

We map the column of the vector onto each row of the matrix, multiplying each element and summing the result.

⎡⎣acebdf⎤⎦∗[xy]=⎡⎣⎢a∗x+b∗yc∗x+d∗ye∗x+f∗y⎤⎦⎥

The result is a vector. The vector must be the second term of the multiplication. The number of rows of the vector must equal the number of columns of the matrix.

An n x m matrix multiplied by an m x 1 vector results in an n x 1 vector.

Matrix-Matrix Multiplication

We multiply two matrices by breaking it into several vector multiplications and concatenating the result

⎡⎣acebdf⎤⎦∗[wyxz]=⎡⎣⎢a∗w+b∗yc∗w+d∗ye∗w+f∗ya∗x+b∗zc∗x+d∗ze∗x+f∗z⎤⎦⎥

An m x n matrix multiplied by an n x o matrix results in an m x o matrix. In the above example, a 3 x 2 matrix times a 2 x 2 matrix resulted in a 3 x 2 matrix.

To multiply two matrices, the number of columns of the first matrix must equal the number of rows of the second matrix.

Matrix Multiplication Properties

* Not commutative. A∗B≠B∗A

* Associative. (A∗B)∗C=A∗(B∗C)

The "identity matrix", when multiplied by any matrix of the same dimensions, results in the original matrix. It's just like multiplying numbers by 1. The identity matrix simply has 1's on the diagonal and 0's elsewhere.

When multiplying the identity matrix after some matrix, the square identity matrix should match the other matrix'scolumns. When multiplying the identity matrix before some other matrix, the square identity matrix should match the other matrix's rows.

Inverse and Transpose

The inverse of a matrix A is denoted A−1. Multiplying by the inverse results in the identity matrix.

A non square matrix does not have an inverse matrix. We can compute inverses of matrices in octave with the pinv(A) function [1].

The transposition of a matrix is like rotating the matrix once clockwise and then reversing it:

A=⎡⎣acebdf⎤⎦

AT=[abcdef]

In other words:

Aij=ATji