# Singular Value Decomposition

**Prerequisites:** Matrix Diagonalization, Eigenvector Computation and Low-Rank Approximations

Before getting in depth into the SVD, let us first briefly understand what Matrix Diagonalization technique is and when it fails to perform efficiently.

**Matrix Diagonalization**

Matrix diagonalization is the process of taking a *square matrix* and converting it into a special type of matrix known as the diagonal matrix. This matrix shares the same fundamental properties of the underlying matrix. Mathematically, any input matrix A can be reduced into any diagonal matrix D if it satisfies:

where, P ->Modal Matrix:It is a (n x n) matrix that consists of eigen-vectors. It is generally used in the process of diagonalization and similarity transformation.

However, the matrix diagonalization technique **fails** for matrices of the form (**m x n)** where **m ≠ n**. (i.e. when the matrix is not a square matrix. This is where *‘Singular Value Decomposition’* comes into picture and provides a good solution to this problem.

**Singular Values (σ)**

Let *A* be any *m x n* matrix with rank *r.* On multiply it with its transpose (i.e. *A ^{T}A*), a

*n x n*matrix is created which is symmetric as well as positive semi-definite in nature. In simpler terms, all the Eigen values

*(λ*of

_{i…r})*A*matrix are non-negative (i.e. greater than 0).

^{T}AThe singular values are defined as the square root of the obtained Eigen values. That is:

**Singular Value Decomposition (SVD)**

Let *A* be any *m x n* matrix. Then the SVD divides this matrix into 2 unitary matrices that are orthogonal in nature and a rectangular diagonal matrix containing singular values till *r*. Mathematically, it is expressed as:

where,Σ ->(m x n) orthogonal matrixU ->(m x m) orthogonal matrixV ->(n x n) diagonal matrix with first r rows having only singular values. (Rest of the values are 0)

Now, It is important to understand how to calculate the matrices **U**, **V** & **Σ**.

**Calculating orthogonal matrix V**

First, we calculate the Eigen vectors x_{i} associated with input matrix A. Then, we find the normalized vectors v_{i} corresponding to x_{i }by dividing each value in vector x_{i }by its magnitude. For example:

Let x = [1,2,4] => mag(x) or |x| = √(1^{2}+ 2^{2}+ 4^{2}) = √21. Therefore, v = [(1/√21), (2/√21), (4/√21)]

We know, *A* is a *m x n* matrix. Therefore, A^{T}A is a n x n symmetric matrix with all Eigen values > 0. So, we can obtain Eigen vectors v_{1…n }of A^{T}A such that:

where,xeigen vector_{i ->}vnormalized eigen vector. and_{i ->}σcorresponding singular value._{i ->}λcorresponding eigen value._{i ->}

Upon calculating the Eigen vectors of *AAT*, matrix *V* will be:

where, v_{1}, v_{2}, ... v_{i}arearranged column-wiseinto matrix V.

**Calculating orthogonal matrix U**

Similarly, for any *A* (*m x n*) matrix, *AA ^{T}* is a

*m x m*symmetric matrix with all eigen values > 0. So, we can obtain eigen vectors x

_{1…n}of

*AA*such that:

^{T}where,xeigen vector. and_{i ->}σcorresponding singular value._{i ->}λcorresponding eigen value._{i ->}

Now, we use the following equation to compute matrix U:

Upon calculating, matrix U will be:

where, u_{1}, u_{2}, ... u_{i}arearranged column-wiseinto matrix U.

**Calculating diagonal matrix Σ**

Here, matrix A has rank(A) = r where r ≤ min (m,n).

**Case 1: **If **m ≤ n**, say m = 2 & n = 4, then assuming all (σ_{i }> 0), Σ can be expressed as:

**Case 2:** If m ≥ n, say m = 5 & n = 3, then assuming all (σ_{i} > 0), Σ can be expressed as:

**Special Case: **When rank of matrix is specified, say r = 3, m = 6 & n = 4. Then Σ can be expressed as:

This implies that **σ _{4} ≤ 0**, hence discarded.

NOTE:The number of singular values whereσcan_{i}> 0determine the rank of the matrix.

**Example Problem**

Consider the following problem. Find the SVD of a (*2 x 3*) matrix A having values:

**Solution**

Let us understand each step required for solving such problems.

Step 1 -Find A^{T}and then compute A^{T}A.

Step 2 -Find the eigen values associated with matrix A^{T}A.(Discussed in the prerequisite articles mentioned above)

**Eigen values associated with A ^{T}A:** λ = 0, 1 & 3.

Step 3 -Find the singular values corresponding to the obtained eigen values using formula:

**Singular values associated with A ^{T}A: **λ = 3, 1 & 0.

λ_{1}= 3->σ_{1}= √3 λ_{2}= 1->σ_{2}= 1 λ_{3}= 0->σ_{3}= 0

Step 4 -Compute diagonal matrix Σ using the values of σ keeping the above discussed cases in mind.

As (m = 2 < n = 3), Case 1 is applied and matrix Σ is:

Step 5 -Find the eigen vectors & corresponding normalized eigen vectors associated with matrix A^{T}A.(Discussed in the prerequisite articles mentioned above)

NOTE:It is important to understand that normalized eigen vectors of A^{T}A define the matrix V.

**Eigen vectors associated with A ^{T}A:**

For λx_{1 = 3 ->}_{1}= [1, 2, 1]For λx_{2 = 1 ->}_{2}= [-1, 0, 1]For λx_{3 = 0 ->}_{3}= [1, -1, 1] where x_{1}, x_{2}and x_{3}are eigen vectors of matrix A^{T}A.

**Normalized eigen vectors associated with A ^{T}A:**

For x_{1}= [1, 2, 1] => v_{1}= [(1/√6), (2/√6), (1/√6)] For x_{2}= [-1, 0, 1] => v_{2}= [(-1/√2), 0, (1/√2)] For x_{3}= [1, -1, 1] => v_{3}= [(1/√3), (-1/√3), (1/√3)] where v_{1}, v_{2}and v_{3}are eigen vectors of matrix A^{T}A.

Step 6 -Use eigen vectors obtained to compute matrix V.

Step 7 -Use the above given equation to compute the orthogonal matrix U.

Therefore, orthogonal matrix U is:

Step 8 -Compute the SVD of A using the equation given below:(As discussed above)

Therefore, using SVD, *A* can be expressed as:

Attention reader! Don’t stop learning now. Get hold of all the important Machine Learning Concepts with the **Machine Learning Foundation Course** at a student-friendly price and become industry ready.