Orthogonal Projections

Read

Discuss

Orthogonal Sets:

A set of vectors $\left \{ u_1, u_2, ... u_p \right \}$ in $\mathbb{R^n}$ is called orthogonal set, if $u_i \cdot u_j =0$ . if $i \neq j$

Orthogonal Basis

An orthogonal basis for a subspace W of $\mathbb{R^n}$ is a basis for W that is also an orthogonal set.

Let S = $\left \{ u_1, u_2, ... u_p \right \}$ be the orthogonal basis for a W of $\mathbb{R^n}$ is a basis for W that is also a orthogonal set. We need to calculate $c_1, c_2, ... c_p$ such that :

$y = c_1 u_1 + c_2 u_2 + ... c_p u_p$

Let’s take the dot product of u_1 both side.

$y \cdot u_1 = (c_1 u_1 + c_2 u_2 + ... c_p u_p) \cdot u_1$

$y \cdot u_1 = c_1 (u_1 \cdot u_1) + c_2 (u_2 \cdot u_1) + ... c_p (u_p \cdot u_1)$

Since, this is orthogonal basis $u_2 \cdot u_1 = u_3 \cdot u_1 = ... = u_p \cdot u_1 =0$ . This gives $c_1$ :

$c_1 = \frac{y \cdot u_1}{ u_1 \cdot u_1}$

We can generalize the above equation

$c_j = \frac{y \cdot u_j}{ u_j \cdot u_j}$

Orthogonal Projections

Suppose {u_1, u_2,… u_n} is an orthogonal basis for W in $\mathbb{R^n}$ . For each y in W:

$y =\left ( \frac{y \cdot u_1}{u_1 \cdot u_1} \right ) u_1 + ... + \left ( \frac{y \cdot u_p}{u_p \cdot u_p } \right )u_p$

Let’s take $\left \{ u_1, u_2, u_3 \right \}$ is an orthogonal basis for $\mathbb{R^3}$ and W = span $\left \{ u_1, u_2 \right \}$ . Let’s try to write a write y in the form $\hat{y}$ belongs to W space, and z that is orthogonal to W.

$y =\left ( \frac{y \cdot u_1}{u_1 \cdot u_1} \right ) u_1 + \left ( \frac{y \cdot u_2}{u_2\cdot u_2 } \right )u_2 + \left ( \frac{y \cdot u_3}{u_3 \cdot u_3} \right ) u_3 \\$

where

$\hat{y} = \left ( \frac{y \cdot u_1}{u_1 \cdot u_1} \right ) u_1 + \left ( \frac{y \cdot u_2}{u_2\cdot u_2 } \right )u_2$

and

$z = \left ( \frac{y \cdot u_3}{u_3 \cdot u_3} \right ) u_3$ [Tex]y= \hat{y} + z[/Tex]

Now, we can see that z is orthogonal to both $u_1$ and $u_2$ such that:

$z \cdot u_1 =0 \\ z \cdot u_2 =0$

Orthogonal Decomposition Theorem:

Let W be the subspace of $\mathbb{R^n}$ . Then each y in $\mathbb{R^n}$ can be uniquely represented in the form:

$y = \hat{y} + z$

where $\hat{y}$ is in W and z in W^{\perp}. If $\left \{ u_1, u_2, ... u_p \right \}$ is an orthogonal basis of W. then,

$\hat{y} =\left ( \frac{y \cdot u_1}{u_1 \cdot u_1} \right ) u_1 + ... + \left ( \frac{y \cdot u_p}{u_p \cdot u_p } \right )u_p$

thus:

$z = y - \hat{y}$

Then, $\hat{y}$ is the orthogonal projection of y in W.

Best Approximation Theorem

Let W is the subspace of $\mathbb{R^n}$ , y any vector in $\mathbb{R^n}$ . Let v in W and different from $\hat{y}$ . Then $\left \| v-\hat{y} \right \|$ also in W.

$z = y - \hat{y}$ is orthogonal to W, and also orthogonal to $v=\hat{y}$ . Then y-v can be written as:

$y-v = (y- \hat{y}) + (\hat{y} -v)$

Thus:

$\left \| y-v \right \|^{2} = \left \| y- \hat{y} \right \|^{2} + \left \| \hat{y}-v \right \|^{2}$

Thus, this can be written as:

$\left \| y-v \right \|^{2} > \left \| y- \hat{y} \right \|^{2}$

and

$\left \| y-v \right \| > \left \| y- \hat{y} \right \|$

References:

Whether you're preparing for your first job interview or aiming to upskill in this ever-evolving tech landscape, GeeksforGeeks Courses are your key to success. We provide top-quality content at affordable prices, all geared towards accelerating your growth in a time-bound manner. Join the millions we've already empowered, and we're here to do the same for you. Don't miss out - check it out now!

Last Updated : 16 Oct, 2021

Boosting in Machine Learning | Boosting and AdaBoost

Robust Correlation

Orthogonal Projections

Orthogonal Sets:

Orthogonal Basis

Orthogonal Projections

Orthogonal Decomposition Theorem:

Best Approximation Theorem

References:

Please Login to comment...