标签：begin right end matrix Orthogonality hat left

4.1 Orthogonal Vectors and Suspaces

• Orthogonal vectors have \(v^Tw=0\)，and \(||v||^2 + ||w||^2 = ||v+w||^2 = ||v-w||^2\).

• Subspaces \(V\) and \(W\) are orthogonal when \(v^Tw = 0\) for every \(v\) in V and every \(w\) in W.

• Four Subspaces Relations:

  • Every vector x in the nullspace is perpendicular to every row of A, because \(Ax=0\).The nullspace N(A) and the row space \(C(A^T)\) are orthogonal subspaces of \(R^n\). \(N(A) \bot C(A^T)\)

    \[Ax = \left[ \begin{matrix} &row_1& \\ &\vdots \\ &row_m \end{matrix} \right] \left[ \begin{matrix} && \\ &x \\ && \end{matrix} \right] = \left[ \begin{matrix} &0& \\ &\vdots \\ &0 \end{matrix} \right] \]

  • Every vector y in the nullspace of \(A^T\) is perpendicular to every column of A, because \(A^Ty=0\).The left nullspace \(N(A^T)\) and the column space \(C(A)\) are orthogonal subspaces of \(R^m\). \(N(A^T) \bot C(A)\)

    \[A^Ty = \left[ \begin{matrix} &(column_1)^T& \\ &\vdots \\ &(column_m)^T \end{matrix} \right] \left[ \begin{matrix} && \\ &y \\ && \end{matrix} \right] = \left[ \begin{matrix} &0& \\ &\vdots \\ &0 \end{matrix} \right] \]

  • The nullspace N(A) and the row space \(C(A^T)\) are orthogonal complements, with dimensions (n-r) + r = n. Similarly \(N(A^T)\) and C(A) are orthogonal complements with (m-r) + r = m.

4.2 Projection

When b is not in the columns space C(A)，then Ax = b is unsolvable，so find the projection p of b onto C(A)，make \(A\hat{x} = p\) is solvable.

Three Steps:

1. Find \(\hat{x}\)
2. Find p
3. Find project matrix P

Projection Onto a Line

A line goes through the origin in the direction of \(a=(a_1, a_2,...,a_m)\).

Along that line, we want the point \(p\) closest to \(b=(b_1,b_2,...,b_m)\).

The key to projection is orthogonality : The line from b to p is perpendicular to the vector a.

Projecting b onto a with error \(e=b-\hat{x}a\) :

\[a \cdot(b-\hat{x}a) = a^T(b-\hat{x}a)=0 \\ \Downarrow \\ \hat{x} = \frac{a \cdot b}{a \cdot a} = \frac{a^T b}{a^T a} \\ \Downarrow \\ p = a \hat{x} =a\frac{a^T b}{a^T a} = Pb \\ \Downarrow \\ P = \frac{aa^T}{a^Ta} \\ \]

\(P=P^T\) (P is symmetric) , \(P^2=P\)

Projection Onto a Subspace

Start with n independent vectors \(A=(a_1,...,a_n)\) in \(R^m\), find the combination \(p=\hat{x}a_1 + \hat{x}a_2 +...+\hat{x_n}a_n\) closest to a givent vector b.

\[A^T(b-A\hat{x}) = \left[ \begin{matrix} -&a_1^T&- \\ &\vdots \\ -&a_n^T&- \end{matrix} \right] \left[ \begin{matrix} && \\ &b-A\hat{x} \\ && \end{matrix} \right] = \left[ \begin{matrix} &0& \\ &\vdots \\ &0 \end{matrix} \right] \\ \Downarrow \\ A^T(b-A\hat{x}) = 0 \ \ 或 \ \ \ A^TA\hat{x} = A^Tb \\ \Downarrow A^TA \ \ invertible\\ p=A\hat{x} = A(A^TA)^{-1}A^Tb \\ \Downarrow \\ P=A(A^TA)^{-1}A^T \]

\(P=P^T\) (P is symmetric) , \(P^2=P\)

4.3 Least Squares Approximations

When b is outside the column space of A, that \(Ax=b\) has no solution. Find the length of \(e=b-Ax\) as small as possible, get a \(\hat{x}\) is a least squares solution.

If A has independent columns, then \(A^TA\) is invertible.

proof :

\[A^TAx = 0 \\ \Downarrow \\ x^TA^TAx = 0 \\ \Downarrow \\ (Ax)^TAx =0 \\ \Downarrow \\ Ax =0 \\ \ A \ \ is \ \ independent \\ \Downarrow \\ x=0 \]

so \(N(A^TA)\) has only zero vector, \(A^TA\) is invertible.

Linear algebra method (projection method)

Sovling \(A^TA\hat{x} = A^Tb\) gives the projection \(p=A\hat{x}\) of b onto the column space of A.

\[Ax=b \ \ (unsolvable) \\ b = p+e \\ \Downarrow \\ A\hat{x} = p \ \ (solvable) \\ \Downarrow \\ \hat{x} = (A^TA)^{-1}A^Tb \]

example: \(A = \left[ \begin{matrix} 1&1 \\ 1&2 \\ 1&3 \end{matrix} \right], x= \left[ \begin{matrix} C \\ D \end{matrix} \right], b=\left[ \begin{matrix} 1 \\ 2 \\ 2 \end{matrix} \right], Ax=b\) is unsolvable.

\[A^TA\hat{x} = A^Tb \\ \Downarrow \\ A^TA = \left[ \begin{matrix} 1&1&1 \\ 1&2&3 \end{matrix} \right] \left[ \begin{matrix} 1&1 \\ 1&2 \\ 1&3 \end{matrix} \right] = \left[ \begin{matrix} 3&6 \\ 6&14 \end{matrix} \right] \\ A^Tb = \left[ \begin{matrix} 1&1&1 \\ 1&2&3 \end{matrix} \right] \left[ \begin{matrix} 1 \\ 2 \\ 2 \end{matrix} \right] = \left[ \begin{matrix} 5 \\ 11 \end{matrix} \right] \\ \Downarrow \\ \left[ \begin{matrix} 3&6 \\ 6&14 \end{matrix} \right] \left[ \begin{matrix} C \\ D\\ \end{matrix} \right] = \left[ \begin{matrix} 5 \\ 11 \end{matrix} \right] \\ \Downarrow \\ C = 2/3 \ , \ D=1/2 \\ \Downarrow \\ \hat{x} = \left[ \begin{matrix} 2/3 \\ 1/2\end{matrix} \right] \\ \]

Calculus method

The least squares solution \(\hat{x}\) makes the sum of errors \(E = ||Ax - b||^2\) as small as possible.

\[E = e_1^2 + e_2^2 +...+e_n^2 \]

example: \(A = \left[ \begin{matrix} 1&1 \\ 1&2 \\ 1&3 \end{matrix} \right], x= \left[ \begin{matrix} C \\ D \end{matrix} \right], b=\left[ \begin{matrix} 1 \\ 2 \\ 2 \end{matrix} \right], Ax=b\) is unsolvable.

\[E = ||Ax - b||^2 = e_1^2 + e_2^2 +e_3^2 = (C +D \cdot 1 -1)^2 + (C +D \cdot 2 -2)^2 + (C +D \cdot 3 -2)^2 \\ \Downarrow \\ \part E / \part C = 0 \\ \part E / \part D = 0 \\ \Downarrow \\ C = 2/3 \ , \ D=1/2 \]

The Big Picture for Least Squares

4.4 Orthonormal Bases and Gram-Schmidt

• The columns \(q_1,q_2,...,q_n\) are orthonormal if \(q_i^Tq_j = \left \{ \begin{array}{rcl} 0 \ \ for \ \ i \neq j \\ 1 \ \ for \ \ i=j \end{array}\right\}, Q^TQ=I\).
• If Q is also square, the \(QQ^T=I\) and \(Q^T=Q^{-1}\), Q is an orthogonal matrix.
• The least squares solution to \(Qx = b\) is \(\hat{x} = Q^Tb\). Projection of b : \(p=QQ^Tb = Pb\).

Projection Using Orthonormal Bases: Q Repalces A

Orthogonal matrices are excellent and stable for computations.

Suppose the basis vectors are actually orthonormal, then \(A^TA\) simplifies to \(Q^TQ\), here is \(\hat{x} = Q^Tb\) and \(p=Q\hat{x}=QQ^Tb\).

\[\\ \ p = \left[ \begin{matrix} |&&| \\ q_1&\cdots&q_n \\ |&&| \end{matrix} \right] \left[ \begin{matrix} q_1^Tb \\ \vdots \\ q_n^Tb \end{matrix} \right] = q_1(q_1^Tb) + ... + q_n(q_n^Tb) \]

The Gram-Schmidt Process

"Gram-Schmidt way" is to create orthonormal vectors.Take independent column vecotrs \(a_i\) to orthonormal \(q_i\).

From independent vectors \(a_1,...,a_n\)，Gram-Schmidt constructs orthonormal vectors \(q_1,...,q_n\).The matrices with these columns satisfy \(A=QR\). The \(R=Q^TA\) is upper triangular because later q's are orthoganal to a's.

Gram-Schmidt Process:

Subtract from every new vector its projections in the directions already set.

• First step : Chosing A=a and \(B = b-\frac{A^Tb}{A^TA}A\) -----> $A \bot B $
• Second step : \(C = c-\frac{A^Tc}{A^TA}A - \frac{B^Tc}{B^TB}B\) -----> $C \bot A \ \ C \bot B $
• Last step (orthonormal): \(q_1 = \frac{A}{||A||},q_2 = \frac{B}{||B||},q_3 = \frac{C}{||C||}\)

The \(q_1,q_2,q_3\) is the orthgonormal base. \(Q=\left[ \begin{matrix} && \\ q_1&q_2&q_3 \\ && \end{matrix} \right]\)

**The Factorization M = QR **：

Any m by n matrix M with independent columns can be factored into \(M=QR\).

\[M=QR=\left[ \begin{matrix} &&\\ q_1&q_2&q_3 \\ && \end{matrix} \right] \left[ \begin{matrix} q_1^Ta&q_1^Tb&q_1^Tc \\ &q_2^Tb&q_2^Tc \\ &&q_3^Tc \end{matrix} \right] \]

example:

Suppose the independent not-orthogonal vectors \(a,b,c\) are

\[a = \left[ \begin{matrix} 1 \\ -1 \\ 0\end{matrix} \right] \ and \ \ b = \left[ \begin{matrix} 2 \\ 0 \\ -2\end{matrix} \right] \ and \ \ c = \left[ \begin{matrix} 3 \\ -3 \\ 3\end{matrix} \right] \]

Then A=a has \(A^TA=2, A^Tb=2\)

First step : \(B=b-\frac{A^Tb}{A^TA}A = \left[ \begin{matrix} 1 \\ 1 \\ -2\end{matrix} \right]\)

Second step : \(C=c-\frac{A^Tc}{A^TA}A -\frac{B^Tc}{B^TB}B = \left[ \begin{matrix} 1 \\ 1 \\ 1\end{matrix} \right]\)

Last step : \(q_1 = \frac{A}{||A||}= \frac{1}{\sqrt{2}}\left[ \begin{matrix} 1 \\ -1 \\ 0\end{matrix} \right],q_2 = \frac{B}{||B||}=\frac{1}{\sqrt{6}}\left[ \begin{matrix} 1 \\ 1 \\ -2\end{matrix} \right],q_3 = \frac{C}{||C||}=\frac{1}{\sqrt{3}}\left[ \begin{matrix} 1 \\ 1 \\ 1\end{matrix} \right]\)

Factorization :

\[M =\left[ \begin{matrix} 1&2&3 \\ -1&0&-3 \\ 0&-2&3 \end{matrix} \right] = \left[ \begin{matrix} 1/\sqrt{2}&1/\sqrt{6}&1/\sqrt{3} \\ -1/\sqrt{2}&1/\sqrt{6}&1/\sqrt{3} \\ 0&-2/\sqrt{6}&1/\sqrt{3} \end{matrix} \right] \left[ \begin{matrix} \sqrt{2}&\sqrt{2}&\sqrt{18} \\ 0&\sqrt{6}&-\sqrt{6} \\ 0&0&\sqrt{3} \end{matrix} \right] = QR \]

Least squares

Instead of solving \(Ax=b\), which is impossible, we finally solve \(R\hat{x}=Q^Tb\) by back substitution which is very fast.

\[A^TA\hat{x}=A^Tb \\ A^TA =(QR)^TQR = R^TQ^TQR=R^TR \\ \Downarrow \\ R^TR\hat{x} =R^TQ^Tb \quad or \quad R\hat{x} =Q^Tb \quad or \quad \hat{x} =R^{-1}Q^Tb \\ \]

标签：begin,right,end,matrix,Orthogonality,hat,left
来源： https://www.cnblogs.com/xiqi2018/p/16101677.html

本站声明： 1. iCode9 技术分享网（下文简称本站）提供的所有内容，仅供技术学习、探讨和分享；
2. 关于本站的所有留言、评论、转载及引用，纯属内容发起人的个人观点，与本站观点和立场无关；
3. 关于本站的所有言论和文字，纯属内容发起人的个人观点，与本站观点和立场无关；
4. 本站文章均是网友提供，不完全保证技术分享内容的完整性、准确性、时效性、风险性和版权归属；如您发现该文章侵犯了您的权益，可联系我们第一时间进行删除；
5. 本站为非盈利性的个人网站，所有内容不会用来进行牟利，也不会利用任何形式的广告来间接获益，纯粹是为了广大技术爱好者提供技术内容和技术思想的分享性交流网站。