10.4 Projection theorem and best approximations

Hilbert spaces allow us to break down complex vectors into simpler parts. The projection theorem shows how any vector can be split into two pieces: one in a subspace and one outside it. This splitting helps us understand and work with complicated objects.

Best approximation is about finding the closest point in a subspace to a given vector. It's like trying to hit a target as closely as possible. This idea is super useful in many areas, from data analysis to signal processing.

Projection Theorem and Orthogonal Projection

Projection Theorem and Orthogonal Projection in Hilbert Spaces

Top images from around the web for Projection Theorem and Orthogonal Projection in Hilbert Spaces

• 

  Math for Machines View original

  Is this image relevant?

• 

  Orthogonal Projections and Their Applications – Advanced Quantitative Economics with Python View original

  Is this image relevant?

• 

  Orthogonal Projections and Their Applications – Advanced Quantitative Economics with Python View original

  Is this image relevant?

• 

  Math for Machines View original

  Is this image relevant?

• 

  Orthogonal Projections and Their Applications – Advanced Quantitative Economics with Python View original

  Is this image relevant?

1 of 3

Top images from around the web for Projection Theorem and Orthogonal Projection in Hilbert Spaces

• 

  Math for Machines View original

  Is this image relevant?

• 

  Orthogonal Projections and Their Applications – Advanced Quantitative Economics with Python View original

  Is this image relevant?

• 

  Orthogonal Projections and Their Applications – Advanced Quantitative Economics with Python View original

  Is this image relevant?

• 

  Math for Machines View original

  Is this image relevant?

• 

  Orthogonal Projections and Their Applications – Advanced Quantitative Economics with Python View original

  Is this image relevant?

1 of 3

• States that given a closed subspace $M$ of a Hilbert space $H$, every vector $x \in H$ can be uniquely decomposed as $x = x_M + x_{M^\perp}$
  • $x_M$ is the orthogonal projection of $x$ onto $M$
  • $x_{M^\perp}$ is the orthogonal projection of $x$ onto $M^\perp$, the orthogonal complement of $M$
• Orthogonal projection $P_M: H \to M$ maps each vector $x \in H$ to its closest point in $M$
  • Characterized by the property that $\langle x - P_M(x), y \rangle = 0$ for all $y \in M$
  • Can be computed using the formula $P_M(x) = \sum_{i=1}^n \langle x, e_i \rangle e_i$, where $\{e_1, \ldots, e_n\}$ is an orthonormal basis for $M$

Properties of Closed Subspaces and Orthogonal Complements

• A subspace $M$ of a Hilbert space $H$ is closed if it contains all its limit points
  • Equivalently, $M$ is closed if and only if it is complete with respect to the induced norm from $H$
• The orthogonal complement of a subspace $M$ is the set $M^\perp = \{x \in H: \langle x, y \rangle = 0 \text{ for all } y \in M\}$
  • $M^\perp$ is always a closed subspace, even if $M$ is not closed
  • If $M$ is closed, then $H = M \oplus M^\perp$ (direct sum decomposition)
• Examples of closed subspaces include:
  • The subspace of continuous functions in $L^2([0,1])$
  • The subspace of polynomials of degree at most $n$ in $L^2([0,1])$

Best Approximation and Least Squares

Best Approximation in Hilbert Spaces

• Given a closed subspace $M$ of a Hilbert space $H$ and a vector $x \in H$, the best approximation to $x$ in $M$ is the unique vector $x_M \in M$ that minimizes the distance $\|x - y\|$ over all $y \in M$
  • The best approximation $x_M$ is precisely the orthogonal projection of $x$ onto $M$
  • Characterized by the orthogonality condition $\langle x - x_M, y \rangle = 0$ for all $y \in M$
• Existence and uniqueness of the best approximation follow from the projection theorem
  • The best approximation always exists and is unique for closed subspaces

Least Squares Approximation and Minimum Norm Solutions

• Least squares approximation is a special case of best approximation, often used in data fitting and regression analysis
  • Given a set of data points $(x_i, y_i)$ and a family of functions $\mathcal{F}$, the least squares approximation is the function $f \in \mathcal{F}$ that minimizes the sum of squared residuals $\sum_{i=1}^n (y_i - f(x_i))^2$
  • Can be formulated as a best approximation problem in a suitable Hilbert space (e.g., $L^2$ space of functions)
• Minimum norm solution refers to the problem of finding the vector $x$ with the smallest norm that satisfies a given set of linear constraints
  • Arises in underdetermined linear systems, where there are infinitely many solutions
  • The minimum norm solution is unique and can be characterized using the orthogonal projection onto the subspace of solutions
• Examples of least squares approximation:
  • Fitting a straight line to a set of data points in the plane
  • Approximating a function by a polynomial of a given degree