5.1 Matrix Notation and Operations

Matrix notation and operations form the backbone of linear regression analysis. They provide a compact way to represent and manipulate data, making complex calculations more manageable. Understanding these concepts is crucial for grasping the mathematical foundations of regression models.

In the context of simple linear regression, matrices allow us to express the relationship between variables efficiently. They enable us to solve systems of equations, estimate coefficients, and analyze model performance using powerful mathematical tools and techniques.

Matrix basics and notation

Matrix fundamentals

Top images from around the web for Matrix fundamentals

• 

  Matrix: Values arranged in rows and columns. View original

  Is this image relevant?

• 

  Introduction to Matrices | Boundless Algebra View original

  Is this image relevant?

• 

  Matrix (mathematics) - Wikipedia View original

  Is this image relevant?

• 

  Matrix: Values arranged in rows and columns. View original

  Is this image relevant?

• 

  Introduction to Matrices | Boundless Algebra View original

  Is this image relevant?

1 of 3

Top images from around the web for Matrix fundamentals

• 

  Matrix: Values arranged in rows and columns. View original

  Is this image relevant?

• 

  Introduction to Matrices | Boundless Algebra View original

  Is this image relevant?

• 

  Matrix (mathematics) - Wikipedia View original

  Is this image relevant?

• 

  Matrix: Values arranged in rows and columns. View original

  Is this image relevant?

• 

  Introduction to Matrices | Boundless Algebra View original

  Is this image relevant?

1 of 3

• A matrix is a rectangular array of numbers, symbols, or expressions arranged in rows and columns
  • Enclosed by square brackets or parentheses
  • Denoted as $A = [a_{ij}]$ or $A = (a_{ij})$, where $a_{ij}$ represents the element in the $i$-th row and $j$-th column
• The size of a matrix is defined by the number of rows and columns it contains
  • Denoted as $m \times n$, where $m$ is the number of rows and $n$ is the number of columns
  • Example: A $3 \times 4$ matrix has 3 rows and 4 columns
• Each element in a matrix is identified by its position, specified by the row and column indices
  • The element in the $i$-th row and $j$-th column is denoted as $a_{ij}$
  • Example: In a matrix $A$, the element $a_{23}$ is located in the 2nd row and 3rd column

Vectors

• A vector is a one-dimensional array of numbers, symbols, or expressions
  • Represented as either a row vector ($1 \times n$) or a column vector ($m \times 1$)
  • Denoted as $\vec{v} = (v_1, v_2, \ldots, v_n)$ for a row vector or $\vec{v} = [v_1, v_2, \ldots, v_m]^T$ for a column vector
• Vectors can be considered special cases of matrices
  • A row vector is a matrix with only one row
  • A column vector is a matrix with only one column

Applications of matrices and vectors

• Matrices and vectors can be used to represent various mathematical and real-world concepts
  • Systems of linear equations
  • Linear transformations and geometric transformations
  • Data in fields such as mathematics, physics, computer science, and economics
• Example: A system of linear equations can be represented using a coefficient matrix and a constant vector
  • $2x + 3y = 5$ and $4x - y = 3$ can be represented as $\begin{bmatrix} 2 & 3 \\ 4 & -1 \end{bmatrix} \begin{bmatrix} x \\ y \end{bmatrix} = \begin{bmatrix} 5 \\ 3 \end{bmatrix}$

Matrix operations

Addition and subtraction

• Matrix addition and subtraction can only be performed on matrices of the same size
  • The resulting matrix has the same size as the input matrices
• To add or subtract matrices, add or subtract the corresponding elements in each position
  • For matrices $A$ and $B$, $C = A + B$ implies $c_{ij} = a_{ij} + b_{ij}$ for all $i$ and $j$
  • Example: $\begin{bmatrix} 1 & 2 \\ 3 & 4 \end{bmatrix} + \begin{bmatrix} 5 & 6 \\ 7 & 8 \end{bmatrix} = \begin{bmatrix} 6 & 8 \\ 10 & 12 \end{bmatrix}$
• Scalar multiplication of a matrix involves multiplying each element of the matrix by a scalar value
  • For a scalar $k$ and matrix $A$, the resulting matrix $B = kA$ has elements $b_{ij} = ka_{ij}$ for all $i$ and $j$
  • Example: $2 \begin{bmatrix} 1 & 2 \\ 3 & 4 \end{bmatrix} = \begin{bmatrix} 2 & 4 \\ 6 & 8 \end{bmatrix}$

Matrix multiplication

• Matrix multiplication can be performed between two matrices $A$ ($m \times n$) and $B$ ($n \times p$)
  • The number of columns in the first matrix must equal the number of rows in the second matrix
  • The resulting matrix $C$ has dimensions ($m \times p$)
• To multiply matrices, multiply each element of a row in the first matrix by the corresponding element of a column in the second matrix and sum the products
  • The element $c_{ij}$ is given by the dot product of the $i$-th row of $A$ and the $j$-th column of $B$
  • $c_{ij} = \sum_{k=1}^n a_{ik}b_{kj}$
• Example: $\begin{bmatrix} 1 & 2 \\ 3 & 4 \end{bmatrix} \begin{bmatrix} 5 & 6 \\ 7 & 8 \end{bmatrix} = \begin{bmatrix} 19 & 22 \\ 43 & 50 \end{bmatrix}$

Solving linear systems

Augmented matrix and Gaussian elimination

• A system of linear equations can be represented using an augmented matrix
  • Consists of the coefficient matrix and the constant terms
  • Example: The system $2x + 3y = 5$ and $4x - y = 3$ can be represented as the augmented matrix $\left[\begin{array}{cc|c} 2 & 3 & 5 \\ 4 & -1 & 3 \end{array}\right]$
• Gaussian elimination is a method for solving systems of linear equations
  • Performs elementary row operations on the augmented matrix to obtain an upper triangular matrix
  • Elementary row operations include swapping rows, multiplying a row by a non-zero constant, and adding a multiple of one row to another
  • These operations do not change the solution set of the system
• Back-substitution is used to solve for the variables once the augmented matrix is in row echelon form
  • Row echelon form is an upper triangular matrix with ones on the diagonal and zeros below the diagonal

Cramer's rule

• Cramer's rule is another method for solving systems of linear equations using determinants
  • Applicable when the system has a unique solution and the coefficient matrix is square and invertible
• For a system of $n$ linear equations with $n$ unknowns, Cramer's rule states that the solution for the $i$-th variable $x_i$ is given by:
  • $x_i = \frac{\det(A_i)}{\det(A)}$, where $A$ is the coefficient matrix and $A_i$ is the matrix formed by replacing the $i$-th column of $A$ with the constant terms
• Example: For the system $2x + 3y = 5$ and $4x - y = 3$, the solution using Cramer's rule is:
  • $x = \frac{\det\begin{bmatrix} 5 & 3 \\ 3 & -1 \end{bmatrix}}{\det\begin{bmatrix} 2 & 3 \\ 4 & -1 \end{bmatrix}} = \frac{-8}{-11} = \frac{8}{11}$ and $y = \frac{\det\begin{bmatrix} 2 & 5 \\ 4 & 3 \end{bmatrix}}{\det\begin{bmatrix} 2 & 3 \\ 4 & -1 \end{bmatrix}} = \frac{-7}{-11} = \frac{7}{11}$

Properties of matrix operations

Commutativity and associativity

• Matrix addition and subtraction are commutative
  • $A + B = B + A$ and $A - B = -(B - A)$ for matrices $A$ and $B$
• Matrix multiplication is associative, but not commutative
  • $(AB)C = A(BC)$, but $AB \neq BA$ in general
• Example: Let $A = \begin{bmatrix} 1 & 2 \\ 3 & 4 \end{bmatrix}$ and $B = \begin{bmatrix} 5 & 6 \\ 7 & 8 \end{bmatrix}$
  • $AB = \begin{bmatrix} 19 & 22 \\ 43 & 50 \end{bmatrix}$, but $BA = \begin{bmatrix} 23 & 34 \\ 31 & 46 \end{bmatrix}$

Identity matrix and inverse matrix

• The identity matrix, denoted as $I$, is a square matrix with ones on the main diagonal and zeros elsewhere
  • It has the property $AI = IA = A$ for any matrix $A$
  • Example: $\begin{bmatrix} 1 & 0 \\ 0 & 1 \end{bmatrix}$ is a $2 \times 2$ identity matrix
• A square matrix $A$ is invertible if there exists a matrix $B$ such that $AB = BA = I$
  • The inverse of $A$ is unique and denoted as $A^{-1}$
  • Not all square matrices have inverses
• Example: The inverse of $\begin{bmatrix} 1 & 2 \\ 3 & 4 \end{bmatrix}$ is $\begin{bmatrix} -2 & 1 \\ \frac{3}{2} & -\frac{1}{2} \end{bmatrix}$

Transpose and determinant

• The transpose of a matrix $A$, denoted as $A^T$, is obtained by interchanging the rows and columns of $A$
  • For matrix multiplication, $(AB)^T = B^T A^T$
  • Example: If $A = \begin{bmatrix} 1 & 2 \\ 3 & 4 \end{bmatrix}$, then $A^T = \begin{bmatrix} 1 & 3 \\ 2 & 4 \end{bmatrix}$
• The determinant of a square matrix $A$, denoted as $\det(A)$ or $|A|$, is a scalar value that provides information about the matrix's properties
  • Invertibility: A matrix is invertible if and only if its determinant is non-zero
  • Linear independence: The columns or rows of a matrix are linearly independent if and only if the determinant is non-zero
• Example: For the matrix $A = \begin{bmatrix} 1 & 2 \\ 3 & 4 \end{bmatrix}$, $\det(A) = 1 \cdot 4 - 2 \cdot 3 = -2$