2.1 Ordinary Least Squares (OLS) Method

The Ordinary Least Squares (OLS) method is a cornerstone of linear regression analysis. It finds the best-fitting line by minimizing the sum of squared residuals between observed and predicted values, providing unique solutions for regression coefficients.

OLS estimates are calculated using normal equations, which can be solved algebraically or through matrix operations. These estimates represent the relationship between predictors and the dependent variable, with their signs and magnitudes indicating direction and strength of associations.

Least Squares Principle in Regression

Minimizing the Sum of Squared Residuals

Top images from around the web for Minimizing the Sum of Squared Residuals

• 

  The Regression Equation | Introduction to Statistics – Gravina View original

  Is this image relevant?

• 

  Linear Regression (2 of 4) | Concepts in Statistics View original

  Is this image relevant?

• 

  Introduction to Assessing the Fit of a Line | Concepts in Statistics View original

  Is this image relevant?

• 

  The Regression Equation | Introduction to Statistics – Gravina View original

  Is this image relevant?

• 

  Linear Regression (2 of 4) | Concepts in Statistics View original

  Is this image relevant?

1 of 3

Top images from around the web for Minimizing the Sum of Squared Residuals

• 

  The Regression Equation | Introduction to Statistics – Gravina View original

  Is this image relevant?

• 

  Linear Regression (2 of 4) | Concepts in Statistics View original

  Is this image relevant?

• 

  Introduction to Assessing the Fit of a Line | Concepts in Statistics View original

  Is this image relevant?

• 

  The Regression Equation | Introduction to Statistics – Gravina View original

  Is this image relevant?

• 

  Linear Regression (2 of 4) | Concepts in Statistics View original

  Is this image relevant?

1 of 3

• The principle of least squares estimates the parameters of a linear regression model by minimizing the sum of the squared residuals
  • Residuals represent the differences between the observed and predicted values of the dependent variable
• The least squares method finds the line of best fit that minimizes the vertical distances (residuals) between the observed data points and the predicted values on the regression line
• The least squares principle assumes that the errors (residuals) are normally distributed with a mean of zero and constant variance (homoscedasticity)

Unique Solution for Regression Coefficients

• The least squares method provides a unique solution for the regression coefficients that minimizes the sum of squared residuals
  • This unique solution makes the least squares approach widely used in linear regression analysis
• The least squares solution is optimal when the assumptions of the linear model are met (linearity, independence, normality, and homoscedasticity of errors)
• The least squares estimates are unbiased and have the lowest variance among all linear unbiased estimators (Gauss-Markov theorem)

Normal Equations for OLS Estimators

Deriving the Normal Equations

• The normal equations are a set of linear equations that can be solved to obtain the OLS estimates for the regression coefficients
• To derive the normal equations, express the sum of squared residuals as a function of the regression coefficients (β₀ and β₁ for a simple linear regression)
• Take the partial derivatives of the sum of squared residuals with respect to each regression coefficient and set them equal to zero
  • This finds the values that minimize the sum of squared residuals
• The resulting normal equations for a simple linear regression are:
  • $\sum(y_i) = n\beta_0 + \beta_1\sum(x_i)$
  • $\sum(x_i * y_i) = \beta_0\sum(x_i) + \beta_1\sum(x_i^2)$

Normal Equations in Matrix Form

• For multiple linear regression with p predictor variables, the normal equations can be expressed in matrix form
  • $(X^T * X)\beta = X^T * y$
  • X is the design matrix containing the values of the predictor variables
  • X^T is the transpose of the design matrix
  • y is the vector of observed values of the dependent variable
• The matrix form of the normal equations simplifies the calculation of OLS estimates in multiple linear regression
• Statistical software packages and programming languages often provide functions or libraries to solve the normal equations efficiently (e.g.,

  lm()

  in R,

  LinearRegression

  in Python's scikit-learn)

Calculating OLS Estimates

Simple Linear Regression

• To calculate the OLS estimates for the regression coefficients in a simple linear regression, solve the normal equations derived earlier
• The OLS estimates for the intercept (β₀) and slope (β₁) can be calculated using the following formulas:
  • $\beta_1 = \frac{\sum(x_i * y_i) - (\sum x_i * \sum y_i) / n}{\sum(x_i^2) - (\sum x_i)^2 / n}$
  • $\beta_0 = (\sum y_i / n) - \beta_1 * (\sum x_i / n)$
• These formulas can be easily implemented in a spreadsheet or programming language to obtain the OLS estimates

Multiple Linear Regression

• In multiple linear regression, the OLS estimates can be obtained by solving the normal equations in matrix form
  • $\beta = (X^T * X)^{-1} * X^T * y$
  • (X^T * X)^(-1) is the inverse of the matrix product X^T * X
• Statistical software packages and programming languages have built-in functions or libraries to calculate the OLS estimates
  • Examples include the

    lm()

    function in R and the

    LinearRegression

    class in Python's scikit-learn library
• These functions and libraries efficiently handle the matrix calculations and provide the OLS estimates along with other relevant statistics (standard errors, t-values, p-values)

Interpreting OLS Estimates

Regression Coefficients

• The OLS estimates of the regression coefficients represent the change in the dependent variable associated with a one-unit change in the corresponding predictor variable, holding all other predictors constant (ceteris paribus)
• The intercept (β₀) represents the expected value of the dependent variable when all predictor variables are equal to zero
  • In some cases, the intercept may not have a meaningful interpretation if zero is not a plausible value for the predictors
• The slope coefficients (β₁, β₂, ..., β_p) indicate the magnitude and direction of the relationship between each predictor variable and the dependent variable, assuming a linear relationship

Sign and Magnitude of Coefficients

• The sign of the slope coefficient indicates whether the relationship between the predictor and the dependent variable is positive (increasing) or negative (decreasing)
  • A positive coefficient suggests that as the predictor variable increases, the dependent variable tends to increase
  • A negative coefficient suggests that as the predictor variable increases, the dependent variable tends to decrease
• The magnitude of the slope coefficient represents the strength of the relationship, with larger absolute values indicating a stronger association between the predictor and the dependent variable
  • For example, a coefficient of 2.5 indicates a stronger positive relationship than a coefficient of 0.5

Considerations for Interpretation

• It is essential to consider the units of measurement for the variables when interpreting the OLS estimates, as the coefficients are scale-dependent
  • For instance, if the dependent variable is measured in thousands of dollars and a predictor variable is measured in years, the coefficient represents the change in thousands of dollars associated with a one-year change in the predictor
• The interpretation of the OLS estimates should be done in the context of the specific problem and the underlying assumptions of the linear model
  • Violations of the assumptions (linearity, independence, normality, and homoscedasticity of the errors) can affect the validity and reliability of the estimates
• Confidence intervals and hypothesis tests can be used to assess the statistical significance and precision of the OLS estimates
  • A 95% confidence interval provides a range of plausible values for the true population coefficient
  • Hypothesis tests (t-tests) can be used to determine if the coefficients are significantly different from zero