🥖Linear Modeling Theory Unit 1 – Linear Models: Intro & Simple Regression

Key Concepts and Definitions

• Linear models represent relationships between variables using linear equations
• Dependent variable (response) is the variable being predicted or explained by the model
• Independent variables (predictors) are the variables used to predict the dependent variable
• Regression coefficients quantify the effect of each independent variable on the dependent variable
• Residuals represent the difference between the observed and predicted values of the dependent variable
• R-squared measures the proportion of variance in the dependent variable explained by the model
• Adjusted R-squared accounts for the number of predictors in the model and penalizes complexity
• Hypothesis testing assesses the significance of individual predictors and the overall model

Linear Models: The Basics

• Linear models assume a linear relationship between the dependent and independent variables
• The general form of a linear model is $y = \beta_0 + \beta_1x_1 + \beta_2x_2 + ... + \beta_kx_k + \epsilon$
  • $y$ represents the dependent variable
  • $\beta_0$ is the intercept (value of $y$ when all predictors are zero)
  • $\beta_1, \beta_2, ..., \beta_k$ are the regression coefficients for each predictor
  • $x_1, x_2, ..., x_k$ are the independent variables (predictors)
  • $\epsilon$ represents the error term (unexplained variation)
• Linear models can include multiple predictors (multiple linear regression)
• Interactions between predictors can be included to capture more complex relationships
• Polynomial terms can be added to model non-linear relationships while still using a linear model framework

Simple Linear Regression Explained

• Simple linear regression involves one dependent variable and one independent variable
• The model equation is $y = \beta_0 + \beta_1x + \epsilon$
  • $\beta_0$ is the intercept (value of $y$ when $x$ is zero)
  • $\beta_1$ is the slope (change in $y$ for a one-unit increase in $x$)
• The goal is to find the line of best fit that minimizes the sum of squared residuals
• Residuals are the differences between the observed and predicted values of the dependent variable
• The line of best fit is determined by estimating the regression coefficients ($\beta_0$ and $\beta_1$)
• The coefficient of determination (R-squared) measures the proportion of variance in the dependent variable explained by the independent variable
• Hypothesis tests can be used to assess the significance of the slope coefficient and the overall model

Model Assumptions and Diagnostics

• Linear models rely on several assumptions for valid inference and prediction
• Linearity assumes a linear relationship between the dependent and independent variables
  • Residual plots can be used to check for non-linearity
• Independence assumes that the errors are independent of each other
  • Durbin-Watson test can be used to detect autocorrelation in the residuals
• Homoscedasticity assumes constant variance of the errors across all levels of the predictors
  • Residual plots can be used to check for heteroscedasticity (non-constant variance)
• Normality assumes that the errors follow a normal distribution
  • Q-Q plots or histograms of the residuals can be used to assess normality
• Outliers and influential points can have a significant impact on the model results
  • Leverage, Cook's distance, and DFFITS can be used to identify influential observations
• Multicollinearity occurs when predictors are highly correlated with each other
  • Variance Inflation Factor (VIF) can be used to detect multicollinearity

Estimation Methods and Least Squares

• Least squares estimation is the most common method for estimating regression coefficients
• The goal is to minimize the sum of squared residuals (SSR) to find the line of best fit
• The normal equations are a set of equations used to solve for the least squares estimates
• The least squares estimates are unbiased and have the smallest variance among all linear unbiased estimators (BLUE)
• Maximum likelihood estimation (MLE) is an alternative method that maximizes the likelihood function
  • MLE estimates are asymptotically efficient and consistent under certain regularity conditions
• Gradient descent is an iterative optimization algorithm used to minimize the SSR
  • It updates the coefficient estimates in the direction of steepest descent until convergence
• Regularization techniques (ridge regression, lasso) can be used to shrink the coefficient estimates and handle multicollinearity

Interpreting Regression Results

• The intercept ($\beta_0$) represents the expected value of the dependent variable when all predictors are zero
• The slope coefficients ($\beta_1, \beta_2, ..., \beta_k$) represent the change in the dependent variable for a one-unit increase in the corresponding predictor, holding other predictors constant
• The standard errors of the coefficients provide a measure of the uncertainty in the estimates
• The t-statistics and p-values are used to assess the significance of individual predictors
  • A low p-value (typically < 0.05) indicates strong evidence against the null hypothesis of no effect
• Confidence intervals provide a range of plausible values for the population parameters
• The F-statistic and its p-value are used to assess the overall significance of the model
• R-squared and adjusted R-squared provide measures of the model's explanatory power
  • R-squared ranges from 0 to 1, with higher values indicating a better fit
  • Adjusted R-squared penalizes the inclusion of unnecessary predictors

Applications and Examples

• Linear regression can be used to predict sales based on advertising expenditure
  • The dependent variable is sales, and the independent variable is advertising expenditure
• Multiple linear regression can be used to model house prices based on various features
  • Predictors can include square footage, number of bedrooms, location, etc.
• Linear models can be used to analyze the relationship between a drug dosage and its effect on patients
  • The dependent variable is the patient's response, and the independent variable is the drug dosage
• Time series regression can be used to forecast future values based on past observations
  • Predictors can include lagged values, trend components, and seasonal indicators
• Linear regression can be used to study the impact of socioeconomic factors on educational attainment
  • Predictors can include parental education, income, and neighborhood characteristics

Common Pitfalls and Limitations

• Overfitting occurs when a model is too complex and fits the noise in the data rather than the underlying pattern
  • Regularization techniques and cross-validation can help mitigate overfitting
• Underfitting occurs when a model is too simple and fails to capture the true relationship between variables
  • Adding more relevant predictors or considering non-linear relationships can improve the model fit
• Extrapolation beyond the range of the observed data can lead to unreliable predictions
  • Models should be used with caution when making predictions outside the range of the training data
• Correlation does not imply causation
  • Observational studies cannot establish causal relationships without additional assumptions and design considerations
• Outliers and influential points can have a disproportionate impact on the model results
  • Robust regression techniques (e.g., least absolute deviations) can be used to mitigate the impact of outliers
• Measurement errors in the variables can lead to biased and inconsistent estimates
  • Instrumental variables and error-in-variables models can be used to address measurement error