10.1 Linear regression models

Linear regression is a cornerstone of econometric analysis in mathematical economics. It allows researchers to model relationships between variables, quantify impacts, and make predictions. This powerful tool helps economists test hypotheses, forecast outcomes, and inform decision-making across various economic contexts.

Understanding linear regression equips economists with essential skills for analyzing trends and relationships in economic data. From simple models with one independent variable to complex multiple regression analyses, these techniques form the foundation for more advanced econometric methods used in modern economic research and policy analysis.

Fundamentals of linear regression

• Linear regression forms the foundation of econometric analysis in mathematical economics, allowing researchers to model relationships between variables
• This statistical method quantifies the impact of one or more independent variables on a dependent variable, crucial for economic forecasting and policy analysis
• Understanding linear regression equips economists with tools to test hypotheses, make predictions, and inform decision-making processes in various economic contexts

Definition and purpose

Top images from around the web for Definition and purpose

• 

  Lab 7. Statistical models. Linear regression [CS Open CourseWare] View original

  Is this image relevant?

• 

  Linear regression analysis: Predicting future performance and explaining observations View original

  Is this image relevant?

• 

  Simple Linear regression algorithm in machine learning with example graph - Codershood View original

  Is this image relevant?

• 

  Lab 7. Statistical models. Linear regression [CS Open CourseWare] View original

  Is this image relevant?

• 

  Linear regression analysis: Predicting future performance and explaining observations View original

  Is this image relevant?

1 of 3

Top images from around the web for Definition and purpose

• 

  Lab 7. Statistical models. Linear regression [CS Open CourseWare] View original

  Is this image relevant?

• 

  Linear regression analysis: Predicting future performance and explaining observations View original

  Is this image relevant?

• 

  Simple Linear regression algorithm in machine learning with example graph - Codershood View original

  Is this image relevant?

• 

  Lab 7. Statistical models. Linear regression [CS Open CourseWare] View original

  Is this image relevant?

• 

  Linear regression analysis: Predicting future performance and explaining observations View original

  Is this image relevant?

1 of 3

• Statistical method used to model the linear relationship between a dependent variable and one or more independent variables
• Aims to find the best-fitting straight line through a set of data points, minimizing the difference between observed and predicted values
• Serves as a powerful tool for economists to analyze trends, make predictions, and test economic theories
• Allows for quantification of the strength and direction of relationships between economic variables

Simple vs multiple regression

• Simple linear regression involves one independent variable and one dependent variable
• Multiple linear regression incorporates two or more independent variables to explain variations in the dependent variable
• Simple regression equation: $Y = β₀ + β₁X + ε$
• Multiple regression equation: $Y = β₀ + β₁X₁ + β₂X₂ + ... + βₖXₖ + ε$
• Multiple regression provides a more comprehensive analysis of complex economic relationships, accounting for multiple factors simultaneously

Assumptions of linear models

• Linearity assumes a straight-line relationship between dependent and independent variables
• Independence of errors requires that residuals are not correlated with each other
• Homoscedasticity assumes constant variance of residuals across all levels of independent variables
• Normality of residuals assumes that errors are normally distributed
• No perfect multicollinearity ensures independent variables are not perfectly correlated with each other

Ordinary least squares method

• Ordinary Least Squares (OLS) serves as the cornerstone estimation technique in linear regression analysis for economic models
• This method provides a framework for finding the best-fitting line by minimizing the sum of squared residuals
• OLS estimators possess desirable statistical properties, making them widely used in econometric research and policy analysis

Minimizing sum of squared residuals

• OLS method finds the line that minimizes the sum of squared differences between observed and predicted values
• Squared residuals are used to penalize both positive and negative deviations equally
• Minimization process involves calculus to find the values of β coefficients that yield the smallest sum of squared residuals
• Graphically represented by the line that passes through the "center" of the data points

OLS estimators and properties

• OLS estimators (β̂) are derived mathematically to minimize the sum of squared residuals
• Unbiasedness property ensures that the expected value of the estimator equals the true population parameter
• Consistency property guarantees that as sample size increases, the estimator converges to the true parameter value
• Efficiency property states that OLS estimators have the smallest variance among all linear unbiased estimators
• Normality of OLS estimators allows for inference and hypothesis testing in large samples

Gauss-Markov theorem

• States that under certain assumptions, OLS estimators are the Best Linear Unbiased Estimators (BLUE)
• BLUE property means OLS estimators have the smallest variance among all linear unbiased estimators
• Assumptions include linearity, random sampling, no perfect multicollinearity, zero conditional mean of errors, and homoscedasticity
• Theorem provides theoretical justification for the widespread use of OLS in econometric analysis
• Violations of Gauss-Markov assumptions may require alternative estimation methods (weighted least squares, generalized least squares)

Model specification

• Model specification plays a crucial role in ensuring the validity and reliability of economic analyses using linear regression
• Proper specification involves selecting appropriate variables, functional forms, and considering potential interactions
• Economists must carefully consider theoretical foundations and empirical evidence when specifying regression models

Dependent vs independent variables

• Dependent variable (Y) represents the outcome or effect being studied in the economic model
• Independent variables (X) are the factors believed to influence or explain variations in the dependent variable
• Selection of variables should be guided by economic theory, previous research, and the specific research question
• Endogeneity occurs when independent variables are correlated with the error term, potentially leading to biased estimates
• Exogeneity assumption requires that independent variables are not influenced by the dependent variable or unobserved factors

Functional form selection

• Linear form assumes a constant change in Y for a unit change in X ($Y = β₀ + β₁X$)
• Log-linear form models percentage changes ($log(Y) = β₀ + β₁X$)
• Log-log form captures elasticities ($log(Y) = β₀ + β₁log(X)$)
• Polynomial forms allow for non-linear relationships ($Y = β₀ + β₁X + β₂X² + β₃X³$)
• Choice of functional form should be based on theoretical considerations and empirical fit to the data

Dummy variables and interactions

• Dummy variables represent categorical information in regression models (0 or 1)
• Used to capture effects of qualitative factors (gender, region, policy changes)
• Interaction terms allow for the effect of one variable to depend on the level of another
• Multiplicative interactions: $Y = β₀ + β₁X₁ + β₂X₂ + β₃(X₁ × X₂)$
• Additive interactions with dummy variables: $Y = β₀ + β₁X + β₂D + β₃(X × D)$

Interpreting regression results

• Interpretation of regression results is essential for drawing meaningful conclusions from economic analyses
• Economists must understand how to translate statistical output into practical insights
• Proper interpretation involves assessing the magnitude, direction, and significance of estimated coefficients

Coefficient interpretation

• Slope coefficients (β) represent the change in Y for a one-unit change in X, holding other variables constant
• Intercept (β₀) indicates the expected value of Y when all independent variables are zero
• In log-linear models, coefficients represent percentage changes in Y for a unit change in X
• Elasticities in log-log models show the percentage change in Y for a 1% change in X
• Standardized coefficients allow for comparison of relative importance across variables with different scales

Statistical significance

• P-values indicate the probability of obtaining the observed results if the null hypothesis (no effect) is true
• Typically, results are considered statistically significant if p < 0.05 or p < 0.01
• Confidence intervals provide a range of plausible values for the true population parameter
• Type I error (rejecting a true null hypothesis) and Type II error (failing to reject a false null hypothesis) must be considered
• Statistical significance does not necessarily imply practical or economic significance

R-squared and adjusted R-squared

• R-squared measures the proportion of variance in Y explained by the independent variables
• Ranges from 0 to 1, with higher values indicating better fit (0.7 might indicate a good fit in some economic applications)
• Adjusted R-squared penalizes the addition of irrelevant variables, addressing overfitting concerns
• Calculated as: $Adjusted R² = 1 - [(1 - R²)(n - 1) / (n - k - 1)]$, where n is sample size and k is number of predictors
• Useful for comparing models with different numbers of independent variables

Hypothesis testing in regression

• Hypothesis testing in regression allows economists to make inferences about population parameters based on sample data
• This process is crucial for validating economic theories, assessing policy impacts, and making informed decisions
• Various tests are employed to evaluate the significance of individual coefficients and overall model fit

T-tests for individual coefficients

• Used to test the null hypothesis that a single coefficient is equal to zero (H₀: βᵢ = 0)
• T-statistic calculated as: $t = (β̂ᵢ - 0) / SE(β̂ᵢ)$, where β̂ᵢ is the estimated coefficient and SE is the standard error
• Compares the t-statistic to critical values from the t-distribution to determine significance
• Two-tailed tests assess whether the coefficient is significantly different from zero in either direction
• One-tailed tests evaluate whether the coefficient is significantly greater than or less than zero

F-test for overall significance

• Tests the null hypothesis that all slope coefficients are simultaneously equal to zero (H₀: β₁ = β₂ = ... = βₖ = 0)
• F-statistic calculated as: $F = (R² / k) / [(1 - R²) / (n - k - 1)]$, where k is the number of predictors and n is sample size
• Compares the F-statistic to critical values from the F-distribution to determine overall model significance
• Significant F-test indicates that at least one independent variable has a non-zero effect on the dependent variable
• Useful for assessing whether the model as a whole has explanatory power

Testing linear restrictions

• Allows for testing hypotheses about relationships between coefficients
• Examples include testing whether two coefficients are equal or if their sum equals a specific value
• Wald test used for testing linear restrictions in large samples
• F-test employed for testing multiple linear restrictions simultaneously
• Likelihood ratio test compares restricted and unrestricted models based on their log-likelihoods

Model diagnostics

• Model diagnostics are essential for assessing the validity and reliability of regression models in economic analysis
• These techniques help identify violations of assumptions and potential issues that may affect the accuracy of results
• Proper diagnostics ensure that the model provides a good fit to the data and yields unbiased, efficient estimates

Residual analysis

• Examines the differences between observed and predicted values (residuals) to assess model fit
• Residual plots help identify patterns or trends that may indicate violations of assumptions
• Normality of residuals can be assessed using Q-Q plots or formal tests (Shapiro-Wilk, Kolmogorov-Smirnov)
• Homoscedasticity checked through plots of residuals against fitted values or independent variables
• Outliers and influential observations identified using leverage, Cook's distance, or DFBETAS statistics

Multicollinearity detection

• Occurs when independent variables are highly correlated, leading to unstable and unreliable coefficient estimates
• Variance Inflation Factor (VIF) measures the increase in variance of an estimated coefficient due to multicollinearity
• VIF values greater than 5 or 10 often indicate problematic multicollinearity
• Correlation matrix of independent variables helps identify pairwise correlations
• Condition number of the design matrix provides an overall measure of multicollinearity in the model

Heteroscedasticity and autocorrelation

• Heteroscedasticity violates the assumption of constant variance of residuals across all levels of independent variables
• Detected using formal tests (Breusch-Pagan, White's test) or graphical methods (residual plots)
• Autocorrelation refers to correlation between residuals, often present in time series data
• Durbin-Watson test used to detect first-order autocorrelation in time series regression
• Remedies include using robust standard errors, weighted least squares, or generalized least squares estimation

Prediction and forecasting

• Prediction and forecasting are crucial applications of linear regression in economic analysis and policy-making
• These techniques allow economists to estimate future values of economic variables based on historical relationships
• Understanding the limitations and uncertainties associated with predictions is essential for informed decision-making

In-sample vs out-of-sample prediction

• In-sample prediction uses the same data used to estimate the model to generate predicted values
• Out-of-sample prediction applies the estimated model to new data not used in the original estimation
• In-sample predictions tend to be overly optimistic about model performance
• Out-of-sample predictions provide a more realistic assessment of the model's predictive power
• Cross-validation techniques (k-fold, leave-one-out) used to evaluate out-of-sample performance

Confidence intervals for predictions

• Provide a range of plausible values for the predicted outcome with a specified level of confidence
• Narrower intervals indicate more precise predictions
• Calculated using the standard error of the prediction: $SE(Ŷ) = s × √(1 + h)$, where s is the residual standard error and h is the leverage
• Prediction intervals account for both model uncertainty and individual observation variability
• Wider than confidence intervals as they include additional uncertainty from individual observations

Limitations of forecasting

• Assumes that historical relationships between variables will continue to hold in the future
• Structural breaks or regime changes can invalidate forecasts based on past data
• Extrapolation beyond the range of observed data may lead to unreliable predictions
• Model misspecification or omitted variables can result in biased forecasts
• Economic shocks or unforeseen events may cause significant deviations from predicted values

Applications in economics

• Linear regression finds widespread application across various fields of economics, providing valuable insights for policy-makers and researchers
• These applications demonstrate the versatility of regression analysis in addressing complex economic questions
• Understanding these applications helps contextualize the importance of regression techniques in economic research and decision-making

Demand and supply analysis

• Estimates price elasticity of demand by regressing quantity demanded on price and other relevant factors
• Supply elasticity calculated by regressing quantity supplied on price and input costs
• Allows for identification of determinants of demand and supply beyond price (income, substitutes, complements)
• Enables forecasting of market equilibrium prices and quantities under different scenarios
• Facilitates analysis of policy impacts (taxes, subsidies) on market outcomes

Production function estimation

• Estimates the relationship between inputs (labor, capital) and output in production processes
• Cobb-Douglas production function: $Y = AL^αK^β$, where Y is output, L is labor, and K is capital
• Logarithmic transformation allows for estimation using linear regression: $log(Y) = log(A) + αlog(L) + βlog(K)$
• Coefficients represent output elasticities with respect to each input
• Enables analysis of returns to scale, technical efficiency, and factor productivity

Policy evaluation models

• Difference-in-differences models assess the impact of policy interventions by comparing treatment and control groups
• Regression discontinuity designs exploit threshold rules to estimate causal effects of policies
• Instrumental variables regression addresses endogeneity issues in policy evaluation
• Panel data models control for unobserved heterogeneity in longitudinal policy studies
• Allows for quantification of policy impacts on various economic outcomes (employment, growth, inequality)

Advanced regression techniques

• Advanced regression techniques extend the basic linear model to address specific challenges in economic data analysis
• These methods provide more flexible and robust approaches to modeling complex economic relationships
• Understanding these techniques equips economists with a broader toolkit for tackling diverse research questions

Weighted least squares

• Addresses heteroscedasticity by assigning different weights to observations based on their variance
• Minimizes the weighted sum of squared residuals: $min Σ wᵢ(Yᵢ - β₀ - β₁Xᵢ)²$, where wᵢ are weights
• Weights are typically inversely proportional to the variance of each observation
• Produces more efficient estimates when heteroscedasticity is present
• Commonly used in cross-sectional studies with varying sample sizes across units

Instrumental variables regression

• Addresses endogeneity issues arising from correlation between independent variables and the error term
• Uses instrumental variables (Z) that are correlated with the endogenous regressor (X) but uncorrelated with the error term
• Two-stage least squares (2SLS) estimation process:
  1. Regress X on Z to obtain predicted values X̂
  2. Use X̂ in place of X in the main regression equation
• Allows for consistent estimation of causal effects in the presence of endogeneity
• Widely used in economics for policy evaluation and causal inference

Panel data models

• Utilize data with both cross-sectional and time series dimensions
• Fixed effects models control for time-invariant unobserved heterogeneity across units
• Random effects models assume unit-specific effects are uncorrelated with regressors
• Hausman test used to choose between fixed and random effects specifications
• Dynamic panel models incorporate lagged dependent variables as regressors
• Enables analysis of both between-unit and within-unit variation in economic variables