7.6 Dealing with multicollinearity and heteroscedasticity

Multicollinearity and heteroscedasticity can mess up your regression analysis. These issues make it hard to figure out which variables are really important and can lead to unreliable results. But don't worry, there are ways to spot and fix these problems.

We'll look at how to detect multicollinearity using things like VIF and correlation matrices. For heteroscedasticity, we'll check out residual plots and statistical tests. Then we'll explore solutions like variable transformations and robust standard errors to get your regression back on track.

Multicollinearity in Regression

Defining and Detecting Multicollinearity

Top images from around the web for Defining and Detecting Multicollinearity

• 

  Determining Multicollinearity for the given ScatterPlot Matrix - Cross Validated View original

  Is this image relevant?

• 

  Detecting Multicollinearity in Regression Analysis View original

  Is this image relevant?

• 

  Determining Multicollinearity for the given ScatterPlot Matrix - Cross Validated View original

  Is this image relevant?

• 

  Detecting Multicollinearity in Regression Analysis View original

  Is this image relevant?

1 of 2

Top images from around the web for Defining and Detecting Multicollinearity

• 

  Determining Multicollinearity for the given ScatterPlot Matrix - Cross Validated View original

  Is this image relevant?

• 

  Detecting Multicollinearity in Regression Analysis View original

  Is this image relevant?

• 

  Determining Multicollinearity for the given ScatterPlot Matrix - Cross Validated View original

  Is this image relevant?

• 

  Detecting Multicollinearity in Regression Analysis View original

  Is this image relevant?

1 of 2

• Multicollinearity occurs when two or more independent variables in a regression model are highly correlated with each other
• Variance Inflation Factor (VIF) detects multicollinearity
  • VIF values greater than 5 or 10 indicate problematic levels
  • Calculate VIF for each predictor variable
• Correlation matrices identify pairwise correlations between independent variables
  • Correlations above 0.8 or 0.9 suggest potential multicollinearity
  • Examine the strength and direction of relationships between predictors
• Eigenvalue analysis of the correlation matrix reveals multicollinearity
  • Small eigenvalues (close to zero) indicate its presence
  • Analyze the spread of eigenvalues to assess multicollinearity severity

Advanced Multicollinearity Metrics

• Condition number detects multicollinearity
  • Calculate as the square root of the ratio of largest to smallest eigenvalue
  • Values exceeding 15 or 30 indicate potential issues
• Tolerance measures multicollinearity
  • Calculate as 1/VIF for each predictor
  • Values below 0.1 or 0.2 indicate potential problems
• Combine multiple detection methods for comprehensive assessment
  • Use VIF, correlation matrices, and eigenvalue analysis together
  • Cross-validate findings using different approaches

Consequences of Multicollinearity

Impact on Regression Coefficients

• Inflates standard errors of regression coefficients
  • Makes coefficients less reliable and potentially insignificant
  • Widens confidence intervals for coefficient estimates
• Regression coefficients become unstable and sensitive to small changes
  • Minor alterations in model specification or data can cause large shifts
  • Coefficients may fluctuate dramatically between similar models
• Can lead to incorrect signs and magnitudes of regression coefficients
  • May contradict theoretical expectations or prior research
  • Example: positive coefficient for price in demand model when negative expected

Model Interpretation Challenges

• Overall model fit (R-squared) may be high, but individual predictors may not be statistically significant
  • Model explains variance well, but individual contributions unclear
  • Example: high R-squared in economic growth model, but insignificant coefficients for education and investment
• Difficult to determine individual importance of predictor variables
  • Effects are confounded due to high correlations
  • Cannot isolate unique contributions of each predictor
• Predictions from the model may still be accurate, but interpretation of individual coefficients becomes problematic
  • Model useful for forecasting, but not for understanding variable relationships
  • Example: accurate sales predictions, but unclear impact of advertising vs pricing

Detecting Heteroscedasticity

Visual Detection Methods

• Heteroscedasticity occurs when variance of residuals is not constant across all levels of independent variables
• Residual plots reveal patterns indicating heteroscedasticity
  • Plot residuals vs. fitted values
  • Plot residuals vs. predictor variables
  • Look for funnel or fan shapes indicating increasing variance
• Scatter plots of squared residuals vs. predictor variables can highlight heteroscedasticity
  • Upward trend suggests increasing variance
  • Example: plot of squared residuals vs. income in wage model shows widening spread

Statistical Tests for Heteroscedasticity

• Breusch-Pagan test detects if variance of errors depends on values of independent variables
  • Regress squared residuals on predictor variables
  • Test if coefficients are jointly significant
• White test uses more general form of Breusch-Pagan test
  • Includes squared terms and cross-products of predictors
  • Allows for non-linear forms of heteroscedasticity
• Goldfeld-Quandt test compares variances of two subsets of data
  • Order data by suspected heteroscedasticity source
  • Compare variances of first and last thirds of data
• Glejser test regresses absolute residuals on independent variables
  • Significant relationships indicate heteroscedasticity
  • Can reveal which variables are associated with changing variance

Addressing Heteroscedasticity

Transformation and Weighting Techniques

• Weighted Least Squares (WLS) assigns weights inversely proportional to variance of each observation
  • Gives less weight to observations with higher variance
  • Example: in time series, recent observations may get higher weights
• Variable transformations can stabilize variance and reduce heteroscedasticity
  • Logarithmic transformations (log income in wage models)
  • Power transformations (square root of population in city growth models)
• Feasible Generalized Least Squares (FGLS) estimates variance function and uses it to transform data
  • Two-step process: estimate OLS, then use residuals to model variance
  • Apply estimated weights to original data and re-estimate

Robust Inference Methods

• Robust standard errors obtain valid inference in presence of heteroscedasticity
  • White's heteroscedasticity-consistent standard errors widely used
  • Huber-White sandwich estimators provide consistent variance estimates
• Bootstrapping techniques obtain valid standard errors and confidence intervals
  • Resample data with replacement to estimate sampling distribution
  • Particularly useful for small samples or complex error structures
• Adding relevant omitted variables or interaction terms to model can help reduce heteroscedasticity
  • Include variables that explain changing variance
  • Example: add firm size variable in financial performance model