8.4 Likelihood ratio tests

Likelihood ratio tests are a powerful tool in theoretical statistics for comparing competing models. By evaluating the ratio of likelihoods under different hypotheses, these tests provide a framework for making inferences about population parameters and assessing model fit.

These tests combine key statistical concepts like hypothesis formulation, test statistics, and critical regions. Understanding their properties, types, and applications enables statisticians to draw meaningful conclusions from data and select appropriate models for analysis.

Definition of likelihood ratio

• Likelihood ratio tests form a cornerstone of hypothesis testing in theoretical statistics
• These tests compare the fit of two competing statistical models to observed data
• Likelihood ratios provide a powerful framework for making statistical inferences about population parameters

Likelihood function basics

Top images from around the web for Likelihood function basics

• 

  Maximum Likelihood Estimate and Logistic Regression simplified — Pavan Mirla View original

  Is this image relevant?

• 

  Likelihood function - Wikipedia View original

  Is this image relevant?

• 

  Maximum Likelihood Estimate and Logistic Regression simplified — Pavan Mirla View original

  Is this image relevant?

• 

  Maximum Likelihood Estimate and Logistic Regression simplified — Pavan Mirla View original

  Is this image relevant?

• 

  Likelihood function - Wikipedia View original

  Is this image relevant?

1 of 3

Top images from around the web for Likelihood function basics

• 

  Maximum Likelihood Estimate and Logistic Regression simplified — Pavan Mirla View original

  Is this image relevant?

• 

  Likelihood function - Wikipedia View original

  Is this image relevant?

• 

  Maximum Likelihood Estimate and Logistic Regression simplified — Pavan Mirla View original

  Is this image relevant?

• 

  Maximum Likelihood Estimate and Logistic Regression simplified — Pavan Mirla View original

  Is this image relevant?

• 

  Likelihood function - Wikipedia View original

  Is this image relevant?

1 of 3

• Likelihood function measures the probability of observing data given specific parameter values
• Expressed mathematically as $L(\theta|x) = P(X=x|\theta)$ for discrete random variables
• For continuous random variables, uses probability density function $L(\theta|x) = f(x|\theta)$
• Maximizes over the parameter space to find most likely parameter values

Ratio of likelihoods

• Compares likelihood of data under null hypothesis to alternative hypothesis
• Calculated as $\lambda = \frac{L(\theta_0)}{L(\theta_1)}$ where $\theta_0$ and $\theta_1$ are parameter values under null and alternative hypotheses
• Values close to 1 indicate little evidence against null hypothesis
• Small values suggest data more consistent with alternative hypothesis

Neyman-Pearson lemma

• Provides theoretical foundation for likelihood ratio tests
• States likelihood ratio test is most powerful test for simple hypotheses
• Guarantees optimal trade-off between Type I and Type II errors
• Applies to tests with fixed significance level

Components of likelihood ratio test

• Likelihood ratio tests combine several key statistical concepts
• These tests require careful formulation of hypotheses and test statistics
• Understanding critical regions enables proper interpretation of test results

Null vs alternative hypotheses

• Null hypothesis (H0) represents status quo or no effect
• Alternative hypothesis (H1) posits a specific deviation from null
• Simple hypotheses specify exact parameter values
• Composite hypotheses involve ranges or inequalities of parameters
• Directionality (one-tailed vs two-tailed) impacts test formulation

Test statistic formulation

• Likelihood ratio test statistic often denoted as $\Lambda$ or LR
• Calculated as $-2\log(\lambda)$ where $\lambda$ is the likelihood ratio
• Transformation improves statistical properties of test statistic
• Under certain conditions, follows chi-square distribution asymptotically
• Degrees of freedom depend on difference in parameter dimensionality between models

Critical region determination

• Critical region defines values of test statistic leading to rejection of null hypothesis
• Determined by chosen significance level (α) of the test
• For chi-square distributed test statistics, uses quantiles of chi-square distribution
• Right-tailed critical region common for likelihood ratio tests
• Rejection occurs when observed test statistic exceeds critical value

Properties of likelihood ratio tests

• Likelihood ratio tests possess several desirable statistical properties
• These properties contribute to their widespread use in statistical inference
• Understanding test properties aids in proper application and interpretation

Asymptotic distribution

• Test statistic approaches chi-square distribution as sample size increases
• Degrees of freedom equal difference in number of free parameters between models
• Wilks' theorem formalizes this asymptotic behavior
• Approximation improves with larger sample sizes
• Allows for straightforward computation of p-values in large samples

Power of the test

• Measures ability to correctly reject false null hypothesis
• Increases with sample size and effect size
• Neyman-Pearson lemma ensures optimal power for simple hypotheses
• Power function describes probability of rejection for different parameter values
• Trade-off exists between power and Type I error rate

Consistency and efficiency

• Likelihood ratio tests are consistent estimators
• Probability of correct decision approaches 1 as sample size increases
• Asymptotically efficient under regularity conditions
• Achieves Cramér-Rao lower bound for variance of estimators
• Efficiency property ensures optimal use of information in data

Types of likelihood ratio tests

• Likelihood ratio tests can be applied to various hypothesis testing scenarios
• Different types of tests address specific statistical questions
• Understanding test types helps in selecting appropriate analysis methods

Simple vs composite hypotheses

• Simple hypotheses specify exact parameter values (H0: θ = θ0 vs H1: θ = θ1)
• Composite hypotheses involve ranges or inequalities (H0: θ ≤ θ0 vs H1: θ > θ0)
• Simple-vs-simple tests often have exact distributions
• Composite hypotheses may require nuisance parameter estimation
• Test construction differs between simple and composite cases

Nested vs non-nested models

• Nested models form subset relationships (linear regression vs quadratic regression)
• Non-nested models cannot be obtained by parameter restrictions (exponential vs power law)
• Likelihood ratio tests directly applicable to nested models
• Non-nested models require modified approaches (Cox test, Vuong test)
• Model selection criteria (AIC, BIC) often used for non-nested comparisons

Parametric vs nonparametric tests

• Parametric tests assume specific probability distributions (normal, Poisson)
• Nonparametric tests make fewer assumptions about underlying distributions
• Parametric likelihood ratio tests often have higher power when assumptions met
• Nonparametric alternatives include empirical likelihood ratio tests
• Rank-based methods provide nonparametric analogues to likelihood ratio tests

Applications in statistical inference

• Likelihood ratio tests find widespread use in various statistical applications
• These tests provide powerful tools for drawing inferences from data
• Understanding applications enhances ability to choose appropriate analysis methods

Parameter estimation

• Used to construct confidence intervals for parameters
• Profile likelihood methods create intervals with desired coverage properties
• Allows testing of specific parameter values or ranges
• Useful for assessing uncertainty in estimated parameters
• Can be extended to multivariate parameter spaces

Model selection

• Compares fit of nested statistical models
• Helps determine appropriate model complexity
• Penalized likelihood approaches (AIC, BIC) extend to non-nested cases
• Stepwise regression uses series of likelihood ratio tests
• Balances model fit against parsimony

Goodness of fit assessment

• Evaluates how well a model fits observed data
• Compares fitted model to saturated model
• Pearson chi-square and deviance tests based on likelihood ratios
• Residual analysis often accompanies likelihood ratio tests
• Helps identify potential model inadequacies or violations of assumptions

Computational aspects

• Implementation of likelihood ratio tests involves several computational considerations
• Efficient algorithms and software tools facilitate practical application
• Understanding computational methods aids in proper test execution and interpretation

Maximum likelihood estimation

• Finds parameter values maximizing likelihood function
• Often requires numerical optimization techniques
• Newton-Raphson method commonly used for smooth likelihood functions
• EM algorithm useful for models with latent variables or missing data
• Gradient descent methods applicable to high-dimensional problems

Numerical optimization techniques

• Iterative methods often necessary to maximize likelihood functions
• Line search algorithms (golden section search, Brent's method) for univariate optimization
• Trust region methods provide alternatives to line search approaches
• Quasi-Newton methods (BFGS, L-BFGS) approximate Hessian matrix
• Stochastic optimization techniques useful for complex likelihood landscapes

Software implementations

• Statistical software packages offer built-in likelihood ratio test functions
• R provides

  anova()

  function for nested model comparisons
• Python's

  statsmodels

  library includes likelihood ratio test capabilities
• SAS implements likelihood ratio tests through PROC GENMOD and other procedures
• Custom implementations possible using optimization libraries (scipy.optimize, optim)

Limitations and alternatives

• Likelihood ratio tests have certain limitations and assumptions
• Understanding these constraints helps in proper test application
• Alternative methods can address specific limitations of likelihood ratio tests

Small sample considerations

• Asymptotic chi-square approximation may be poor for small samples
• Exact tests or Monte Carlo methods provide alternatives
• Bartlett correction improves chi-square approximation for some models
• Bootstrap techniques can assess sampling distribution of test statistic
• Permutation tests offer distribution-free alternatives for small samples

Robustness issues

• Sensitivity to outliers or model misspecification
• M-estimators provide robust alternatives to maximum likelihood
• Influence functions assess impact of individual observations on test results
• Robust likelihood ratio tests based on density power divergence
• Sandwich estimators correct for certain types of model misspecification

Bayesian alternatives

• Bayes factors compare posterior odds of competing models
• Posterior predictive p-values assess model fit in Bayesian framework
• Deviance information criterion (DIC) for Bayesian model selection
• Bayesian model averaging addresses model uncertainty
• Integrated nested Laplace approximations (INLA) for efficient Bayesian inference

Extensions and variations

• Several extensions and variations of likelihood ratio tests exist
• These modified tests address specific statistical scenarios or assumptions
• Understanding extensions broadens applicability of likelihood ratio testing framework

Generalized likelihood ratio test

• Applies to composite hypotheses where nuisance parameters present
• Maximizes likelihood over nuisance parameters for each hypothesis
• Test statistic based on ratio of maximized likelihoods
• Asymptotically follows chi-square distribution under regularity conditions
• Useful for testing subsets of parameters in multiparameter models

Conditional likelihood ratio test

• Used when full likelihood factorizes into marginal and conditional components
• Tests based on conditional likelihood given sufficient statistics
• Eliminates nuisance parameters through conditioning
• Applicable to stratified data or matched case-control studies
• Provides exact tests for some discrete data models (Fisher's exact test)

Profile likelihood ratio test

• Addresses nuisance parameters by profiling over them
• Maximizes likelihood with respect to nuisance parameters for each value of parameter of interest
• Resulting profile likelihood treated as function of parameter of interest
• Useful for constructing confidence intervals with correct coverage
• Can be computationally intensive for models with many nuisance parameters