🎣Statistical Inference Unit 11 – Maximum Likelihood & Sufficiency

Key Concepts

• Maximum likelihood estimation (MLE) is a method for estimating the parameters of a probability distribution by maximizing the likelihood function
• The likelihood function quantifies the probability of observing the data given a set of parameter values
• MLE provides a consistent approach to parameter estimation for a wide range of statistical models
• Sufficient statistics contain all the information relevant to estimating the parameters of a distribution
• The sufficiency principle states that the information contained in sufficient statistics is equivalent to the information in the full data set for making inferences about the parameters
• MLE and sufficiency are fundamental concepts in statistical inference and are used in various applications such as regression analysis, hypothesis testing, and model selection
• Understanding the properties and limitations of MLE and sufficiency is crucial for making valid statistical inferences and interpreting results accurately

Probability Foundations

• Probability is a measure of the likelihood of an event occurring and is expressed as a number between 0 and 1
• Joint probability is the probability of two or more events occurring simultaneously and is calculated by multiplying the individual probabilities of each event
• Conditional probability is the probability of an event occurring given that another event has already occurred and is calculated using Bayes' theorem
• Independence of events means that the occurrence of one event does not affect the probability of another event occurring
• Random variables are variables whose values are determined by the outcome of a random experiment and can be discrete (taking on a countable number of values) or continuous (taking on any value within a range)
• Probability distributions describe the likelihood of different outcomes for a random variable and can be represented by probability mass functions (PMFs) for discrete random variables or probability density functions (PDFs) for continuous random variables
• Expected value is the average value of a random variable over a large number of trials and is calculated by summing the product of each possible value and its probability

Likelihood Function Basics

• The likelihood function is a function of the parameters of a statistical model given the observed data and is proportional to the probability of the data given the parameters
• For discrete random variables, the likelihood function is the product of the probabilities of each observed data point given the parameter values
• For continuous random variables, the likelihood function is the product of the probability densities of each observed data point given the parameter values
• The likelihood function is not a probability distribution itself but rather a function of the parameters that measures how well the model fits the data
• The maximum likelihood estimate (MLE) of a parameter is the value that maximizes the likelihood function
• The log-likelihood function is often used instead of the likelihood function for mathematical convenience and is the natural logarithm of the likelihood function
• The shape of the likelihood function provides information about the precision and uncertainty of the parameter estimates, with narrower peaks indicating more precise estimates

Maximum Likelihood Estimation (MLE)

• MLE is a method for estimating the parameters of a statistical model by finding the parameter values that maximize the likelihood function
• The MLE is the parameter value that makes the observed data most probable under the assumed statistical model
• MLE is used in a wide range of applications, including linear regression, logistic regression, and Gaussian mixture models
• The MLE is obtained by setting the derivative of the log-likelihood function with respect to each parameter equal to zero and solving the resulting system of equations
• In some cases, the MLE can be obtained analytically, but in many cases, numerical optimization methods such as gradient descent or Newton's method are used
• MLE is a consistent estimator, meaning that as the sample size increases, the MLE converges to the true parameter value
• MLE is asymptotically efficient, meaning that as the sample size increases, the MLE achieves the lowest possible variance among all consistent estimators

Properties of MLE

• Consistency: As the sample size increases, the MLE converges to the true parameter value
• Asymptotic normality: As the sample size increases, the distribution of the MLE becomes approximately normal with mean equal to the true parameter value and variance equal to the inverse of the Fisher information matrix
• Efficiency: The MLE achieves the lowest possible variance among all consistent estimators asymptotically
• Invariance: The MLE is invariant under parameter transformations, meaning that if $\hat{\theta}$ is the MLE of $\theta$, then $g(\hat{\theta})$ is the MLE of $g(\theta)$ for any function $g$
• Asymptotic unbiasedness: As the sample size increases, the bias of the MLE tends to zero
• Equivariance: The MLE is equivariant under transformations of the data, meaning that if $\hat{\theta}$ is the MLE based on the original data, then $\hat{\theta}$ is also the MLE based on the transformed data
• Asymptotic efficiency: The MLE achieves the Cramér-Rao lower bound asymptotically, meaning that it has the smallest possible variance among all unbiased estimators

Sufficiency Principle

• The sufficiency principle states that if a statistic is sufficient for a parameter, then any inference about the parameter should depend only on the sufficient statistic and not on the full data set
• A statistic is sufficient for a parameter if the conditional distribution of the data given the statistic does not depend on the parameter
• The sufficiency principle implies that if two different data sets have the same value for a sufficient statistic, then they contain the same information about the parameter
• The sufficiency principle allows for data reduction, as it suggests that only the sufficient statistic needs to be retained for inference about the parameter
• The Rao-Blackwell theorem is a consequence of the sufficiency principle and states that if an estimator is not a function of a sufficient statistic, then it can be improved by conditioning on the sufficient statistic
• The sufficiency principle is related to the likelihood principle, which states that all the information about a parameter contained in the data is captured by the likelihood function
• The sufficiency principle is a fundamental concept in statistical inference and is used in various applications such as hypothesis testing, point estimation, and interval estimation

Sufficient Statistics

• A statistic is a function of the data that is used to estimate a parameter or make inferences about a population
• A sufficient statistic is a statistic that contains all the information about a parameter that is contained in the full data set
• Formally, a statistic $T(X)$ is sufficient for a parameter $\theta$ if the conditional distribution of the data $X$ given $T(X)$ does not depend on $\theta$
• The factorization theorem provides a way to identify sufficient statistics by factoring the joint probability density or mass function of the data into a product of two functions, one that depends only on the data and the parameter and one that depends only on the data
• The minimal sufficient statistic is the sufficient statistic with the smallest possible dimension and is unique up to one-to-one transformations
• Sufficient statistics can be used to construct point estimators, such as the MLE, and to perform hypothesis tests and construct confidence intervals
• Examples of sufficient statistics include the sample mean for the normal distribution with known variance, the sample proportion for the binomial distribution, and the sample mean and sample variance for the normal distribution with unknown mean and variance

Applications and Examples

• MLE is widely used in linear regression to estimate the coefficients of the regression model by maximizing the likelihood function of the observed data assuming normally distributed errors
• In logistic regression, MLE is used to estimate the coefficients of the model by maximizing the likelihood function of the observed binary outcomes given the predictor variables
• MLE is used in Gaussian mixture models to estimate the parameters (means, variances, and mixing proportions) of a mixture of Gaussian distributions by maximizing the likelihood function of the observed data
• In hypothesis testing, the likelihood ratio test is a powerful test that uses the ratio of the maximum likelihood under the null and alternative hypotheses to make a decision
• Sufficient statistics are used in the Rao-Blackwell theorem to improve the efficiency of estimators by conditioning on a sufficient statistic
• The sample mean is a sufficient statistic for the mean of a normal distribution with known variance, and the sample proportion is a sufficient statistic for the probability of success in a binomial distribution
• In Bayesian inference, the posterior distribution of the parameters given the data is proportional to the product of the prior distribution and the likelihood function, which emphasizes the importance of the likelihood function in Bayesian analysis