🎣Statistical Inference Unit 5 – Point Estimation: Methods & Properties

What's Point Estimation?

• Point estimation involves using sample data to calculate a single value that serves as a "best guess" or estimate of an unknown population parameter
• Aims to find an estimator, a sample statistic, that can be used to infer the true value of the parameter
• Estimators are functions of the sample data, often denoted with a "hat" symbol (e.g., $\hat{\theta}$ for an estimator of the parameter $\theta$)
• Differs from interval estimation, which provides a range of plausible values for the parameter rather than a single point
• Example: using the sample mean ($\bar{X}$) to estimate the population mean ($\mu$)
  • If the sample mean from a random sample of 100 individuals is $\bar{X} = 25$, the point estimate for the population mean would be $\hat{\mu} = 25$
• The goal is to find estimators that are as close as possible to the true parameter value
• Involves a trade-off between accuracy and precision
  • Accuracy refers to how close the estimator is to the true value on average
  • Precision refers to how much variability there is in the estimates across different samples

Key Concepts to Know

• Population parameter: a numerical summary of a characteristic of the entire population, usually unknown and denoted by Greek letters (e.g., $\mu$, $\sigma$, $\pi$)
• Estimator: a sample statistic used to estimate a population parameter, often denoted with a "hat" symbol (e.g., $\hat{\mu}$, $\hat{\sigma}$, $\hat{\pi}$)
• Sampling distribution: the probability distribution of an estimator, describing its behavior over repeated samples
• Bias: the difference between the expected value of an estimator and the true parameter value
  • An unbiased estimator has an expected value equal to the parameter it's estimating
• Efficiency: a measure of the variability of an estimator
  • A more efficient estimator has a smaller variance and thus provides more precise estimates
• Consistency: an estimator is consistent if it converges in probability to the true parameter value as the sample size increases
• Sufficiency: an estimator is sufficient if it captures all the relevant information about the parameter contained in the sample
• Mean squared error (MSE): a measure of the quality of an estimator, equal to the sum of its variance and the square of its bias

Methods of Point Estimation

• Method of moments: equates sample moments (e.g., mean, variance) to their population counterparts and solves for the parameter
  • Example: for a normal distribution, set $\bar{X} = \mu$ and $S^2 = \sigma^2$ to obtain method of moments estimators $\hat{\mu} = \bar{X}$ and $\hat{\sigma}^2 = S^2$
• Maximum likelihood estimation (MLE): chooses the parameter value that maximizes the likelihood function, the probability of observing the sample data given the parameter
  • Involves setting the derivative of the log-likelihood equal to zero and solving for the parameter
  • Often results in estimators with desirable properties like consistency and asymptotic efficiency
• Bayesian estimation: incorporates prior information about the parameter in the form of a prior distribution, updating it with the sample data to obtain a posterior distribution
  • Point estimates can be obtained from the posterior, such as the mean, median, or mode
• Least squares estimation: chooses the parameter value that minimizes the sum of squared differences between observed and predicted values
  • Commonly used in regression analysis to estimate coefficients
• Estimating equations: sets up a system of equations based on the sample data and solves for the parameter
  • Generalized estimating equations (GEE) extend this approach to correlated data, such as in longitudinal studies

Properties of Good Estimators

• Unbiasedness: an unbiased estimator has an expected value equal to the true parameter value
  • Example: the sample mean is an unbiased estimator of the population mean, i.e., $E(\bar{X}) = \mu$
• Efficiency: an efficient estimator has the smallest possible variance among all unbiased estimators
  • The Cramér-Rao lower bound provides a theoretical limit for the variance of an unbiased estimator
  • An estimator that achieves this lower bound is called a minimum variance unbiased estimator (MVUE)
• Consistency: a consistent estimator converges in probability to the true parameter value as the sample size increases
  • Ensures that the estimator becomes more accurate with larger samples
  • Example: the sample proportion is a consistent estimator of the population proportion
• Sufficiency: a sufficient estimator captures all the relevant information about the parameter contained in the sample
  • The sample mean is a sufficient estimator for the mean of a normal distribution
  • Sufficient estimators can be used to construct uniformly minimum variance unbiased estimators (UMVUEs)
• Robustness: a robust estimator is not heavily influenced by outliers or deviations from model assumptions
  • Example: the median is a more robust estimator of central tendency than the mean
• Asymptotic normality: many estimators, such as MLEs, are asymptotically normal, meaning their sampling distribution approaches a normal distribution as the sample size increases
  • Allows for the construction of confidence intervals and hypothesis tests based on the normal distribution

Bias and Efficiency

• Bias is the difference between the expected value of an estimator and the true parameter value
  • Positive bias means the estimator tends to overestimate the parameter, while negative bias means it tends to underestimate
  • Unbiased estimators have a bias of zero, i.e., $E(\hat{\theta}) = \theta$
• The bias of an estimator can be calculated as $Bias(\hat{\theta}) = E(\hat{\theta}) - \theta$
  • Example: the sample variance $S^2 = \frac{\sum_{i=1}^n (X_i - \bar{X})^2}{n}$ is a biased estimator of the population variance $\sigma^2$, with $E(S^2) = \frac{n-1}{n}\sigma^2$
    • The unbiased estimator is $S^2 = \frac{\sum_{i=1}^n (X_i - \bar{X})^2}{n-1}$
• Efficiency refers to the precision of an estimator, with more efficient estimators having smaller variances
  • The Cramér-Rao lower bound provides a theoretical limit for the variance of an unbiased estimator
  • An estimator that achieves this lower bound is called an efficient or minimum variance unbiased estimator (MVUE)
• The relative efficiency of two estimators $\hat{\theta}_1$ and $\hat{\theta}_2$ is the ratio of their variances, $RE(\hat{\theta}_1, \hat{\theta}_2) = \frac{Var(\hat{\theta}_2)}{Var(\hat{\theta}_1)}$
  • If $RE > 1$, $\hat{\theta}_1$ is more efficient than $\hat{\theta}_2$
• There is often a trade-off between bias and efficiency
  • Biased estimators can sometimes have lower variance than unbiased ones
  • The mean squared error (MSE) takes into account both bias and variance: $MSE(\hat{\theta}) = Var(\hat{\theta}) + [Bias(\hat{\theta})]^2$
  • Minimizing the MSE can lead to a biased estimator with better overall performance

Consistency and Sufficiency

• Consistency is a large-sample property of an estimator, ensuring that it converges to the true parameter value as the sample size increases
  • Formally, an estimator $\hat{\theta}$ is consistent for $\theta$ if, for any $\epsilon > 0$, $\lim_{n \to \infty} P(|\hat{\theta} - \theta| < \epsilon) = 1$
  • Consistency is a weak requirement, as it doesn't specify the rate of convergence or the estimator's behavior for finite sample sizes
• Checking for consistency often involves applying the Law of Large Numbers or the Central Limit Theorem
  • Example: the sample mean $\bar{X}$ is a consistent estimator of the population mean $\mu$ by the Law of Large Numbers
• Sufficiency is a property that ensures an estimator captures all the relevant information about the parameter contained in the sample
  • A statistic $T(X)$ is sufficient for $\theta$ if the conditional distribution of the sample $X$ given $T(X)$ does not depend on $\theta$
  • Intuitively, this means that once we know the value of $T(X)$, the remaining data provides no additional information about $\theta$
• The Factorization Theorem provides a way to identify sufficient statistics
  • If the joint pdf or pmf of the sample can be factored as $f(x|\theta) = g(T(x), \theta) \cdot h(x)$, where $g$ depends on $x$ only through $T(x)$ and $h$ does not depend on $\theta$, then $T(X)$ is a sufficient statistic for $\theta$
• Sufficient statistics can be used to construct uniformly minimum variance unbiased estimators (UMVUEs) using the Rao-Blackwell Theorem
  • If $\hat{\theta}$ is an unbiased estimator of $\theta$ and $T(X)$ is a sufficient statistic, then $\hat{\theta}^* = E(\hat{\theta}|T(X))$ is a UMVUE of $\theta$
  • This process is called Rao-Blackwellization and can be used to improve the efficiency of estimators

Real-World Applications

• Survey sampling: point estimation is used to estimate population means, proportions, and totals from sample data
  • Example: estimating the average income of a city's residents based on a random sample of households
• Quality control: point estimation is used to monitor process parameters and ensure that products meet specifications
  • Example: estimating the proportion of defective items in a manufacturing batch based on a sample
• Econometrics: point estimation is used to estimate economic parameters such as elasticities, marginal effects, and returns to scale
  • Example: estimating the price elasticity of demand for a product based on historical sales and price data
• Biostatistics: point estimation is used to estimate treatment effects, disease prevalence, and other health-related parameters
  • Example: estimating the average reduction in blood pressure due to a new medication based on a clinical trial
• Machine learning: point estimation is used in various algorithms, such as linear regression, logistic regression, and neural networks
  • Example: estimating the coefficients in a linear regression model to predict housing prices based on features like square footage and number of bedrooms
• Finance: point estimation is used to estimate risk measures, such as value at risk (VaR) and expected shortfall
  • Example: estimating the 95% VaR of a portfolio based on historical returns data
• Actuarial science: point estimation is used to estimate parameters in mortality tables and loss distributions
  • Example: estimating the parameters of a Weibull distribution to model claim sizes in property insurance

Common Pitfalls and How to Avoid Them

• Overfitting: occurs when an estimator is too complex and fits the noise in the sample data rather than the underlying pattern
  • Can lead to poor performance on new, unseen data
  • Avoid by using cross-validation, regularization techniques, or model selection criteria like AIC or BIC
• Underfitting: occurs when an estimator is too simple and fails to capture the true relationship between the variables
  • Can lead to biased estimates and poor predictive performance
  • Avoid by considering more complex models or adding relevant features
• Outliers: extreme values that can heavily influence some estimators, particularly those based on least squares
  • Can lead to biased and unstable estimates
  • Avoid by using robust estimators (e.g., median instead of mean) or removing outliers based on domain knowledge
• Non-representative samples: when the sample is not randomly selected or does not adequately represent the population of interest
  • Can lead to biased estimates and incorrect conclusions
  • Avoid by using probability sampling techniques and ensuring that the sample covers all relevant subgroups
• Violation of assumptions: when the data does not meet the assumptions of the estimation method (e.g., normality, linearity, independence)
  • Can lead to biased, inefficient, or inconsistent estimators
  • Avoid by checking assumptions using diagnostic plots or tests and considering alternative methods if assumptions are violated
• Multicollinearity: when predictor variables in a regression model are highly correlated with each other
  • Can lead to unstable and difficult-to-interpret estimates
  • Avoid by removing redundant variables, combining related variables, or using regularization techniques like ridge regression or lasso
• Ignoring data structure: when the estimation method does not account for the inherent structure of the data (e.g., clustering, time series, spatial dependence)
  • Can lead to biased and inefficient estimates, as well as incorrect standard errors
  • Avoid by using methods specifically designed for the data structure, such as mixed models, time series models, or spatial regression