🎲Mathematical Probability Theory Unit 7 – Limit Theorems

Key Concepts and Definitions

• Limit theorems study the asymptotic behavior of sequences of random variables as the sample size or number of random variables increases
• Convergence describes how a sequence of random variables approaches a limit in various senses (distribution, probability, or almost surely)
• Random variables are functions that map outcomes of a random experiment to real numbers
  • Discrete random variables take on countable values (integers)
  • Continuous random variables take on uncountable values (real numbers)
• Probability distributions assign probabilities to events or outcomes
  • Probability mass functions (PMFs) define discrete probability distributions
  • Probability density functions (PDFs) define continuous probability distributions
• Expected value $\mathbb{E}[X]$ represents the average value of a random variable $X$ over its distribution
• Variance $\text{Var}(X)$ measures the spread or dispersion of a random variable $X$ around its expected value
• Characteristic functions uniquely determine probability distributions and are defined as $\varphi_X(t) = \mathbb{E}[e^{itX}]$

Types of Convergence

• Convergence in distribution (weak convergence) occurs when the cumulative distribution functions (CDFs) of a sequence of random variables converge to a limiting CDF
  • Denoted as $X_n \xrightarrow{d} X$ or $X_n \xrightarrow{D} X$
• Convergence in probability happens when the probability of the absolute difference between a sequence of random variables and a limit being greater than any positive value approaches zero
  • Denoted as $X_n \xrightarrow{p} X$
• Almost sure convergence (strong convergence) takes place when a sequence of random variables converges to a limit with probability one
  • Denoted as $X_n \xrightarrow{a.s.} X$
• Convergence in mean (L^p convergence) occurs when the expected value of the absolute difference between a sequence of random variables and a limit raised to the power $p$ approaches zero
• Relationships between types of convergence
  • Almost sure convergence implies convergence in probability
  • Convergence in probability implies convergence in distribution
  • Convergence in mean (for $p \geq 1$) implies convergence in probability

Law of Large Numbers

• The law of large numbers (LLN) states that the sample mean of a sequence of independent and identically distributed (i.i.d.) random variables converges to the population mean as the sample size increases
• Weak law of large numbers (WLLN) asserts convergence in probability
  • If $X_1, X_2, \ldots$ are i.i.d. with $\mathbb{E}[X_i] = \mu$, then $\bar{X}_n \xrightarrow{p} \mu$ as $n \to \infty$
• Strong law of large numbers (SLLN) asserts almost sure convergence
  • If $X_1, X_2, \ldots$ are i.i.d. with $\mathbb{E}[X_i] = \mu$, then $\bar{X}_n \xrightarrow{a.s.} \mu$ as $n \to \infty$
• LLN justifies the use of sample means to estimate population means in statistics
• Applies to various scenarios (insurance claims, polling, Monte Carlo methods)

Central Limit Theorem

• The central limit theorem (CLT) states that the standardized sum of a sequence of i.i.d. random variables with finite variance converges in distribution to a standard normal random variable as the sample size increases
• If $X_1, X_2, \ldots$ are i.i.d. with $\mathbb{E}[X_i] = \mu$ and $\text{Var}(X_i) = \sigma^2 < \infty$, then $\frac{\sum_{i=1}^n X_i - n\mu}{\sigma\sqrt{n}} \xrightarrow{d} N(0, 1)$ as $n \to \infty$
• CLT explains why many real-world phenomena follow a normal distribution (heights, IQ scores)
• Enables the construction of confidence intervals and hypothesis tests in statistics
• Generalizations of the CLT (Lyapunov CLT, Lindeberg-Feller CLT) relax the assumptions of identical distributions and finite variance

Weak Convergence and Characteristic Functions

• Weak convergence (convergence in distribution) can be characterized using characteristic functions
• Lévy's continuity theorem states that a sequence of random variables converges in distribution to a limit if and only if their characteristic functions converge pointwise to the characteristic function of the limit
• Characteristic functions are powerful tools for proving limit theorems and studying the properties of probability distributions
  • Uniquely determine probability distributions
  • Convolution of independent random variables corresponds to the product of their characteristic functions
• Characteristic functions can be used to derive moments and cumulants of probability distributions

Applications in Statistics

• Limit theorems provide the foundation for many statistical methods and techniques
• Law of large numbers justifies the use of sample means and proportions to estimate population parameters
  • Enables the construction of point estimators (sample mean, sample variance)
• Central limit theorem allows for the construction of confidence intervals and hypothesis tests
  • Used in t-tests, z-tests, and ANOVA
• Limit theorems are crucial in the development of asymptotic theory in statistics
  • Maximum likelihood estimation
  • Efficiency of estimators
• Applications in various fields (finance, physics, engineering)

Proofs and Derivations

• Proofs of limit theorems rely on various mathematical tools and techniques
  • Characteristic functions
  • Moment generating functions
  • Truncation and approximation arguments
• Proofs often involve showing convergence of moments or characteristic functions
• Techniques for proving the law of large numbers
  • Chebyshev's inequality for the WLLN
  • Borel-Cantelli lemma for the SLLN
• Proofs of the central limit theorem
  • Lindeberg's condition
  • Stein's method
• Derivations of the characteristic functions of common probability distributions (normal, Poisson, exponential)

Common Misconceptions and Pitfalls

• Assuming that the law of large numbers implies convergence to a constant value rather than the expected value
• Misinterpreting the central limit theorem as a statement about the distribution of individual random variables rather than their standardized sum
• Applying the central limit theorem to dependent or non-identically distributed random variables without justification
• Confusing the different types of convergence and their implications
• Neglecting the assumptions and conditions required for limit theorems to hold
  • Independence
  • Identical distributions
  • Finite moments
• Misusing limit theorems in situations where the sample size is not sufficiently large
• Overreliance on asymptotic results without considering finite-sample behavior