📊Probability and Statistics Unit 4 – Expectation, Variance, and Moments

Key Concepts and Definitions

• Random variable represents a numerical outcome of a random experiment can be discrete (countable outcomes) or continuous (uncountable outcomes)
• Probability distribution function (PDF) defines the probability of each possible outcome for a discrete random variable
  • Denoted as $P(X = x)$ where $X$ is the random variable and $x$ is a specific value
• Cumulative distribution function (CDF) gives the probability that a random variable is less than or equal to a certain value
  • Defined as $F(x) = P(X \leq x)$
• Probability density function (pdf) describes the likelihood of a continuous random variable taking on a specific value
  • Area under the pdf curve between two points represents the probability of the variable falling within that range
• Expected value (mean) of a random variable is the average value obtained if an experiment is repeated many times
  • Denoted as $E(X)$ or $\mu$
• Variance measures the average squared deviation of a random variable from its mean
  • Calculated as $Var(X) = E[(X - \mu)^2]$ or $\sigma^2$
• Standard deviation is the square root of variance provides a measure of dispersion in the same units as the random variable
• Moments are mathematical expectations of powers of a random variable used to characterize its probability distribution

Probability Distributions and Random Variables

• Bernoulli distribution models a single trial with two possible outcomes (success or failure) with probability $p$ for success and $1-p$ for failure
• Binomial distribution describes the number of successes in a fixed number of independent Bernoulli trials with the same success probability
  • Denoted as $X \sim B(n, p)$ where $n$ is the number of trials and $p$ is the success probability
• Poisson distribution models the number of events occurring in a fixed interval of time or space when events occur independently at a constant average rate
  • Denoted as $X \sim Poisson(\lambda)$ where $\lambda$ is the average number of events per interval
• Normal (Gaussian) distribution is a continuous probability distribution characterized by its bell-shaped curve
  • Denoted as $X \sim N(\mu, \sigma^2)$ where $\mu$ is the mean and $\sigma^2$ is the variance
• Exponential distribution models the time between events in a Poisson process or the waiting time until the first event occurs
  • Denoted as $X \sim Exp(\lambda)$ where $\lambda$ is the rate parameter
• Uniform distribution assigns equal probability to all values within a specified range $(a, b)$
  • Denoted as $X \sim U(a, b)$
• Joint probability distribution describes the probabilities of two or more random variables occurring together
• Marginal probability distribution is obtained by summing or integrating the joint distribution over the values of the other variables

Understanding Expectation (Mean)

• Expectation is a key concept in probability theory and statistics represents the average value of a random variable over many trials
• For a discrete random variable $X$ with probability mass function $P(X = x_i)$, the expected value is calculated as $E(X) = \sum_{i} x_i P(X = x_i)$
  • Example: For a fair six-sided die, $E(X) = 1 \cdot \frac{1}{6} + 2 \cdot \frac{1}{6} + \ldots + 6 \cdot \frac{1}{6} = 3.5$
• For a continuous random variable $X$ with probability density function $f(x)$, the expected value is calculated as $E(X) = \int_{-\infty}^{\infty} x f(x) dx$
• Linearity of expectation states that for random variables $X$ and $Y$, $E(X + Y) = E(X) + E(Y)$, even if $X$ and $Y$ are dependent
• Expected value of a constant is the constant itself: $E(c) = c$
• If $X$ is a random variable and $a$ and $b$ are constants, then $E(aX + b) = aE(X) + b$
• The expected value of a function $g(X)$ of a random variable $X$ is given by $E(g(X)) = \sum_{i} g(x_i) P(X = x_i)$ for discrete $X$ and $E(g(X)) = \int_{-\infty}^{\infty} g(x) f(x) dx$ for continuous $X$

Exploring Variance and Standard Deviation

• Variance measures the average squared deviation of a random variable from its mean indicates the spread of the distribution
  • Calculated as $Var(X) = E[(X - \mu)^2]$ where $\mu = E(X)$
• Standard deviation is the square root of variance provides a measure of dispersion in the same units as the random variable
  • Denoted as $\sigma = \sqrt{Var(X)}$
• For a discrete random variable $X$ with probability mass function $P(X = x_i)$, variance is calculated as $Var(X) = \sum_{i} (x_i - \mu)^2 P(X = x_i)$
• For a continuous random variable $X$ with probability density function $f(x)$, variance is calculated as $Var(X) = \int_{-\infty}^{\infty} (x - \mu)^2 f(x) dx$
• Variance has several important properties:
  • $Var(aX + b) = a^2 Var(X)$ for constants $a$ and $b$
  • If $X$ and $Y$ are independent, then $Var(X + Y) = Var(X) + Var(Y)$
• Chebyshev's inequality relates the variance to the probability of a random variable deviating from its mean by a certain amount
  • States that $P(|X - \mu| \geq k\sigma) \leq \frac{1}{k^2}$ for any $k > 0$
• Standard deviation is often used to construct confidence intervals and test hypotheses about population parameters

Higher Moments: Skewness and Kurtosis

• Skewness is a measure of the asymmetry of a probability distribution
  • Positive skewness indicates a longer right tail, while negative skewness indicates a longer left tail
  • Calculated as $Skewness(X) = E\left[\left(\frac{X - \mu}{\sigma}\right)^3\right]$
• Kurtosis measures the heaviness of the tails of a distribution compared to a normal distribution
  • Higher kurtosis indicates heavier tails and more extreme values
  • Calculated as $Kurtosis(X) = E\left[\left(\frac{X - \mu}{\sigma}\right)^4\right]$
• Moments are mathematical expectations of powers of a random variable used to characterize its probability distribution
  • The $n$-th moment of a random variable $X$ is defined as $E(X^n)$
  • Central moments are calculated using deviations from the mean: $E[(X - \mu)^n]$
• The first moment is the mean, the second central moment is the variance, the third standardized moment is skewness, and the fourth standardized moment is kurtosis
• Higher moments provide additional information about the shape and properties of a probability distribution beyond the mean and variance

Properties and Theorems

• Law of Large Numbers states that the sample mean converges to the population mean as the sample size increases
  • Implies that the average of a large number of independent trials will be close to the expected value
• Central Limit Theorem (CLT) states that the sum or average of a large number of independent and identically distributed random variables will be approximately normally distributed, regardless of the distribution of the individual variables
  • Enables the use of normal distribution for inference in many situations
• Markov's Inequality provides an upper bound on the probability that a non-negative random variable exceeds a certain value
  • States that for a non-negative random variable $X$ and any $a > 0$, $P(X \geq a) \leq \frac{E(X)}{a}$
• Jensen's Inequality relates the value of a convex function of an expectation to the expectation of the convex function
  • For a convex function $g$ and a random variable $X$, $E(g(X)) \geq g(E(X))$
• Wald's Equation states that the expected value of the sum of a random number of independent and identically distributed random variables is equal to the product of the expected number of terms and the expected value of each term
• Moment Generating Function (MGF) is a way to uniquely characterize a probability distribution
  • Defined as $M_X(t) = E(e^{tX})$ for a random variable $X$
• Properties of expectation, variance, and moments can be used to simplify calculations and derive relationships between random variables

Practical Applications

• Portfolio optimization in finance uses expected returns, variances, and covariances of assets to construct portfolios with desired risk-return characteristics
• Quality control in manufacturing relies on the mean and variance of product characteristics to ensure consistency and identify deviations from specifications
• Insurance companies use probability distributions and moments to model claim sizes and frequencies, set premiums, and manage risk
• Hypothesis testing and confidence intervals in statistical inference rely on the properties of expectation, variance, and the Central Limit Theorem
• Regression analysis uses the expected value of the response variable conditional on the predictors to model relationships and make predictions
• Time series analysis and forecasting employ moments and autocorrelations to characterize the dependence structure and predict future values
• Machine learning algorithms, such as Gaussian Naive Bayes and Gaussian Mixture Models, use the properties of normal distributions and moments to model and classify data
• Monte Carlo simulations rely on the Law of Large Numbers and Central Limit Theorem to estimate probabilities, expectations, and quantiles of complex systems

Common Pitfalls and Tips

• Remember that expectation is a linear operator, but variance is not: $Var(X + Y) \neq Var(X) + Var(Y)$ unless $X$ and $Y$ are independent
• Be cautious when using the sample variance to estimate the population variance, as it is a biased estimator
  • Use the unbiased sample variance $s^2 = \frac{1}{n-1} \sum_{i=1}^n (x_i - \bar{x})^2$ instead
• Check the assumptions of the Central Limit Theorem (independence, identical distribution, finite variance) before applying it
• Consider the effect of outliers on the sample moments, as they can heavily influence the estimates
  • Use robust measures like the median and interquartile range when outliers are present
• Be aware of the limitations of Chebyshev's and Markov's inequalities, as they provide bounds but not exact probabilities
• Remember that skewness and kurtosis are sensitive to the units of measurement
  • Standardize the variable before computing these moments
• Interpret the moments in the context of the problem and the underlying distribution
  • High kurtosis may indicate the need for a heavy-tailed distribution, while skewness may suggest a transformation
• Use the properties of expectation and variance to simplify calculations whenever possible
  • Example: $Var(aX + b) = a^2 Var(X)$ for constants $a$ and $b$