🎣Statistical Inference Unit 3 – Joint Distributions & Independence

Key Concepts

• Joint distributions describe the probability distribution of two or more random variables simultaneously
• Marginal distributions represent the probability distribution of a single random variable, ignoring the others
• Conditional distributions describe the probability distribution of one random variable given the values of other random variables
  • Conditional distributions are derived by fixing the values of the conditioning variables and normalizing the joint distribution
• Independence between random variables implies that the joint distribution is the product of the marginal distributions
  • Independent random variables have no influence on each other's probability distributions
• Covariance measures the linear relationship between two random variables
  • Positive covariance indicates a direct relationship, while negative covariance suggests an inverse relationship
• Correlation is a standardized version of covariance, ranging from -1 to 1
  • Correlation of 0 implies no linear relationship, while -1 and 1 indicate perfect negative and positive linear relationships, respectively

Types of Joint Distributions

• Discrete joint distributions involve random variables that can only take on a countable number of values
  • Joint probability mass function (PMF) is used to describe discrete joint distributions
  • Example: The joint distribution of the number of heads and tails in a series of coin flips
• Continuous joint distributions involve random variables that can take on any value within a range
  • Joint probability density function (PDF) is used to describe continuous joint distributions
  • Example: The joint distribution of the heights and weights of a population
• Mixed joint distributions involve a combination of discrete and continuous random variables
  • A mix of PMFs and PDFs is used to describe mixed joint distributions
• Multivariate normal distribution is a common continuous joint distribution characterized by a mean vector and a covariance matrix
  • Assumes a bell-shaped curve and symmetric distribution for each variable
• Bivariate distributions are a special case of joint distributions involving only two random variables
  • Easier to visualize and analyze compared to higher-dimensional joint distributions

Marginal Distributions

• Marginal distributions are obtained by summing (for discrete variables) or integrating (for continuous variables) the joint distribution over the other variables
  • Marginal PMF: $P(X=x) = \sum_y P(X=x, Y=y)$
  • Marginal PDF: $f_X(x) = \int_{-\infty}^{\infty} f(x,y) dy$
• Marginal distributions provide information about the individual behavior of each random variable
• The sum or integral of a marginal distribution over its entire range is equal to 1
  • This property ensures that the marginal distribution is a valid probability distribution
• Marginal distributions do not contain information about the relationship or dependence between the random variables
• Marginal distributions can be used to calculate probabilities, expected values, and other statistics for individual random variables

Conditional Distributions

• Conditional distributions describe the probability distribution of one random variable given the values of other random variables
  • Conditional PMF: $P(Y=y|X=x) = \frac{P(X=x, Y=y)}{P(X=x)}$
  • Conditional PDF: $f_{Y|X}(y|x) = \frac{f(x,y)}{f_X(x)}$
• Conditional distributions allow us to update our knowledge about one variable based on the observed values of other variables
• The denominator in the conditional distribution formula is the marginal distribution of the conditioning variable
  • This ensures that the conditional distribution integrates or sums to 1 over the range of the conditioned variable
• Conditional distributions are essential for making predictions and inferring relationships between variables
• The law of total probability expresses the marginal distribution as a weighted sum or integral of conditional distributions
  • Discrete case: $P(Y=y) = \sum_x P(Y=y|X=x) P(X=x)$
  • Continuous case: $f_Y(y) = \int_{-\infty}^{\infty} f_{Y|X}(y|x) f_X(x) dx$

Independence: Definition and Properties

• Two random variables X and Y are independent if and only if their joint distribution is the product of their marginal distributions
  • For discrete variables: $P(X=x, Y=y) = P(X=x) P(Y=y)$
  • For continuous variables: $f(x,y) = f_X(x) f_Y(y)$
• Independence implies that the occurrence of one event does not affect the probability of the other event
• If X and Y are independent, their conditional distributions are equal to their marginal distributions
  • $P(Y=y|X=x) = P(Y=y)$ and $P(X=x|Y=y) = P(X=x)$
  • $f_{Y|X}(y|x) = f_Y(y)$ and $f_{X|Y}(x|y) = f_X(x)$
• The expected value of the product of independent random variables is the product of their individual expected values
  • $E[XY] = E[X] E[Y]$
• The variance of the sum of independent random variables is the sum of their individual variances
  • $Var(X+Y) = Var(X) + Var(Y)$
• Independence is a stronger condition than uncorrelatedness
  • Independent variables are always uncorrelated, but uncorrelated variables may not be independent

Covariance and Correlation

• Covariance measures the linear relationship between two random variables
  • $Cov(X,Y) = E[(X-E[X])(Y-E[Y])]$
  • Positive covariance indicates a direct relationship, while negative covariance suggests an inverse relationship
• Covariance is affected by the scale of the random variables
  • Changing the units of measurement can alter the magnitude of covariance
• Correlation is a standardized version of covariance, ranging from -1 to 1
  • $Corr(X,Y) = \frac{Cov(X,Y)}{\sqrt{Var(X)Var(Y)}}$
  • Correlation is unitless and not affected by the scale of the random variables
• A correlation of 0 implies no linear relationship between the variables
  • However, a non-linear relationship may still exist
• A correlation of -1 or 1 indicates a perfect negative or positive linear relationship, respectively
• The square of the correlation coefficient, known as the coefficient of determination ($R^2$), represents the proportion of variance in one variable explained by the other variable

Applications and Examples

• Joint distributions are used in various fields, such as finance, engineering, and social sciences, to model and analyze the relationship between multiple variables
• Example: In finance, the joint distribution of stock returns can be used to assess the risk and diversification of a portfolio
  • The correlation between stock returns helps determine the optimal asset allocation
• Example: In quality control, the joint distribution of product dimensions can be used to ensure that the products meet the required specifications
  • The conditional distribution of one dimension given the others can help identify the source of defects
• Example: In medical research, the joint distribution of risk factors (age, blood pressure, cholesterol) can be used to predict the likelihood of developing a disease
  • The marginal distributions of risk factors can help identify high-risk populations for targeted interventions
• Example: In machine learning, the joint distribution of features and target variables is used to train models for prediction and classification tasks
  • The conditional distribution of the target variable given the features is the basis for many supervised learning algorithms

Common Pitfalls and Misconceptions

• Confusing independence with uncorrelatedness
  • Independence is a stronger condition than uncorrelatedness
  • Variables can be uncorrelated but still dependent (e.g., non-linear relationships)
• Misinterpreting conditional distributions as causal relationships
  • Conditional distributions describe the association between variables but do not necessarily imply causation
  • Confounding factors or reverse causation may lead to spurious associations
• Assuming normality for joint distributions without verification
  • Many statistical methods assume that the joint distribution is multivariate normal
  • Violating this assumption can lead to incorrect inferences and predictions
• Ignoring the importance of marginal and conditional distributions
  • Focusing solely on the joint distribution may overlook important insights from the marginal and conditional distributions
  • Analyzing the marginal and conditional distributions can provide a more comprehensive understanding of the relationships between variables
• Mishandling missing data in joint distributions
  • Missing data can introduce bias and affect the estimation of joint, marginal, and conditional distributions
  • Appropriate methods (e.g., multiple imputation) should be used to handle missing data in joint distributions