📉Statistical Methods for Data Science Unit 5 – Hypothesis Testing & Statistical Inference

Key Concepts and Definitions

• Hypothesis testing assesses the validity of a claim or hypothesis about a population parameter based on sample data
• Null hypothesis $(H_0)$ represents the default or status quo position, typically stating no effect or no difference
• Alternative hypothesis $(H_a)$ represents the claim or research question, suggesting an effect or difference
• Type I error (false positive) occurs when rejecting a true null hypothesis, denoted by $\alpha$ (significance level)
• Type II error (false negative) occurs when failing to reject a false null hypothesis, denoted by $\beta$
• Statistical power is the probability of correctly rejecting a false null hypothesis $(1-\beta)$
• Effect size measures the magnitude of the difference or relationship between variables

Types of Hypothesis Tests

• One-sample tests compare a sample statistic to a known population parameter (e.g., one-sample t-test, z-test)
• Two-sample tests compare two independent samples to determine if they come from populations with different parameters (e.g., two-sample t-test, Mann-Whitney U test)
  • Independent samples have no relationship or influence on each other
• Paired-sample tests compare two related or dependent samples (e.g., paired t-test, Wilcoxon signed-rank test)
  • Dependent samples have a one-to-one correspondence or come from the same individuals
• Analysis of Variance (ANOVA) tests compare means across three or more groups or conditions (e.g., one-way ANOVA, two-way ANOVA)
• Chi-square tests assess the relationship between categorical variables (e.g., chi-square test of independence, chi-square goodness-of-fit test)
• Correlation tests measure the strength and direction of the linear relationship between two continuous variables (e.g., Pearson's correlation, Spearman's rank correlation)

Steps in Hypothesis Testing

• State the null and alternative hypotheses clearly, specifying the population parameter of interest
• Choose an appropriate test statistic and distribution based on the type of data and hypothesis
• Set the significance level $(\alpha)$ to determine the threshold for rejecting the null hypothesis (common levels: 0.01, 0.05, 0.10)
• Collect sample data and calculate the test statistic
• Determine the p-value associated with the test statistic
  • p-value represents the probability of observing a test statistic as extreme as or more extreme than the one calculated, assuming the null hypothesis is true
• Compare the p-value to the significance level and make a decision to reject or fail to reject the null hypothesis
• Interpret the results in the context of the research question and consider the practical significance of the findings

Statistical Significance and p-values

• Statistical significance indicates the likelihood that the observed results are due to chance rather than a true effect
• p-value is the probability of obtaining the observed results or more extreme results, assuming the null hypothesis is true
  • Smaller p-values provide stronger evidence against the null hypothesis
• Significance level $(\alpha)$ is the predetermined threshold for rejecting the null hypothesis
  • If p-value $\leq \alpha$, reject the null hypothesis; if p-value $> \alpha$, fail to reject the null hypothesis
• Statistically significant results do not necessarily imply practical or clinical significance
• Multiple testing and p-value adjustment methods (e.g., Bonferroni correction, false discovery rate) help control for Type I errors when conducting multiple hypothesis tests

Confidence Intervals and Estimation

• Confidence intervals provide a range of plausible values for a population parameter based on sample data
• Level of confidence (e.g., 95%, 99%) represents the proportion of intervals that would contain the true population parameter if the sampling process were repeated many times
• Confidence intervals are constructed using the sample statistic and its standard error
  • For a population mean: $\bar{x} \pm z_{\alpha/2} \cdot \frac{s}{\sqrt{n}}$ or $\bar{x} \pm t_{\alpha/2, n-1} \cdot \frac{s}{\sqrt{n}}$
• Wider confidence intervals indicate greater uncertainty in the estimate, while narrower intervals suggest more precise estimates
• Confidence intervals can be used to test hypotheses by examining whether the hypothesized value falls within the interval
• Margin of error is the half-width of the confidence interval and represents the maximum expected difference between the sample estimate and the true population parameter

Common Statistical Tests

• t-tests assess the difference between means (one-sample, two-sample, or paired)
  • Assumes normally distributed data or large sample sizes (n > 30)
• ANOVA tests compare means across three or more groups
  • One-way ANOVA examines the effect of one categorical factor on a continuous response variable
  • Two-way ANOVA examines the effects of two categorical factors and their interaction on a continuous response variable
• Chi-square tests evaluate the association between categorical variables
  • Chi-square test of independence assesses whether two categorical variables are independent
  • Chi-square goodness-of-fit test compares the observed frequencies of categories to the expected frequencies based on a hypothesized distribution
• Correlation tests measure the strength and direction of the linear relationship between two continuous variables
  • Pearson's correlation assumes normally distributed data and a linear relationship
  • Spearman's rank correlation is a non-parametric alternative that assesses the monotonic relationship between variables
• Non-parametric tests (e.g., Mann-Whitney U, Wilcoxon signed-rank, Kruskal-Wallis) are used when data do not meet the assumptions of parametric tests or when dealing with ordinal data

Interpreting Test Results

• Determine whether to reject or fail to reject the null hypothesis based on the p-value and significance level
• Interpret the direction and magnitude of the effect, if applicable (e.g., positive or negative correlation, size of the mean difference)
• Consider the confidence interval for the parameter estimate to assess the precision and plausible range of values
• Evaluate the practical or clinical significance of the results, not just the statistical significance
• Discuss the limitations of the study and potential sources of bias or confounding factors
• Avoid overgeneralizing the results beyond the scope of the study or population sampled
• Consider the context of the research question and the implications of the findings for future research or decision-making

Real-world Applications and Examples

• A/B testing in marketing compares the effectiveness of two versions of a website or advertisement (e.g., click-through rates, conversion rates)
• Clinical trials in medicine evaluate the efficacy and safety of new treatments or interventions compared to a placebo or standard treatment
• Quality control in manufacturing uses hypothesis testing to assess whether a product meets specified standards or tolerances
• Polling and surveys use confidence intervals to estimate population proportions or means based on sample data (e.g., political polls, customer satisfaction surveys)
• Psychological research employs various hypothesis tests to study the relationships between variables or the effects of interventions on behavior or mental health outcomes
• Environmental studies use hypothesis testing to assess the impact of human activities or interventions on ecosystems or species populations (e.g., comparing biodiversity in protected and unprotected areas)
• Social science research applies hypothesis testing to investigate the relationships between demographic, social, or economic factors and various outcomes (e.g., education, health, income)