Fiveable
Fiveable
Fiveable
Fiveable
Unlock Cram Mode

Linear Modeling Theory

11.4 Post-hoc Analysis for Two-Way ANOVA

4 min readLast Updated on July 30, 2024

Two-Way ANOVA helps us spot differences between groups, but it doesn't tell us which ones. That's where post-hoc analysis comes in. It lets us compare specific groups and figure out exactly where the differences lie.

Post-hoc tests are crucial because they control for errors when making multiple comparisons. They help us avoid false positives and draw more accurate conclusions about our data. Understanding post-hoc analysis is key to getting the most out of Two-Way ANOVA results.

Post-Hoc Analysis in ANOVA

Purpose and Importance

Top images from around the web for Purpose and Importance
Top images from around the web for Purpose and Importance
  • Conducted after a significant main effect or interaction effect is found in a two-way ANOVA to determine which specific group means differ significantly from each other
  • Control the familywise error rate, the probability of making at least one Type I error (false positive) when conducting multiple pairwise comparisons
  • Essential for identifying the specific differences between group means that contribute to the overall significant effect found in the two-way ANOVA
  • Without post-hoc analysis, researchers cannot determine which specific group means differ significantly, limiting the interpretability and practical implications of the findings
  • Provide a more detailed understanding of the nature of the significant effects found in the two-way ANOVA, allowing researchers to draw more precise conclusions and make more targeted recommendations based on the results

Controlling Type I Error

  • Post-hoc tests are designed to control the familywise error rate, which increases with the number of pairwise comparisons conducted
  • Familywise error rate is the probability of making at least one Type I error (false positive) across all pairwise comparisons
  • Without controlling for the familywise error rate, the likelihood of finding a significant difference by chance alone increases as more comparisons are made
  • Post-hoc tests adjust the significance level for each comparison to maintain the overall Type I error rate at the desired level (usually 0.05)
  • Examples of post-hoc tests that control the familywise error rate include Tukey's HSD, Bonferroni correction, and Scheffe's test

Choosing Post-Hoc Tests

Factors to Consider

  • Number of pairwise comparisons: Some post-hoc tests (Bonferroni) are more conservative and appropriate when the number of comparisons is relatively small, while others (Tukey's HSD) are suitable for a larger number of comparisons
  • Sample size: Post-hoc tests may have different performance depending on the sample size; some tests (Tukey's HSD) are more robust to unequal sample sizes than others
  • Assumption of homogeneity of variances: Some post-hoc tests (Tukey's HSD) assume equal variances across groups, while others (Games-Howell) are more robust to violations of this assumption
  • Research question and specific comparisons of interest: Some post-hoc tests (Dunnett's test) are designed for comparing treatment groups to a control group, while others (Tukey's HSD) compare all possible pairs of means

Commonly Used Post-Hoc Tests

  • Tukey's Honestly Significant Difference (HSD) test: Compares all possible pairs of group means while controlling the familywise error rate; appropriate when sample sizes are equal and the assumption of homogeneity of variances is met
  • Bonferroni correction: Adjusts the significance level for each pairwise comparison to control the familywise error rate; more conservative than Tukey's HSD and appropriate when the number of comparisons is small
  • Scheffe's test: A more conservative post-hoc test that is robust to violations of the assumption of homogeneity of variances; suitable when the number of comparisons is large
  • Dunnett's test: Compares each treatment group to a control group while controlling the familywise error rate; appropriate when the research question specifically involves comparisons to a control condition

Interpreting Post-Hoc Results

Presenting Results

  • Post-hoc test results are typically presented as a matrix or table showing the pairwise comparisons between group means and their corresponding p-values
  • A significant p-value (p < .05) indicates that the difference between the two group means is statistically significant, while a non-significant p-value suggests that the difference is not significant
  • In addition to p-values, post-hoc test results may include mean differences, standard errors, confidence intervals, and effect sizes to provide a more comprehensive understanding of the findings

Drawing Conclusions

  • When interpreting post-hoc test results, researchers should focus on the specific pairwise comparisons that are relevant to their research question and hypotheses
  • Consider the magnitude of the differences between group means, in addition to their statistical significance, to assess the practical importance of the findings
  • Be cautious not to overinterpret non-significant differences or to make causal inferences without proper experimental design
  • Discuss the implications of the post-hoc test results in the context of the research question, previous literature, and the limitations of the study
  • Clear and concise reporting of post-hoc test results, along with effect sizes and confidence intervals, can enhance the interpretability and replicability of the findings
© 2025 Fiveable Inc. All rights reserved.
AP® and SAT® are trademarks registered by the College Board, which is not affiliated with, and does not endorse this website.


© 2025 Fiveable Inc. All rights reserved.
AP® and SAT® are trademarks registered by the College Board, which is not affiliated with, and does not endorse this website.

© 2025 Fiveable Inc. All rights reserved.
AP® and SAT® are trademarks registered by the College Board, which is not affiliated with, and does not endorse this website.
Glossary
Glossary