2.3 Selection bias and confounding factors

Selection bias and confounding factors can seriously mess up our impact evaluations. They make it hard to figure out if our program actually caused the changes we see. We need to watch out for these sneaky issues when designing studies and analyzing results.

There are lots of ways selection bias can creep in, like when people choose to join a study or when researchers influence who gets treatment. Confounding factors are tricky variables that affect both who gets treatment and the outcomes we care about. We've got some tools to deal with these problems though.

Selection bias in causal inference

Understanding selection bias

Top images from around the web for Understanding selection bias

• 

  Why It Matters: Inference for Means | Concepts in Statistics View original

  Is this image relevant?

• 

  Sequentially mixing modes in an election survey | Survey Methods: Insights from the Field (SMIF) View original

  Is this image relevant?

• 

  Why It Matters: Inference for Means | Concepts in Statistics View original

  Is this image relevant?

• 

  Sequentially mixing modes in an election survey | Survey Methods: Insights from the Field (SMIF) View original

  Is this image relevant?

1 of 2

Top images from around the web for Understanding selection bias

• 

  Why It Matters: Inference for Means | Concepts in Statistics View original

  Is this image relevant?

• 

  Sequentially mixing modes in an election survey | Survey Methods: Insights from the Field (SMIF) View original

  Is this image relevant?

• 

  Why It Matters: Inference for Means | Concepts in Statistics View original

  Is this image relevant?

• 

  Sequentially mixing modes in an election survey | Survey Methods: Insights from the Field (SMIF) View original

  Is this image relevant?

1 of 2

• Selection bias occurs when the sample or treatment group is not representative of the target population, leading to systematic differences between groups
• Compromises the validity of causal inferences by violating the assumption of exchangeability, crucial for establishing causal relationships
• Can result from self-selection (participants choosing to join a study), researcher selection (investigators influencing group assignment), or attrition (differential dropout rates)
• Leads to over- or underestimation of treatment effects, potentially invalidating the results of an impact evaluation
• Quantify selection bias by comparing baseline characteristics between groups and conducting sensitivity analyses (Rosenbaum bounds)

Impact on study design and interpretation

• Essential for designing robust impact evaluations and interpreting their results accurately
• Influences sample selection methods, data collection strategies, and analytical approaches
• Requires careful consideration of potential sources of bias throughout the research process
• Affects the generalizability of study findings to the broader population
• Necessitates the use of appropriate statistical techniques to mitigate its effects (propensity score matching, inverse probability weighting)

Sources of selection bias

Participant-driven biases

• Self-selection bias occurs when individuals choose whether to participate in a program or treatment, potentially based on unobserved characteristics (motivation levels)
• Attrition bias results from systematic differences between those who remain in a study and those who drop out (higher dropout rates among less engaged participants)
• Survivorship bias emerges when only successful or surviving units are included in the analysis, overlooking those that failed or disappeared (studying only companies that survived an economic crisis)
• Healthy worker effect is a form of selection bias in occupational studies where healthier individuals are more likely to be employed and remain in the workforce

Researcher and study design biases

• Researcher or program implementer selection bias arises when they consciously or unconsciously influence who receives the treatment (assigning more motivated participants to the intervention group)
• Non-response bias occurs when there are systematic differences between those who respond to a survey or participate in a study and those who do not (only highly satisfied customers responding to feedback surveys)
• Berkson's bias, or collider bias, can arise in case-control studies when the selection of cases and controls is influenced by a common effect of the exposure and outcome (studying the relationship between a risk factor and a disease only in hospitalized patients)

Confounding factors and bias

Understanding confounding factors

• Confounding factors are variables that influence both the treatment assignment and the outcome, potentially leading to spurious associations
• Create a backdoor path between the treatment and outcome, violating the assumption of conditional independence
• Can be observed (measurable variables) or unobserved (unmeasured characteristics), with the latter being particularly challenging to address in impact evaluations
• Lead to over- or underestimation of the true causal effect, depending on the direction and magnitude of their influence
• Require thorough understanding of the causal mechanisms and context of the intervention being evaluated for proper identification

Analyzing confounding relationships

• Directed Acyclic Graphs (DAGs) serve as useful tools for visualizing and analyzing potential confounding relationships in a causal framework
• Help identify minimal sufficient adjustment sets to control for confounding
• Assist in recognizing potential sources of bias, including collider bias and mediator bias
• Time-varying confounders present additional challenges, as they may be affected by previous treatment and simultaneously influence future treatment and outcomes
• Require specialized analytical techniques (marginal structural models) to address their complex temporal relationships

Addressing selection bias and confounding factors

Experimental approaches

• Randomization serves as a powerful tool for addressing selection bias and confounding by creating comparable groups at baseline
• Ensures that observed and unobserved characteristics are balanced between treatment and control groups
• Difference-in-differences and fixed effects models control for time-invariant unobserved confounders in panel data settings
• Regression discontinuity designs exploit exogenous cutoffs to estimate causal effects in the presence of selection bias (assigning treatment based on a threshold score)

Quasi-experimental and statistical methods

• Matching techniques, such as propensity score matching, create balanced comparison groups in observational studies
• Instrumental variables estimation addresses selection bias and confounding when a valid instrument is available (using rainfall as an instrument for agricultural productivity)
• Collecting rich baseline data on potential confounders and using appropriate statistical controls helps mitigate bias in impact evaluations
• Inverse probability weighting and marginal structural models address time-varying confounding in longitudinal studies
• Sensitivity analyses, such as Rosenbaum bounds, assess the robustness of results to potential unobserved confounding