📈Applied Impact Evaluation Unit 7 – Econometric Methods for Impact Analysis

Key Concepts and Terminology

• Impact evaluation assesses the causal effects of interventions, policies, or programs on specific outcomes of interest
• Counterfactual thinking involves considering what would have happened in the absence of the intervention
• Treatment group consists of individuals or units that receive the intervention being evaluated
• Control group serves as a comparison and does not receive the intervention
• Selection bias occurs when treatment and control groups differ systematically in ways that affect the outcome
• Confounding variables are factors that influence both the treatment assignment and the outcome, potentially biasing the results
• Randomization assigns treatment randomly to ensure that treatment and control groups are balanced on observable and unobservable characteristics
  • Reduces selection bias and allows for causal inference
• Heterogeneous treatment effects refer to variations in the impact of an intervention across different subgroups or contexts

Theoretical Foundations

• Potential outcomes framework is a conceptual approach that defines causal effects in terms of potential outcomes under different treatment conditions
• Rubin causal model formalizes the idea of potential outcomes and provides a framework for causal inference
• Stable unit treatment value assumption (SUTVA) states that the potential outcomes for one unit are unaffected by the treatment assignment of other units
  • Ensures no interference between units and allows for well-defined causal effects
• Conditional independence assumption (CIA) implies that treatment assignment is independent of potential outcomes given a set of observed covariates
  • Justifies the use of matching and regression methods for causal inference
• Ignorability assumption combines SUTVA and CIA, stating that treatment assignment is independent of potential outcomes conditional on observed covariates
• Overlap assumption requires that there is a positive probability of receiving each treatment level for all values of the covariates
  • Ensures that there are comparable units in both treatment and control groups
• Instrumental variables approach relies on finding a variable that affects treatment assignment but not the outcome directly
  • Allows for causal inference in the presence of unmeasured confounding

Research Design Principles

• Clearly define the research question and the causal effect of interest
• Identify the appropriate unit of analysis (individuals, households, communities)
• Determine the relevant outcome variables and how they will be measured
• Consider potential sources of bias and confounding factors
• Choose a research design that maximizes internal validity while balancing external validity
  • Internal validity refers to the ability to make causal claims within the study sample
  • External validity relates to the generalizability of the findings to other contexts or populations
• Randomized controlled trials (RCTs) are considered the gold standard for causal inference
  • Random assignment of treatment eliminates selection bias
• Quasi-experimental designs (difference-in-differences, regression discontinuity) can be used when randomization is not feasible
  • Rely on natural experiments or discontinuities in treatment assignment

Data Collection and Preparation

• Develop a comprehensive data collection plan that captures all relevant variables
• Use reliable and valid measurement instruments to ensure data quality
• Pilot test the data collection tools and procedures to identify and address any issues
• Train data collectors to ensure consistency and minimize measurement error
• Implement quality control measures (double data entry, spot checks) to maintain data integrity
• Clean and preprocess the data to handle missing values, outliers, and inconsistencies
  • Document all data cleaning steps for transparency and reproducibility
• Create a codebook that provides clear definitions and coding schemes for all variables
• Merge and link data from different sources if necessary (survey data, administrative records)

Statistical Methods and Models

• Descriptive statistics summarize key features of the data (means, standard deviations, correlations)
• Hypothesis testing assesses whether observed differences between groups are statistically significant
• Regression analysis estimates the relationship between the outcome and treatment while controlling for other factors
  • Ordinary least squares (OLS) regression is commonly used for continuous outcomes
  • Logistic regression is appropriate for binary outcomes
  • Multilevel models account for clustered or hierarchical data structures
• Propensity score methods match or weight observations based on their likelihood of receiving treatment
  • Helps balance treatment and control groups on observed covariates
• Difference-in-differences (DID) compares changes in outcomes over time between treatment and control groups
  • Assumes parallel trends in outcomes prior to the intervention
• Regression discontinuity design (RDD) exploits a cutoff or threshold that determines treatment assignment
  • Compares outcomes for units just above and below the threshold

Causal Inference Techniques

• Randomization inference tests the sharp null hypothesis of no treatment effect for any unit
  • Permutes treatment assignments to generate a distribution of test statistics under the null
• Instrumental variables (IV) estimation uses an exogenous source of variation in treatment assignment to identify causal effects
  • Two-stage least squares (2SLS) is a common IV estimation method
• Mediation analysis investigates the mechanisms through which an intervention affects outcomes
  • Decomposes the total effect into direct and indirect effects
• Sensitivity analysis assesses the robustness of the results to violations of key assumptions
  • Examines how the estimates change under different scenarios or alternative specifications
• Bounds analysis provides a range of plausible effect sizes when assumptions are partially violated
  • Worst-case bounds make no assumptions about the direction of the bias
• Meta-analysis combines results from multiple studies to provide a more precise estimate of the overall effect
  • Accounts for heterogeneity across studies and potential publication bias

Practical Applications and Case Studies

• Impact evaluations have been conducted in various fields (education, health, labor, environment)
• Case study: Conditional cash transfer programs (Progresa in Mexico) have been rigorously evaluated using RCTs
  • Found positive effects on school enrollment, health outcomes, and poverty reduction
• Case study: Microfinance impact evaluations (Banerjee et al. 2015) have used RCTs to assess the effects on borrowers' socioeconomic outcomes
  • Results suggest modest positive impacts on business investment and consumption smoothing
• Case study: Deworming interventions (Miguel and Kremer 2004) have been evaluated using cluster-randomized trials
  • Showed significant improvements in school attendance and health outcomes
• Replication and external validity assessments are important for understanding the generalizability of findings
  • Replication studies test the robustness of the original results
  • External validity studies examine whether the effects hold in different contexts or populations

Limitations and Ethical Considerations

• Impact evaluations can be costly and time-consuming, requiring significant resources and expertise
• Ethical concerns arise when withholding potentially beneficial interventions from the control group
  • Equipoise principle suggests that there should be genuine uncertainty about which treatment is superior
• Informed consent and data privacy are critical ethical considerations in impact evaluations
  • Participants should be fully informed about the study and their rights
  • Data should be securely stored and accessed only by authorized personnel
• Spillover effects can occur when the treatment affects outcomes for untreated units
  • Can bias the estimates of the treatment effect if not properly accounted for
• Hawthorne effects arise when participants change their behavior because they know they are being observed
  • Can lead to overestimation of the treatment effect
• Generalizability of the findings may be limited by the specific context, population, or implementation of the intervention
  • Extrapolating the results to other settings requires careful consideration of the underlying assumptions and boundary conditions
• Publication bias can distort the evidence base if studies with null or negative results are less likely to be published
  • Preregistration of study protocols and reporting of all results can mitigate this bias