🪚Public Policy Analysis Unit 11 – Quantitative Methods for Policy Analysis

Key Concepts and Terminology

• Quantitative methods involve using numerical data and statistical techniques to analyze and inform policy decisions
• Variables can be classified as independent (explanatory) or dependent (response) based on their role in the analysis
• Measurement scales include nominal, ordinal, interval, and ratio, each with increasing levels of precision and mathematical properties
• Reliability refers to the consistency of measurements, while validity assesses whether a measure accurately captures the intended concept
• Sampling is the process of selecting a subset of a population for analysis, with various techniques such as simple random sampling, stratified sampling, and cluster sampling
  • Simple random sampling ensures each unit has an equal probability of being selected
  • Stratified sampling divides the population into homogeneous subgroups before sampling from each stratum
• Hypothesis testing involves formulating null and alternative hypotheses and using statistical tests to determine the likelihood of observed results under the null hypothesis
• Statistical significance indicates the probability that observed results are due to chance, with common thresholds being 0.05 and 0.01
• Effect size measures the magnitude of a relationship or difference, providing practical significance beyond statistical significance

Data Collection and Measurement

• Primary data is collected directly by the researcher for a specific purpose, while secondary data is pre-existing data collected by others
• Surveys are a common method of primary data collection, involving questionnaires administered to a sample of respondents
  • Survey design considerations include question wording, order, and response formats to minimize bias and maximize response rates
  • Modes of survey administration include in-person, telephone, mail, and online, each with advantages and limitations
• Experiments involve manipulating one or more variables while controlling others to establish causal relationships
  • Random assignment to treatment and control groups helps ensure internal validity by minimizing confounding variables
• Observational studies collect data without manipulating variables, making it more difficult to establish causality but offering greater external validity
• Measurement error can arise from various sources, such as instrument error, respondent error, and processing error
• Reliability can be assessed through test-retest, parallel forms, and internal consistency methods
• Validity types include face validity, content validity, criterion validity, and construct validity
  • Face validity is a subjective assessment of whether a measure appears to capture the intended concept
  • Content validity assesses whether a measure covers all relevant aspects of a construct

Descriptive Statistics and Data Visualization

• Measures of central tendency summarize the typical or average value of a dataset, including the mean, median, and mode
  • The mean is sensitive to outliers, while the median is more robust
• Measures of dispersion quantify the spread or variability of a dataset, such as the range, variance, and standard deviation
  • The range is the difference between the maximum and minimum values
  • The standard deviation is the square root of the variance and is in the same units as the original data
• Frequency distributions organize data into categories or intervals and display the count or percentage of observations in each
• Histograms are graphical representations of frequency distributions, with bars representing the count or percentage of observations in each interval
• Box plots display the median, quartiles, and outliers of a dataset, providing a concise summary of its distribution
• Scatterplots show the relationship between two continuous variables, with each point representing an observation
• Correlation coefficients measure the strength and direction of the linear relationship between two variables, ranging from -1 to 1
  • Pearson's correlation coefficient is commonly used for continuous variables
• Data visualization principles include choosing appropriate chart types, using clear labels and legends, and avoiding clutter and distortion

Probability and Statistical Inference

• Probability is the likelihood of an event occurring, expressed as a value between 0 and 1
• Probability distributions describe the probabilities of different outcomes for a random variable
  • Discrete probability distributions (binomial, Poisson) are used for countable outcomes
  • Continuous probability distributions (normal, exponential) are used for measurable outcomes
• The normal distribution is a symmetric, bell-shaped distribution characterized by its mean and standard deviation
  • The standard normal distribution has a mean of 0 and a standard deviation of 1
• The Central Limit Theorem states that the sampling distribution of the mean approaches a normal distribution as the sample size increases, regardless of the shape of the population distribution
• Confidence intervals estimate a population parameter with a specified level of confidence, typically 95%
  • A 95% confidence interval means that if the sampling process were repeated many times, 95% of the intervals would contain the true population parameter
• Hypothesis testing involves comparing a sample statistic to a hypothesized population parameter to determine the likelihood of the observed results under the null hypothesis
  • The p-value is the probability of observing results as extreme as the sample results, assuming the null hypothesis is true
  • If the p-value is less than the chosen significance level (e.g., 0.05), the null hypothesis is rejected in favor of the alternative hypothesis
• Type I error (false positive) occurs when the null hypothesis is rejected when it is actually true, while Type II error (false negative) occurs when the null hypothesis is not rejected when it is actually false

Regression Analysis Techniques

• Simple linear regression models the relationship between one independent variable and one dependent variable
  • The slope coefficient represents the change in the dependent variable for a one-unit change in the independent variable
  • The intercept represents the value of the dependent variable when the independent variable is zero
• Multiple linear regression extends simple linear regression to include multiple independent variables
  • Partial regression coefficients represent the effect of each independent variable on the dependent variable, holding other variables constant
• Assumptions of linear regression include linearity, independence, normality, and homoscedasticity
  • Linearity assumes a straight-line relationship between the independent and dependent variables
  • Independence assumes that observations are not related to each other
  • Normality assumes that residuals are normally distributed
  • Homoscedasticity assumes that the variance of residuals is constant across all levels of the independent variables
• Residuals are the differences between the observed and predicted values of the dependent variable
• R-squared measures the proportion of variance in the dependent variable explained by the independent variables, ranging from 0 to 1
• Adjusted R-squared accounts for the number of independent variables in the model, penalizing the addition of variables that do not significantly improve the model fit
• Logistic regression is used when the dependent variable is binary or categorical, modeling the probability of an event occurring
  • Odds ratios represent the change in the odds of the event for a one-unit change in the independent variable

Time Series and Forecasting Methods

• Time series data consists of observations collected at regular intervals over time
• Components of time series include trend, seasonality, cyclical patterns, and irregular fluctuations
  • Trend refers to the long-term increase or decrease in the series
  • Seasonality refers to regular patterns that repeat over fixed periods (e.g., monthly, quarterly)
  • Cyclical patterns are longer-term fluctuations that do not have a fixed period
• Moving averages smooth time series data by averaging observations within a specified window
  • Simple moving averages assign equal weights to all observations in the window
  • Weighted moving averages assign different weights to observations based on their recency or importance
• Exponential smoothing methods assign exponentially decreasing weights to past observations, with more recent observations having greater influence
  • Simple exponential smoothing is appropriate for series with no trend or seasonality
  • Holt's linear trend method accounts for series with a trend but no seasonality
  • Holt-Winters' method accounts for series with both trend and seasonality
• Autoregressive Integrated Moving Average (ARIMA) models combine autoregressive, differencing, and moving average components to capture complex patterns in time series data
  • Autoregressive terms model the relationship between an observation and a certain number of lagged observations
  • Differencing removes trend and seasonality by computing the differences between consecutive observations
  • Moving average terms model the relationship between an observation and past forecast errors
• Stationarity is a key assumption of many time series models, requiring the mean, variance, and autocorrelation structure to remain constant over time
• Forecasting involves predicting future values of a time series based on past observations and patterns
  • Forecast accuracy can be assessed using measures such as mean absolute error (MAE), mean squared error (MSE), and mean absolute percentage error (MAPE)

Policy Evaluation and Impact Assessment

• Policy evaluation assesses the effectiveness, efficiency, and impact of public policies and programs
• Process evaluation examines the implementation and delivery of a policy or program, identifying strengths, weaknesses, and areas for improvement
• Outcome evaluation measures the extent to which a policy or program achieves its intended goals and objectives
• Impact evaluation assesses the causal effects of a policy or program on targeted outcomes, using counterfactual analysis to estimate what would have happened in the absence of the intervention
  • Randomized controlled trials (RCTs) are the gold standard for impact evaluation, randomly assigning units to treatment and control groups
  • Quasi-experimental designs, such as difference-in-differences and regression discontinuity, can be used when randomization is not feasible
• Cost-benefit analysis compares the monetary costs and benefits of a policy or program, calculating net present value and benefit-cost ratio
• Cost-effectiveness analysis compares the costs and outcomes of different interventions, identifying the most efficient option for achieving a given objective
• Sensitivity analysis examines how changes in key assumptions or parameters affect the results of an evaluation or analysis
• Stakeholder engagement involves incorporating the perspectives and input of various stakeholders, such as policymakers, program staff, and target populations, throughout the evaluation process

Practical Applications and Case Studies

• Education policy: Evaluating the impact of class size reduction on student achievement using a difference-in-differences approach
  • Comparing changes in test scores between schools that implemented class size reduction and those that did not, before and after the intervention
• Healthcare policy: Assessing the cost-effectiveness of different screening strategies for colorectal cancer
  • Estimating the incremental cost-effectiveness ratio (ICER) for each strategy, measuring the additional cost per quality-adjusted life year (QALY) gained
• Environmental policy: Analyzing the relationship between air pollution levels and respiratory health outcomes using multiple linear regression
  • Controlling for confounding variables such as age, gender, and smoking status to isolate the effect of air pollution on health
• Transportation policy: Forecasting traffic volume using an ARIMA model to inform infrastructure planning and investment decisions
  • Incorporating seasonal patterns and trends in traffic data to generate accurate long-term projections
• Social welfare policy: Conducting a randomized controlled trial to evaluate the impact of a job training program on employment and earnings outcomes
  • Randomly assigning eligible participants to treatment and control groups and comparing their labor market outcomes over time
• Criminal justice policy: Using logistic regression to identify factors associated with recidivism among released offenders
  • Estimating the odds ratios for variables such as age, criminal history, and participation in rehabilitation programs to inform risk assessment and resource allocation
• International development policy: Employing propensity score matching to evaluate the impact of a microcredit program on household income and consumption in a developing country
  • Matching program participants to similar non-participants based on observable characteristics to create a valid comparison group
• Urban planning policy: Applying spatial regression techniques to analyze the relationship between land use patterns and housing prices across a city
  • Accounting for spatial dependence and heterogeneity to capture the effects of neighborhood characteristics on property values