📈Applied Impact Evaluation Unit 5 – Data Collection and Measurement

Key Concepts and Definitions

• Data collection involves gathering information from various sources to answer research questions or test hypotheses
• Measurement refers to assigning numerical values or categories to observed phenomena in a systematic way
• Sampling is the process of selecting a subset of a population to represent the entire group
• Validity assesses whether a measurement tool accurately captures the intended construct or concept
  • Face validity evaluates if a measure appears to be measuring what it claims to measure based on its content
  • Construct validity examines if a measure behaves as expected in relation to other variables
• Reliability refers to the consistency and stability of measurements across time, raters, or items
• Bias occurs when systematic errors or deviations from the truth are present in the data collection process
• Confounding variables are extraneous factors that influence both the independent and dependent variables, obscuring the true relationship

Types of Data Collection Methods

• Surveys involve administering questionnaires to a sample of respondents to gather self-reported information
  • Can be conducted through various modes such as online, phone, mail, or in-person
  • Allows for collecting data from a large number of individuals relatively quickly and cost-effectively
• Interviews entail asking open-ended questions to participants to gain in-depth insights and perspectives
  • Can be structured with a fixed set of questions, semi-structured with a flexible guide, or unstructured with a free-flowing conversation
• Observations involve systematically watching and recording behaviors, events, or interactions in natural settings
  • Can be participant observation where the researcher engages in the activities being studied or non-participant observation where the researcher remains detached
• Experiments manipulate one or more independent variables to examine their effects on dependent variables while controlling for other factors
• Secondary data analysis utilizes existing data sources (census records, administrative databases) that were originally collected for other purposes
• Physiological measurements capture biological or physical indicators (heart rate, brain activity) using specialized equipment
• Ecological momentary assessment (EMA) collects real-time data on participants' experiences, behaviors, or states in their natural environments through repeated sampling

Sampling Techniques and Strategies

• Probability sampling ensures that each unit in the population has a known, non-zero chance of being selected
  • Simple random sampling assigns an equal probability of selection to each unit using a random mechanism (random number generator)
  • Stratified random sampling divides the population into mutually exclusive subgroups (strata) and randomly samples from each stratum
  • Cluster sampling involves selecting naturally occurring groups (clusters) and then sampling units within the selected clusters
• Non-probability sampling does not rely on random selection and may be subject to bias
  • Convenience sampling selects participants based on their accessibility or willingness to participate
  • Purposive sampling chooses participants deliberately based on specific characteristics or criteria
  • Snowball sampling identifies initial participants who then refer other potential participants from their networks
• Sample size determination balances statistical power, precision, and feasibility considerations
  • Larger samples generally provide more precise estimates and greater power to detect effects
  • Power analysis can help determine the minimum sample size needed to detect an effect of a given magnitude with a desired level of confidence
• Sampling weights can be applied to adjust for unequal probabilities of selection or non-response bias

Measurement Tools and Instruments

• Questionnaires are standardized sets of questions used to collect self-reported data from respondents
  • Can include closed-ended questions with predetermined response options (multiple choice, Likert scales) or open-ended questions that allow for free-text responses
  • Should be carefully designed to ensure clarity, relevance, and absence of leading or biased language
• Scales are measurement instruments that assign numerical values to represent the level or intensity of a construct
  • Likert scales use a series of statements with agree-disagree response options to measure attitudes or opinions
  • Visual analog scales (VAS) use a continuous line with anchors at each end to represent the extremes of a construct
• Tests are standardized assessments used to evaluate knowledge, skills, or abilities
  • Can be criterion-referenced (measuring performance against a fixed standard) or norm-referenced (comparing performance to a reference group)
• Observational protocols provide structured guidelines for systematically recording behaviors or events
  • Should specify the behaviors to be observed, the coding scheme, and the recording procedures
• Physiological measures capture biological or physical indicators using specialized equipment
  • Examples include heart rate monitors, electroencephalography (EEG), or functional magnetic resonance imaging (fMRI)
• Qualitative data collection tools such as interview guides or focus group protocols outline the topics and questions to be explored in a flexible manner

Data Quality and Validity

• Validity refers to the extent to which a measurement tool accurately captures the intended construct or concept
  • Content validity assesses whether the items or questions adequately represent the domain of interest
  • Criterion validity examines the relationship between the measure and an external criterion or gold standard
  • Construct validity evaluates whether the measure behaves as expected in relation to other variables or constructs
• Reliability assesses the consistency and stability of measurements across time, raters, or items
  • Test-retest reliability measures the agreement between scores obtained at different time points
  • Inter-rater reliability evaluates the agreement between multiple raters or observers
  • Internal consistency assesses the homogeneity or coherence of items within a scale
• Measurement error refers to the discrepancy between the observed value and the true value
  • Random error is unsystematic and tends to cancel out across multiple measurements
  • Systematic error is consistent and introduces bias in a particular direction
• Response bias occurs when participants provide inaccurate or distorted responses
  • Social desirability bias involves responding in a way that presents oneself favorably
  • Acquiescence bias refers to the tendency to agree with statements regardless of their content
• Data cleaning involves identifying and correcting errors, inconsistencies, or missing values in the dataset
• Data triangulation compares and integrates data from multiple sources or methods to enhance validity and credibility

Ethical Considerations in Data Collection

• Informed consent ensures that participants are fully informed about the study purpose, procedures, risks, and benefits before agreeing to participate
  • Should be voluntary, free from coercion, and based on a clear understanding of the research
  • Special considerations apply for vulnerable populations (children, individuals with cognitive impairments)
• Confidentiality protects participants' identities and personal information from unauthorized disclosure
  • Involves using secure data storage, encryption, and access controls
  • May require de-identification or anonymization of data
• Privacy safeguards participants' right to control access to their personal information and space
  • Includes respecting participants' boundaries and minimizing intrusion into their lives
• Beneficence obligates researchers to maximize benefits and minimize harm to participants and society
• Justice ensures fair and equitable treatment of participants and distribution of research benefits and burdens
• Ethical review boards (IRBs) evaluate research proposals to ensure compliance with ethical standards and protect participant rights and welfare
• Cultural competence involves being sensitive to and respectful of participants' cultural backgrounds, values, and practices

Practical Challenges and Solutions

• Non-response occurs when sampled individuals do not participate or provide incomplete data
  • Can introduce bias if non-responders differ systematically from responders
  • Strategies to improve response rates include incentives, reminders, and multiple contact attempts
• Attrition refers to the loss of participants over time in longitudinal studies
  • Can compromise the representativeness and power of the study
  • Retention strategies include maintaining regular contact, providing incentives, and minimizing participant burden
• Missing data can occur due to item non-response, attrition, or data entry errors
  • Can be handled through various techniques such as listwise deletion, pairwise deletion, or imputation
• Resource constraints (time, budget) may limit the scope and design of data collection
  • Efficient sampling strategies, streamlined instruments, and leveraging existing data can help optimize resources
• Technological challenges may arise with electronic data capture, data storage, or data security
  • Robust software, backup systems, and technical support can help mitigate risks
• Logistical challenges can include coordinating data collection across multiple sites, languages, or cultures
  • Clear communication, standardized protocols, and local partnerships can facilitate smooth implementation
• Participant burden should be minimized to ensure compliance and data quality
  • Concise instruments, user-friendly interfaces, and reasonable time commitments can enhance participant experience

Real-World Applications

• Program evaluation assesses the implementation, effectiveness, and impact of interventions or policies
  • Involves collecting data on program inputs, activities, outputs, and outcomes
  • Helps inform decision-making, resource allocation, and program improvement
• Needs assessment identifies the strengths, weaknesses, and gaps in a community or population
  • Collects data on demographics, health status, service utilization, and stakeholder perspectives
  • Guides the development and targeting of interventions to address identified needs
• Monitoring and evaluation (M&E) systems track progress, performance, and results of development projects
  • Involves regular data collection, analysis, and reporting against predefined indicators
  • Supports accountability, learning, and adaptive management
• Surveillance systems continuously collect, analyze, and interpret data on health events or risk factors
  • Enable early detection, response, and prevention of public health threats
  • Examples include infectious disease surveillance, behavioral risk factor surveillance, and injury surveillance
• Market research gathers data on consumer preferences, behaviors, and trends to inform business strategies
  • Utilizes surveys, focus groups, and customer feedback to understand market dynamics and inform product development
• Quality improvement initiatives use data to identify areas for enhancement and monitor the impact of changes
  • Involves collecting process and outcome measures to assess the efficiency, effectiveness, and safety of healthcare delivery
• Social science research investigates human behavior, social phenomena, and societal issues
  • Employs various data collection methods to test theories, explore patterns, and inform policy recommendations