5.1 Designing surveys and questionnaires

Surveys and questionnaires are crucial tools in impact evaluation, helping researchers gather data to assess program effectiveness. Designing these instruments requires careful planning to ensure data validity and reliability, from choosing methods to developing sampling strategies and addressing ethical concerns.

Crafting effective questions is key to obtaining accurate, useful information. This involves selecting appropriate question types, wording items clearly, and structuring the survey logically. Techniques like randomization and attention checks help minimize bias and improve response quality, enhancing the overall evaluation process.

Survey Design Considerations

Planning and Methodology

Top images from around the web for Planning and Methodology

• 

  Planning, monitoring and evaluation - supporting constructive change and social learning View original

  Is this image relevant?

• 

  The Planning Cycle | Principles of Management View original

  Is this image relevant?

• 

  EEP impacts 1 - Framework for impact evaluation - HAMK Unlimited View original

  Is this image relevant?

• 

  Planning, monitoring and evaluation - supporting constructive change and social learning View original

  Is this image relevant?

• 

  The Planning Cycle | Principles of Management View original

  Is this image relevant?

1 of 3

Top images from around the web for Planning and Methodology

• 

  Planning, monitoring and evaluation - supporting constructive change and social learning View original

  Is this image relevant?

• 

  The Planning Cycle | Principles of Management View original

  Is this image relevant?

• 

  EEP impacts 1 - Framework for impact evaluation - HAMK Unlimited View original

  Is this image relevant?

• 

  Planning, monitoring and evaluation - supporting constructive change and social learning View original

  Is this image relevant?

• 

  The Planning Cycle | Principles of Management View original

  Is this image relevant?

1 of 3

• Survey design critically impacts impact evaluation by ensuring data validity and reliability
• Choose between quantitative and qualitative methods based on evaluation objectives and required information
• Develop sampling strategy to affect data representativeness and generalizability
• Select survey administration mode (face-to-face, telephone, online) considering response rates, data quality, and cost-effectiveness
• Address ethical considerations including informed consent and data privacy
• Conduct pilot testing to identify and rectify potential issues before full implementation
• Consider timing and frequency of survey administration to influence data quality and usefulness

Practical Implementation

• Tailor questions to target population's literacy level, cultural context, and subject familiarity
• Include filter questions to direct respondents to relevant follow-up questions, improving survey efficiency
• Implement skip logic and branching to reduce respondent fatigue and improve question relevance
• Provide clear instructions and definitions for complex terms to improve response accuracy
• Include attention check questions to identify and filter out low-quality responses
• Employ cognitive interviewing techniques during development to identify potential response errors
• Use visual aids (progress bars, estimated completion times) to manage expectations and reduce dropout rates

Outcome Measurement Questions

Question Types and Formats

• Craft closed-ended questions (multiple choice, Likert scales) for standardized, easily quantifiable responses
• Utilize open-ended questions for detailed and nuanced responses, considering analysis resources
• Avoid double-barreled questions to ensure clarity and accuracy in responses
• Select appropriate response scales (nominal, ordinal, interval, ratio) to capture intended measurement level
• Incorporate a mix of positively and negatively worded items to detect response sets and improve data quality
• Utilize reverse-coded items to detect and mitigate acquiescence bias (tendency to agree regardless of content)

Question Wording and Structure

• Prioritize clear and concise question wording to directly impact respondent comprehension and response accuracy
• Apply the funnel approach, moving from general to specific questions, to ease respondents into the survey
• Group related questions in thematic sections to enhance logical flow and coherence
• Strategically place sensitive or controversial questions later in the survey to build rapport
• Incorporate transition statements between sections to improve narrative flow and maintain interest
• Vary question types and formats throughout the survey to maintain engagement and reduce monotony
• Craft a clear, concise introduction explaining the survey's purpose to motivate high-quality responses

Minimizing Response Bias

Randomization and Balancing Techniques

• Randomize question order to reduce order effects and primacy/recency bias
• Balance the use of positively and negatively worded items to detect response sets
• Implement counterbalancing of question blocks to control for order effects across different survey versions
• Utilize randomized response techniques for sensitive questions to encourage honest responses
• Employ item randomization within matrix questions to reduce straight-lining behavior

Response Quality Improvement

• Use attention check questions to identify and filter out low-quality responses (inattentive or disengaged participants)
• Implement soft and hard validation checks to ensure data quality and consistency
• Utilize forced-choice questions to reduce satisficing behavior
• Employ branching logic to personalize question paths and improve response relevance
• Use hover-over definitions or tooltips for complex terms to ensure consistent understanding across respondents

Survey Structure and Engagement

Logical Flow and Organization

• Apply the funnel approach to question ordering (general to specific)
• Group related questions into thematic sections for improved coherence
• Place sensitive or controversial questions strategically later in the survey
• Incorporate clear transition statements between sections
• Implement a modular design to allow for flexible survey administration (core and optional modules)

Respondent Experience Enhancement

• Vary question types and formats to maintain engagement and reduce monotony
• Use visual aids (progress bars, completion time estimates) to manage expectations
• Craft a clear, concise introduction explaining the survey's purpose and importance
• Implement mobile-friendly design for surveys administered on smartphones or tablets
• Provide estimated time for completion at the beginning of the survey
• Offer breaks or save-and-resume functionality for longer surveys
• Use conditional display logic to show only relevant questions based on previous responses