Surveys

Question 1

How do I generate items for a survey?

Answer 1

1. Qualitative exploration
2. Question Writing
3. Pre-testing

Question 2

What is the aim of qualitative exploration?

Answer 2

Uncover attitudes of respondents that deserve further exploration.
Participants representing survey population.
Purposeful sampling.

Question 3

What are the keys to success in question writing?

Answer 3

ability to anticipate:

• the cognitive processes of respondents
• the likely distribution of responses under different conditions
• the analytical work that might be performed on the reponses
• The “so what” / decision value attached to the statistical result

Question 4

What is the aim of pre-testing responses?

Answer 4

Uncover ambiguities, confusions, inappropriate assumptions.

Question 5

What is the general checklist for writing a survey?

Answer 5

1. Langauge.
   Simple, familiar.
   Avoid abbreviations and slang.
2. Statements should be short and contain only one complete construct.
3. Avoid
   - vague concepts, or provide examples when needed.
   - complicated syntax.
   - double negatives.
   - universals (all, always, none, never)
   - “only”
   - “charged” words like conservative/liberal.
   - Statements that are irrelevant
   - Statements likely to be endorsed by almost everyone or no one
4. Structure
   Group questions by category or objective.
   Group questions that are similar in structure.
   Put the most interesting questions at the beginning, save personal or problematic questions for last.

Question 6

Advantages and different types of questions

Answer 6

Open-ended: Hardest to analyse, selection bias
Yes/No: Miss the shades.
Scales/Rankings: Require more work in creation

Question 7

Open vs Close Ended Questions

Answer 7

Open-ended better if:

• the researcher wants to learn more.
• Responders must be more motivated.

Closed-ended better if:

• Classification of behaviour
• Researcher has good knowledge of existing frame of reference.
• Already coded
• Responses expected to require psychological effort

Question 8

Validity in Surveys & Solutions

Answer 8

– Validity of Findings:

Internal Validity: Theory is consistent
main risk: omitted variable bias.
- actively try to conceptualise and measure possible confounds.

External Validity: Generalise outside your sample

• Deliberate sampling for heterogeneity
• Replicate on independent samples

– Validity of Measures

//Conclusion Validity:
main risk: lack of reliability of measure across time.
- Use different measures, use multi-scale items, (calculating Cronbach’s alpha)
- compare across settings

Threat: Common method bias

//Construct Validity:
—Do measures represent the constructs?
• Before data collection:
- search for already validated measures
- Measure potential confounds
- Use multiple measures per construct (convergent validity)
• After data collection:
- Control for multicollinearity
- Use factor analytic procedures to validate scales (convergent
validity)
- Highlight overlap of constructs and measures

—Is the sample random?
• Avoid convenience sampling
• Prevent self-selection

—Do respondents represent the group from which they are sampled?
• Report response rate
• Actively try to increase response rate
• Compare descriptive demographic data and observable
characteristics

—Do non-respondents (or late respondents) differ significantly from
respondents?
• Identify measures that are available for the whole population and
include them in the questionnaire
• Control for significant differences on potentially confounding
variables (e.g., t-tests)

Question 9

What is the common method bias?

Answer 9

Is when variance is attributable to the measurement method rather than to the constructs the measure represents.

Potential Sources of Common method bias:

• Common rater effect (social desirability bias, acquiescence biases)
• Item characteristic effect (ambiguity, wording)
• Item context effect (context induced mood)
• Measurement context effect (association of X and Y because they are measured in the same way)

Solution:

• Triangulate with different data sources (, times, places)
• Convince subjects of confidentiality, importance of response and subjectivity of answer.
• Randomly re-order scales.
• Discard subjects who display systematic biases.