Brainscape's Knowledge GenomeTM

Browse over 1 million classes created by top students, professors, publishers, and experts.


See full index

Research 101 > Survey Methodology > Flashcards

Survey Methodology Flashcards

You may prefer our related Brainscape-certified flashcards:
Physics 101 flashcards Psychology 101 flashcards Chemistry 101 flashcards Biology 101 flashcards Organic Chemistry 101 flashcards Sociology 101 flashcards
1
Q

Why is questionnaire pretesting important?

A
  • To reduce measurement error and increase validity

- An opportunity to cross-check typical assumptions researchers make when drafting a survey

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
2
Q

Examples of assumptions we make when designing questionnaires (4)

A
  • Questions asked are appropriate and relevant, and have answers
  • Questions are posed so that respondents understand what is requested
  • Respondents can retrieve any information requested from memory
  • Respondents can communicate their information using the response options provided
How well did you know this?
1
Not at all
2
3
4
5
Perfectly
3
Q

Types of measurement errors

A
  • Validity error
  • Measurement error
  • Processing error
How well did you know this?
1
Not at all
2
3
4
5
Perfectly
4
Q

Types of representation errors

A
  • Coverage error
  • Sampling error
  • Non-response error
  • Adjustment error
How well did you know this?
1
Not at all
2
3
4
5
Perfectly
5
Q

Validity error

A

When the measurement doesn’t properly capture the construct of interest

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
6
Q

Frame error

A

Frame error typically results from the frame construction process. For example, some units may be omitted or duplicated an unknown number of times, or some ineligible units may be included on the frame, such as businesses that are not farms in a farm survey.

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
7
Q

Non-response error

A

Nonresponse error occurs at both the survey unit and participant level when some people in the sample are not interviewed, or skip certain sections.

Example: Certain types of respondents refusing to participate in the study.

If there is a difference between respondents and non-respondents, or if the response rate is low, this becomes a big issue.

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
8
Q

Measurement error

A

Occurs when respondents are not providing answers that they should, given researcher’s intentions.

Example: People will answer questions even when they don’t know the meaning the terms or question wording.

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
9
Q

Sampling error

A

Occurs when only a subset of the population is included in a survey and sampled units differ from the full population.

Example: non-probability based sampling introduces potential biases because the sampling error cannot be estimated.

You end up uncertain about how representative your sample is of your population.

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
10
Q

Processing error

A

processing error refers to errors that arise during the data processing stage, including errors in the editing of the data, data encoding, the assignment of survey weights, and tabulation of the survey data.

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
11
Q

Coverage error

A

Coverage error results when there are differences between the target population and the sample frame.

Undercoverage may occur if not all voters are listed in the phone directory. Overcoverage could occur if some voters have more than one listed phone number. Bias could also occur if some phone numbers listed in the directory do not belong to registered voters

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
12
Q

Sampling frame

A

Subset of population, taken from a sampling frame or source of people

Usually smaller than the population since there usually isn’t a list available of all population members. I.e., the difference between all Facebook users and those using the Facebook web interface.

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
13
Q

Sample

A

People who you expose the survey to

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
14
Q

Respondents

A

The people who answer your survey

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
15
Q

Probability-based sampling techniques (examples)

A

Intercept surveys
List-based samples (e.g., email invitations)
Pre-recruited probability-based panels of Internet users

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
16
Q

Non-probability based sampling techniques (examples)

A

Snowball sampling
Convenience samples (e.g., students on campus)
Volunteer opt-in panels
Links on blogs, social networks

17
Q

Margin of error

A

The uncertainty around an estimate of a population measure

Expresses the amount of sampling error in the survey

Example: 60% parameter, 5% MOE, actually 55-65%

18
Q

Confidence level

A

The likelihood the metric falls within the margin of error if repeated

A 95% confidence level means that if we were to replicate a survey 100 times, 95% of all sample parameters would fall within the MOE interval.

19
Q

Advantages to Internet Surveys

A

Advantages

  • Easy access to large audience
  • Cost savings (invitations and analysis)
  • Shorter fielding periods
  • Lower bias due to respondent anonymity
  • Ability to customize questionnaire
20
Q

Disadvantages to Internet Surveys

A

Disadvantages

  • Coverage error (people without Internet?)
  • Lower motivation to respond (satisficing)
  • Computer skills required
  • Little non-inferred information about the respondent
21
Q

Common Survey Biases

A
  • Satisficing
  • Acquiesence Bias
  • Response Order Bias
  • Social Desirability Bias
  • Leading Information
  • Priming effect (use a funnel)
  • Move sensitive questions to the end
22
Q

Why weight responses?

A

Even when you have probability-based samples, there can be errors that you need to account for.

For example, you may encounter error with non-response bias within a probability sample, which becomes an issue if respondents substantively differ from non-respondents, or if the response rate is low.

23
Q

Three steps to weighting

A
  1. Design weights to account for differential selection probabilities
  2. Subweights to account for nonresponse and coverage errors
  3. Calibration weights to borrow power from external sources
24
Q

Design weights

A

You use this in order to help adjust for the fact that you may need to oversample certain sub-populations that are smaller (e.g., coverage error).

For example, you oversample states with lower populations in order to achieve an adequate sample.

That said, you need to apply design weights in order to generalize your findings to the population.

Design weights in this case would be the inverse of the selection probability.

25
Q

Subweights for nonresponse

A

You use this in order to adjust for non-response bias.

For example - a survey incentive is $20. People making $10/hr are more likely to complete it than people making $100/hr.

The subweight applied would be the inverse of the segments response rate.

26
Q

Calibration

A

Sometimes design weights and subweights are all you need to correct for representativeness.

Sometimes you want to further calibrate your sample based on information known about the true value of the population (say, with the census).

So say 5% of the sample (post design weights and subweights) is 18-24 female, and the census population value is 6%, you’d multiple by 6/5 or 1.2 to participants in this group.

27
Q

Pros and Cons to Weights

A

Weighting multiple times can reduce bias (get us closer to the real world actuals), but this also tends to introduce more variance.

Unweighted samples carry more bias but less variance. Overweighted samples carry less bias but more variance. Ideally, you want to apply the optimal combination of weights such that you minimize the total error (i.e., total of bias and variance).

Also, do avoid any weights are extreme e.g., 1/99 or 99/1. You can help to avoid this by bounding or trimming weights to a reasonable range.

Limit the number of calibrations you use (e.g., the number of characteristics you account for) and only use reliable official information to calibrate.

28
Q

Paradata

A

Data that is collected about respondents who engaged with the survey. Examples of paradata include response time, mode of completion, number of invitations sent to participant.

29
Q

Ways to pretest a survey

A
  • Cognitive interviewing
  • Focus groups
  • Pilot tests (sending out to a small subset, ask questions at end)
  • Vignettes
  • Paradata analysis (e.g., task time, unit non-response)
  • Comparing with big data (e.g., from sensors, logs data), as well as with other forms of research
30
Q

Cognitive Pretesting

A

Interview-based administration of the survey with 1-2 participants where you can them to follow a concurrent think-aloud protocol.

  • What does this question mean
  • What’s the thought process behind your answer?
  • Question interpretation
  • Construct validity
  • Terminology comprehension
  • Missing answer options
31
Q

Data Cleaning Practices

A
  • Remove satisficers based on parametric data
  • Straighlighting across questions
  • Gibberish open-ended responses
  • Treating Outliers
32
Q

Metrics to evaluate response quality

A
  • Click through rate (from invitation)
  • Completion rate (% completing of those who started)
  • Response rate (% completing of those invited)
  • Completion rate and time
  • Break-off rate (of those who started survey, where did they drop off)
33
Q

Red flags in response quality metrics

A
  • High response rate - people outside intended sample received survey)
  • Low response rate - issue with invitation, incentives
  • Low completion rate - fatigue
  • High completion rate - satisficing
  • Break off rates - problematic units/questions
34
Q

Survey Internationalization Steps

A

Translate questions and answers carefully

Cultural response effects
Interpretation effects
Product differences
Userbase differences
Competitive marketplace
Nonresponse variance/effects

Research 101 (10 decks)

Brainscape helps you reach your goals faster, through stronger study habits.
© 2024 Bold Learning Solutions. Terms and Conditions