Q5 Surveys

Question 1

Two ways that surveys breed skepticism

Answer 1

• Aren’t surveys often wrong?
• How can a small sample represent an entire nation?

Question 2

How can a small sample be fine?

Answer 2

If the sample is representative and diverse, since size is less important than diversity when it comes to a representative sample

Question 3

Is a survey different from a poll?

Answer 3

To us, surveys and polls are identical because if they’re good then they’re done the same way

Question 4

Four key principles to lessen interpretation errors

Answer 4

• Surveys are snapshots, not predictions (can only caption a person’s current thinking)
• Opinions can change
• A recent, high-profile event can influence survey responses
• Survey results are in a range, not a point

Question 5

How are survey results in a range?

Answer 5

The percentages given as survey results are the center of the range, and extend + or - however many percentage points in the sampling error

Question 6

What do presidential polls measure?

Answer 6

National preferences, not electoral votes

Question 7

Why aren’t presidential polls done for each state?

Answer 7

• It’s too expensive
• We use Electoral College so small states can have more clout than they would have if each person’s vote was counted equally

Question 8

How to get a higher confidence level

Answer 8

The range must be wider

Question 9

How to get a smaller range

Answer 9

Lower the confidence level

Question 10

Confidence level

Answer 10

The expectation that repeated surveys will be within the sampling error

Question 11

Sampling error

Answer 11

The expected variance between a population and a representative sample

Question 12

If you double the size of a sample of 1,200, and the sampling error is + or - 3 percentage points, what would the new sampling error be?

Answer 12

• Only 1 percentage point higher because of standard deviation
• Reducing sampling error by 1 percentage point requires a big, expensive increase in sample size

Question 13

Sample subgroups

Answer 13

• Carry larger error margins
• A larger sample is needed if a subgroup analysis is important

Question 14

Questions we should always ask

Answer 14

• Who did the survey?
• Who paid for the survey?
• Which survey mode was used?

Question 15

How can you tell which organizations doing surveys are good?

Answer 15

• Should be transparent about its method and survey details (response rate, why some respondents were removed)
• Should publicly disclose all details about questions and answers (number of responses, whether it was randomized)

Question 16

When does the issue of payment surface?

Answer 16

Whenever an advocacy group pays for a survey

Question 17

Does the involvement of an advocacy group automatically invalidate the survey?

Answer 17

No, because if it’s done properly, then the survey can still be reliable

Question 18

How are advocacy groups often selective?

Answer 18

Tend to release only results that support their position

Question 19

How to make sure an advocacy groups’ survey is valid

Answer 19

See the entire survey and all the details about the methodology, the questions and answers

Question 20

Survey modes

Answer 20

Written
- Mail
- In person
- Online panel
Interview
- Telephone

Question 21

Mail survey mode

Answer 21

Cheap, but few respond

Question 22

In-person survey mode

Answer 22

Reach is hard to find, and costly

Question 23

Online panel survey mode

Answer 23

• Efficient, but answers could be insincere and have straightlining
• Not available to offline people

Question 24

Telephone interview survey mode

Answer 24

• Effective, but hard to get responses
• Proven to be reliable and likely to elicit more sincere answers
• Cost is an issue

Question 25

Does written or interview surveys have an advantage for sensitive subjects?

Answer 25

- Written has an advantage because it's best for subjective subjects - People are more candid in writing and more guarded when speaking with a human being

Question 26

Random digit dial

Answer 26

Draws a representative sample by dialing phone numbers at random

Question 27

What is the problem with telephone interviews today?

Answer 27

Fewer people these days are willing to answer telephone surveys (lower response rate)

Question 28

Response rate

Answer 28

The percentage of valid potential respondents who agree to take a survey

Question 29

What is the problem with lower response rate?

Answer 29

- Raises costs because more people must be called to get 1 response - Invokes non-response bias

Question 30

Non-response bias

Answer 30

- Occurs when those who choose to respond to a survey differ in a meaningful way from those who decline - Potential risk, but not yet an issue

Question 31

Best alternative for telephone surveys

Answer 31

Online panels

Question 32

Some organizations offer polls on their websites or social media feeds

Answer 32

- Don't represent the people who visit the websites, let alone the public - Rely on non-representative samples, so they don't matter

Question 33

How does a valid online panel survey draw a representative sample?

Answer 33

Randomly chooses among panelists

Question 34

Types of panels

Answer 34

- Volunteer, opt-in panel - Invitation

Question 35

Volunteer panel

Answer 35

Most common because it's the cheapest

Question 36

Invitation panel

Answer 36

Better to invite panelists at random by sending letters or dialing phones

Question 37

How are survey panelist participants categorized?

Answer 37

By demographics to reflect the population, then assigned at random to different surveys

Question 38

Straightlining

Answer 38

- When respondents speed through a survey without giving sincere answers - Greater for opt-in panels and lower for invitation panels

Question 39

What to do if the sample rate is only half the population rate

Answer 39

Assign the survey answers from those respondents twice the weight

Question 40

Answers from overrepresented groups

Answer 40

Would be given proportionally less weight

Question 41

Weighting

Answer 41

Adjusting the value of answers if the sample demographics differ substantially from the population

Question 42

Social desirability bias

Answer 42

Tendency to give the "correct" answer to questions involving civic or social responsibility

Question 43

Instead of asking "Do you plan to vote in the upcoming election?" what is better?

Answer 43

- "Identify your polling place" - If they name it quickly, its evidence they do vote - Does not work in states like FL with voting by mail

Question 44

Pollsters use lists of registered voters to know if they're likely to vote in the upcoming election

Answer 44

Does not remove social desirability bias, but it at least narrows the population by excluding those not registered

Question 45

A slight change in wording in questions

Answer 45

Can change results, such as saying "forbid" versus "allow"

Question 46

How does offering a choice in a survey matter?

Answer 46

- We are wired to say yes and want to be agreeable when given a "yes" or "no" question - Asking for a preference produces a more valid answer

Question 47

Acquiescence bias

Answer 47

A tendency to select a favorable response or agree when in doubt

Question 48

How can different surveys, done properly, produce different results with different question order?

Answer 48

Asking other questions before can make people change their opinion on the core question

Question 49

Priming

Answer 49

- When exposure to a stimulus unconsciously influences a response - Not neutral!

Question 50

Answer order in writing

Answer 50

The first one written will be chosen the most often

Question 51

Answer order spoken

Answer 51

The last one spoken will be chosen most often

Question 52

Primacy effect

Answer 52

- A tendency to choose or remember items offered first - Advantage seen in written surveys

Question 53

Recency effect

Answer 53

- A tendency to choose or remember items offered last - Advantage seen in oral surveys

Question 54

How can a telephone oral survey make sure no one answer choice has an advantage?

Answer 54

On the next call, move the last question to be the first, and so forth for the next calls

Question 55

Likert scale

Answer 55

A consistent sequence that always begins with Agree option, a Neutral center, and ends with Disagree

Question 56

Closed-ended vs. open-ended answer choices

Answer 56

Open-ended answers better reflects true public opinion but cannot make all surveys open-ended because it's slow and costly

Question 57

Adding in an answer choice saying the respondent "doesn't know enough"

Answer 57

- When giving the option to admit lacking information, social desirability bias drops out - Majority usually chooses it