Measuring Sensitive Topics

Question 1

What makes a question sensitive?

Answer 1

• Depends on culture and context
– Information related to identity
– Illegal activities
– Socially unacceptable behaviour

Question 2

Why may respondents not answer truthfully?

Answer 2

– Social desirability bias – Respondents may feel embarrassed
– Strategic: respondent perceives their answer may influence a later service they receive
– Other

Question 3

3 things to measure sensitive questions

Answer 3

• Asking the question directly
• Indirect techniques
• Understanding social norms through direct and indirect techniques

Question 4

Direct Techniques: Privacy

Answer 4

• ensure people’s privacy
• explain the confidentiality of all the information given
• have a very good and strong data security plan afterwards, where individual identifying information is stripped out of data and kept separately and encrypted

Question 5

Direct Techniques - Ways to Collect Data

Answer 5

Online surveys reduce desire to not say these things face-to-face to someone else

And/or self-administered modules

Question 6

Framing the question - deliberate loading

Answer 6

a term used for asking a question in a way that suggests that maybe isn’t so bad to do something, whereas otherwise the social norm might say it’s not so good to do that.

the basic idea is to set the question in a context where the behavior is acceptable

Question 7

Placement of questions - tips

Answer 7

Avoid placing sensitive questions too early or too late in the survey
• Too early: May influence other questions, may cut the
interview short
• Too late: Don’t do it last. May leave the respondent feeling bad.
• Remember the rule: do no harm, and (try to) leave the world untouched.
• But, later in the survey likely best, just not last. If it does cut the interview short, you lose less.

Question 8

Surveyor quality in asking sensitive questions

Answer 8

• Hire appropriate surveyors, and train well
• Not personally known to the respondent
• Matching surveyor to respondent may matter (i.e. by gender)

Question 9

List Randomization

Answer 9

What list randomization does is allows you to ascertain for a group of people (not individual)… what proportion of them are saying yes to a specific yes/no question.

One group is asked 3 innocuous questions, other group is asked 3 innocuous questions and 1 sensitive question - you add up the number of yeses and compare the two groups. You don’t know WHICH questions they answered yes to - it just gives the individual a way of hiding their answer in a way that’s very salient and obvious that they’re not giving you the answer to every single question.

Question 10

random response method

Answer 10

Pair a sensitive question along with a neutral question with a known distribution
• Use chance to dictate which question the respondent
answers (i.e. flip a coin, if you get heads you just say yes, and if you get tails, you answer truthfully)
• Surveyor does not know whether the sensitive question was answered or not, just the response option - improves chances of truthful response
• The two questions must have the same type or number of responses (like Yes-No)

Question 11

polling booth survey

Answer 11

You can literally set up a little box in the same way that voting works in the United States, and you can ask them to fill out that form, and go put their form in the box. And on that form, you have no individual identifying information– no name, nothing–just the sensitive question.

And then they put it in and now you know the answer on average for everybody in that community and the distribution. But you don’t know any one particular person’s information

Question 12

Revealed Preference Approach

Answer 12

Measures risk aversion, time preferences, political attitudes, etc.

Revealing peoples preferences about something that has monetary stakes or something real in the outside world

(If you do this, I’ll give you $5…) And then you can also change that amount of money that is offered and see how much money does it need to be before people are saying they will reject the offer?

Question 13

Mystery clients

Answer 13

Mostly used for assessing firm behavior

here, an enumerator visits a location and basically pretends to be a typical client or a citizen, observes the quality of services received, and any other indicator in interest.

And this is often useful in measuring antisocial or illegal activities, or discrimination, corruption, and has been used in many different domains, discrimination being one of the main domains. This is a bit more labor intensive and requires a lot of training of surveyors to make sure that they are hopefully behaving the same way, regardless of who they’re visiting and regardless of who they are. And then you’re testing how they are treated. And you’re maybe comparing, across your surveyors, how they get treated.

Question 14

Incognito enumerators (ride along)

Answer 14

again, it is not appropriate in a lot of contexts but sometimes, the cost of just actually having an enumerator be there while the world is happening, recording what is happening, and recording what they are observing might actually be the best way to observe true behavior

Question 15

Certain modes of data collection such as having a respondent record their answer themselves on a tablet can help us collect sensitive information principally by:

Answer 15

Saving the respondent the embarrassment of telling a human

Question 16

Which of the following is the best description of “framing” sensitive questions to make them appear less sensitive?

Answer 16

Load the question with qualifying language suggesting that the “undesirable answer” is not wrong or unusual

Question 17

List randomization protects respondents from revealing sensitive information about themselves because:

Answer 17

We can not discern whether the respondent is admitting to objectionable actions or to other actions in the list correct

Question 18

Random response is similar to list randomization because

Answer 18

In both, the surveyor is unaware of whether the respondent’s response relates to the sensitive question(s) or the “placebo” (non-sensitive) question(s).

In both, we estimate aggregate proportions of who engaged in the “socially undesirable” behavior correct

Question 19

For which of the indirect techniques is it possible to identify each respondent’s individual response to the sensitive question we care about?

Answer 19

In revealed preference, we observe whether the person is willing to “endorse” a particular group (for example), based on some related decision they make.

Question 20

Mystery clients and incognito enumerators are similar in that both (

Answer 20

Record observations of behaviors and actions
Require some deception

Both mystery clients and incognito enumerators record observations and conceal their identity as surveyors. However, neither are known to be conducting a survey. And only mystery clients engage directly with the “respondent”.

Question 21

With a list randomization, respondents who received the list without a sensitive question reported that on average 6.1 statements on the list were true. Those with the “undesirable component” on the list answered that 6.3 statements were true. Based on this, what proportion of respondents likely engaged in the undesirable behavior?

Answer 21

20 percent

Notice that the question states “respondents who received the list without a sensitive question” this means there was only one (1) sensitive question asked. Those in the group without the sensitive question listed 6.1 statements as true on average whereas those with the sensitive question listed 6.3 as true.To calculate the proportion of students who engaged in undesirable behavior we need: a) First to calculate the difference between the group who received the extra sensitive question and the group that didn’t’ since this would give us the average of people who answered yes to the sensitive question. That gives us 6.3-6.1=0.2. b) To get the ratio (proportion) we then divide this first difference by the difference between the number of questions asked to the group with sensitive questions and the group without the sensitive question. This difference corresponds to the number of extra questions the group with sensitive questions received. We know it is 1. We then have (6.3-6.1)/1=0.2 or 20%.

Question 22

Respondents were told to roll a six-sided die. If they rolled a 1, they were to answer a sensitive question truthfully. If they rolled 2-6, they were to answer yes, regardless of their actual behavior. 90 percent of people responded yes.

Assuming respondents correctly followed this protocol, what proportion actually engaged in the undesirable behavior?

Answer 22

40 percent

We know that all those that rolled 2-6 had to answer yes regardless of their actual behavior, this represents 5/6 of all respondents or around 0.83. We know 1/6 of all respondents answered truthfully about their behavior (either yes or no): this represents around 0.17. We know that 90% (0.9) of all respondents answered yes. We know that around 0.9-0.83= 0.07 of all respondents answered yes and engaged in the undesirable behavior. What we don’t know is what this 0.07 represents as a share of those who answered truthfully (0.17) about engaging in undesirable behavior. The calculation is then 0.07/0.17 which gives us around 0.4, i.e. 40% of those who answered truthfully answered yes (engaged in undesirable behaviors).

Question 23

In which of the methods are (at least some of the) respondents asked about a sensitive behavior or attitude?

List randomization
Random Response
Polling booth surveys
Revealed preference
Incognito enumerators

Answer 23

List randomization
Random Response
Polling booth surveys

With Revealed preference, respondents reveal whether they are willing to “endorse” a socially undesirable group, but they do not necessarily know they are being asked about that endorsement. With Mystery clients and incognito enumerators, the undesirable behavior is observed rather than asked.