Questionnaire Design & Research Ethics

Question 1

Tips for Improving Questionnaires (9 items)

Answer 1

1. Remember your survey’s purpose
2. If in doubt, throw it out
3. Keep your questions simple
4. Stay focused - avoid vague issues
5. If a question can be misinterpreted, it will be
6. Consider alternative ways to ask sensitive questions
7. Keep open-ended questions to a minimum
8. Consider a “don’t know” response
9. Provide a meaningful scale

Question 2

1. Remember your survey’s purpose

Answer 2

• All other rules and guidelines are based on this one
• There is a reason you decided to spend time and effort doing a survey and you should ensure that every question you ask supports that reason
• If you start to get lost while writing your questions, refer back to this rule
• If you notice a mistake in qualitative interviews, you can fix it in the next interview and it’s not a major problem
• For quantitative, you have to run the interviews in exactly the same way; so there’s no room for forgiveness and the interviews are already out there and happening
• Questionnaires need to be perfect before they go into the field
• Run a pilot study
• • Useful for every research endeavor
• • Even when repeating the research, run a pilot
• • If you make major changes, run a pilot study again
• • Repeat until you’re sure things are perfect
• Remove questions that aren’t relevant; if in doubt, remove

Question 3

2. If in doubt, throw it out

Answer 3

• This is another way of stating the first rule
• A question should never be included in a survey just because you can’t think of a good reason to discard it
• If you can’t come up with a concrete research benefit that will result from the question, don’t use it
• • May also distract people answering because they’ll start to focus on non-key questions

Question 4

3. Keep your questions simple

Answer 4

• Compound sentences force respondents to keep a lot of information in their heads and is likely to produce unpredictable results
• Complex questions should be broken down into component parts
• If you start writing into a second line, stop and reevaluate
• If questions are too short, people may be understanding in different ways, causing a mistake in your study; must find the right balance
• Don’t use academic language in your questionnaires; don’t use jargon
• • Questionnaires are not for your colleagues, they’re for the people you’re asking
• • Always depends on your audience

Question 5

4. Stay focused - avoid vague issues

Answer 5

• If you ask “please rate your satisfaction with the school’s discipline policy” the answers will not lead to any specific action steps
• Particular elements of the school’s discipline policy must be probed if responses are to result in specific recommendations

Question 6

5. If a question can be misinterpreted, it will be

Answer 6

• “What time do you normally eat dinner” will be answered differently by people living in different regions because it can refer to the midday or evening meal
• Be clear, concise, always be aware of imprecise language, and avoid double negatives

Question 7

6. Consider alternative ways to ask sensitive questions

Answer 7

• Income, drug or alcohol consumption, sexual habits, religious beliefs, and political views are obvious examples of sensitive topics
• Questions like “did you vote in the last election” forces respondents into a corner
• • People might be unwilling to admit they did not vote because of civic pride or embarrassment
• People can write answers, put in an envelope, and turn in answers anonymously

Question 8

7. Keep open-ended questions to a minimum

Answer 8

• While open-ended questions are a valuable tool, they should not be overused
• They can result in respondent fatigue where you’ll only get short answers
• These short answers can be avoided by using a set of well-designed, closed-ended questions
• Open-ended questions also pose problems in terms of coding and analysis
• Open-ended questions are often used out of laziness

Question 9

8. Consider a “don’t know” response

Answer 9

• It’s useful to allow people to say they simply don’t have an opinion about a topic
• Some researchers worry that people will opt for that choice, reducing the ability to analyze responses
• Evidence shows this fear is largely unfounded
• If you only want information from those with an informed opinion about an issue or interest in a topic, offer a “don’t know” choice

Question 10

9. Provide a meaningful scale

Answer 10

• Endpoints of the scale should be anchored with meaningful labels
• Number of scale points can have an effect on conclusions you draw later
• An odd number of points provides a middle alternative
• • Must define your midpoint
• With smaller samples, it’s better to have an even number of categories; forced to choose answers closer to one side

Question 11

Codes of Ethics (3 primary)

Answer 11

1. ESOMAR Guidelines (1948) ESOMAR
2. Standards and Best Practices (American Psychological Association)
3. Ethical Guidelines of Social Research Association

• Can use these as a textbook; god comprehensive info
• If you needed to study children, disabled, or elderly for example

Question 12

Researcher responsibilities (4)

Answer 12

1. To respondents and participants
2. To clients
3. To other researchers
4. To the public

Question 13

Researcher responsibilities

1. To respondents and participants

Answer 13

• You can’t force people to take part in research; if not willing you can only say ok and move on
• Respondent has a right to discard their questionnaire
• If there’s any recording from the interview, you must have written consent
• Must describe how you’ll handle data and that record
• Up to you to prove you have consent; might be sufficient to have the verbal ‘yes’ recorded; depends on different court rulings; should also have a copy of that consent for the respondent
• Better to have more than less
• If people filling in a questionnaire, in some cases it implies consent
• Just because you have consent, doesn’t mean you can do whatever you want
• Be sensitive, because it matters under what circumstances you ask the question
• You can’t be careless with data and records; if other people get access you are still responsible

Question 14

Researcher responsibilities

2. To clients

Answer 14

• Must specify your rights and obligations to other stakeholders; may have an impact on the clients
• • Clients may ask to provide primary records from respondents, but if you promise you’ll use responses only for analysis then you can’t
• Clients have a strong argument because they’re paying for you but your code of conduct is agreed between scientific institutions but client is not responsible to higher scientific associations
• A client once used focus group recordings in their ad; very unethical behavior

Question 15

Researcher responsibilities

3. To other researchers

Answer 15

• You can use scales that are already published

* Be careful when using questions from other surveys

Question 16

Researcher responsibilities

4. To the public

Answer 16

• You may be working for a client and they publish the data
• You may form a public opinion with your research; must be very careful
• Even if not your intention, you may still have consequences of your research and you are still responsible

Question 17

Transparency Initiatives

Answer 17

1. Who sponsored the research study and who
   conducted it. If different from the sponsor, the
   original sources of funding will also be disclosed.
2. The exact wording and presentation of
   questions and response options whose results are
   reported. This includes preceding interviewer or
   respondent instructions and any preceding
   questions that might reasonably be expected to
   influence responses to the reported results.
3. A definition of the population under study and
   its geographic location.
4. Dates of data collection.
5. A description of the sampling frame(s) and
   its coverage of the target population, including
   mention of any segment of the target population
   that is not covered by the design.
6. The name of the sample supplier, if the
   sampling frame and/or the sample itself was
   provided by a third party.
7. The methods used to recruit the panel or participants, if the sample was drawn from a pre-recruited panel or pool of respondents.
8. A description of the sample design, giving a
   clear indication of the method by which the
   respondents were selected, recruited, along with
   any eligibility requirements and/or oversampling.
9. Method(s) and mode(s) used to administer
   the survey (e.g. CATI, CAPI, mail survey, web
   survey) and the language(s) offered.
10. Sample sizes and a discussion of the
    precision of the findings.
11. A description of how the weights were
    calculated, including the variables used and the
    sources of weighting parameters, if weighted
    estimates are reported.
12. Contact for obtaining more information
    about the study.

Question 18

1. Who sponsored the research study and who
   conducted it. If different from the sponsor, the
   original sources of funding will also be disclosed.

Answer 18

• Should be the actual name of a company or benefactor

* You must name names

Question 19

2. The exact wording and presentation of
   questions and response options whose results are
   reported. This includes preceding interviewer or
   respondent instructions and any preceding
   questions that might reasonably be expected to
   influence responses to the reported results.

Answer 19

• Not necessarily whole questionnaire but should publish the ones used for published research
• Question wording really matters so it’s important for correct and transparent interpretation to have the original questions as they were phrased to participants

Question 20

3. A definition of the population under study and

its geographic location.

Answer 20

• What were your goals?
• Where was this conducted and why?
• How could location have influenced your results?

Question 21

4. Dates of data collection.

Answer 21

• Important for voting behavior

* Nice to know for interpretation or observation of rate at which field work is complete

Question 22

5. A description of the sampling frame(s) and
   its coverage of the target population, including
   mention of any segment of the target population
   that is not covered by the design.

Answer 22

• Most of the public won’t understand but that doesn’t matter
• Important for transparency regardless and good for people who do understand

Question 23

6. The name of the sample supplier, if the
   sampling frame and/or the sample itself was
   provided by a third party.

Answer 23

• Could be handled by smaller agencies

Question 24

7. The methods used to recruit the panel or participants, if the sample was drawn from a pre-recruited panel or pool of respondents.

Answer 24

• Not representing general public well because they’re part of a panel or were recruited for some reason
• Internet provider company did panel study of preferred internet provider on group of people who used their internet (lolz); biased research

Question 25

8. A description of the sample design, giving a
   clear indication of the method by which the
   respondents were selected, recruited, along with
   any eligibility requirements and/or oversampling.

Answer 25

• How did you select your respondents?

* Why was this study designed the way it was designed?

Question 26

9. Method(s) and mode(s) used to administer
   the survey (e.g. CATI, CAPI, mail survey, web
   survey) and the language(s) offered.

Answer 26

• Another straightforward aspect of research design

Question 27

10. Sample sizes and a discussion of the

precision of the findings.

Answer 27

• Summary of number of respondents and analysis of findings

Question 28

11. A description of how the weights were
    calculated, including the variables used and the
    sources of weighting parameters, if weighted
    estimates are reported.

Answer 28

• Calculations, etc.

Question 29

12. Contact for obtaining more information

about the study.

Answer 29

• Researchers who participated, created, wrote, and published this research