Survey Design (Reliability & Participant Biases)

Question 1

Identify:

The TWO main categories of considerations when designing a psychological survey?

Answer 1

Participant Characteristics

&

Survey Characteristics

Question 2

Define:

Scale items

(In the context of surveys/questionnaires)

Answer 2

The various questions/statements used to measure the same construct.

Question 3

Define:

Scale anchors

Answer 3

Labels used to provide meaning to numerical or point-based rating systems.

(e.g. in a Likert scale, ‘1’ might be ‘Strongly Disagree’ and ‘5’ might be ‘Strongly Agree’)

Question 4

Define:

‘Ceiling & Floor Effects’

Answer 4

Phenomena in the data that occurs when it is respectively too easy to score highly or score lowly on your test/scale, resulting in skewed results.

(i.e. most data-points at the high end or low end of the spectrum respectively)

Question 5

What two methods help you avoid ceiling and/or floor effects in a survey?

Answer 5

1. Using clear and strong question phrasing.
2. Giving scale anchors to your scale.

Stronger questions (e.g. “I have a lot of admiration for celebrities”) help prevent making it too easy to score highly in either direction (i.e. Agree vs Disagree).

Scale anchors ensure participants know how extreme/what each end of the scale in a rating system represents (e.g. ‘Strongly Agree’ vs. ‘Agree’)

Question 6

Fill-in-the-Blank:

Use ____ ____ to ensure there is directionality to participants’ answers.

Answer 6

clear wording

Ambiguous phrasing can lead to participants interpretting questions in highly varied ways to each other, and therefore no longer providing meaningful responses for the intended/same construct your research is interested in

Question 7

List:

THREE key categories of response biases

(In surveys/questionnaires)

Answer 7

1. Demand characteristics & social desirability.
2. Acuiescence bias.
3. Priming effects.

In order to combat these, it is vital to consider the psychology of participants.

Question 8

How do ‘demand characteristics’ and ‘social desirability’ affect participants?

Answer 8

These sources of response bias cause some participants to behave/respond during the research in ways that they think…

1. …the researcher wants/hypothesises
2. …are socially acceptable/favoured

This can be mitigated by research design components such as ‘double-blind testing, ensuring participant confidentiality/anonymity, keeping the original and exact hypothesis obscured, etc.

It is also important to check your final data against other related external data as this can help identify discrepencies indicating any bias that may have occurred.

Question 9

Define:

Convergent validity

Answer 9

This arises when data and findings from one study/research is cohesive or correlates with other external studies dealing with the same concept(s)/construct(s)

Lack of convergence can sometimes highlight bias that has occurred (e.g. response bias from participants).

Question 10

Explain:

The Bogus Pipeline methodology and what its purpose is in research design

Answer 10

This is where you hook participants up to a fake ‘lie detector’ (i.e. ‘physical-arousal monitor’) in order to get them to be more concerned with being seen to tell truthful/honest answers rather than answers that align with demand characteristics and social desirability.

(i.e. they will hypothetically be more concerned about being perceived as dishonest than going against what they think the researcher hypothesises and/or what they deem ‘socially acceptable’)

Question 11

Fill-in-the-Blank:

One way to combat effects of ‘social desirability’ in your survey is to increase the ‘____’ of questions, and design them to have ____ ‘____’ ____.

Answer 11

• ‘ambiguity’
• no ‘correct’ answers

Question 12

Define:

Double-Blind Testing/Studies

Answer 12

A study where key information is obscured from both the participants and the researcher(s).

Key information could be whether or not a participant is part of the ‘control’ or ‘treatment’ group.

By obscuring this from the participant, you reduce the likelihood of various biases/confounding factors such as ‘placebo effects’ on their part.

Then, by obscuring this information from the researcher too, you simultaneously reduce the probability of bias on their behalf during aspects of the research such as analysing the data (i.e. so they do not allow their desire to prove/disprove ideas to overly influence their recognition of patterns and trends in the data).

Question 13

Define:

Acquiescence Bias

Answer 13

This is the tendency of participants to be more likely to agree with research statements, even if it does not accurately or fully reflect their true thoughts and beliefs.

(or ‘agree more strongly’)

Note: This effect tends to be greater in collectivist societies/groups.

In rarer instances, participants may be more likely to respond negatively/with disagreement, particularly if they distrust the researcher/intentions of the study.

Question 14

What are TWO key ways to minimise acquiescence bias?

Answer 14

1. Create many items to later average into one scale
2. Use reverse-wording

Reverse-wording is more effective when you avoid simply using the same item with ‘never’/’not’ simply added to it.

An example of reverse-wording for a scale measuring extroversion vs. introversion may be “I need to be around others to feel energised” and “Being around large groups of people drains my energy”.

Question 15

What does Cronbach’s Alpha measure and represent?

Answer 15

It measures the internal validity and represents the reliability of a scale in a survey.

(i.e. the extent to which different items of a scale measure the same construct in a consistent way, or how closely related they are).

Items with ‘reverse-wording’ are factored by sorting them into the ‘reverse-scaled items box’.

Question 16

Fill-in-the-Blank:

A Cronbach’s Alpha test value ranges from ____ to ____, and the ____ the value, the more the items of a scale are ____ ____ and consistently measure a particular construct.

Answer 16

1. 0.00
2. 1.00
3. higher
4. closely related

However, getting a value of exactly 1.00 should ring alarm bells - as it is highly unlikely for you to get perfect consistency without having made a mistake somewhere!

In Psychological studies, generally:
- >.70 = acceptable
- >.80 = good
- >.90 = excellent

Question 17

Define:

Priming Effects

(In Psychological Scientific research/surveys)

Answer 17

The phenomena where exposure to and the answering of earlier questions in the survey influences answers to later questions.

There a wide variety of priming effects, such as ‘stereotype threat’ and ‘affective priming’.

Question 18

Define:

Stereotype Threat

(In Psychological Scientific research/surveys)

Answer 18

A type of priming effect whereby participants’ answers are influenced by questions early in the survey relating to demographics/identity.

An influence on later answers may be the conscious/unconscious belief they will be ‘stereotyped’ based on their particular demographic (i.e. gender, ethnicity, age, political-stances, etc.).

The priming effects of ‘stererotype threat’ can be mitigated somewhat by putting questions about demographics towards/at the end of the survey instead.

Question 19

Define:

Affective Priming

(In Psychological Scientific research/surveys)

Answer 19

The use of strong/emotive language in questions leading to participants having stronger emotional affects and thus bias towards these in their answers.

This priming effect can be seen carried into later questions that don’t even have as ‘emotively-charged’ language in them.

Ways to lessen the likelihood of affective priming biasing the results of a survey are to…

• Put more emotive/impactful questions towards the end.
• Break up these questions with ‘distraction tasks’.
• Randomise question order.
  (However, this will not eliminiate the error from affective priming, but rather average it across participants).

Question 20

Fill-in-the-Blank:

Some biases may increase the ____ but always lower the ____ of data and results.

(In Psychological Scientific research/surveys)

Answer 20

1. reliability/consistency
2. validity

Question 21

What THREE examples of errors in survey design can lead to your results being completely invalid?

Answer 21

1. Scale-Labelling Mistakes
2. Double-Negative/Barrelled Questions
3. Leading Questions

Scale-Labelling errors could include overlapping measures/numbers such as ‘0-3 hrs’, ‘3-6 hrs’ and ‘6+ hrs’ (which would mean participants could choose multiple answers since they fall into more than one category etc.).

Double-Negatives make questions difficult to understand (and therefore answer!), whilst double-barrelled questions can make it confusing as to which part of the question you should prioritise in your answer/response.