Week 4: Survey Methods

Question 1

What is descriptive research?

Answer 1

• Descriptive research: description of individual variables
• Describing how things are, rather than explaining why they are like that
• Observational & Survey Research
• RQ: What is the typical number of hours spent studying each week?
• Survey the participants then perform descriptive statistics

Question 2

Advantages of surveys?

Answer 2

• Limits of behavioural observation
• Allows gathering large amounts of information fairly easily
• On-line surveys & global research

Question 3

Disadvantages of surveys?

Answer 3

• Based on self-report
• Biases eg. social desirability (intentional or un)
• Memory errors & limits of insight

Question 4

An example of a research survey that measures individual differences?

Answer 4

E.g., Five factor model of personality (Costa & McCrae, 1992)
Measures Psychological traits or characteristics

Question 5

An example of a research survey that measures ability?

Answer 5

Intelligence tests

Question 6

An example of a research survey that measures attitudes?

Answer 6

E.g. a likert scale questionnaire about how much you like work
(Measure particular beliefs toward something e.g. work)
e.g. I am satisfied with the work I do.
Strongly disagree 1 2 3 4 5 6 7 Strongly agree

Question 7

What two things should you consider ethically/morally about surveys?

Answer 7

1) Need to ensure that the measures being used and reliable and valid.
2) Aware of equal opportunities and cultural biases.
Knife example

Question 8

What are open format questions?

Adv and Dis?

Answer 8

Introduces a topic and allows participants to respond in their own words. Have no determined set of responses.

“Tell us about the occasions when you have been academically vindictive.”

Adv: Lead to more qualitative data, gives participant greatest flexibility. May find out things interviewer hadnt thought about before.

Dis: Time consuming AND difficult to analyse. Difficult to compare answers with different content across participants. Sometimes researcher might have to impose own subjective interpretation on an answer to code.

Question 9

What are closed format questions?

Adv and Dis?

Answer 9

are short questions or statements followed by a number of options. Has limited number of response alternatives. Like a multiple choice.

E.g. I feel bitter towards those who do better than me on my course.
Strongly disagree Disagree Not certain Agree Strongly agree

Adv: easy to analyse and summarise. Can code things numerically.

Dis: you wont find anything out beyond what you were expecting

Question 10

What are three ways you could get advice on how to write your questions for your survey?

Answer 10

Theoretical literature: Ideas that appear in the theoretical literature should be used as a basis.
Experts: Recruit experts in the area to suggest items.
Colleagues: Can help you to generate more items.

Question 11

What is the problem with this survey question and how could you fix it?

“If I had the opportunity, resources and ability to change other students’ exam grades so that mine was the best, I would do it.”

Answer 11

Problem: asking about too many factors. Respondents may concentrate on opportunity, resources, and ability to different extents.

Solution - remove ambiguity:
If I had the opportunity to change other students’ exam grades so that mine was the best, I would do it.

Question 12

Should we use or avoid leading questions?

Answer 12

Avoid.

Question 13

What are leading questions?

Answer 13

a question phrased in such a way you’re leading
the person to answer the question in a certain way

e.g. Mr Woolley, are you worried about the danger
of war?

e.g. “Do you agree with the majority of Australians that it is wrong to falsely tell a classmate the wrong exam date, so that they miss the exam?”

Problem:
Leads the respondent in a particular direction by indicating what “a majority of Australians” think.

Solution:
“Is it wrong to falsely tell a classmate the wrong exam date, so that they miss the exam?”

Question 14

What is reverse wording?

Answer 14

Reversing the wording/phrasing on some questions to get a stronger and more valid measure.

For example, if we want to measure retirement confidence we might ask respondents how much they agree with the statement “I am confident that I will be able to live comfortably in retirement” and “I worry about being able to make ends meet in retirement.”

Question 15

What does reverse wording reduce the problem of?

Answer 15

Positivity bias (someone who tends to agree with all questions)

Negativity bias. (someone who tends to agree with all questions)

Also raises questions about data for those people just clicking random answers to finish the survey.

Question 16

What is a dichotomous response format?

Answer 16

only two opposite answers/options

e.g. (yes/no true/false)

Problem: forced choice as well. But this can be a bad thing, removes that nuances, if they genuinely dont know
or care, then forcing them isnt necessarily good either.
e.g. Do you suffer from nerves? yes or no?

Question 17

What is a ‘frequency of behaviour’ response format?

Answer 17

Asks about activities/behaviour

e.g. I get upset and let my emotions out.
• I usually don’t do this at all
• I usually do this a little bit
• I usually do this a medium amount • I usually do this a lot

Question 18

What response format has “Strongly agree – strongly disagree”

Answer 18

Likert scale

e.g. Overall, I expect more good things to happen to me than bad.
Strongly Disagree Disagree Not certain Agree Strongly Agree

Question 19

What is a numerical scale response format?

How does it differ from likert scale?

Answer 19

It is a likert scale actually and works pretty similar to agree, strongly agree etc.
BUT end points are anchored semantically.

e.g. I feel unsure of myself.
Not at all like me Very much like me
1 2 3 4 5 6 7

Question 20

Why are instructions important for a test?

Answer 20

Because they:

• can for example ask to not overthink questions and be intuitive e.g. personality test, or can ask to consider responses carefully.
• can give an idea of state vs trait questionnaire. State means “Indicate to what extent you feel this way right now, that is, at the present moment” VS trait which is testing something generally stable over time.

Question 21

Explain this equation to me:

OBSERVED score = TRUE score + Error

What is it even referring to?

Answer 21

It is referring to the Classical theory of error in measurement. It is related to measuring reliability (consistency of a measure).

E.g. trying to measure your intelligence with IQ test.
Score you get is partially determined by your true score (actual level of intelligence, quite stable) + your error (a bunch of other factors like fatigue, level of health, current mood, hunger, luck).

Note: over a multiple number of tests, the increases and decreases caused my error should average to zero.
Note: observed score can also be called measured score.

More notes:
• Standard error of measurement
• Universe of items
• All items correlate to some extent with the true score
• Reliability is related to the average correlation between items and test length

Question 22

OBSERVED score = TRUE score + Error

If error is small, my scores would be relatively …..
from one measurement to another, and therefore……..

Answer 22

consistent

reliable

Question 23

OBSERVED score = TRUE score + Error

If error is big, my scores would be relatively …..
from one measurement to another, and therefore…….

Answer 23

inconsistent

NOT FUCKING RELIABLE

Question 24

What are four measures of internal reliability?

Answer 24

– Split-half reliability
– Parallel forms
– Cronbach’s Alpha
– KR-20

Question 25

What is split half internal reliability?

Answer 25

internal reliability: are all questions measuring the same thing?
Split half: can take first half and second half and calculate
scores on first half and second half should correlate btn eachother.

Note:
Items are split into two halves, based on:
• Odd vs. even numbers
• Randomly selecting items for each half
• First half vs. second half of the test

Correlate the total scores for each half
Pearsons R Correlation of 0.80 or higher for good/adequate reliability

Question 26

What is parallel forms and how does it test internal reliability?

Answer 26

• Create a large pool of items.
• Randomly divide the items into two separate tests.
• Administer the two tests to the same participants.
• Calculate the correlation between the two forms.
Problem: Difficult to generate the large number of items required.

Question 27

What is Cronbach’s alpha? What does it measure?

Answer 27

Measures internal reliability

• Cronbach’s Alpha is mathematically equivalent to the average of all possible split- half estimates.
• Usually a figure of +0.7 or greater indicates acceptable internal reliability.
• Calculates correlation of every possible combination of
half one and half two of test.

Question 28

What does this do:

Kuder-Richardson Formula 20 (KR – 20)

Answer 28

DOES THE SAME THING BASICALLY AS CRONBACH BUT ONLY FOR DICHOTOMOUS SCALES

• Measures internal reliability for measures with dichotomous choices (i.e., 2 choices Yes/No).
• Usually a figure of +0.7 or greater indicates acceptable internal reliability.

Question 29

Which test of internal reliability involves the creation of two different versions of a questionnaire?

a) Split-half reliability
b) Cronbach’s alpha
c) Parallel forms reliability
d) KR-20

Answer 29

Parallel forms bitches

Question 30

External reliability vs internal reliability? Difference?

Answer 30

Internal reliability assesses the consistency of results across items within a test. External reliability refers to the extent to which a measure varies from one use to another.

Question 31

What is a measure of external reliability. Describe it.

Answer 31

Test restest reliability measure: measures stability of test over time.

• Perform the same survey, with the same respondents, at different points in time.
• The closer the results, the greater the test- retest reliability of the survey.
• The correlation coefficient between the two sets of responses is often used as a quantitative measure of the test-retest reliability.

Question 32

Problems with test retest reliability ?

Answer 32

practice effect

Question 33

What is INTER-rater reliability?

Note this doesnt seem to be classified as external or internal in lecture (but in tb it appears to be ext).

Answer 33

Inter-rater reliability determines the extent to which two or more raters obtain the same result when coding the same response.

Cohen’s Kappa: larger numbers indicate better reliability, used when there are two raters.
Fleiss’ Kappa: an adaptation which works for any fixed number of raters.
NOTE: Measures agreement, not accuracy. IMP DISTINCTION

Question 34

What is INTRA-rater reliability?

what is a drawback?

Answer 34

The same assessment is completed by the same rater on two or more occasions.
These different ratings are then compared, generally by means of correlation.
Problem: Since the same individual is completing both assessments, the rater’s subsequent ratings are contaminated by knowledge of earlier ratings.

Question 35

What are some sources of unreliability in a survey?

Answer 35

• Guessing
• Ambiguous items
• Test length
• Instructions
• Temperature, illness
• Item order effects
• Response rate
• Social desirability

Question 36

Which measure of reliability measures stability over time?

a) Split-half reliability b) Cronbach’s Alpha c) Test-retest reliability d) Cohen’s Kappa

Answer 36

Test retest r

Question 37

what is validity?

Answer 37

the degree to which the measurement process measures the variable that it claims to measure

Question 38

Name 5 types of validity

Answer 38

• Faith
• Face
• Content
• Construct
     - Convergent
     - Discriminant 
• Predictive

Question 39

what is FAITH validity?

Answer 39

Faith Validity is the least defensible type of validity but the most difficult to influence. It is simply a conviction, a belief of blind faith that a selection test is valid. There is no empirical evidence and, what is more, none is wanted.

Question 40

What is FACE validity?

Answer 40

Again the least scientific type of validity along with faith.
The superficial appearance, face value of a measurement procedure. e.g. asking yourself does the measurement technique loooook like it measures the variable we want to measure? yeah.

Can have high face validity: as in its obvious what is being tested through the measure (problem participants may adjust their answers to appear socially desirable).
Low face validity: ambiguous

In relation to the academic vindictiveness scale:
Find experts in academic vindictiveness, and ask them to judge whether the questionnaire represents a good measure of that construct.

Question 41

What is content validity?

Answer 41

The extent to which a measure represents all facets of the phenomena being measured.

So, in the case of academic vindictiveness:
There might be different types: 
Academic vindictive behaviours 
Academic vindictive attitudes 
Academic vindictive feelings

Content would need to measure all three of these factors for instance.

Question 42

What is construct validity? What are the two components that make up this type of validity?

Answer 42

Construct v: Seeks to establish a clear relationship between the construct at a theoretical level and the measure that has been developed.

TWO SUBTYPES:
Convergent validity:
That the measure shows associations with measures that it should be related to, e.g., academic vindictiveness should be related to other aspects of vindictiveness; such as a tendency to seek revenge, or spitefulness.

Discriminant validity:
That the measure is NOT related to things that it should not be related to. If they dont correlate strongly with eachother then thats good in this instance.

e.g. Academic vindictiveness (revenge) should be measuring something different from Five Factors of personality, so should not correlate highly with extraversion, neuroticisim, agreeableness, openness and conscientiousness.

Question 43

What is predictive validity?

Answer 43

Assesses whether a measure can accurately predict future behaviour.

Scores on the academic vindictiveness scale should be able to predict people acting in an academically vindictive way in the future:
Not sharing notes
Not helping other people revise

Question 44

Fill in sentence:

“Reliability is a necessary but …. …………. condition for validity

Answer 44

not sufficient

Question 45

Distinguish once more for me between reliability and validity.

Answer 45

Reliability:
A questionnaire is reliable if all of the questions in your test are consistently measuring the same underlying concept, and that this remains stable over repeated times that the test is administered

Validity:
A test is valid if it is actually measuring what you intend it to measure.

Question 46

Which measure of validity states that a test should represent all facets of the phenomena being measured?

a) Face validity
b) Predictive validity
c) Content validity
d) Faith validity

Answer 46

Content validity

Question 47

Tell me what these scales do:

Nominal
Ordinal
Interval
Ratio

Answer 47

Nominal: tells us only that difference exists

Ordinal: tells us the direction of the difference (which is more and which is less)

Interval: can determine the direction and the magnitude of the difference.

Ratio: tells us the direction, magnitude and ratio of the difference.

(remember this order, it will help you, as we go down list gets more informed and tells us more). NOIR (black)

Question 48

Non response bias ?

random q im throwing in from tb

Answer 48

The idea that youre only testing maybe the motivated people who respond, instead of the lazier ones who dont respond to surveys.