Q2 reliability & validity

Question 1

what is true of a good measure?

Answer 1

it assesses behavioral variability accurately

Question 2

observed score =

Answer 2

true score + measurement error

Question 3

true score

Answer 3

systematic variance

Question 4

measurement error

Answer 4

error variance

Question 5

where does measurement error come from? (2)

Answer 5

participant
not the participant

Question 6

2 ways participants may introduce measurement errror

Answer 6

transient factors
stable attributes

Question 7

transient factors

Answer 7

anxiety, tiredness (not stable)

Question 8

stable attributes

Answer 8

differ across participants but are stable within a given individual; IQ, personality traits, general motivation

Question 9

3 ways (aside from the participant) that measurement error can be introduced

Answer 9

situation factors
measure factors
mistakes

Question 10

situation factors

Answer 10

where are participants taking a questionnaire? lighting, temperature, etc.

Question 11

measure factors

Answer 11

questions aren’t clear or the questionnaire isn’t good

Question 12

what kind of mistakes can contribute to error variance?

Answer 12

participant is given the wrong test, experimenter says something wrong, computer error, etc.

Question 13

reliability

Answer 13

consistency and dependability in scores across time; undermined by measurement error

Question 14

how do we estimate reliability?

Answer 14

via correlations between measures of the same attribute

Question 15

satisfactory reliability coefficient

Answer 15

0.7+ or 70%

Question 16

3 kinds of reliability

Answer 16

test-retest
inter-item
inter-rater

Question 17

test-rest reliability

Answer 17

will you get the same score of behavior if you measure at two different times? wouldn’t test non-stable traits

Question 18

inter-item reliability

Answer 18

ensures consistency among scale items aiming to measure the same construct (internal consistency)
make sure there are high enough correlations between similar questions

Question 19

number used to assess inter-item reliability and satisfactory threshold

Answer 19

chronbach’s alpha >0.7

Question 20

what does chronbach’s alpha measure?

Answer 20

inter-item reliability
systematic variance

Question 21

how do we assess inter-rater reliability in nominal data?

Answer 21

kappa coefficient (chi-square)
0.6+ is reliable

Question 22

how do we assess inter-rater reliability in ordinal data?

Answer 22

spearman’s r (0.7+)
kendall’s tau (0.45+)

Question 23

how do we assess inter-rater reliability in interval/ratio data?

Answer 23

pearson’s r (0.7+)

Question 24

how do we assess inter-rater reliability in 3+ raters’ data?

Answer 24

intraclass correlation coefficient (0.75+)

Question 25

validity

Answer 25

are you actually measuring what you want to measure? accuracy

Question 26

why is validity a concern in psychological research?

Answer 26

psychological constructs are not directly observable

Question 27

construct validity

Answer 27

when multiple measures correlate with each other

Question 28

convergent validity

Answer 28

finding measures that correlate with each other that should correlate with each other (happiness and positive affect)

Question 29

discriminant validity

Answer 29

no correlation with unrelated measures (happiness and negative affect)

Question 30

criterion-related validity

Answer 30

correlation between measure and relevant behavior; should have predictive power in behavior (SAT -> graduation rates, GPA)

Question 31

concurrent criterion-related validity

Answer 31

correlation between measure and behavior at the current time

Question 32

predictive criterion-related validity

Answer 32

correlation between measure and behavior at a future time

Question 33

2 kinds of criterion-related validity

Answer 33

concurrent
predictive

Question 34

how can we maximize reliability and validity?

Answer 34

provide specific operational definitions (precise, appropriate)
no ambiguity on how you measure your variable (backed up by the literature)
standardize procedures (scripts, etc.)
inclusion and assessment of subject variables (sex, age, etc.)
random and/or standardized sampling to wash out potential systematic individual differences between groups