RELIABILITY AND VALIDITY

Question 1

Define reliability

Answer 1

the consistency or repeatability of your measurement

Question 2

what are the three types of reliability

Answer 2

stability of the measure (test-retest)

internal consistency of the measure (split-half, cronbach;s alpha)

Agreement or consistency across raters (inter-rater reliability

Question 3

what does test-retest reliability look at?

Answer 3

whether your test measures the the same thing every time you use it

same Q. given on two occasions and data correlate

Question 4

test-retest - how do you address the stability of the measure?

Answer 4

• you administer the measure at one point in time (test)
• you then give the same measure to the same participant at a later point in time (retest)
• correlate the scores on the two measures

Question 5

what are the two main problems with test-retest

Answer 5

Memory Effect -
participants may remember the experiment > will improve their second measure
- to short time between = greater risk of memory effects

Practice effect-
performance improve because of practice in test taking
- too long time between = risk of other variables (additional learning)

Question 6

what does split half reliability look at?

Answer 6

whether your measure is internally consistent

Split Q in half and correlate data from two halves

Question 7

split half reliability - how do you test whether your measure is internally consistent?

Answer 7

• administer a single measure at one time to a group of participants
• split the measure into two halves and correlate the scores
• higher correlation means greater reliability

e.g. 20 item, score one half (10 items) and second half (10 items), test correlation between the two halves

Question 8

pros and cons of split- half reliability

Answer 8

PRO
eliminates memory/practice effects

CON
are the two halves really equivalent

Question 9

two methods of assessing internal consistency

Answer 9

split-half method

cronbach’s alpha

Question 10

what does cronbach’s alpha assess

Answer 10

internal consistency of your measure

tells you how well the items or questions in your measure appear to reflect the same underlying construct

good internal consistency = when individuals respond the same way

Question 11

how is cronbach’s alpha measured

Answer 11

mathematically equivalent to average of all possible split-half reliabilities

coefficient alpha can range from 0-1 >closer to 1 = better reliability

Question 12

what does inter-rater reliability look at?

Answer 12

whether different raters measure the same thing

checking the match between two or more raters or judges

e.g. coding videos for infants “looking time” - need to check agreement amongst the coders

Question 13

how is inter-rater reliability calculated

Answer 13

nominal/ordinal scale
- the percentage of times different raters agree

interval or ratio scale
- correlation coefficient

Question 14

define Validity

Answer 14

the credibility of the measure

are we measuring what we think we are?

Question 15

why is validity an issue

Answer 15

many variables in social research cannot be directly observes

• motivation, satisfaction, helplessness

Question 16

types of validity

Answer 16

face validity

content validity

criterion validity (concurrent, predictive)

construct validity (convergent, discriminant/divergent)

Question 17

what is face validity

Answer 17

items appear to relate to construct

a weak, subjective method for assessing validity

a good first step to validity assessment

Question 18

what is content validity

Answer 18

the extent to which the measure is representative of a sampling of relevant dimensions

does it cover all aspects of the construct that its meant to measure

how much does the measure cover the content of the definition?

Question 19

what is criterion-related validity

Answer 19

checking the performance of your measure agains an external criterion

agree with external sources

Question 20

what are the two types of criterion-related validity

Answer 20

concurrent

predictive

Question 21

define concurrent criterion validity

Answer 21

a means in establishing validity of your measurement by comparing to a gold standard
>i.e. existing validated measure of the same construct

agrees with pre-existing “gold standard” measure

Question 22

what is predictive criterion validity

Answer 22

assessing the validity of your measure against what you theoretically predict to happen

agrees with future behaviour

Question 23

define construct validity

Answer 23

how well the measure and other constructs relate to each other (consistent with a theory)

Question 24

what are the two types of construct validity

Answer 24

convergent

divergent

Question 25

define divergent construct validity

Answer 25

assessing validity by comparing measures of construct that theoretically should not be related to each other and are observed to not relate to each other > theoretically should not and in fact are not related < i.e. you should be able to discriminate/diverge between dissimilar constructs

Question 26

define convergent construct validity

Answer 26

assessing validity by comparing measures of constructs that theoretically should be related that are observed to relate to each other > theoretically should and in fact are related < i.e. there is correspondence or convergence between similar constructs