Week 4 (reliabilty & validity)

Question 1

Explain measured scores?

Answer 1

Measured scores = true score+noise

The assumption is everything we want to measure has a true value or score

The goal is that measurement will give us a pure score

Reality: measurement will always include some amount of random noises
The measurement decide or person measuring is going to have some degree to which there will be fluctuations around true score which will be different from time to time

Question 2

Explain the relationship between consistency (reliability) and variability

Answer 2

Consistency is measure by variability
INVERSE RELATIONSHIP

high variability = low reliability
Low variability= high reliability

Question 3

What are the three types of reliability?

Answer 3

Test-retest reliability (consistency over time): if we do measurements on multiple occasions and the true score doesn’t change

Internal reliability (consistency across items): eg. Surveys need more than one item to measure abstract constructs

Inter-rater reliability (consistency across researcher): different people collecting the data are consistent. Assessed using a statistic called cohens kappa (k) which is analogous to chronbachs alpha

Question 4

Is a reliable measure valid?

Answer 4

A measure that is reliable is not automatically valid

Question 5

Why does variability occur?

Answer 5

1. Casual relationships between the IV and DV (effect) - when IV changes DV changes
2. Variability due to random factors (noise) - unknown factors affect the DV

Question 6

Can research design be used to seperate effect from noise?

Answer 6

Noise can be reduced but not eliminated, there is always some amount

Question 7

What is the split-half reliability?

Answer 7

Rosenberg self-esteem scale
Divide items into two halves and correlate scores on the two halves
Should be strong and positive

Question 8

What is cronbach’s alpha?

Answer 8

Measures internal consistency

Conceptually related to split half reliability
Calculate all possible split-half correlations for a set of items

Cronbachs is the mean of the set of correlations

Question 9

Explain parallel forms reliability?

Answer 9

Two versions of test for same test
Scores for the two versions should be highly and positively correlated

Need different set of questions due to testing someone twice

Question 10

What is validity?

Answer 10

Validity is the extent to which measures represent what they are intended to

Loosely the truthfulness of a measure

Question 11

What are the forms of validity?

Answer 11

Construct
Internal
External
Face validity

Question 12

What is construct validity?

Answer 12

Construct validity is “the degree to which a test measures what it claims, or purports, to be measuring.”

Question 13

What is internal validity?

Answer 13

Extent to which casual statements about the relations among variables can be made

Dependent on study design

High control over extraneous variables have high internal validity

Question 14

Why is external validity?

Answer 14

Extent to which you can generalise your conclusions to other settings, participants, populations etc

Sample size doesn’t necessarily affect

Question 15

When does externally invalid research occur?

Answer 15

When conclusions are limited to the original setting, sample etc

Question 16

What is face validity?

Answer 16

Isn’t a true form of validity but is used
Do IV and DV appear to reflect or measure what the ya re intended to

Research is often unfairly challenged when it lacks face validity - less accepted by general public

Question 17

What are the threats towards internal validity?

Answer 17

History 
Maturation 
Instrumentation decay 
Statistical regression 
Selection 
Testing 
Morality

Question 18

How does history threaten internal validity?

Answer 18

Longitudinal studies use risk- measurements that occur overtime are susceptible to effects from factors other than IV. Risk increases as time between measures increases

Question 19

How does maturation threaten internal validity?

Answer 19

Change may be a result of natural maturation processes

Question 20

How does instrumentation decay threaten internal validity?

Answer 20

Changing the measurement instrument during a study can affect measurement of the DV

Includes questionnaires and researchers

Question 21

How does statistical regression threaten internal validity?

Answer 21

Regression to the mean when groups are formed by selecting extreme scores at pretest these scores tend to be less extreme at post test they move closer to true scores

Question 22

How does selection (biased assignment) threaten internal validity

Answer 22

When groups are not formed through random assignment there is potential bias in group formation

Question 23

How does testing (protest sensitisation) threaten internal validity

Answer 23

Some studies required participants to do a pre-test (for task familiarisation or baseline measure) after an intervention participants are tested again using the same test this pre-test my influences post test outcome it may alert participants to the expected outcome affecting behavioural test maybe sensitive to learning affects

Question 24

How does mortality (differential attrition) threaten internal validity?

Answer 24

Differential attrition in studies with more than one group - nonrandom factor that results in greater drop out in one group than the other

Question 25

Explain subject roles

Answer 25

Good subject: behave in ways that they believe will help the researcher

Faithful subject: they will do precisely what you’re asking them to do

Negative subject: will try to undermine the study and do the opposite of what the researcher wants

Apprehensive-subject: anxious and trying to do things well but anxiety may interfere and they second guess themselves etc

Question 26

What are some additional forms of validity?

Answer 26

Convergent validity: tests whether constructs/measures that should be related, are related

Discriminant (divergent) validity: tests whether constructs/measures that should be related are unrelated

Predictive validity: tests whether current performance in some measure predicts future performance on another

Concurrent validity: test whether performance on a validated test correlated with performance on a new test

Question 27

What can be used to test construct validity?

Answer 27

Can be tested through convergent and discriminant (divergent) validity

Question 28

How do we quantify test-retest reliability?

Answer 28

Using the correlation coefficient
In excel =CORREL

Strong test-retest reliability is .8 or larger

Question 29

What is used to calculate convergent validity statistically?

Answer 29

Same as test-retest reliability
=CORREL In excel
r >.8 is good

Question 30

Explain correlation of split-half reliability

Answer 30

Should be strong and positive