reliability and validity

Question 1

what is reliability?

Answer 1

Very broadly, the dependability of a measurement instrument. Eg weighing scales, should expect to weight roughly the same/ consistently across time of day

Question 2

what is validity?

Answer 2

The property that a measurement instrument measures what it claims to measure. Eg clock measures timeHarder to establish in science

Question 3

Internal consistency (Cronbach’s α):

Answer 3

• Extent to which each of the tests measures the same thing
• Sampling distribution is unknown s bench mark values have been proposed, where 0-1, higher value = greater reliability and ability of an instrument.

Question 4

Alternate-forms reliability:

Answer 4

• Involves the construction of two tests of equal length by randomly sampling from the same domain(‘universe of items’).
• The Pearson correlation between these two tests (forms) is an estimate of the reliability

Question 5

Parallel-forms reliability:

Answer 5

• Involves the construction of two tests comprising items of the same difficulty, which leads to similar score distributions.
• The Pearson correlation between these two tests (forms) is an estimate of the reliability.

Question 6

Split-half reliability

Answer 6

• Randomly splits the test into two halves, calculates the Pearson correlation between them, and applies the Spearman-Brown prophecy formula to estimate reliability. - Still used, but not recommended as can just use computer methods now.

Question 7

Test-retest reliability:

Answer 7

• Also known as ‘temporal stability’. Same test administered on two different occasions spaced about one month or more apart.
• The Pearson correlation between the two scores is an estimate of the reliability. Note this is very different from the foregoing types of reliability.

Question 8

Construct validity:

Answer 8

• The degree to which a test measures a specified construct as determined by the interpretation of the psychological meaning of test scores and the implications of this interpretation.
• Term introduced by Cronbach and Meehl (1955).
• Construct validity is important from a scientific perspective and rests on the psychological theory underpinning an instrument.
• Not associated with a number as reliability is

Question 9

Criterion validity:

Answer 9

• The degree to which scores on a test correlate with scores on a relevant external criterion.
• It is a broad type of validity encompassing several more specific types (e.g., convergent, predictive, concurrent).

Question 10

Convergent validity

Answer 10

• The degree to which scores on a test correlate with variables they are supposed to correlate with, given the nature of the construct.

Question 11

Discriminant validity:

Answer 11

• The degree to which scores on a test do NOT correlate with (are ‘independent of’ or ‘orthogonal to’) variables they are NOT supposed to correlate with, given the nature of the construct.
• Eg happiness with IQ, if test of happiness correlates with IQ then it is not good! So if it doesn’t correlate then that is good.

Question 12

Predictive validity:

Answer 12

• The degree to which scores on a test predict future behaviour on a criterion variable.

Question 13

Concurrent validity

Answer 13

• Is based on the correlation between predictor (test) and criterion scores obtained at approximately the same time (e.g., self-reported and clinically diagnosed depression).

Question 14

Congruent validity:

Answer 14

• The degree to which a new test correlates with extant measurement instruments of the construct.
• It is a weak type of validity because extant measures may themselves have low validity.
• Good if establishing a new test if there is already a gold standard that can be used to compare to.

Question 15

Face validity:

Answer 15

• ‘experts’ review test contents to determine if they are appropriate ‘on their face’.
• Eg assessing leadership potential –> experts in the field and see if valid on face value.

Question 16

Internal validity:

Answer 16

The extent to which the observed effect on the dependent variable is caused only by the experimental treatment condition.
- Threats to internal validity; confounding that must be controlled:

Question 17

Factor analysis:

Answer 17

• Called PCA and PFA
• Statistical methods to reduce the dimensionality of a correlation matrix
• Which variables in a particular set for coherent and independent structures
• Summarise quantitave information eg NEO personality test which has a lot of correlations.. Can reduce to the important/crucial ones.
• Developed by spearman

Question 18

Kaiser eigenvalue criterion: FA

Answer 18

This is the default in SPSS and extracts all factors with eigenvalues > 1. It is often
–Very misleading and should never be adopted

Question 19

Scree plot: FA

Answer 19

Eigenvalues are plotted against the principal components. The cut-off point for factor extraction is where the line changes slope. We extract all the factors that lie above the break. If there are two breaks in the
slope, we extract all the factors above the leftmost break.

Question 20

why is FA good

Answer 20

• Allows to investigate concepts that are not easily measured directly by collapsing a large number of variables into few interpretable underlying factors.

Question 21

what is an eigenvalue?

Answer 21

measure of how much of the variance of the observed variables a factor explains: if under or equal to 1 explains more than single observed variable

• If the factor for SES had an eigenvalue of 2.3=2.3 explain as much variance as 2.3 of the varibles
• The FACTOR which captures most variance in those three vs could be used in other analyses
• Factors of the least variance get discarded

Question 22

what is factor loading?

Answer 22

• Relationship of each v to the underlying factor
• Strongest association variable can be put in hierarchy
• Can also be interpreted like standardised regression coeffs SO if load is .65 then can say it has a corr of .65 with factor 1 which is strong.
• Depending on the highly loaded variable; can deduce what the overall underlying factor(s) are