validity part 2

Question 1

explain incremental validity

Answer 1

Does using this test improve upon other tests that we are already using?
-Yes, if the correlation between the criterion and the new test is significantly greater in magnitude than the correlations between the criterion and the other tests

Does adding this test to a battery of tests that we are already using allow us to predict the criterion more accurately?
-Use Hierarchical Multiple Regression

Question 2

what is the regression formula

Answer 2

Allows us to estimate the score on a dependent variable (Y) from an independent variable (X)
Yest = BX + C

• B = “beta weight” (how much to weight the score on the independent variable; the number of units Y increases for each unit increase in X)
• C = constant (the value of the dependent variable when the independent variable is 0).

NOTE: The dependent variable (Y) is the variable we want to predict from the independent variable (X)

B = slope of the line
C = y-axis intercept

Question 3

how to calculate regression formula

Answer 3

-To calculate a regression formula, we must have measured values for both X and Y in our sample (this is known as the derivation sample)

The regression formula gives us the best equation for predicting Y from X

• B = by what number should we multiply each value of X
• C = what is the value of Y when X = 0

We then hope to be able to use the formula in a new sample where we have also measured values of X and Y

We want to see how well the formula works in the new sample

But ….SHRINKAGE will occur

Question 4

what is shrinkage

Answer 4

• Always occurs when we apply a regression formula calculated on a sample to a new sample
• This happens because the calculated values of B and C are optimal for the derivation sample, and therefore could be capitalizing on chance relationships within this sample

Question 5

what is cross validation

Answer 5

• To estimate the degree of shrinkage, we apply the regression formula to a new sample (cross-validation sample)
• We examine the correlation between the actual values of Y (which were measured when the original MR formula was developed) and the values of Y predicted by the formula (Yest) in both samples
• If the correlation is significantly lower in the cross-validation sample (i.e., if substantial shrinkage has occurred), then the formula cannot be generalized beyond the derivation sample
• Important: For a fair test of cross-validation, the derivation sample and cross-validation sample must be matched on relevant demographic factors.

Question 6

cross validation: calculating shrinkage

Answer 6

• 1. Calculate the values of Yest, B, and C in Sample 1 using the measured values for Y and X in Sample 1
• 2. Calculate the correlation between estimated values of Y and the actual values of Y in Sample 1
• 3. Recruit Sample 2 which is matched in important respects to Sample 1
• 4. Measure the independent variable (X) in Sample 2
• 5. Calculate the estimated values of Y in Sample 2 by applying the values of B and C that were calculated in Sample 1 (i.e., enter into the formula BX + C).
• 6. Compare the estimated values of Y (Yest) to the actual values of Y in sample 2 by calculating the correlation between Yest and Y in Sample 2
• 7. Compare the correlation between Yest and Y in Sample 2 (Step 6) to the correlation between Yest and Y in Sample 1 (Step 2). The difference between these two correlation coefficients is the shrinkage

Question 7

double cross validation

Answer 7

• Common procedure = recruit a large sample then split the sample into two (Samples 1 and 2)
• Sample 2 is the cross-validation sample for Sample 1 (weights calculated in Sample 1 are applied to Sample 2 and shrinkage estimated)
• Sample 1 is the cross-validation sample for Sample 2 (weights calculated in Sample 2 are applied to Sample 1 and shrinkage estimated)

formula calculated on sample 1, cross validated on sample 2 and vice versa

Question 8

regression simple vs. multiple

Answer 8

simple: one IV
multiple: two or more IVs

Question 9

multiple correlation

Answer 9

(R) = correlation between the actual values of Y and the values of Y that are predicted by the multiple regression (MR) formula.

Question 10

squaring the multiple correlation

Answer 10

(R2) tells us the amount of variance in the DV that can be predicted by the combined set of IVs

Question 11

what to do when satisfied with the results of the shrinkage analysis

Answer 11

then re-combine the samples and calculate a new regression formula on the entire sample

Question 12

multiple regression: sample size

Answer 12

Absolute minimum is 10 participants per IV
-e.g., if we have 3 IV’s we need at least 30 participants

Preferable to have 20, some sources say 50 or even up to 100 participants per IV

The larger the sample, the smaller the shrinkage

Question 13

multiple regression: multiplecollinearity

Answer 13

• Occurs when IV’s are highly correlated with one another
• In this case, two variables are really measuring the same construct and so might be redundant

Why a problem?

• Requires a larger sample size
• Could result in larger shrinkage

Question 14

hierarchical: multiple regression

Answer 14

• We determine by how much R-squared increases when an additional IV is added to the regression formula
• For example, when we add an additional test to an existing battery of tests

Question 15

different methods of multiple regression

Answer 15

hierarchical, stepwise, logistic, discriminant

Question 16

hierarchical multiple regressions

Answer 16

Experimenter determines the order of entry of the variables;
used to determine incremental validity

Question 17

stepwise multiple regressions

Answer 17

Computer algorithm determines the order of entry of the

variables; will select the subset of IVs that will result in the best estimate of the DV

Question 18

logistic regressions

Answer 18

Used when the purpose is classification into two groups; Y
is a binary (0, 1) variable; reports the “odds ratio” that a case
belongs in one group vs the other

Question 19

discriminant function

Answer 19

Also used for classification into groups; can accommodate
more than two groups (Multiple Discriminant Function);
reports the percentage of cases correctly classified