Multi-trait multi-method validity analysis

Question 1

Validity

Answer 1

Does a test actually measure what it claims to measure?

-we need to have reliability to establish validity

Question 2

Construct

Answer 2

are things we assume exist – but they are things which cannot be directly observed or measured
E.g., depression, love, personality, anxiety, intelligence, empathy, etc.

Question 3

convergent validity

Answer 3

When a construct that is measured by a test is similar to other similar constructs

Question 4

divergent or discriminant validity

Answer 4

When a construct that is measured by a test is different from similar constructs

Question 5

To work out convergent validity

Answer 5

We need to take a test that measures something, and correlate scores on this test with scores on test that measures similar things
E.g., if you say that emotional intelligence (EI) is a new form of intelligence you need to demonstrate high correlations between a test of EI and traditional IQ tests

Question 6

To work out divergent validity

Answer 6

We need to take a test that measures something, and correlate scores on this test with scores on tests that measure different things
This is particularly important if you say you have discovered a new construct
E.g., if you say that EI is a new form of intelligence, you have to show that it differs from social cognition and/or emotionality

Question 7

Why do we say that psychological tests are trait-method units?

Answer 7

-because scores on tests are influenced by actual difference between people (true variance) and also method variance.

trait= a factor that the test measures
method= the item format that you use

Question 8

Why use multiple traits/factors and multiple method/item formats?

Answer 8

Can see the correlations between different constructs

Can see the correlations between different types of measures/item formats

You can see the correlations between different constructs measured by the same item formats
And correlations between different constructs measured by different item formats

Question 9

How do we establish construct validity properly?

Answer 9

Campbell & Fiske (1959) proposed that this is done through a Multitrait-Multimethod (MTMM) Matrix
A correlation matrix which shows correlations between tests measuring different traits/factors, measured according to different methods

have 4 hierarchical rules

Question 10

4 rules of MTMM

Answer 10

Rule 1: The values in the validity diagonal should be more than 0, and large enough to encourage further exploration of validity

Rule 2: A value in the validity diagonal should be higher than the values lying in its column and row, in the heterotrait-heteromethod triangles

Rule 3: A value in the validity diagonal should be higher than the values lying in its column and row, in the heterotrait-monomethod triangles

Rule 4: There should be more or less the same pattern of correlations in all the different triangles

Question 11

The reliability diagonals

Answer 11

The reliability diagonals are the values in parentheses (brackets) in the MTMM matrix
Remember that we need to have reliability before you can establish validity – so these values should be pretty good

also called monotrait-monomethod values

Question 12

The validity diagonal

Answer 12

These are the correlations of the same factors, measured by different methods/questionnaires

• E.g., the correlation of Anxiety in Everyday Situations, measured by the SEAL and the CADS
• High correlations values here mean good convergent validity

Question 13

another name for validity diagonal

Answer 13

• also called monotrait-heteromethod values

- Because the traits/factors are the same – but the method/questionnaire used to measure them are different

Question 14

Interpreting an mtmm matrix: rule 1

Answer 14

• The entries in the validity diagonal should be different from 0
• And large enough to encourage further examination of validity
• This is evidence of convergent validity
• Of course you can’t establish convergent validity if you don’t have convergent validity to begin with

Question 15

Interpreting an mtmm matrix: rule 2

Answer 15

A value in the validity diagonal should be higher than the values lying in its column and row in the heterotrait-heteromethod triangles

The validity values are evidence for convergent validity – the heterotrait-heteromethod values are divergent validity values
So what we are saying here is that convergent validity should be higher than divergent validity

Question 16

What is a heterotrait-heteromethod triangle?

Answer 16

• This is the group of correlation values which come from the correlations of different traits/factors measured by different methods/questionnaires
• These values are a collection of divergent validity values

Question 17

Interpreting an Mtmm matrix: rule 3

Answer 17

A value in the validity diagonal should be higher than the values lying in its column and row
In the heterotrait-monomethod triangles

The validity values are evidence for convergent validity – the heterotrait-monomethod values are divergent validity values
So like Rule 2, what we are saying here is that convergent validity should be higher than divergent validity

Question 18

What is a heterotrait-monomethod triangle?

Answer 18

This is the group of correlation values which come from the correlations of the different traits/factors measured by the same methods/questionnaire
These values are also a collection of divergent validity values

Question 19

Interpreting an mtmm matrix: rule 4

Answer 19

There should be the same pattern of correlations in the two different types of triangles

Question 20

Summary of the 4 rules

Answer 20

Rule 1: The values in the validity diagonal should be more than 0, and large enough to encourage further exploration of validity

Rule 2: A value in the validity diagonal should be higher than the values lying in its column and row, in the heterotrait-heteromethod triangles

Rule 3: A value in the validity diagonal should be higher than the values lying in its column and row, in the heterotrait-monomethod triangles

Rule 4: There should be more or less the same pattern of correlations in all the different triangles