MTMM Analysis

Question 1

What is validity?

Answer 1

• Does a test actually measure what it claims to measure?
• The aim of establishing validity is that it speaks directly to the usefulness of the test. If the test doesn’t measure what it actually wants to measure, then it is not useful.

Question 2

What are the different types of validity?

Answer 2

• Content validity
• Criterion-related validity
• Construct Validity
• Face validity

Question 3

What comes first; reliability or validity?

Answer 3

We need to have reliability to establish validity?

Question 4

What are the different types of criterion-related validity?

Answer 4

• Concurrent
  -Predictive

Question 5

What are the different kinds of construct validity?

Answer 5

• Convergent
• Divergent.

Question 6

What is face validity?

Answer 6

• This is based on expert opinion.
• On the surface, the measure seems to measure what it claims to measure.
• You can have good face validity but not be a valid test overall.
• Some people don’t consider this a measure of validity at all because no statistical analysis is required. However, it is important to establish it early on in your test creation.
• Tests with low face validity also have low reliability.

Question 7

What is content validity?

Answer 7

Content validity refers to how well the test or scale items represent the domain of the construct being measured.

Question 8

How do we establish content validity?

Answer 8

• By getting judgement fro expert judges
• Through statistical methods such as factor analysis.

Question 9

What are two aspects of content validity?

Answer 9

1. Construct under-representation: this occurs when a test does not capture important component of the construct. E.g. a test of PTSD that does not have questions relating to vividly re-experiencing the traumatic event.
2. Construct-irrelevant variance: this occurs when test scores are influenced by things other than the construct the test is supposed to measure. E.g. test scores may be influenced by reading ability or performance anxiety. This might be caused by poorly constructed or administered tests.

Question 10

What is criterion validity?

Answer 10

Criterion-related validity refers to how well the scale or test anticipates a criterion behaviour or outcome, either at the present time (concurrent validity) or in the future (predictive validity). E.g. cut-off scores for a questionnaire matching clinician diagnosis.

Question 11

What are the two types of criterion validity?

Answer 11

• Concurrent criterion validity: The extent to which test scores can correctly identify the current state of individuals
• Predictive criterion validity: refers to how well the test predicts a criterion behaviour in the future.
• The test is the predictor and the future event is the criterion.
• If you correlate the two scores, a high correlation means your test is a good predictor.

Question 12

What is the most difficult type of validity to establish and why?

Answer 12

• Construct validity because constructs are things we assume exist- but they are things which cannot be directly observed or measured. E.g. love, depression, anxiety, intelligence.

Question 13

How do we establish construct validity?

Answer 13

• Because construct validity is an abstract concept, constructs are best defined by looking at their relationship to other constructs. i.e. in what way is this construct similar to other constructs? and in what way is this construct different from other constructs.
• In order for a test to have construct validity, it needs to have both convergent and divergent validity.

Question 14

What is convergent and divergent validity?

Answer 14

• Convergent validity ascertains construct validity through comparing the scale
  with a measure of an allied construct.
• Divergent validity ascertains construct validity through contrasting the scale with a measure of an opposing construct.

Question 15

How do we work out convergent validity?

Answer 15

• We need to take a test that measures something, and correlate scores on this test with scores on test that measures similar things
• E.g., if you say that emotional intelligence (EI) is a new form of intelligence you need to demonstrate high correlations between a test of EI and traditional IQ tests

Question 16

How do we work out divergent validity?

Answer 16

• We need to take a test that measures something, and correlate scores on this test with scores on tests that measure different things
• This is particularly important if you say you have discovered a new construct
• E.g., if you say that EI is a new form of intelligence, you have to show that it differs from social cognition and/or emotionality.

Question 17

What factors affect validity?

Answer 17

• Reliability. Any form of measurement error can reduce validity.
• Social diversity. Tests may not be equally valid for different social/cultural groups. E.g. superstition might be a test of religiosity and something else in another. We need to use tests that have already been shown to be valid for the population we’re looking at.
• Look at table summarising different types of validity on the slides.

Question 18

What do we know about measuring constructs and measurement methods?

Answer 18

• Often when we measure constructs, we use the same measurement methods, but sometimes we use different methods as well.
• So any variance in test scores could be due to actual differences in scores on test, or due to error associated with various measurement methods we use (method variance). E.g. Likert scales are prone to things such as response sets whereas Q-sorts arent.

Question 19

What is true variance and method variance?

Answer 19

• Scores on tests are influenced by actual differences between people (true variance) and also (method variance).
• If the methods we use to measure something are too error-prone, we are not going to be measuring what we want to measure properly.
• i.e. it will not have validity as scores on the test will be influenced too much by method error.

Question 20

Method variance and validity

Answer 20

• when you are trying to establish construct validity, you need to work out not only convergent and divergent validity for the constructs/traits/factors that you are trying to measure, you need to work this out using tests that use different formats.

Question 20

Why do we say that psychological tests are trait-method units?

Answer 20

• Trait: a factor that the test measures
• Method: the item format that you use.
• A psychological test measures a specific trait via a specific method (item format)

Question 21

Why do we use multiple traits/factors and multiple methods/item formats?

Answer 21

1. Can see the correlations between different constructs
2. Can see the correlations between different types of measures/item formats
3. You can see the correlations between different constructs measured by the same item formats.
4. And you can see correlations between different constructs measured by different item formats.

Question 22

How do properly establish construct validity, accounting for method variance?

Answer 22

• By doing a multitrait-multimethod matrix.
• This is a correlation matrix which shows correlations between tests measuring different traits/factors, measured according to different methods.
• There is a set of 4 rules used in the interpretation of the MTMM matrix. These rules follow a hierarchical pattern.

Question 23

What are the four rules of doing an MTMM analysis?

Answer 23

• Before starting, look at the reliability diagonal. The values should be above the threshold.
• Rule 1: The values in the validity diagonal should be more than 0, and large enough to encourage further exploration of validity.
• Rule 2: A value in the validity diagonal should be higher than the values in its similar column and row in the heterotrait-heteromethod triangles
• Rule 3: A value in the validity diagonal should be higher than the values in its column and row in the heterotrait-monomethod triangles.
• Rule 4: There should be more or less the same pattern of correlation in all the different triangles.

Question 24

What is some important vocabulary for MTMM analysis?

Answer 24

• Methods: the different questionnaires
• Traits/factors: the different dimensions of each test.
• Monotrait-monomethod values: the values in the reliability triangle. they are one trait measure by one method.
• Heterotrait-heteromethod triangle: this is the group of correlation values which come from the correlations of different traits measured by different methods.
• Heterotrait-monomethod triangle: This is the group of correlation values which come from the correlations of the different traits measured by the same methods.

Question 25

What do the values in the reliability diagonal represent?

Answer 25

• These values represent the internal consistency of each dimension within each measure. So it is how well all the items pertaining to that factor fit together.
• These are also called monotrait-monomethod values.

Question 26

What do we know about the validity diagonal?

Answer 26

• These are correlations of the same factor, measured by different methods/questionnaires.
• High correlation values here mean good convergent validity.
• The values in this diagonal are also called monotrait-hetermethod values because the traits are the same but the method used to measure them are different.

Question 27

More about Rule 1 of the MTMM matrix

Answer 27

• Rule: the values in the validity diagonal should be different from zero, and large enough to encourage further examination of validity.
• This is evidence of convergent validity
• Note: cannot establish construct validity if you don’t have convergent validity to begin with.

Question 28

More on Rule 2

Answer 28

• Rule 2: value in the validity diagonal should be higher than the values lying in its column and row in the heterotrait-heteromethod triangles
• The values in the heterotrait-heteromethod triangle are a collection of divergent validity values.
• So this rule is trying to see if the convergent validity is higher than the divergent validity.
• We want things that are similar to have a bigger relationship than those that are different.

Question 29

More on rule 3

Answer 29

• Rule: a value in the validity diagonal should be higher than the values in its column and row in the heterotrait-monomethod triangles.
• The values in the heterotrait-monomethod triangle are also a collection of divergent validity values.
• So again here we want convergent validity to be greater than divergent validity.
• Rule 2 is assessing evidence within a block of values
• Rule 3 is assessing evidence across a block of values.

Question 30

What are blocks in the MTMM matrix?

Answer 30

• The collection of triangles and diagonals form different blocks in the matrix.
• The validity diagonals and the heterotrait-heteromethod triangles form the heteromethod blocks.
• The reliability diagonals and heterotrait-monomethod values form the monomethod blocks.
• The monomethod block looks at the same test
• The heteromethod block looks at correlations across measures.

Question 31

More on rule 4

Answer 31

• Rule: There should be the same pattern of correlations in the two different types of triangles.
• Using rule 1-3 we have established whether or not there was convergent and divergent validity.
• By looking at the pattern of correlations in the triangles, this lets us see if the pattern of convergent and divergent validity is about the same. So we can see how stable the pattern of validity is.
• i.e. is the pattern of validity about the same for all the factors measured by the tests?
• Look for patterns by either looking at: a) are the numbers roughly the same across columns. or b) are the numbers in the columns greater than the single digit in the corner.