Introduction to Psychometrics & Validity (Week 2)

Question 1

state what it is meant by the key term - ‘construct validity’

Answer 1

the degree to which evidence and theory support the interpretation of test scores

(American Educational Research Association)

Question 2

construct is an ongoing process, meaning…

3 points

Answer 2

1) it takes many years
2) many studies must be done
3) new additions to the theory

Question 3

how do we measure psychological constructs, and why do we do it that way?

Answer 3

• self-reported measures (e.g. - questionnaires)

- this is because we don’t have hard outcome measures like we do in other sciences

Question 4

state what it is meant by the key term - ‘construct’

Answer 4

a construct is a psychological phenomenon (e.g. - SE, neuroticism, SDT)

Question 5

state 3 benefits of using questionnaires

Answer 5

1) cost effective
2) time effective
3) easy to administer

Question 6

what is the issue with the use of questionnaires? (4 points)

Answer 6

• to what extent can we trust the data obtained?
• confusing word choices
• scaling (more or less needed?)
• no reading of instructions if too long (too much effort)

Question 7

what 3 questions do we need to ask ourselves if there is construct validity?

Answer 7

1) does the data obtained align with the theory/evidence?
2) what is the meaning of the test scores?
3) can we trust the data obtained?

Question 8

give 2 examples to why we may not trust obtained data

Answer 8

1) did the respondent answer 100% accurately

2) were you measuring the correct construct or another one by accident?

Question 9

state, in order, the 6 aspects of construct validity

Answer 9

1) content validity
2) substantive validity
3) structural validity
4) external validity
5) consequential validity
6) generalisability validity

Question 10

state what it is meant by the key term - ‘validity’

Messick., 1989

Answer 10

an overall evaluative judgement of the degree to which empirical evidence and the theoretical rationales support test scores and other modes of assessment (Messick., 1989)

Question 11

what 3 things are test scores a function of?

Answer 11

1) items/stimulus
2) person responding
3) assessment context

Question 12

what did (Cronbach., 1971) say about what needs to be valid?

Answer 12

‘what needs to be valid is the interpretation of the test score’ (Cronbach., 1971)

Question 13

state what it is meant by the key term - ‘score’

Messick., 1995

Answer 13

any coding or summarising observed consistencies or performance regularities on assessment devises (Messick., 1995)

Question 14

what did (Cronbach & Meehl., 1955) say about the comprehensiveness of construct validity

Answer 14

“construct validity is based on an integration of any evidence that bears on the interpretation or meaning of the test scores” (Cronbach & Meehl., 1955)

Question 15

state what it is meant by the key term - ‘contstruct representation’

(Embreston., 1983)

Answer 15

put simply, Construct representation is concerned with identifying the theoretical mechanisms that underlie responses (Embreston., 1983)

Question 16

the 6 aspects of validity apply to all psychological and educational measurement, including performance assessment…

Answer 16

taken together, they provide a way of addressing the multiple and interrelated validity questions that need to be answered to justify some interpretation and use (Messick., 1995)

Question 17

what are the two threats to construct validity ?

Answer 17

1) construct underrepresentation

2) construct irrelevant variance (difficulty + easiness)

Question 18

state what it is meant by the key term - ‘construct underrepresentation’

Answer 18

the assessment is too narrow and fails to include important dimensions and facets of the construct (Messick., 1995)

Question 19

state what it is meant by the key term - ‘construct irrelevant variance’

Answer 19

the assessment is too broad, containing excess variance associated with other distinct constructs as well as method variance (such as guessing) that affect responses irrelevant to the interpreted construct (Messick., 1995)

Question 20

state what it is meant by the key term - ‘construct irrelevant difficulty’

Answer 20

aspects of the task that are extraneous to the focal content make the task irrelevantly difficult for some responders (e.g. - word choice)

Question 21

state an issue with ‘construct irrelevant variance’

Answer 21

can cause bias (Holland & Wainer., 1993)

Question 22

state what is it meant by the key term - ‘construct irrelevant easiness’

Answer 22

occurs when extraneous clues in item or task formats permit some individuals to respond correctly or appropriately in ways irrelevant to the construct being assessed (Messick., 1995)

Question 23

state and explain an issue with ‘construct irrelevant easiness’

Answer 23

can occur when some test material is familiar to others which leads to incredibly high scores for affected individuals (Messick., 1995)

Question 24

what is the primary measurement concern with respect to adverse consequences ?

Answer 24

the primary measurement concern is that any negative impact on individual’s or groups should not come from any test invalidity, such as construct underrepresentation or construct irrelevant variance (Messick., 1989)

low scorers should not occur because the assessment is missing something relevant to the construct

low scorers should not occur because the assessment is containing something irrelevant to the construct

Question 25

state the (Wiggins., 1993) and (Messick., 1994) statements on ‘construct irrelevant variance’

Answer 25

construct irrelevant variance is important in all educational and psychological measurements, including performance assessment (Wiggins., 1993)

however, what constitutes construct irrelevant variance is a tricky and continuous issue (Messick., 1994). typically, many skills are higher order in sports performance

Question 26

state the (Cacio et al., 1991) and (Gottfredson., 1994) statements on ‘construct irrelevant validity’

Answer 26

another issue arises when construct irrelevant variance is capitalised upon to produce desired outcomes (Cacio et al., 1991) eg - score adjustments for minority groups

recognising that adjustments distort the meaning of the construct can help psychologists ensure it doesn’t happen (Gottfredson., 1994)

Question 27

what does (Kane., 1992) say about scores and construct validity?

Answer 27

“almost any kind of information about a test can contribute to an understanding of score meaning, but the contribution becomes stronger if the degree of fit of the information with the theoretical rationale underlying the score interpretation is explicitly valued” (Kane., 1992)

Question 28

potential and actual consequences of test use must be considered for 2 reasons (Gallikson et al., 1950), what are they?

Answer 28

1) applications of likely outcomes may guide one to look for side effects
2) such anticipation may alert one to take timely steps to capitalise on some positive effects and to forefall negative effects

Question 29

what did (Messick., 1995) say which contradicts (Gallikson et al., 1950) about potential and actual consequences?

Answer 29

unintended consequences are also strands in the constructs nomological network that needs to be taken into account in the constructs theory, score interpretation, and test use (Messick., 1995)

Question 30

what is content validity ? (2 points)

Answer 30

‘what should the questionnaire include?’

measures the relevance, representativeness, and the technical quality of the measure

Question 31

state 3 items that negatively affect content validity

Answer 31

1) unclear, overly complicated, overly-colloquial questions
2) no variability in responses
3) double barrelled questions

Question 32

what are the 3 steps in achieving content validity ?

Answer 32

1) review literature to determine:
- construct conceptualisation, dimension of construct, current measures

2) develop preliminary questionnaire based on review
- items, scaling, instructions, presentation, language

3) obtain feedback from experts in the field
- all aspects assessed?
- clear instructions?

Question 33

what is a key issue with obtaining content validity?

Answer 33

the specification of the boundaries of the construct that need to be assessed (Brunswik., 1956)

Question 34

what did (Brunswik., 1956) come up with to overcome the issue of not knowing what to assess for content validity? (3 points)

Answer 34

Ecological Sampling (Brunswik., 1956)

• an in depth analysis of the domain
• intent is to ensure that all aspects of the domain are covered

Question 35

what is substantive validity? (2 points)

Answer 35

1) how do respondents interpret the questionnaire and its items?

2) is there an alignment between what’s supposed to be measured and the respondents interpretations?
- e.g. are members of your team on the same page?

Question 36

how is substantive validity obtained?

Answer 36

1) test preliminary questionnaire with sample from target population
- what do you think this question is asking?
- did the scale allow you to answer properly?
- did all questions apply to you?

Question 37

(Messick., 1995) stated that there are 2 important points to consider when talking about substantive validity, what are they?

Answer 37

1) the need for tasks providing appropriate sampling of domain processes in addition to traditional coverage of domain content
2) the need to move beyond traditional judgement of domain content to accrue empirical evidence from responders

Question 38

what did (Loevinger., 1957) say about substantive validity?

Answer 38

“the substantive aspect adds empirical evidence from the domain to content validity” (Loevinger., 1957)

Question 39

what bridges content and substantive validity ?

Answer 39

representativeness (Messick., 1995)

Question 40

what is structural validity?

Answer 40

does the data fit/align with the theoretical propositions of that construct?

date = theory/evidence?

Question 41

state and explain 3 points to how we achieve structural validity ?

Answer 41

1) larger sample of your participants complete your questionnaire

2) quantitive analysis of data
- e.g.) factor analysis, relationships between questions
- does data ‘fit’ the theory?

3) does the measure need refinement?
- be careful, if not removed in content validity, then surely it was important to include?
- if we remove it, we may miss something theoretically relevant

Question 42

what did (Loevinger., 1957) say about structural validity ?

Answer 42

structural validity should aid the selection of tasks, and also the rationale development of construct-based scoring criteria (Loevinger., 1957)

Question 43

what is external validity?

Answer 43

how does your data relate to the data from other related constructs?

date = theory/evidence?

Question 44

how do we get external validity? (3 points)

Answer 44

1) participants complete your measure and other relevant measures
- e.g.) personality and self-efficacy

2) is the data from your measure related to the other measures as expected?
- e.g.) positive, negative, no relationship?

3) come back to the theory and/or previous research

Question 45

both convergent and discriminant correlation patterns are important for external validity. what are ‘convergent patterns’?

Answer 45

the convergent pattern indicating a correspondance between measures of the same construct (Campbell & Fiske., 1957)

Question 46

both convergent and discriminant correlation patterns are important for external validity. what are ‘discriminant patterns’?

Answer 46

the discriminant evidence is particularly critical for discounting plausible rival alterations to the focal construct interpretation (Messick., 1995)

Question 47

what is generalisability validity ?

Answer 47

to what extent do the results generalise across populations, contexts, or tasks

Question 48

how do we get generalisability validity? (3 points)

Answer 48

1) participants from samples of various populations complete your measure and other measures
2) does the data align with your original data? why? why not?

3) testing generalisability is an ongoing process
- you don’t have it, you are on a low to high scale with it

Question 49

what did (Feldt & Brennen., 1989) say about generalisability validity?

Answer 49

“the limit of score meaning affects generalisability across tasks. Error in the sampling of tasks, occasions, and scorers underlie traditional reliability concerns” (Feldt & Brennen., 1989)

Question 50

what does (Wiggins., 1993) say about generalisability validity?

Answer 50

“because of the time required to achieve generalisability, it may be depicted that a trade off occurs between validity and power of interpretation. such a conflict signals a design problem that needs to be carefully negotiated in performance assessment” (Wiggins., 1993)

Question 51

what is consequential validity?

Answer 51

what are the implications that may result from using a questionnaire?

1) can the measure be used for a ‘basis for action’?
2) are there any potential consequences of test use?
- follow-up appointments
- use data from other sources

Question 52

what is Bull’s mental skills questionnaire? (2 points)

Answer 52

1) sports psychologists could use this to guide their treatment with an athlete
- i.e. constructs with the worse test score from the athlete can be the initial focus of an intervention