lecture 4- ecological and construct validity

Question 1

what is ecological validity?

Answer 1

• Ecological validity refers to the degree to which test performance corresponds to
  real-world performance
• Very relevant to applied situations – what are the functional consequences of
  stroke, brain injury, dementia etc.
• Has been suggested that some neuropsychological tests have limited validity

Question 2

how can ecological validity be assessed?

Answer 2

Test scores can be compared with rating of everyday behaviour for the domain of
interest using:

-Self and informant questionnaires
-Clinical rating scales
-Observation of simulated tasks

Question 3

what does executive function refer to?

Answer 3

executive function refers to cognitive skills in problem solving, decision making, planning and completing tasks, and reflecting on activity

Question 4

dysexecutive problems:

Answer 4

starting or finishing tasks, planning ahead, making decisions, thinking through problems and forming solutions, behaving appropriate and controlling emotions such as anger

Question 5

tests to measure dysexecutive problems-

Answer 5

• Range of tests that measure dysexecutive problems, e.g. Behavioral Assessment
  of the Dysexecutive Syndrome (BADS: Wilson et al., 1996).
• Dysexecutive Questionnaire (DEX; Burgess et al., 1998)
  =>20 items measuring behavioral, cognitive, motivational and emotional
  changes from pre-morbid functioning generating.
  -Neuropsychological tests of executive function can be compared with DEX scores

Question 6

burgess et al., 1998

Answer 6

• Burgess reports modest correlations

Better correlations when using informant, clinician, parent/carer ratings
Test results will be affected by:
* Environment
* Limited behaviour observed
* Compensatory strategies

Ratings of everyday functioning are by no means perfect
* Relatives may be protective, or may be overly negative in their
appraisals

Taking these into consideration, ability to predict is improved

Question 7

what is construct validity?

Answer 7

construct validity is the broadest form of validity: does a test measure what its meant to test

Question 8

state- constructs are unobservable but the tests measure them- the WAIS measures intelligence

Question 9

Researchers can assess construct validity by making predictions on how scores may
change in various ways?

Answer 9

◦ Test Homogeneity
◦ Evidence from changes with age
◦ Evidence from pre-test and post-change changes
◦ Evidence from distinct groups
◦ Convergent and divergent evidence
◦ Factor Analysis

Question 10

what is factor analysis?

Answer 10

factor analysis determines the underlying relationships between sets of variables such as test scores

Question 11

state- relationships are called factors (ie the construct such as intelligence or personality traits)
- here we are not interested in the statistical procedures

Question 12

state- In psychometrics FA is used as a data reduction technique – it take individual test and the correlations between them. Patterns of scores clustering together suggests they are measuring the same thing
◦ E.g. Do you like going to parties / socialising / are you the life and soul of a party etc., and
similar scores may all be measuring – extraversion

Question 13

so what this got to do with CV?

Answer 13

researchers can collect data and make specific predictions on how scores should correlate based on their theory

Question 14

state- factor analysis has been used to study construct validity

Question 15

what is confirmatory factor analysis?

Answer 15

• Confirmatory factor analysis is a relatively recent technique for evaluating the
  construct validity of psychological tests
• It is now very widely used in psychological research
• Therefore you should become familiar with it if you want to make sense of the
  literature on psychological assessment and in other areas of psychology
• As a starting point we will use the example of the WAIS-IV and ask whether
  intelligence is made up of one, two, or more constructs

Question 16

assessing the fit of CFA models

Answer 16

• We get a number of statistics that tell us which model the data best fits – these
  are collectively called fit statistics. For example:
• A large chi square value ( 2 ) tells you that a model has poor fit.
• Cumulative Fit Indices (CFI): there are a variety of these. All have the characteristic that they range from 0 to 1.
• High values (i.e. at least above 0.9) for a CFI suggest that the model has good fit

-There are other measures of fit but we will limit ourselves to these two

Question 17

conclusions from testing CFA models

Answer 17

-The findings support the construct validity of the WAIS-IV

• That is, the subtests are combined into four indexes plus an overall IQ, and CFA tells us
  that the model that specifies that structure has the best fit
• As the data shown were for a UK sample, the results are particularly supportive

-That is they replicate the results found in the USA standardization sample

• Shows the WAIS-IV is measuring the same broad dimensions of ability in UK and USA

-Psychologists have more confidence in a measure if initial findings have been
replicated