Issues of Measurement and Testing

Question 1

What is dimensionality?

Answer 1

Whether an instrument (test) measures a single constuct (dimension) or many

Question 2

What is reliability?

Answer 2

Whether a test measures a construct consistently

Question 3

What are psychometric properties expressed as?

Answer 3

An index, coefficient or other numerical quantity

Question 4

What is standardisation?

Answer 4

• The process of establishing norms for a test
• The use of uniform procedures, same conditions, scored by same criteria allowing results to be compared
• The transformation of data into a distribution of standardised scores, often having a mean of 0 and SD of 1

APA Dictionary of Psychology (2023)

Question 5

What are the 2 common standardisation methods?

Answer 5

Norm-referencing and Criterion -referencing

Question 6

What is norm-referencing?

Answer 6

“Compares the score of an individual with those of other candidates who took the test under similar conditions (norm group) “ - (Rust,2007)
- norm group should be representative of whole population
- Allows for meaningful comparisons

Question 7

What is criterion-referencing?

Answer 7

• Scores compared with some objectively assessed reference point or standard
• Not commonly used in personality testing, but potentially relevant for psychopathology diagnosis

Question 8

What is a percentile rank?

Answer 8

A number where a certain % of scores fall below

Question 9

What are z-scores?

Answer 9

The value of a z score tells you how many standard deviations you are away from the mean

Question 10

What is validity?

Answer 10

The degree to which empirical evidence and theoretical rationales support the interpretations based on test scores or other measures (West & Finch, 2007)

Question 11

What is reliability?

Answer 11

A measure of reproducibility or dependability of measurements, free from error

Question 12

What is test-retest reliability?

Answer 12

Stability over time/repeatability

Question 13

What is internal consistency?

Answer 13

Whether all items are measuring the same thing

Question 14

What is inter-rater reliability?

Answer 14

The degree to which different raters’ scores/codes/ratings are correlated

Question 15

What factors influence test retest reliability?

Answer 15

• Characteristics of test takers e.g. illness, tiredness
• Characteristics within tests - poor instructions, complexity
• Differences in conditions - time of day, distractions
• Time gaps - must be minimum of 3 months
• Difficulty level - floor/ceiling effects
• Sample size and sampling

Question 16

What’s the most used index of internal consistency?

Answer 16

Cronbach’s Alpha - reliability should never be below.7

Question 17

How can bias be avoided?

Answer 17

Several raters and ensure assessments are consistent across raters

Question 18

What are the 2 common ways to measure inter-rater reliability?

Answer 18

Percent agreement and Cohen’s Kappa

Question 19

What is a percent agreement?

Answer 19

% of items the judges agree on, between 0 and 1

Question 20

What is Cohen’s Kappa?

Answer 20

Calculates the % of items raters agree on, while accounting for the fact that raters may happen to agree on items due to chance
- Ranges between 0 and -1, with -1 indicating systematic disagreement between raters
- 0.7-0.8 is acceptable

Question 21

What is construct validity?

Answer 21

The degree to which an assessment tool adequately measures a hypothesised psychological construct

Question 22

What are the different types of construct validity?

Answer 22

• Face
• Content
• Convergent
• Discriminant
• Predictive

Question 23

What is face validity?

Answer 23

The extent to which a test appears to measure what it claims to based on fave value

Question 24

What is content validity?

Answer 24

Concerned with a test’s ability to include or represent all of the contents of a particular construct

Question 25

What is convergent validity?

Answer 25

The extent to which scores from a new test correlate with other measures of the same phenomenon

Question 26

What is discriminant validity?

Answer 26

Refers to the extent to which a test score does not correlate with the scores of theoretically unrelated measures

Question 27

What is predictive validity?

Answer 27

Evidence that a test score or other measurement correlates with another variable that can only be assessed at some point after the test has been administrated

Question 28

What features of psychometric tools might influence the degree to which they are valid and reliable?

Answer 28

• Methods of data collection and bias
• Purposes of tests
• Cross culture validity?

Question 29

What are the strengths of psychometric tests?

Answer 29

• Objective and scientific way of describing people and their behaviour
• Usually quick and easy
• Allows for statistical analysis

Question 30

What are the weaknesses of psychometric tests?

Answer 30

• Difficult to make valid and reliable
• Culture bias

Question 31

What are some sources of innacuracy and bias in self report measures?

Answer 31

Extreme responding, dissent bias (agree/disagree with questtionaire irrespective of content), SDB, Recall bias, hostility bias

Question 32

How did Crowne (1964) theorise SDB?

Answer 32

Socially desirable responses reflect a repressive defense against vulnerable self-esteem

Question 33

What did Zemore find about SDB and when?

Answer 33

Higher SDB scores have been linked with outcomes such as better attendence of drugs/alcohol treatment programmes

Question 34

What is the Lees-Haley Fake Bad Scale (FBS)/ MMPI Symptom Validity Scale?

Answer 34

• 43 items in the Minnesota Multiphasic Personality Inventory selected by Less-Haley (1991) to detect malingering in personal injury claimants

Question 35

What is impression management?

Answer 35

Ways people attempt to control how they are perceived by others (Goffman, 1959)

Question 36

How do people mitigate against impression management?

Answer 36

• Lie scales to flag who is lying
• Forced choice items
• Inconsistency scales
• Multiple assessment methods (other than self-report)