Psychometrics; Lec 9 & 11 (no 10 due to bank hol); Lab 5

Question 1

What is a psychometric test?

Answer 1

A psychometric test is a standardised procedure for sampling behaviour and describing it using scores or categories

Question 2

Most tests are ‘norm-referenced’ what does this mean?

Answer 2

They describe the behaviour in terms of norms, test results gathered from a large group of subjects.

Question 3

While most tests are norm-referenced, some are ‘criterion-referenced’, what does this mean?

Answer 3

The objective is to see if the subject can attain some pre-specified criterion

Question 4

What are 5 things one should consider in writing a test?

Answer 4

1. Ensure that all aspects of the construct are dealt with e.g. anxiety - all aspects
2. Need to be long enough to be reliable - start with 30 questions and reduce to 20
3. Should assess only one trait
4. Should be culturally neutral
5. Should not be the same item rephrased (mentioned during FA)

Question 5

In terms of establishing item suitability. There should not be too many items which are…?

Answer 5

There should not be too many items which are either very easy or very hard.

i.e., 10% of items with scores .8 is questionable

Question 6

In terms of establishing item suitability; items should have an acceptable standard deviation, what does this mean?

Answer 6

If the SD is too low then it is not tapping into individual differences.

Question 7

In terms of establishing item suitability; if there are different constructs then…

Answer 7

… it is important that an equal number of items refers to each construct.

Question 8

What is criterion keying and how is it used to establish item suitability?

Answer 8

Criterion keying - items are chosen based on their ability to differentiate the population in general from a specific group (e.g. surgeons, pilots).

Criterion keying is atheoretical

Groups must be well defined

Question 9

Why should you interpret measures that have been established using criterion keying liberally?

Answer 9

Because there will be overlap in response distribution.

Question 10

How is factor analysis used to establish item suitability?

Answer 10

Based on FA, items that have a low loading (

Question 11

Classical item analysis is used to establish item suitability and improve reliability, how does this work?

Answer 11

Based on classical item analysis, the correlation of an item’s score with the score on the whole test (excluding that item) is calculated.

Removing items with low correlations improves reliability. Although, because reliability is also a product of the number of items, there is a balance.

Each time an item is removed the correlation of each item to the main score must be recalculated since this will change as items are removed.

Question 12

How many psychological constructs should each scale measure?

Answer 12

One

Question 13

What does ‘measurement error’ mean?

Answer 13

That for any one item, the psychological construct only accounts for low % of the respondent’s variation. (other factors cause most of variation - age, religious beliefs, sociability, peer-group pressure)

Question 14

How do you get rid of random variation (e.g. age, religious beliefs) when building a scale?

Answer 14

Use several items and this variation should cancel itself out such that measured variance is due to the underlying construct

Question 15

One important measure of reliability for psychometric instruments is that of temporal stability - what is this? What is it often referred to as?

Answer 15

Temporal stability is often referred to as ‘test-retest reliability’.

Temporal stability involves administering the same test to people over a time span. It measures whether the measure produces the same outcomes over time.

e.g. If a respondent scores strongly as an extrovert on a particular day and then, 2 weeks later, scores strongly as an introvert, we may begin to question whether the instrument is measuring anything useful.

Question 16

What are two ways of measuring the extent to which a scale measures one construct only? (a form of reliability testing)

Answer 16

1. Split half reliability

2. Chronbach’s Alpha

Question 17

What is split half reliability?

Answer 17

Split half testing measures internal consistency reliability.

Steps:

1. Adminster the test to a large group of students
2. Randomly divide the test questions into two parts. For example, separate even questions from odd questions.
3. Score each half of the test for each student
4. Find the correlation coefficient (Pearson’s) for the two halves - a reliable test will have high correlation.

Question 18

Split-half reliability and Cronbach’s Alpha are measures of internal consistency, what does this mean?

Answer 18

Internal consistency reliability is a way to gauge how well a test or survey is actually measuring what you want it to measure.

Question 19

What is Cronbach’s Alpha?

Answer 19

Cronbach’s Alpha measures internal consistency reliability (how well a test or survey is actually measuring what you want it to measure) for multiple question likert scales.

Question 20

What are some problems with Cronbach’s Alpha?

Answer 20

1. It is influenced by the average correlation between the items and the number of items in the test
2. It can be artificially boosted by asking the same question twice
3. Test should not be used if alpha is below .7

Question 21

What are 3 types of validity?

Answer 21

1. Logical
2. Statistical
3. Construct

Question 22

What are two types of logical validity?

Answer 22

1. Face

2. Content

Question 23

What are two types of statistical validity?

Answer 23

1. Concurrent

2. Predictive

Question 24

When first trying to decide upon a research methodology and a suitable instrument, what are 3 very practical issues that need to be considered?

Answer 24

1. Is the research question well established or more exploratory in nature?
2. Should the questions conform to an open or closed format?
3. Are the instruments of interest licensed or freely available?

Question 25

With regards to psychometrics, what are the practical implications of the research question being well established?

Answer 25

Concepts will mostly already be defined, meaning you can point to an existing psychometric measure.

Questions can build on previous research meaning they can be more precise than they would be in exploratory research

Question 26

With regards to psychometrics, what are the practical implications of the research question being more exploratory in nature?

Answer 26

If the latent constructs you are measuring are less well-studied you may need to adapt measures, or use multiple measures to try to reach the construct you are looking for.

Question 27

With regards to psychometrics, what are the practical implications of using questions that are open ended?

Answer 27

Probably more appropriate to use interview style, giving participants time and space to speak at length

Design likely to be qual/content analysis

If using FA would be EFA

Question 28

With regards to psychometrics, what are the practical implications of using questions that are closed format?

Answer 28

Closed questions are more quant - this numerical data is likely to be analysed through correlations of EFA.

Question 29

With regards to psychometrics, what are the practical implications of using questionnaires that are licensed as opposed to freely available?

Answer 29

If something is licensed it can cost a lot of money; this can be prohibitive (e.g., for students).

Question 30

When we have decided what kind of scale we want to use, how do we choose from a range of options?

Answer 30

We rate potential scales based on their psychometric properties with respect to validity and reliability.

Question 31

What are the 4 different types of validity?

Answer 31

1. Face validity
2. Content validity
3. Construct validity
4. Criterion validity

Question 32

What is face validity?

Answer 32

Considering what the test is supposed to be measuring, do the questions seem to be addressing the issues that they should be? (a somewhat superficial form of validity, but usually quite obvious)

Question 33

What is content validity?

Answer 33

Does the scale measure the full breadth of the concept?

Question 34

What is construct validity?

Answer 34

The extent to which the test measures the theoretical construct that it is supposed to measure.

Question 35

What is criterion validity? What are two types of criterion validity?

Answer 35

A more quantitative form of validity - it compares the scale against a direct measure of the construct.

Can be divided into:

1. Convergent
2. Divergent

Question 36

Convergent validity is a sub-type of criterion validity (comparing the scale against a direct measure of the concept) - what question does it attempt to answer?

Answer 36

Does the scale give similar results to those of other measures of the same concept?

Question 37

Divergent validity is a sub-type of criterion validity (comparing the scale against a direct measure of the concept) - what question does it attempt to answer?

Answer 37

Does the scale give results different from a scale that is supposed to measure a different concept?

Question 38

Why can good face validity be conceived of as both a good and bad thing?

Answer 38

It is good because it likely won’t be confusing to participants - i.e. they will understand what is being asked of them.

It is bad because it makes it potentially easier to manipulate by providing socially desirable answers.

Question 39

How could you measure content validity for the GHQ-12?

Answer 39

You could have it reviewed by mental health experts and compared to the DSM to see if it measures all the symptoms of dysphoria.

Question 40

How could you measure the construct validity of the GHQ-12?

Answer 40

A commonly used method to measure construct validity is confirmatory factor analysis (Atkinson et al., 2011). You can use CFA to see whether the measure actually just measures dysphoria as it is supposed to, or whether it measures any other latent variables.

Question 41

How could you measure the convergent criterion validity of the GHQ-12?

Answer 41

You could compare results to a similar, validated measure on depression e.g. Becks inventory of depression.

People who score high in depression on GHQ should also score high in depression on Becks.

Question 42

How could you measure the divergent criterion validity of the GHQ-12?

Answer 42

You could compare results to a different, validated measure on a similar, but different construct e.g., anxiety - Becks anxiety inventory

Assuming the individual does not have comorbid anxiety (something that should be assessed), someone who scores high on the GHQ should not have a similar score on the BAI.

Question 43

What is the most basic form of reliability?

Answer 43

Test-retest reliability - it involves determining whether if you repeat a test you get similar results across both sets of trials

Question 44

What is the internal consistency of a scale?

Answer 44

It measures whether several items that propose to measure the same general construct produce similar scores.

Question 45

Using reliability analysis you can do 3 things, what are they?

Answer 45

1. Determine the extent to which items in your questionnaire are related to each other
2. Get an overall measure of the internal consistency of the scale as a whole
3. Identify problem items that should be excluded from the scale

Question 46

What 5 models of reliability are available in SPSS?

Answer 46

1. Alpha (Cronbach)
2. Split-half
3. Guttman
4. Parallel
5. Strict parallel

Question 47

What is Cronbach’s Alpha? How does it work?

Answer 47

It is a model of internal consistency, based on the correlation between the sums of items taken from the two halves of the questionnaire. The Cronbach method produces all possible split-half combinations, finds the correlation between them, and quotes an average of these correlations.

Question 48

Why is it important to compute a reliability measure every time a questionnaire is used?

Answer 48

Because reliability is sample dependent (for example, hungover university students might not care as much about giving accurate answers as someone who is paying a mental health professional to deliver the same test.

Question 49

How do you compute Cronbach’s Alpha (1951) in SPSS?

Answer 49

Analyze –> Scale –> Reliability Analysis –> move all items of the scale to be analysed into the ‘item’ dialogue box –> OK

Question 50

What is the acceptable score for Cronbach’s Alpha?

Answer 50

> 0.7

Question 51

What is reliability?

Answer 51

The reliability of any given measurement refers to the extent to which it is a consistent measure of a concept