Week 10 - Psychometrics and Test Construction

Question 1

What are the 6 steps in test/questionnaire construction?

Answer 1

Define the test
select a scaling method
construct the test items
test the items
revise the test
publish the test

Question 2

What is a test?

Answer 2

A set of items that allows measurement of some attribute

Question 3

What is an item?

Answer 3

generic word for the various forms of content in a psychological test (eg a question)

Question 4

2 things to consider when defining a test?

Answer 4

what is it that i am seeking to measure

has there already been a test developed

Question 5

Kaufman and Kaufman model of the test definition process

Answer 5

Theoretical and research basis
distinguishable
translatable
novel questions
easy to administer
sensitive to needs

Question 6

What are the 2 types of data?

Answer 6

Categorical and Numerical

Question 7

What are examples of Categorical data?

Answer 7

gender, age, political party etc

Question 8

What are the 2 types of Numerical data?

Answer 8

Discrete and Continuous

Question 9

What is an example of discrete data?

Answer 9

number of children, assignment mark

Question 10

What is an example of continuous data?

Answer 10

weight, voltage, length

Question 11

What are the 4 types of measurement?

Answer 11

Nominal, ordinal, interval and ratio

Question 12

What is Nominal measurement

Answer 12

Just a group you put someone in (categorical)

The numerical value assigned to each group is meaningless (eg gender)

Question 13

What is Ordinal measurement?

Answer 13

A ranking order (eg first place, second place)

We cannot tell the distance between each point

Question 14

What is an Interval measurement?

Answer 14

Continuos data eg temperature
there is NO true 0
Often referred to as Likert-type scale eg 1 = strongly agree

Question 15

What is a Ratio measurement?

Answer 15

This is a continuous measurement with a starting point of 0.

Differences between starting points on the scale are meaningful (eg weight and length)

Question 16

What is homogeneity?

Answer 16

Does it allow for attributes to be reliably measured

Question 17

What is heterogeneity?

Answer 17

Does it allow for adequate differentiation of people?

Question 18

2 things that are important in constructing the items of a test?

Answer 18

initial questions

item format

Question 19

4 item formats

Answer 19

MCQ
True/false Qs
Forced-Choice Method
Likert-Type Scales

Question 20

Problems with MCQ

Answer 20

doesn’t assess free recall

Question 21

Problems with T/F (2)

Answer 21

Does not permit much variability

answers may reflect social desirability more than personality traits

Question 22

Problems with FCM

Answer 22

People do not always fit in to one category

Question 23

Positive of LTS

Answer 23

Can account for individual differences

Question 24

Tips for writing good tests

Answer 24

Qs should be simple
Use clear words
Offer an out for Qs
Avoid offering too many options

Question 25

More common events =

Answer 25

shorter window of recall

Question 26

Steps to consider when testing the items (5)

Answer 26

Pilot study
Administer to large sample
Do number crunching
Investigate difficulty levels of each item
Determine reliability and validity

Question 27

What does cross-validate mean?

Answer 27

Testing whether the newly refined test performs just as well in a new sample

Question 28

Revising the test should consist of what? (3)

Answer 28

Cross validating
Making refinements
Obtaining feedback

Question 29

Publishing the test should consist of

Answer 29

developing a technical and users manual (background info, development history, instructions etc)

Question 30

3 types of Validity

Answer 30

Content, Construct and Criterion related Validity

Question 31

What is Content Validity?

Answer 31

Determined by the degree to which items are representative of the domain of behaviour the test purports to measure
(low content validity would be if the exam was made up entirely of Q’s about WAIS)

Question 32

What is Construct Validity

Answer 32

The appropriateness of inferences about the underlying construct (is extraversion item in your test match extraversion/outgoingness items in other tests)

Question 33

What is Correlation?

Answer 33

Statistical measure to indicate the extent to which two variables are related

Question 34

Correlation measurement

Answer 34

-1 to 1
0 = no correlation
1 = high positive
-1 = high negative

Question 35

What is Factor Analysis?

Answer 35

Statistical technique used to determine the pattern of correlations or variability amongst the items

Question 36

What do factors represent?

Answer 36

Underlying abilities

Question 37

What is Criterion Validity?

Answer 37

The extent to which the test predicts or is related to an outcome (eg does performance in IQ test predict academic success)

Question 38

What are the 2 types of Reliability?

Answer 38

Internal and External

Question 39

What does Internal Reliability mean?

Answer 39

The extent to which a measure is consistent within itself (eg are the items within the MMPI related and measuring the same construct?)

Question 40

What is External Reliability?

Answer 40

The extent to which a measure varies from one use to another use

Question 41

What are the 2 types of External Reliability?

Answer 41

Test-Retest

Inter-Rater

Question 42

What does Test-retest reliability mean?

Answer 42

The stability over time (if i do a test today and then in 6 months will i get the same score)

Question 43

What does Inter-Rate reliability mean?

Answer 43

The degree to which different raters give consistent estimates of same behaviour (do examiner A and B observe and score the same behaviour similarly)

Question 44

What are 3 sources of error?

- Things about the test x 3

Answer 44

Test construction
Test administration
Test scoring and interpretation

Question 45

What is meant by test construction error?

Answer 45

item sampling or content sampling (variation among items)

Question 46

What is meant by test administration error?

Answer 46

external factors that affect test-taker attention and motivation

Question 47

What is meant by test scoring and interpretation error?

Answer 47

rigidity and rigorousness of scoring criteria (meaning of test results in relation to psychological functioning)