Week 10 - Psychometrics and Test Construction Flashcards
What are the 6 steps in test/questionnaire construction?
Define the test select a scaling method construct the test items test the items revise the test publish the test
What is a test?
A set of items that allows measurement of some attribute
What is an item?
generic word for the various forms of content in a psychological test (eg a question)
2 things to consider when defining a test?
what is it that i am seeking to measure
has there already been a test developed
Kaufman and Kaufman model of the test definition process
Theoretical and research basis distinguishable translatable novel questions easy to administer sensitive to needs
What are the 2 types of data?
Categorical and Numerical
What are examples of Categorical data?
gender, age, political party etc
What are the 2 types of Numerical data?
Discrete and Continuous
What is an example of discrete data?
number of children, assignment mark
What is an example of continuous data?
weight, voltage, length
What are the 4 types of measurement?
Nominal, ordinal, interval and ratio
What is Nominal measurement
Just a group you put someone in (categorical)
The numerical value assigned to each group is meaningless (eg gender)
What is Ordinal measurement?
A ranking order (eg first place, second place)
We cannot tell the distance between each point
What is an Interval measurement?
Continuos data eg temperature
there is NO true 0
Often referred to as Likert-type scale eg 1 = strongly agree
What is a Ratio measurement?
This is a continuous measurement with a starting point of 0.
Differences between starting points on the scale are meaningful (eg weight and length)
What is homogeneity?
Does it allow for attributes to be reliably measured
What is heterogeneity?
Does it allow for adequate differentiation of people?
2 things that are important in constructing the items of a test?
initial questions
item format
4 item formats
MCQ
True/false Qs
Forced-Choice Method
Likert-Type Scales
Problems with MCQ
doesn’t assess free recall
Problems with T/F (2)
Does not permit much variability
answers may reflect social desirability more than personality traits
Problems with FCM
People do not always fit in to one category
Positive of LTS
Can account for individual differences
Tips for writing good tests
Qs should be simple
Use clear words
Offer an out for Qs
Avoid offering too many options
More common events =
shorter window of recall
Steps to consider when testing the items (5)
Pilot study Administer to large sample Do number crunching Investigate difficulty levels of each item Determine reliability and validity
What does cross-validate mean?
Testing whether the newly refined test performs just as well in a new sample
Revising the test should consist of what? (3)
Cross validating
Making refinements
Obtaining feedback
Publishing the test should consist of
developing a technical and users manual (background info, development history, instructions etc)
3 types of Validity
Content, Construct and Criterion related Validity
What is Content Validity?
Determined by the degree to which items are representative of the domain of behaviour the test purports to measure
(low content validity would be if the exam was made up entirely of Q’s about WAIS)
What is Construct Validity
The appropriateness of inferences about the underlying construct (is extraversion item in your test match extraversion/outgoingness items in other tests)
What is Correlation?
Statistical measure to indicate the extent to which two variables are related
Correlation measurement
-1 to 1
0 = no correlation
1 = high positive
-1 = high negative
What is Factor Analysis?
Statistical technique used to determine the pattern of correlations or variability amongst the items
What do factors represent?
Underlying abilities
What is Criterion Validity?
The extent to which the test predicts or is related to an outcome (eg does performance in IQ test predict academic success)
What are the 2 types of Reliability?
Internal and External
What does Internal Reliability mean?
The extent to which a measure is consistent within itself (eg are the items within the MMPI related and measuring the same construct?)
What is External Reliability?
The extent to which a measure varies from one use to another use
What are the 2 types of External Reliability?
Test-Retest
Inter-Rater
What does Test-retest reliability mean?
The stability over time (if i do a test today and then in 6 months will i get the same score)
What does Inter-Rate reliability mean?
The degree to which different raters give consistent estimates of same behaviour (do examiner A and B observe and score the same behaviour similarly)
What are 3 sources of error?
- Things about the test x 3
Test construction
Test administration
Test scoring and interpretation
What is meant by test construction error?
item sampling or content sampling (variation among items)
What is meant by test administration error?
external factors that affect test-taker attention and motivation
What is meant by test scoring and interpretation error?
rigidity and rigorousness of scoring criteria (meaning of test results in relation to psychological functioning)
