Psychometric tests

Question 1

what is a psychometric test??

Answer 1

a way in which one can gather a standardised sample of behaviour in a standardised environment and assess behaviour objectively

Question 2

what are the limitations of psychometric tests

Answer 2

small sample of behaviour
at a particular point in time
removed from individuals’ usual environment - particularly important for people with autism/ anxiety etc

ignores influence of person on environment
e.g high social intelligence, shape environment in a productive

Question 3

what different types of dedicated psychometric test are there

Answer 3

self-report questionnaire: personality, clinical symptoms, life quality (requires metacognition, self-awareness, and honesty)

ability tests: IQ, numeracy, social skills

open-ended interviews: qualitative research where the interviewee sets the agenda and decides what is important

structured interviews: used heavily in clinical research (DSM-V and Autism Diagnostic Observation Schedule)

Question 4

how is standardisation ensured in psychometric tests

Answer 4

uniformity of procedure in administering and scoring test
testing conditions must be the same to make valid comparison
exact materials, time limits, oral instructions, preliminary demonstrations, ways of handling queries from examinees etc

Administer to a large, representative sample of the type of persons for whom it is designed
normalise

Question 5

why does one size not fit all

Answer 5

subjective differences in nerves, sensory symptoms and social anxiety

response to reward: relative value

motivation in general: volunteers for trials often passionate about issue if personal, while others just need credit/money

expectancy effects: Prosoagnosia: believe they will fail on facial recognition so are less inclined to try to work problems out

Question 6

what is validity

Answer 6

the extent to which a test measures what it is meant to
different validities depending on the specific purposes of the test and varying wiht the method of assessing validity

analysing test content
relating scores on the test to scores on a criterion of interest
investigating the particualr psychological characteristics of constructs being measured by the test

Question 7

what is content validity

Answer 7

face validity: looks like it measures what it is supposed to - for wide use

expertise in selecting items
stimulus materials or situations comprising the test must call for a range of resposnes that respresent the entire domain or universe of skills, understanding or other behaviours that the test is supposed to measure

Question 8

what is criterion related validity

Answer 8

relating scores on the test to performance on criterion measures
usually restricted to validation procedures in which test scores of a group of examinees are compared with ratings, classifications, other test scores

Question 9

what is concurrent validitiy

Answer 9

when new and established tests are administered at the same time to check correlation

Question 10

what is predictive validity

Answer 10

measured whenever the criterion measure is not available until some time after the test is administered
correlation between the test/predictor and a measure of performance/criterion with which test scores predict criterion scores
important in aptitude and intelligence test
correlation rarely over .6
limits interpretation

Question 11

what is incremental validity

Answer 11

the degree to which a new test can predict the criterion over and above existing measures
useful in judging whether a test should be deployed

Question 12

what is construct validity

Answer 12

process of gathering evidence from many experiments and observations on how the test is functioning
can predict direction and strength of correlation - comapare with actual results
convoluted correlatiosn indicate that our measure may not be specific

Question 13

how can we establish construct validity

Answer 13

Check internal consistency of the test (see internal reliability).
Lab and field studies of scores from groups thought to differ on construct.
Check correlations with other tests of construct and those thought to be related. Factor analyse these inter-correlations.
Ask participants how they did the test.
Check test doesn’t correlate with other constructs (discriminant validity).
E.g test for autistic traits: big differences between ASD and control groups - check for discriminant validity
May not be because of autistic traits but associated traits such as anxiety, or depression.

Question 14

what is reliability

Answer 14

to be useful, tests and other measuring instruments must be fairly consistent, or reliable, in what they measure.
a test is its relative freedom from unsystematic errors of measurement.
reliable if it produces consistent measurement under varying conditions that can produce measurement errors.
Difficulty is determining variability of measurement from variability of what is measured (effects of mood, concentration, learning effects etc).
People are not consistent - huge variations which according to a strict textbook interpretation makes tests unreliable

Question 15

what different types of errors are there

Answer 15

Unsystematic errors:
Affect test scores in a random, unpredictable manner from situation to situation
Lower test reliability e.g fMRI
Systematic errors
Constant, may inflate or deflate test scores
Do so in a fixed wat so do not affect the reliability of thest
Does affect accuracy of test
Error variance
Conditions irrelevant to the purpose of the test
Error variance for one purpose may be classified as true variance for another

Question 16

what sources of error are there in variance and reliability

Answer 16

time sampling
content sampling
content heterogeneity
observer differences

Question 17

what is time sampling: test-retest reliability

Answer 17

Administer the test to the same group on two separate occasions
Test-retest reliability coefficient is the correlation between the scores on the two administrations
Takes into account errors produced by differences in conditions associated with teh two occasions on which the test is administered
Interval between test should rarely exceed 6 months
• Big Five Terracciano, Costa, & McCrae (2006). 6 – 42 years interval N= .78, E= .83, O= .85, A= .80, C= .81
What do low correlations mean?
Unreliable
Or what you are measuring is changing

Question 18

what is content sampling: parallel forms

Answer 18

Administer two tests that are equivalent in the sense that they contain the same kinds of items of equal difficulty but not the same items
Correlation between 2 sets of test scores is computed
Takes into account error variance produced by using different samples of items
Hardly ever happens but is important

Question 19

what is the best procedure for estimating relaibility

Answer 19

Assess performance at Time 1 on Parallel Form A.
Correlate with performance at Time 2 on Parallel Form B
Include covariates

Question 20

what is content heterogeneity

Answer 20

Internal Consistency Coefficient
Parallel forms are expensive to produce and extremely difficult to create. Matching difficulty exactly requires lots of time and money.
Instead, tests sometimes considered as consisting of two parts, measuring the same thing (20-item questionnaire considered to be two ten-item qu’s).
Calculate split-half reliability. Dependent on how you split test…
Cronbach’s alpha: (roughly) average item inter-correlation (>.7 acceptable).
CA is not a measure of dimensionality.
Factor analysis does this

Question 21

what is the observer effect

Answer 21

Inter-rater / Inter-scorer Reliability
Objective tests should be given the same score for a given test paper
scoring of essays and other non-objective tests is usually fairly subjective, and so it is wise to have such tests scored independently by at least 2 people.
correlation between the scores assigned by the two scorers provides an index of their agreement, known as an inter-scorer or inter-rater reliability coefficient
Useful but not everything you want to know

Question 22

how should you interpret reliability coefficients

Answer 22

Separate variance into a true measure + noise
Range from 0 (no reliability) to 1 (perfect reliability).
.85 means that 85% of the variance in test scores depends on true variance in the traits measured (not claimed to be measured), and 15% depends on error variance.
Experimental designs that yield more than one type of reliability coefficient allow breakdown of total score variance into different components.

Question 23

how can you increase reliability

Answer 23

Design better instruments / select better items.
Select items of moderate difficulty (p of around .5, corrected for guessing).
Increase the length of the test (vs Boredom / Fatigue effects, may vary by group).
Test in a more homogeneous group.
Will break down as soon as heterogeneous groups are introduced
Removing differences can be problematic as it limits what we can measure

Question 24

summary

Answer 24

Validity = Does a test measure what it claims to?
Face validity = Does it look like it measures what it claims to?
Content validity = Does it measure all it should measure or a subset?
Criterion validity = Does new test correlate with establish test(s)?
Concurrent validity = As above but when tests are administered concurrently.
Predictive validity = Does test predict future criterion values?
Incremental validity = Does it add predictive validity to existing tests?
Construct Validity = Pretty much all of the above (general measure).