Week 7: Tests Flashcards
What are psychometrics?
The branch of psychology concerned with testing and measurement
What is classical test theory?
Observed score = true score + error
What is the goal of testing?
To produce useful tests and reduce the capacity for error as much as possible
List some examples of test construction that could influence error
- poorly worded questionnaires
- extreme statements
- confusing words and expression
- culturally confusing terms
What are some factors that could influence errors in test administration?
- deviation from instructions
- testing environment
- test administrator
- participant derived error
Examples of participant derived error
- hostile or uncooperative
- donkey vote
- misunderstanding of instructions
- temporary illness or condition that might impact test scores
- contrast your motivations in collecting data with participant motivations in taking part in research
What are the types of validity
- face
- construct
- content
- criterion
Face validity
Does the test meet participants expectations and appear to measure what it is supposed to?
Content validity
Does the test assess the whole content area of what it’s meant to be assessing?
List some useful non-specific bases to cover in generating content
- emotions
- thoughts
- behaviours
- interpersonal relationships
What are four ways of categorising constructs?
- component analysis
- factor analysis
- cluster analysis
- taxometric analysis
Describe component analysis
About taking a bunch of things and reducing them into one thing
Describe factor analysis
Finding the underlying constructs of items
Describe cluster analysis
Used for grouping participants based on current data or predicted outcomes
What is the aim of cluster analysis?
To create groups that are homogenous as possible
Describe taxometric analysis
- attempts to answer continuum vs category questions
What is null hypothesis significance testing?
Process of logic examining the probability that a selection of data indicating an effect is drawn from the same sampling distribution as a selection of data not demonstrating an effect (the hypothesis that there is no effect occurring)
When do we reject the null?
When the probability is small (e.g.
What are some short comings of significance?
- higher sample size likely to bias towards lower p value
- lower p value tied to stronger effect, so doesn’t necessarily detect small but consistent effects
Discuss how to cope with assumption violations
- run the original tests and qualify this in the discussion
- modify the distribution using a transformation
- identify tests that have different assumptions the might be more appropriate (e.g. non parametric)
What is the ‘optimal’ solution for assumption violations?
- run the statistics in a raw form and one where you compensate for violated assumptions
- only report a modified data if the pattern of results substantially differs
Why is significance controversial?
- significant data is reported and biased in publication
How do computerised tasks typically operate?
Examining number of errors (accuracy) or reaction time (speed)
List some methodological decisions for testing
- number of items
- evenness of content distribution
- format of response scale
- mid point option in response scale
- use of reverse coding
- instructions framing questionnaire
- time period responses is asked to reflect on
What is ipsatisation?
When we normalise each participant’s responses so that they reflect their distance from an individual midpoint
What is the issue with ipsatisation?
It controls for differences in how the response scale is perceived, but can be viewed as removing useful variance, as perhaps the way that the response scale is perceived is an important part of the trait in the first place
What are some options to adapt a test to your needs?
- deleting items
- adding items
- changing response styles
- making slight alterations to wording
Test-restest reliability
- how are scores expected to change over time?
- need to consider whether change in these scores is legitimate or due to error
Inter-rater reliability
- concordance in agreement between raters
- can be affected by a defective scale, or poorly trained raters
Split-half reliability
- test is split in two, scores for each half compared
- consistency suggests reliability
Parallel-forms reliability
Administering diffrent versions of an assessment tool to the same people - both versions must reflect the same construct and content
Internal consistency
- cronbach’s alpha
- the degree to which a set of items measures a single, unidimensional construct
Convergent construct validity
- is the test outcome correlated with the variables that we could expect?
- can be a negative or positive correlation
- can be an identical construct
- can be different constructs
Discriminant construct validity
- does a measure lack correlation with variables that we would not expect it to correlate with?
Criterion related validity
- should be an observable, behavioural or group based characteristic
Concurrent
When you assess it at the same time as the measure you are validating
Predictive
When you assess the criterion later on
Why does psych have a higher error rate than other sciences?
Because we look at abstract variables and infer meaning
What are some ways to improve computerised testing?
- utilise metrics
- engage/motivate the participant
- ask them about the quality when they’re done
