L1: Intro to psych measurement Flashcards
What is a psychological construct? + give an example
aka latent variable
- theoretical concept of psych differences between individuals
- unobservable
ex: working memory
What is a psychological test?
- Measurement instrument to quantify individual differences in the psych construct
- Observable
- operational definitions/ operationalizations
what are the 3 important components of a psychological test?
- involve behavioural samples of some kind
- samples are collected in a systematic way
- purpose is to detect differences between ppl (or within a person across time or situations)
What’s the difference between a psychological construct and a psychological test?
psych construct = unobservable
psych test = observable
How do psych tests differ?
- content (depends on the construct being measured)
- response required (open ended vs closed ended)
- method of administration (individual vs group)
- use (criterion referenced vs norm referenced)
- timing (speeded vs power)
- the meaning of indicators (formative vs reflective)
What are criterion referenced vs norm referenced psych tests used for?
criterion referenced: to determine if someone passes a certain cut-off (ex: exam)
norm referenced: to compare score to population (aka the reference sample) (ex: intelligence test)
What are the different types of timing that can be used for psych tests?
speed test (time limit): relatively easy questions, see how far one comes
power tests (no time limit): different difficulty level, see what one can solve
What is formative measurement?
- the item responses (indicators) define the construct
- items are not necessarily correlated (items independently contribute to the construct)
- ex: the value of a car (construct) is determined by its age, condition, size (items)
What is reflective measurement?
most psych tests involve reflective measurement:
- construct is assumed to cause differences in the test scores
- item responses are indicators of the construct
- items are necessarily correlated (higher values on the construct increase score son all items)
- ex: differences in arithamtic skills cause differences in how well children solve 18+6
What are some challenges in psych measurement?
- Complexity of psych concepts
- Participant reactivity
- Observer expectancy/bias
- Composite scores (++ items scores need to be combined into one score)
- Score Sensitivity: (unknown beforehand how sensitive a scale should be)
- (lack of) Awareness of psychometrics (by for ex test administrators)
What do you need to watch out for in the score sensitivity of a test?
- Too few response categories: you may miss out on individual differences
- Too many response categories: people can’t meaningfully distinguish the different categories
not known beforehand how many there should be
What is participant reactivity?
ppl respond differently as they know that they are being measured
-> threatens the validity of the measure
What is observer expectancy bias?
expectation of the researcher may affect the test
define (score) sensitivity
a measures ability to discriminate between meaningful amounts of dimension being measured
what are the different forms of participant reactivity?
- demand characteristics
- social desirability
- malingering
What is malingering?
when ppl change their behaviour to convery a poor impression to the person doing the measurment
What are demand characteristics?
participants change their behaviour to accommodate the researcher
What is the social desirability effect?
change their behaviour to try to impress the person doing the measurement
define psychometrics
science concerned w evaluating the attributes of psych tests
what are the main test attributes that psychometrics care about?
- type of info generated by the use of psych tests
- reliability of data obtained from psych tests
- validity of data obtained from psych tests
define variability
degree of differences within a set of test scores or among the values of a psych attribute
define covariability
degree to which varaibility in one set of scores corresponds with (or is consistent with) variability in another set of scores
define interindividual variability
differences that exist between ppl
define intraindividual variability
differences that emerge in one person over time or under different circumstances
define distribution of scores
a set of test scores
define variance
statistical way of quantifying variability of individual differences
define covariance
way of quantifying covariability aka the connection between variability in one set of scores and variability in another set of scores
what is the mean?
definition, main concept, symbol, equation
- the average persons test score
- concept: central tendency: which single score best represents the set of scores
- x
- equation: sum of all numbers / N
how can you calculate the differences among peoples test scores? what is the concept behind this?
concept: variability: to what degree do the scores differ from each other
2 statistics:
1. variance (s squared)
2. standard deviation (s)
what are the equations for variance & standard deviation?
variance = sum of all (x’s - x mean) ^2/ N
standard dev = square root of variance
what is the skew?
explanation & equation
- shape of a distribution of test scores
- to what degree is the set of scores evenly / symmetrically distributed around the mean?
- perfectly symmetrical distribution has skew value = 0 (normal distribution)
- distributions w long right tail = positively skewed so skew value > 0
- distributions w long left tail = negatively skewed so skew values < 0
- equation : sum of all (x’s - x mean) ^3 / N*s^3
how can you calculate the consistency between 2 sets of scores from the same ppl?
concept & symbols
- concept: covariability: to what degree are the differences in one set of scores consistent w the differences in another set of scores?
2 stats: - covariance (c xy)
- correlation (r xy)
what is the equation for covariance?
sum of (x - x mean) (y - y mean) / N
what is the equation for correlation?
c xy / sx * sy
what is a normal distribution?
a symmetrical, bell shaped distribution, where most values cluster around the mean
what is a good way to visually represent the association between 2 variables?
scatterplot
shows both magnitude & direction
- upward trend of points (from lower left to upper right) indicates positive association
- downward trend of point (from upper left to lower right) indicates negative association
how do you interpret covariance?
positive covariance ? positive association so both variables from both distributions move in the same direction
negative covariance ? inverse relationship: one variable increases as the other decreases
what is the con of covariance?
only gives info about the direction of the relationship between the distributions/variables, not the magnitude (unlike correlation coefficients)
define covariance
degree of association between variability in 2 distributions of scores
what is the variance covariance matrix?
- organizes variances & covariances for multiple variables in a structured, symmetric format
- variances presented in diagonal cells, and covariances appear identically above and below the diagonal
how do you calculate the variance of a composite score?
variance of item a + variance of item b + variance of item etc + 2* correlation between scores on the two items * sd of a * sd of b * sd of etc
= the total test score variance will depend solely on the item variability and the correlation between item pairs
what does the variance of a composite score look like if the items are uncorrelated?
sum of the item variances s^2 of i + s^2 of j
what is a composite score?
summed or averaged responses to multiple test items
how do you calculate the covariance between 2 composite test scores?
c (composite 1 and composite 2) = c(ik) + c(il) + c(jk) + c(jl)
aka the sum of the covariances between the items from composite 1 (with items i and j) and composite 2 (with items k and l)
what is the mean of a binary item?
proportion (p) of positive responses
what is the variance of a binary item?
p(1-p) where p is the proportion of positive responses and 1-p is proportion of negative responses
when is the variance of a binary item highest?
when half the respondenst answer positively and the other half negatively because p = 1-p = 0.5 resulting in a max variance of 0.25
when is the variance of a binary item lowest?
if all respondents provide the same response (either all yes aka 1, or all no aka 0) then theres no variance so it cannot be correlated w other scores
what are the 2 ways we need to interpret psych test scores?
- quantitative interpretation: understanding whether a raw score is relatively high or low, requires comparing individuals score to entire distribution using means, sds, and normal distributions
- psych interpretion: the psych implications of the score
what does a z score mean?
expresses how far an individuals test score deviates from the mean in terms of sd units which allows you to see if a score is high/medium/low compared to others (z score of 0.5 says score is half an sd above mean)
how do you calculate a z score?
(individual score - mean)/sd
what are the advantages of z scores?
- provide context to raw scores, making them easier to interpret
- allow for comparisons across different tests and metrics
- essential in stats analyses like correlation computations
what are the limitations of z scores?
- may not be intuitive for everyone
- negative z scores or z scores in decimals may be confusing
what are converted standard scores aka standardized scores?
z scores that have been rescaled to make them easier for ppl to understand
how do you standardize z scores?
changing the mean and sd of the scores
1. select a new mean (Xnew) and new sd (SDnew) (for example X = 50, Sd = 10)
2. z score converted using formula T = z x (SDnew) x Xnew
what is the advantage of standardizing z scores?
- reframe the z score in more understandable and intuitive context w/o losing their meaning
- key to interpreting them is knowing the mean and sd of the converted distribution
what are percentile ranks?
indicates the percentage of scores below a specific score
ex: scoring at 85th percentile means test taker scored higher than 85% of others
what are the 2 methods for determining percentile ranks?
- direct method: if you have access to all individual scores, can calculate by counting nr of scores below yours, dividing by total number of scores x 100
- using z scores: if u only have mean and sd, first convert raw score to z score, then use standard normal distribution to estimate percentile rank (see table for the corresponding percentile to z rank)
what is the mean & sd of standard normal distirbution?
mean = 0
sd = 1
how can you normalize a test score to form a normal distribution ?
- compute percentile ranks: transform raw test scores into percentile ranks based on the distribution fo scores
- convert to z scores: use the percentile ranks to find corresponding z scores from the standard normal distribution
- transform to standard scores: convert these z scores into a new scale w a desired mean and standard deviation
what is the point of normalizing test scores?
to transform a non normal distribution of scores into a distribution that approximates a normal (bell shaped) curve.
done when test developers believe that psych attribute being measured is normally distributed in pop but actual scores from test are not
what is the difference between standardization and normalization?
standardization: focuses on individual score comparison within the same distribution (you change an individual score to see how their position is relative to the mean (in sds)
normalization: adjusts entire distribution to fit a normal curve
Define dimensionality
the nr. of constructs a test measures
What is unidimensionality vs multidimensionality?
uni: test measures a single construct (test of attention)
multi: test measures ++ constructs (ex: Big 5 test)
What are the 3 main types of multidimensionality?
- multidimensional test w uncorrelated dimensions
- multidimensional tests w correlated dimensions (ex: Big 5)
- multidimensional tests w higher order factor (ex: WAIS IV, IQ test)
What is factor analysis?
statistical method to study dimensionality of a test
What are the 2 types of factor analysis?
- exploratory factor analysis (EFA): theres no theory about the factor structure (used in PTPR)
- confirmatory factor analysis (CFA): theres a clear theory about the factor structure (not used in PTPR)
What are the 4 steps fo EFA (exploratory factor analysis)?
- correlation matrix: check where theres high correlations (2 items that have high correlations could form a factor)
- check the eigenvalues
- select a number of factors: use all factors before inflection point in scree plot (eigenvalue on y axis, factor number on x axis)
rotate the factors (orthogonal for uncorrelated factors, oblique for correalted, oblique is preferred) to see clearly what item goes with what factor - interpret the factors: see factor loading (item factor association)
what does an eigenvalue mean?
it shows how much variance one factor can account for in the data (so the more variance, the more important the factor/dimension)
what are the 2 methods to choose how many factors to select from the eigenvalues?
- kaiser criterion: select all factors with an eigenvalue greater than 1.0 (dont use)
- scree plot: eigenvalue on y axis and factor number on x axis, find the inflection point in the line and choose the factors above
what are factor loadings?
correlation between item and a factor (between -1 and +1)
helps us interpret factors (the higher the correlation the more likely the item belongs to the factor) then based on what items go with what factor you can name the factor
what is the goal of orthogonal & oblique rotation?
helps us interpret the factors by making a simpler structure of them (it helps you see which items go with which factor)
how do you do orthogonal rotation?
y axis represents factor 2
x axis represents factor 1
plot the 4 items on the graph based on how closely they are correlated to each factor (factor loading)
rotate the axis but keeping the 90degree angle between them to see how close the items are to each axis
remove items that dont clearly correlate with either factor
how do you do oblique rotation?
same as orthogonal, except you dont have to keep 90 degree angle
so the factors are correlated
what are factor scores?
calculate score of how much ppl score on the 2 or however many factors ppl have (for example language & science)
what are the advantages of factor scores?
- reduces the data
- has advantage over the composite/sum score
- avoids multicollinearity
what is the pattern matrix vs structure matrix?
the 2 tables you get when doing oblique rotation
pattern matrix: factor loadings controlled for the correlation between the factors (best interpretable)
structure matrix: factor loadings (more difficult to interpret cause includes correlation between factors)
what are the 3 key questions regarding test dimensionality?
- how many dimensions are reflected in the test items?
- are the dimensions correlated?
- what are the dimensions
The human development index (HDI) is a composite measure of human development consisting of health, education and income. It is used by the UN to measure the social and economic development per country. Are the scores on a HDI test reflective /effect indicators or formative /causal indicators, and why?
Formative/causal indicators, because the scores on health, education and income indicators make up a country’s HDI level
What are the 4 main differences in timing, completion, scoring, and difficulty between speeded tests and power tests?
power tests:
- not time limited
- completion of th entire test
- score by counting nr of items answered correctly
- pretty diffcult
speeded tests:
- time limited
- completion of part of the test
- scored by counting nr of items answered
- less difficult questions
A researcher conducts a longitudinal study measuring the language ability of a group of children. Over the past years, during various measuring occasions, the language ability of the girls tended to increase more than that of the boys. Therefore, the researcher expects that also at the current measuring occasion, the girls’ language ability has increased more than the boys’. What challenge of psychological testing might come into play here?
Observer/scorer bias
Tom participates in an experiment. During this experiment, he has to fill in a questionnaire about sustainable behavior. Tom doesn’t want others to think of him as an environmentally unfriendly person. How do we call it when Tom’s awareness that he is being assessed influences his answers on the questionnaire?
Participant reactivity
On which of the variables below, you expect the most individual differences in a sample of first-year university students Psychology ? Intelligence, age, or personality
Personality
Nathan conducts research into social anxiety for his master thesis. He uses a social anxiety questionnaire to assess the social anxiety levels of a student sample. The results show that participant 57 has a z score of 0.7. What does that mean?
That participant has a social anxiety test score that is 0.7 sds above the mean
Scores on the Wechsler Adult Intelligence Scale IV (WAIS-IV) have a mean of 100 and a standard deviation of 15. What kind of scores are these?
Standardized scores
In what situation do we use normalization transformation to obtain normalized scores?
When the raw test scores are not normally distributed
What is the relation between correlation and covariance?
a correlation is a standardized covariance
for which kind of test is it meaningful to compute a total test score (aggregating over all dimensions)?
multidimensional test w higher order factor
The Novaco Anger Scale is part of the two-piece Novaco Anger Scale-Provocation Inventory (NAS-PI; Novaco, 1994) and consists of 48 items measuring an individual’s anger experiences. The NAS has three subscales: cognitive, arousal and behavior. The test produces scores on each of the subscales, as well as a total test score.
What kind of test is the Nocavo Anger Scale?
Multidimensional test with a higher-order factor
What is a critique of using the eigenvalues-greater-than-one rule to determine the number of factors in factor analysis?
It likely results in an overestimation of the number of factors
what are the differences between factor matrix, pattern matrix, and structure matrix?
factor matrix:
- no rotation
- factors are uncorrelated
pattern matrix
- rotation
- controlling for factor correlations
structure matrix
- not controlling for factor correlations
