Statistics Flashcards
What does the validity of a test refer to?
Validity of a test refers to the extent to which the test measures what it was designed to measure
What is internal validity?
Internal validity is the extent to which the intervention or manipulation of the IV accounts for changes in the DV
What are the most common risks to internal validity?
- History (event that occurs in the experiment or outside of the experiment other than the manipulation of IV that could account for change in the DV)
- Maturation (development/decay of biological or psychological factors)
- Testing effect, e.g., taking the test more than once, eg, pre test effect
- Instrumentation ( change in instrument/measuring procedures during the course of the experiment)
- Regression to the mean- tendency of extreme scores to regress to the mean
- Selection bias
What is the Rosenthal effect/Pygmalion effect?
The tendency for participants’ performance to be effected by expectations of the tester ( eg, students perform better than other students because the teacher expects them too)
How do researchers safeguard internal validity?
- Random assignment
- Matching subjects on possibly relevant characteristics ( less powerful than random assignment, but necessary when groups cannot be randomly assigned).
- Blocking ( study the effects of extraneous subject characteristics by treating the extraneous characteristics as IVs)
- ANCOVA ( analysis of co-variance)- a statistical procedure developed to account for group differences in extraneous characteristics
What is external validity?
Refers to the generalizability of a study/test results. It refers to the limits or boundaries of the findings.
What are threats to external validity?
- Interactions between subject selection and treatment (treatment effects do not generalize to other members of the population)
- Testing and treatment effects ( treatment effects do no generalize to individuals who did not participate in the pre-testing, eg, from demand characteristics)
- History and treatment effects ( treatment effects depend on history of testing period)
- Demand characteristics (cues in research the alert subjects
- Hawthorne effect-tendency of subjects to respond differently based on expectations of the researcher
- order effects (risk in repeated measure designs)
What is Hawthorne effect?
Tendency of subjects to behave differently when they are in a research study
How do researchers safeguard external validity?
- Random selection from population of interest
- Conduct naturalistic/field research
- Use single or double blind research designs
- Counterbalancing (eg, vary the order of treatment strategies among participants to eliminate order effects)
- Stratified random sampling (select random sample from each of several subgroups of target population)
- Cluster sampling ( the unit of sampling is naturally occurring in groups of individuals
What is content validity?
The degree to which items on a test represent the domain that test is supposed to measure (eg, does a test of depression include items that measure vegetative Sxs)
What is face validity?
The extent to which a test APPEARS to measure what it states it measures. Face validity can be misinterpreted content validity.
What is criterion-related validity?
Refers to the relationship between test items and a criterion of interest (correlation of a measure with a criterion of interest, eg, correlation of performance on an academic test with grades). 2 types of criterion validity: 1. concurrent validity 2. predictive validity
What is concurrent validity?
One type of criterion related validity. Refers to correlation of performance on 2 measures at the same point in time (eg, correlation of performance on an achievement test with current grades)
What is predictive validity?
A type of criterion validity. Refers to the relation between test scores and future performance on a criterion of interest.
What is construct validity?
here are 2 ways to think about Construct Validity:
- Construct validity of an experiment: If a causal relationship has been determined between the IV and DV (i.e., internal validity has been supported), construct validity refers to the experimentor’s explanation for why the causal relationship exists (i.e., what aspect of the intervention was the causal agent).
- More common use of construct validity is in the context of test development. It refers to the extent to which a measure assesses the psychological construct or trait it was designed to measure. In this context there are 2 types of Construct Validity: convergent validity, divergent validity
Convergent validity
A type of construct validity. The extent to which 2 measures assess similar constructs. Construct validity is supported when measures of the same construct are correlated. It also refers to the similarity among measures of the same construct that are administered in different formats (eg, self and parent report of sxs, essay response vs multiple choice)
What is divergent validity?
Another type of construct validity, also known as discriminant validity. Correlation between measures of different constructs. CONSTRUCT VALIDITY IS SUPPORTED BY LOW CORRELATIONS BETWEEN MEASURES OF DIFFERENT CONSTRUCTS
How do convergent and divergent validity relate to construct validity?
Construct validity is supported when a measure correlates highly with other measures of the same construct (convergent validity) and does not correlate with measures of different constructs (divergent validity)
How can construct validity of a test be established?
- Establish a relationship between performance on the test and the theoretical construct it was designed to measure (eg, examine concurrent validity and predictive validity 2. Conduct a factor analysis of test items and assess whether factors relate to constructs of interest
- Perform factor analysis of items on test with items from established tests with construct validity.
- Establish that performance on a test rates more highly with tests of similar constructs (convergent validity)
What is reliability
Reliability is the consistency of a measure. It refers to the extent to which random, unsystematic factors affect the measure of a construct/trait.
4 types of reliability
- Internal reliability ( consistency among items in a test)
- test-retest reliability
- Alternate form reliability
- Interrater reliability
How are validity and reliability related?
A test can be reliable but NOT valid, but not the other way around. Reliability is necessary but not sufficient for validity.
What are the 2 components of a test score according to psychometric theory
- Test score
- Error score
Test score refers to all systematic information that contributes to the score. Error refers to all the random “noise” in the score. Note: true score does not refer to the underlying construct of interest, only to systematic variation in score.
How can true score be ID’d?
True score is a hypothetical construct and cannot be directly observed. It is best estimated as the means of repeated measures on the same test (each score includes error and the true score, so the true score is the average of these repeated measures). It is also assumed that the distribution of scores is normal.