Statistics and Measurement Flashcards
How is a quasi-experimental research design different from a true experimental design?
In a quasi-experimental design random assignment doesn’t/can’t occur.
What contributes to a non-experimental research design?
The independent variable is pre-existing and there is no manipulation or intervention.
What is required in a true experimental design?
There must be at least 1 manipulated independent variable
Subjects must be randomly assigned to their group
What is another name for a nomothetic research design?
Group Research Design
What are the 3 group research design strategies?
Between-Groups
Within-Subjects
Mixed
What is a between-groups research design?
Comparing groups where the groups and data are independent of one another
What is a within-subjects research design?
comparing groups in which the data is related/correlated/repeated
What type of research design is used when two groups of subjects are matched on some characteristic/variable?
Within-Subjects Design
What are carryover effects?
Occurs in a within-subjects design when subjects are exposed to multiple conditions and their performance naturally changes
Carryover effects occur in what type of research design?
Within-subjects
How should a research counteract carryover effects?
Use a latin square design
What is a mixed research design?
Contains components of both between-groups and within-subjects designs. One component is independent and the other is repeated
What is another name for an idiographic research design?
Single Subject
What is one problem of a single subject research design?
Autocorrelation
What is the AB research design?
A is baseline
B is treatment
What is the biggest threat to an AB treatment design?
History
How is an ABAB treatment design better than an AB design?
It removes the threat of history
What is one ethical concern about the ABAB design?
Treatment B may be effective and then you remove it for a return to baseline
What is one general concern about the ABAB design?
Participants may not return to baseline a second time
What is a drawback of a multiple baseline design?
Time Consuming
Expensive
What is a benefit of a multiple baseline design?
Avoids the problems of AB and ABAB designs
What is the simultaneous treatment approach?
Simultaneous treatment occurs, but at two different times of day
What is a changing criterion research design?
Used when gradually trying to change behavior
work towards achieving a criterion and the criterion changes as time progresses
When would you use interval sampling?
When the behavior being measured does not have a specific or identifiable beginning or end
What are the two types of interval sampling?
Momentary Sampling
Whole Interval Sampling
What is momentary interval sampling?
Choose specific times and if the behavior is being exhibited at that time you get a tally
What is whole interval sampling?
Choose and interval of time and only get a check if the behavior is begin exhibited that entire time interval
What is analog research?
Approximates a clinical situation
What is a problem with analog research?
Limits generalizability
What are the three time frames of research design?
Cross-Sectional
Longitudinal
Cross-Sequential
What is one problem with a cross-sectional research design?
Cohort Effects
What is a problem with longitudinal research?
Expense
Dropout Rates
What is a cross-sequential research design?
Break groups into cross-sections and then monitor them over shorter periods of time
Simple Random Sampling
Everyone in the population has an equal change of being chosen
Stratified Random Sampling
Dividing the population into strata and then sampling equal numbers within each strata
Proportional Sampling
Sample people in proportion to their representation in the population
Systematic Sampling
Creating a system and sampling that way (every 10, odd SSN’s, etc.)
Cluster Sampling
Sampling naturally occuring clusters
Name some threats to internal validity
History Maturation Testing/Test Practice Instrumentation Statistical Regression/Regression to the Mean Selection Bias Attrition/Experimental Validity Diffusion
What is the best way to control for threats of history and maturation?
Having a control group
What is the threat of history?
Any incident or occurrence that happens while in treatment, but outside the experiments that affects the outcome
What is the threat of maturation?
Something happening within the subject due to the passage of time
What is the treat of testing?
People do better due to familiarity with the test instrument
What is the best way to control for the threat of testing?
Use a Solomon 4 Group Design
What are the four groups in a Solomon 4 Group Design?
Pre-test, Post-test, with Intervention
Pre-test, Post-test, without Intervention
Intervention and Post-test
Post-Test Only
What is the threat of instrumentation?
Calibration of instruments causing change
What is the threat of Statistical Regression?
Tendency for extreme scores to move closer to the mean
How can you control for the threat of statistical regression?
Use a control group
What is the threat of selection bias?
Non-random assignment to groups
What is the threat of attrition?
When people withdraw during the experiment
What is the threat of diffusion?
When the treatment diffuses into the control group
What are some threats to external validity?
Sample Characteristics
Stimulus Characteristics
Contextual Characteristics
How can sample characteristics be a threat to external validity?
Most research relies on volunteers, so there may be a certain type of person that volunteers versus one who does not.
How can stimulus characteristics be a threat to external validity?
The characteristics of the research set-up may not generalize to the population.
How can contextual characteristics affect be a threat to external validity?
People are participating in research and being observed so they may respond differently (Hawthorne Effect)
What are some threats to Construct Validity?
Experimenter Expectancies/Rosenthal Effect
Demand Characteristics
John Henry Effect/Compensatory Rivalry
What can experimenter expectancies be a threat to construct validity?
The experimenter may inadvertently communicate something to the subjects about how to respond
How can a researcher control for experimenter expectancies?
Double-blind studies
How can demand characteristics be a threat to construct validity?
Subjects might pick up on something that cause them to respond differently
How can a researcher control for demand characteristics?
Don’t tell a participant which group they are in
How can the John Henry Effect be a threat to construct validity?
When people in the control group try even harder than the experimental group
How can a research control for the John Henry Effect?
Don’t tell a participant which group they are in
What is a threat to statistical conclusion validity?
Low Power
What might cause low power in a research study?
Small Sample Size
Unreliable Measures
Inconsistent Procedures
Heterogeneous Subjects
The higher the internal validity, the ____________ the external validity
Lower
Positively skewed data has a higher proportion of ________ scores.
Low
Negatively skewed data has a higher proportion of ___________ scores.
High
Which measure of central tendency is highest in positively skewed data?
Mean
Which measure of central tendency is the lowest in positively skewed data?
Mode
Which measure of central tendency is the highest in negatively skewed data?
Mode
Which measure of central tendency is the lowers in negatively skewed data?
Mean
What percent of the population falls within 3SD of the mean?
99%
What percent of the population falls within 2SD of the mean?
95%
What percent of the population falls within 1 SD of the mean?
68%
What does a z-score tell you?
How many SD’s above of below the mean someone scores?
What are the Mean and SD of T scores?
Mean=50
SD=10
What are the mean and SD of IQ scores?
Mean=100
SD=15
What is the central limit theorum?
If we take equal sized samples from the population, and we take an infinite number of them and plot the mean, we get the normal distribution
What is Type I Error?
Incorrectly Rejecting the Null Hypothesis
What represents the probability of making a Type I error?
Alpha
What is Alpha?
The probability of making a Type I error
What is Type II Error?
Incorrectly accepting the null hypothesis
What represents Type II Error?
Beta
What is Beta?
Type II Error
What is Power?
Correctly rejecting the null hypothesis
1-Beta=?
Power
What are some examples of non-parametric tests?
Chi Square
Mann Whitney U Test
Wilcoxin
What are the three requirements of parametric tests?
Interval/Ratio Data
Homogeneity of Variance/Homoscedasticity
Normal Distribution of Data
What is Homoscedasticity?
Homogeneity of Variance
What are 3 types of T-tests?
Single Samples T-Test
Independent Samples T-Test
Matched Samples T-Test
What type of post-hoc tests are available when significance is found on an ANOVA?
Pairwise Comparisons
Sheffe
Tukey
Post-hoc tests for ANOVAs lead to an increased chance for what?
Type II Error
When would you use ANCOVA?
When you find that something affects your outcome that you weren’t planning for, you can statistically remove it using ANCOVA
When do you use trend analysis?
When an ANOVA is significant
What is a trend analysis?
Allows you to look at patterns of change so you can see how the DV changes with various continuous levels of the IV
What is the coefficient of determination?
The square of the correlation (r squared); it determines shared variability
What is a 0 order correlation?
X and Y are correlated and we don’t think anything else is interfering with that relationship
What is a first order correlation?
There is a relationship between X and Y, but you think variable Z also affects this relationship
What is a semi-partial correlation?
When you have a first order correlation, but remove the effects of Z on either X or Y.
What is used to create the line of best fit?
Least Squares Criterion
What type of data is needed for a Pearson R correlation?
X and Y both need to be interval or ratio data
What type of data is needed for a Point Biserial correlation?
One variable is dichotomous and the other is interval or ratio data
What is the difference between point biserial and biserial?
Point biserial is a true and naturally occurring dichotomy. Biserial is an artificial dichotomy.
What is the problem of multicollinearity?
When there are multiple predictor variables (Xs) and they are all highly correlated with one another
What is a stepwise regression?
The computer decides which predictor variable is processed first, second third etc. based on how strongly related each variable is with the criterion
What is Hierarchical regression?
The researcher uses an underlying theory to determine the order of predictor variables
What is canonical correlation?
Two or more predictors and two or more criteria
What is discriminant function analysis?
A special case of multiple regression equation used when Y is nominal
What is logilinear analysis?
Looks like discriminant analysis with nominal Y data, but X’s are all nominal
What is a moderator variables?
Affects the strength/nature of the relationship between X and Y
What is a mediator variable?
Explains why the relationship between X and Y exits
If you remove the mediator variable, the correlation between X and Y no longer exist
What is a path analysis?
A way of using correlations to determine if there’s a cause/effect relationship
What’s the most common method of structural equation modeling?
LISERAL
What is LISERAL?
A method of structural equation modeling
What is a factor analysis?
Look at all factors and determine which cluster together to explain the outcome
What is an eigenvalue?
Tells you the strength of a factor in factor analysis
What are the two rotations used in factors analysis?
Orthogonal and Oblique
What is the difference between orthogonal rotation and oblique rotation?
Orthogonal rotation leaves you with factors that are uncorrelated with one another. Oblique rotation leaves you with correlated factors
When would you use principal components analysis?
When the researcher has no theoretical expectation of what the factors may be
What is classical test theory?
Any given score is the combination of truth and error
What is true score variability?
Reliability
What are 3 sources of test error?
Content sampling
Time Sampling
Test Heterogeneity
How does content sampling lead to test error?
When items on the instrument happen to be something you do or do not know
How does time sampling lead to test error?
Passage of time can naturally affect scores
How can test heterogeneity affect test error?
When an item taps more than on domain it can lead to test error
What affects instrument reliability?
Number of test items Homogeneity of items Range of Scores Population Sample Ability to Guess
How does the number of items on an instrument affect its reliability?
More items=More reliable
How does the homogeneity of items affect an instrument’s reliability?
More homogeneous=more reliable
How does score range affect instrument reliability?
larger range of scores=more reliable
How does the population sample affect instrument reliability?
Heterogeneous Sample=more reliable
How does the ability to guess the answer to questions affect a test’s reliability?
If items are harder to guess, the test is more reliable
What are the ways of measuring reliability?
Test-Retest
Parallel Forms
Internal Consistency
Interrater Reliability
What causes error in test-retest reliability?
Time
What causes error in parallel forms reliability?
Time
Content Sampling
How does split half reliability affect overall reliability?
It artificially reduces reliability
What is the Spearman-Brown Prophecy Formula used for?
Uses the split-half reliability, but tells you how much more reliable your test would be if you used the full number of items
What is the primary source of error is split half reliability?
Content Sampling
Split half reliability should never be used with what type of test?
Speeded tests
What is Kuder Richardson Coefficient Alpha?
It’s a way of determining split-half reliability. The computer takes the test and splits it in every possible half, generates a correlation for each of these, and averages them
What is the standard error of measurement?
The average change in scores if the same person were to take the same test repeatedly
What are some types of validity?
Content Validity
Criterion Related Validity
Construct Validity
What is content validity?
Does the test cover all the basic content knowledge/skills of what you want to measure?
What is criterion related validity?
Does the test accurately predict what we are trying to predict?
What are some types of criterion related validity?
Concurrent Validity
Predictive Validity
What is an expectancy table?
Tells you the probability that a specific criterion score will fall in a range given the range in which a predictor score falls?
What are Taylor-Russel Tables?
Tells you how much of an improvement you will make in a selection/hiring decisions when using a predictor test as compared to using no test
What do you need to know to use a Taylor-Russel table?
Base Rates of successful hires and selection ration
What is selection ratio?
Number of opening to number of applicants
What impacts incremental validity?
Criterion Related Validity of the Instrument
Base Rate
Selection Ratio
How would you optimize incremental validity?
Have a moderate base rate (.5) and low selection ratio (.1)
What is decision-making theory?
It looks at predictions based on using a predictor test and compares them to actual results
What are the four possible outcomes of a predictor test?
True Positive
False Positive
True Negative
False Negative
what is easier to change, the predictor or the criterion?
The predictor. Sometimes you cannot change the criterion.
What is item discrimination?
How well an items discriminates between high scorers and low scorers
What is item validity?
How well an individual scored on that item and the entire test
What is the item characteristic curve?
Relationship between how you do on an item and your total score
What is item response theory?
It allows the researcher to calculate to what extent a specific item correlates to the underlying trait being measured and is used to create individually tailored adaptive tests
What creates shrinkage?
Cross-validation
What factors affect the validity coefficient?
Range of scores
Reliability of the predictor
Correction for Attenuation
Criterion Contamination
How does the range of scores affect validity?
Broad Range=Better Validity
How does the reliability of the predictor affect validity?
Reliability puts an upper limit on validity
Validity is less than or equal to the square root of reliability
What is correction for attenuation?
Because X and Y are both not perfectly reliable, validity is lessened. This formula tells us, if X and Y were perfectly reliable, how much better our validity would be
What is Criterion Contamination?
This occurs when Y is subjectively scored and the rater knows how people did on the predictor. This artificially inflates validity
In the multitrait multimethod matrix, what is convergent validity?
Monotrait Heteromethod
In the multitrait multimethod matrix, what is divergent vaidity?
Heterotrait Monomethod
What are the subtypes of construct validity?
Convergent Validity
Divergent Validity
What is the Standard Error of the Estimate?
How much error there is in estimating what we are trying to predict
