Midterm 1 Flashcards
Research variables
Independent: typically manipulated
Dependent: always measured
Control variable
Experiment (3 steps)
Hypothesis
Experimental manipulation (independent variable)
Interesting outcome
Categorical
Levels are categories
Ordinal
Quantitative variable
Numerals represent a rank order, but distance is not equal
Interval
Quantitative variable
Numerals have an equal distance, but no real zero
Ratio
Quantitative variable
Numerals have an equal distance but there is a real zero
Reliability
Consistency of results over time
Less spread: more reliable
How well we’ve measured something in a technical sense irrespective of what it is we’re measuring
Internal reliability
Internally consistent
Correlate with itself
Test-retest
Same results over time
Correlate with itself on 2 occasions
Inter-rater
Two or more observers agree (scores correlate)
Validity (definition)
We have properly captured the construct of interest
Link between validity and reliability
For a measure to be valid, it has to be reliable
For a measure to be reliable, it doesn’t have to be valid
Assessing validity (4 questions to ask yourself)
- Is it possible you’re measuring something else
- Are you measuring some part innacurently
- Are you measuring all important aspects
- Are you measuring it well enough to suit your intended argument
Internal validity
Is there a better alternative explanantion?
Construct validity
Capture the right thing
Statistical validity
How big is the effect
External validity
Representative sample
Face validity
Does it seem to be valid
Content validity
Capture all the aspects
Criterion validity
Relate to a concrete outcome
Convergent validity
Relate to other things it should
Discriminant validity
Not relate to other things it shouldn’t
Good design
Captured construct well
Good number of items
Good answers
Order effects
Response sets
Faking
Order effect
Order of items in a questionnaire matter, so the order of questionnaires within a study, establishing a context for the participant
Acquiescence - Response set
Tendency to answer positively to all questions
Threatens to construct validity
Fence-sitting - Response set
Selecting neutral answers
Weakens construct validity
Social desirability - Faking
Trying to look better in someone’s eyes
Malingering - Faking
People try to look bad
May try to say the survey is anonymous, but participants may not take it seriously
Include some reverse questions
Self-reports (4 reasons why they are not good)
Untrustworthy and useless because
- The order of questions can affect the participants
- The order of questionnaires can affect the participants
- People don’t answer thoughtfully
- People try to emphasize that they are good or bad
Can we believe participants
Yes if we ask good questions
Correlations
- what is it
- the famous quote
The simplest form of association claim (bivariate correlation, zero-order correlation, linear regression)
A strong association doesn’t imply causation
If there’s is no correlation there’s no causation
Features of correlations (4)
Only 2 variables
Both variables are measured
Min of -1 and max of +1
0 indicates no correlation
Understanding correlations (value of person’s r)
Use Pearson's r 0,10= small 0,30= medium 0,50= strong If it's smaller or bigger than expected, this might be a sign that there's a problem with our theory
If one variable is an IV and the other is DV, can correlation demonstrate causation?
Yes
This is true even if your IV was only measured, not both measured and manipulated
Questioning validity (what’s the assumption)
Assume that our sample is one homogeneous group, but they can vary relatively normally on both measures
Not homogeneous sample
May have some important problems
The sample has two or more subgroups that are different from each other
It can happen without any sign and detecting when it’s a problem can be difficult
Internal validity of the correlation (sample size)
Outliers can influence internal validity, especially with small samples
Better to have large samples so we can increase our confidence
Rules of causation (3)
Covariance: cause (IV) must be related to the effect (DV)
Temporal precedence: cause before the effect
Internal validity: no other plausible explanation
Problems establishing covariance (think about the graph)
Correlations detect straight line association even when there is no association
Threats to experimental validity (3)
Design confounds, selection effects, order effects
Design confounds (variable)
Does another variable vary systematically with the IV (need for control variable)
Selection effects
Are different kinds of participants in the groups (independent-group design)
Error while selecting the participants to participate in a study
Order effects
Are later responses systematically affected by earlier ones (within-groups design)
Managing selection effects (2)
Random assignment
Matching
Preventing order effects
For within-group design, we can encounter fatigue, practice effects, or carryover effects
Best solution is counterbalancing
Multiple groups
Posttest only
Pretest and posttest
Single group
Concurrent measures
Repeated measures
Independent-groups
Requires more people (control group)
No contamination across IV levels
Within-groups
Requires fewer people
Participants are their own controls (power)
Order and demand effect
May not be possible
Confidence intervals
Confidence around the mean, correlation
If it includes 0, it’s not statistically valid
Large sample for statistical validity
When IV doesn’t have an effect
When manipulation is not effective at producing a change in the outcome, but it seems like it is
Manipulation seemed somewhat effective or not at all when it should have been
Type I error
False positive
Reject the hypothesis when it’s true
Type II error
False negative
Fail to reject the hypothesis when it’s false
Threats to internal validity
Can’t always eliminate them all, need to recognize their impact
Threats can overlap each other
Testing effects
Special case of order effect
Testing process itself changes the outcome
Can happen in within-subject design
History effects
Unrelated event occurs during the study affecting the whole study and group
Can happen any time, impossible not to happen in longitudinal
Maturation effects
Natural or spontaneous change occurring
Especially with kids
Comparisons groups
Best way to identify the presence of threats to internal validity
Eliminate if plausible alternatives occurred
Attrition
Outliers dropped out of the study, normalizing one group
Can’t include the scores of people out dropped out of the study
Not applicable because we get rid of someone who didn’t complete the whole study
Regression to the mean
Extreme scores at T1 are normal at T2
Can happen if the mean shifts overtime
Instrument effects
Decay of equipment accuracy or coding method
Can happen over a long period of time
Attrition vs regression to the mean
Pre rest looks about the same
1 person vs entire group
Group looks equivalent based on means but the number of dots doesn’t (unusual score doesn’t mean unusual mean)
Unusual score has a big effect on the mean, need to take out the unusual score of BOTH groups
Need to justify every time we take out a score
Observer bias
Ratings (or data) become adjusted to fit the hypothesis, researchers manipulate the data
Double-blind condition
Depends if we have observers
Demand effects
Participants attempt to produce the hypothesized outcome (participants change their answers to help the researchers)
Need to ask them at the end of the study their idea of the hypothesis
Between subject-design instead of within
Can happen if participants have more time to figure out the purpose of the study
Placebo effects
Participants can will themselves to improve, change
Control condition can tell us how big the placebo was
The downward slope doesn’t guarantee a placebo effect
Depends if we are manipulating something or not
Probabilities
Not all threats are equally likely
Regression to the mean, testing, and demand effects are VERY likely
History, attrition, and instrumentation effects are pretty unlikely
Maturation and observer bias can be likely in the right contexts
Placebo is very likely in the right context
Recognizing threats (2 reasons why they help us)
Understanding the threats is very important
Help us to avoid accepting bad arguments as valid ones
Help us recognize ways to improve for follow-up studies
Third variable
Plausible alternative explanations can come in the form of alternative causal variables
Try to hold them constant as control variables
Multiple regression
- what does it help us with
- link with internal validity
- when is it used
- whats the letter
Helps us answer questions about an association while controlling some other association If we take into account a potential confound, we improve internal validity If we determine our preferred causal variable as a stronger influence than another cause, we improve internal validity It is used when we want to predict the value of a variable based on the value of two or more other variables Standardized beta (same principle as r) R= combination of all factors
Recognizing the use of MR (3 expressions)
controlling for the effect of Z
taking into account the effect of Z
correcting for Z
Third variable problems
- role of MR
- allow us to do what
- how is the argument?
MR lets us control for other unwanted influences on the association of our IV and DV (Take the bivariate association and rule out any other influences we can think to measure )
Allows us to make more complex arguments
Argument is always causal when we use MR (always a predictor and outcome variable)
Other uses of MR
Not only good for ruling out third variable
Mediation
Moderation
Mediation
Occurs when some other variable helps us better understand the nature of the causal association between 2 variables
Third variable is present but it’s not a problem
Mediators are involved in the causal association
Mediation vs third variable problem
Mediators are a part of the causal chain that connects an IV to a DV
A third variable is the only cause that makes an IV appear to be linked to a DV
Moderation
Occurs when some other variable controls whether your association does or doesn’t appear
Moderators dictate what kind of association can exist between the IV and DV
Correlation and temporal precedence
Correlations are good at establishing covariance (MR will improve internal validity)
Correlations are not good at establishing temporal precedence
Longitudinal designs
- problem it addresses
- what does it do
- when is it useful
Help address the problem of temporal precedence with correlations
Follow participants over a long period of time
More work than cross-sectional design
Useful when experimental design is not possible
If 2 variables use correlations across variables
Cross-sectional correlations
Only calculate correlation at one-time point
Used to examine if changes in one or more variable are related to changes in another variable(s)
Autocorrelations
Self-correlations, correlation with same variable
Correlation with itself
Cross-lagged correlation
Across time and different variables
Study in which two variables are measured once and then again at a later time
Parsimony
The degree of simplicity in a theory
Occam’s Razor states that all else being equal, the solution that makes the fewest assumptions is usually the correct one
Pattern
Demonstrating that a series of different findings (preferably different studies) converge on the same solution
Pattern and parsimony
If we combine the two we really have something
If multiple studies show support for the same simple conclusion, it’s much more likely that your proposed causal association is correct
