Experimental Exam #1 Flashcards
Types of data in studies
1) experiments
2) quasi-experiments
3) correlational studies
4) observational studies
5) surveys
Basic research
Fundamental questions of behavior (usually human behavior)
Ex: what is the capacity of memory?
Applied research
finding solutions to problems/ is related to specific situations
Ex: a new treatment for depression
Often in a clinical landscape
Characteristics of scientists
1) Scientists are empiricists: the practice of basing ideas and theories on testing and experience
2) Scientists test theories
3) Scientists tackle both basic and applied problems
4) Scientists make science public
5) Scientists talk to the world in popular media
What is good evidence?
1) Should be peer reviewed → should catch flaws in results and study design
2) Want to isolate cause and effect → manipulate IV then measure DV
3) Rule out potential alternative explanations (confounding variables)
4) Can show you what would have happened (can make predictions about the future and evaluate the validity of those)
How to ensure control
Groups are identical in every possible way except for the condition which you are manipulating
- Same demographic, ages, education statuses, genders, socioeconomic status
Theory definition
systematic body of ideas about a particular topic/phenomenon
Qualities of a theory
Describes the relationship among variables
Organizes / summarizes knowledge or findings
Describes, explains or predicts behavior
Supported by data
Falsifiable: a principle or theory can only be considered scientific if it is even possible to establish it as false
Parsimonious (occam’s razor): all else being equal, we want the simplest solution
How do we know what we know?
1) based on experience
2) using intuition
3) trusting authority figures
Problems with basing science on experience
1) Not probabilistic → only one data point
2) No comparison group: you need a control/placebo group to have something to compare your results to
3) Has confounds
4) Not systematic: need to hold everything constant and change one thing
Confound definition
Confounds: plausible alternative explanation for the fining → when a second variable varies systematically along with the IV and provides an alternative explanation for the results
Problems with using intuition
1) Often uses colloquial phrases / trendy phrases (ex: “toxins”)
2) Sometimes intuitions are inconsistent
3) Sometimes intuitions describe the past
4) Intuitions can lead us astray
5) biases (x3)
Availability bias
- things that come to mind easily can bias our thinking
Ex: recent or vivid memories (often from the news)
Present/present bias
- Examples that are easier to call to mind are more “available” to call to mine and guide our thinking
- Very similar to availability but more specifically deals with the fact that we often fail to look at absences
- Can be from family and friends, stories, culture at large
Confirmation bias
- When people are asked to test a hypothesis, they gather evidence that supports their previous thoughts
- Can be conscious or unconscious
- Collaboration can help overcome this
Problems with trusting authority figures
Ex: jenny mccarthy went on opera and said vaccines caused autism in her son → gave a boost to the antivax community
No evidence and her son didn’t even have autism
Ex: Dr. Oz and Dr. Phil → often used status to sell products (unethical)
Dr. Oz: did have credentials but was not using/applying them in an ethical way
Empirical articles
a first-time published study
Review articles
summarize and integrate all the published studies that have been done in one research area
Components of a scientific paper
1) Title
2) Abstracts
3) Introduction
4) Methods
5) Results
6) Discussion
7) References
Title
should tell you the main idea of the article
Abstracts
brief summary of the articles content (summary paragraph)
Introduction
introduces the problem and explain why it is important
Mostly described what other research have found and explains why their research is relevant to your → last paragraph: usually introduces the method you used, your variables and hypothesis
Methods
explains how you conducted your study
Usually so detailed that someone else could conduct a direct replica of the study
Includes: participants, materials, procedures
Results
present the study’s numerical results, including any statistical tests, tables or figures
Discussion
summarize the results of your study; describe how they relate to your hypothesis or answer your research question
Evaluate your study, advocating for the strengths and defending the weaknesses
Suggest what the next step might be for the theory-data cycle
Measured variable
something that is observed or recorded
ex: Height, weight,
Manipulated variable
- controlled for
- Depression: can not force you to be depressed but you can make it manipulated by having a control group
Conceptual variables
Called constructs: abstract, general, theoretical
ex: Depression, pain
Operational definitions
A specific way to measure something
abstract – Not directly measuring the construct as it is theoretical, but helps us tap into it
Operationalizing
taking something invisible and turning it into something we can measure
Types of measures
1) self report
2) Observational/behavior: spatial reasoning tasks, working memory tasks
3) Physiological: fMRI BOLD signal, Physiological measures are not inherently better than behavioral measures
Self-report pros and cons
Pros: you know yourself best
Cons:
- Can be influenced by emotional/physical states (biased) and life events
- Might play up symptoms for experimenter (experimenter effects)
- Might not be accurate in certain groups (dementia, kids)
Pros and cons for observational and behavior
Pros:
- Can’t play up symptoms
Cons:
- Need lots of things: technology, software, instructions
Scales of measurement
1) nominal
2) ordinal
3) interval
4) ratio
Nominal variables
“names variables”
- categories; not continuous; can not be added or subtracted
Ordinal variables
“rankings”
- Inherent order to the variables
- spacing between not necessarily the same
Interval variables
“equally spaced numbers”
- Temp in fahrenheit → 0 doesn’t mean anything
- Equally spaced
- Continuous
Ratio variables
“has a meaningful zero”
- When zero is the absence of something
- Continuous
Types of claims
1) frequency claims
2) association claims
3) causal claims
Frequency claim
- One variable that is measured
- Usually in percentages (surveys)
Association claims
- 2 variables are linked, both measured
- “Correlation”
Causal claims
- One variable causes change in the other
- One must be manipulated
Things you need to make a causal claim
1) covariance:
As A changes, B also needs to change (can be in the same direction or different direction)
2) temporal precedence:
Have to verify the order in which the variables come
A causes B → A has to happen earlier in time than B
NOT bidirectional → correlations can be bidirectional but causal claims can not be
3) internal validity:
The study’s method ensures that there are no plausible alternative explanations for the change in B; A is the only thing that is changed
4 types of validity
1) construct validity
2) external validity
3) internal validity
4) statistical validity
Construct validity
“quality of measures and manipulations”
- How did you measure your DV?
- How well did you operationalize the construct
- How reliable are your measurements
How well did you measure what you claim to measure
Reliability
- Refers to consistency: if you are given a measurement on personality, it should be the same next week
- Reliability is necessary but not sufficient for validity
External validity
“does it generalize to other situations / populations “
- Generalize: how do we extrapolate our findings to other settings, entities, groups, operationalizations of the same construct (ex: depression inventories), etc.
- May not be the goal of the study: if studying a rare disease, you may not care if it generalizes; it can still be a valid claim
Internal Validity
“no alternative causal explanation for the outcome”
- Was the study free of confounds
- Was there random assignment
- Was there controls and/or counterbalancing
- Was there no different between the condition other than the IV
Statistical validity
“appropriate and reasonable statistical conclusions”
How well do the numbers support the claim
- P-value
- Effect size
- Well-powered
Do you believe the number to think the stats are lying to you
Control group
Control: closest to a null condition
- The neutral or no treatment level of the IV
Comparison group
something else - not a null but just something different
experimental group
group exposed to a manipulation
Placebo conditions
when the control group is exposed to an inert treatment (no active ingredient)
Treatment conditions
the non-neutral level of levels of the IV
Prioritizing external and internal validity
Doing experiments in a lab allows you to control for confounds; BUT a lab is not the real world - how do you know it will generalize?
Often: do in a controlled environment first, and then do it in a less controlled environment and see if results hold
Systematic variability
when confounds trends together → threatens internal validity
Unsystematic variability
random/haphazard trends that affect both groups → NOT a confound
Ex: a random amount of people from both groups drop out of the study
Individual differences study
what makes us different → where is our variance?
- Hard to make a causal claims when there are individual
- Don’t use an experiment → use a correlational design; harness the variability differences
Experiments (in relation to individual differences)
when we break people into groups we are treating them as one unit
1) Should be hard to detect differences within the groups
2) Should be easy to detect differences across the groups (due to manipulation)
Selection effects
when the kinds of participants in 1 group are systematically different than another group
How can we combat selection effects?
Random assignment: a way of assigning participants to levels of the IV such that each participant has an equal chance of being in each group
All participants should be equivalent on all important dimensions (age, eduction, race, income, sex etc.) → called group 1 and 2 matching
**can use a t-test to ensure this
Independent groups (between groups)
Each group has different participants
Posttest design (+pros and cons)
Posttest design: one measure at the end of the study
Pros:
- No attrition (participant drop out) → only gathering one thing
- One-time snapshot
Cons:
- No baseline: can not tell if something improved or got worse
Pretest /posttest design
test before and after manipulation
Pros:
- You get a baseline
- Allows you to study change over time
Cons:
- Practice effect: get better over time solely because of practice, not the manipulation
- Fatigue effect: get worse over time, not due to the manipulation but due to fatigue
- Attrition (drop outs): two sessions required
- Costs more: have to pay double (more for two sessions)
Within groups (repeated-measures)
All participants are exposed to all levels of the independent variable
Repeated measures design
participants respond to a dependent variable at least twice → after exposure to each independent variable
Pros:
- Do not have to worry about assignment issues or selection effects (don’t even assign people)
- Less recruitment
- The group is its own control (fewer participants needed for a study)
- Increases statistical power
Cons:
- Change in behavior → might figure out what the experiment is looking for
- Includes practice behavior and fatigue behavior
- Carryover effects: the first trial affects later trials
Concurrent-measures design
do it at the same time
Order effects
“exposure to one conditions changes participant response to a later condition”
1) item effects
2) carryover effects
Item effects
maybe the previous item gives you information or influences the next one
Carryover effects
contamination carrying over from one condition to the next
Ex: you drink caffeine and then take a test then you drink decaf and take a test → caffeine is still in your system for the second test
1) Practice effects: participants get better at a task over time
2) Fatigue effects: participants get worse at a task over time
Counterbalancing
presenting the levels of the IV to participants in different sequences
Can help fix some order effects
12 threats to internal validity
1) Design confounds
2) Selection effect
3) Order effect
4) Maturation
5) History
6) Regression to the mean
7) Attrition / mortality
8) Testing effect
9) Instrumentation
10) Observer bias
11) Demand characteristics
12) Placebo effects
History
Refers to any event that occurs between the beginning of treatment and the measurement of outcome that night have produced the observed effects
*Nothing to be done → can not fix this effect; have to just put it in the discussion of your research paper or start over
Maturation
A change in behavior that emerges spontaneously over time
Changes in the organism that occurs regardless of treatment might masquerade as treatment effects
*usually in development
Attrition
- Refers to who is dropping out of your study (or dying → that is mortality)
- Makes the designs complicated: the N at pretest doesn’t necessarily match the N at posttest
- As long as attrition happens at random it is ok → if all people in one group drop out and others don’t, this is a confound
Regression to the mean
Regression to the mean: when extreme scores become less extreme over time
Example: do a study and find a HUGE effect size → if you run this again, the extreme scores you saw previously will likely regress to the mean
*replication studies
Testing effects
- Testing effects: refer to a change in participants as a result of experiencing the DV more than once
- Example: practice and fatigue effects
Instrumentation
- Example: Goodhart’s law → when a measure becomes the target it ceases to be a good measure
- Factory asked workers to create x amount of nails → people made really really tiny nails to get a big number → asked to fo it by weight instead to avoid this → made three really heavy nails
- Example: scales that are designed for one population
Observer bias
performance/behavior might change because you are being watched by the researcher
Demand characteristics
participants guess what the study is supposed to be about and then change their behavior in the expected direction
- You may also be bad at self reported measures → you may not realize you are angry because you are hungry so you just report angry even though that’s not the root of the problem
Placebo effects
- When people are not getting the treatment but improve
- Can be a limitation of the study
Why might there be a null effect?
1) Weak manipulations
2) No variability/variance
3) Individual differences
4) Measurement error
5) Statistical power
Ceiling effects
scores are all on the high end
Floor effects
scores of all on the low end
Statistical power
the ability to detect an effect if one is there
Ways to increase power
Increase sample size (N)
- When you are underpowered, can only detect effect size if it is massive → only thing you can detect
P-value definition
the probability of getting our data or something more extreme IF the null hypothesis is true
Linear regression
Y = b0 + b1X + e
b0 = intercept
b1 = coefficient
X = IV
Y = DV
e = error
Interactions
test whether the effect of one IV depends on the level of the other IV
ANOVA
Multiple IVs
If two IVS: ____ x _____
In each blank, you put how many levels of each IV there are:
First IV: 2 levels
Second IV: 3 levels
2 x 3 ANOVA
Main effects
effects of each IV alone → ignoring / averaging across the other IV
Number of main effects is the number of IV’s
