Exam 2 Lecture 3 Flashcards
Poor _____ _________ is the #1 way that people lie with or botch statistics. To be able to detect the junk, you must understand the basic principles of statistics!
Data quality
How can you identify garbage?
Validity testing
Validity (designing a study that studies what you think you’re studying)
Before you believe a conclusion, you need to believe the process
Studying how well antidepressants work by measuring depression symptoms for 1 week after starting antidepressants
The DESIGN! Wrong timeline! (Need to measure later on)
Studying whether anti-anxiety medications help people sleep by looking at sleep hours on nights when meds are taken versus nights when meds aren’t taken (but forgetting to ask about alcohol, cannabis, and other drug use)
The DESIGN! Confounding factors
What are the two categories under study validity?
Internal validity and external validity
What does study validity aim to ask?
Is your study design relevant to answer the question you are asking?
Study Validity
Refers to how likely the results that you get from an experiment/research study reflect the reality of the larger population
Study validity ensures that ____ you measure something is valid, not just ______ you measure
Example
Study validity ensures that HOW you measure something is valid, not just WHAT you measure.
Example
- You can ask someone how often they get high with a validated cannabis use survey, but would you get different results from these study designs:
- Anonymous online survey
- Survey completed in police station
- Survey completed in a comfortable lab setting with research assistants
Internal Validity
I believe the results of the study because it was done well.
Careful consideration of construct validity (same data are collected on everyone)
- Face validity -> I believe that participants are going to understand what to do and do it correctly.
- Criterion validity-> I believe that the measures are equivalent to a ‘gold standard’ and will predict what you want to predict.
- Content validity -> I believe that your measures will ensure complete and consistent data
Careful consideration of the study design so that I believe that possible other explanations are limited.
- Within the study, there is limited noise.
Did you do your study correctly? Things to keep in mind
- Is your result “real” or is it due to a methodological error or an artifact of how the study was designed/carried out?
Errors can come in many forms.
This is about establishing cause and effect
- What are possible alternative explanations?
- How well does your study rule out alternative explanations?
A good study takes a lot of time to design because you have to imagine all possible outcomes and make sure you account for things that could interfere
Internal Validity- Do people who exercise more have lower resting HRs?
Designing an experiment to test this requires a lot of thinking
1. All people? A certain age group, ethnicity, health status, fitness level?
2. All exercise? Physical activity too? Strength vs. endurance?
3. How much rest is truly resting? When should you measure it?
You’re in charge, so make some decisions (& have a reason)
1. Age 35-45. Because this is when cardiovascular disease starts becoming an issue.
2. All physical activity- calculated prospectively for a month. Because quantifying exercise is complicated; there are some surveys.
3. Sit quietly for 3 minutes; count your breaths. Then record HR. Do this 3 times per day. Because it fluctuates; want to control mind-wandering; breathing is key to HR.
Data Literacy!!!!!!! What are some issues?
Researchers publish their METHODS but most of us just hear about it through social media
- This means that a 5000- word scientific finding is distilled into a few characters or seconds. Problem is: the devil is in the details.
What if….
- I included all different ages in my study?
- I had asked people to self-report how much they thought they exercised last month?
- If I had people measure HR at 12pm - 4pm - 9pm
These design factors would increase alternative explanations for my results!
Confounding variable
A unmeasured third factor that could explain a result
- A variable that you didn’t account for, but that confounds the result
- Something that influences an outcome or result that was not measured or was not intended to be measured
External validity
This is about whether your sample is representative and whether your study design would translate to different settings.
Things to keep in mind:
- Is your outcome broadly applicable to other people in other settings?
- Do the results from your sample parallel the results you’d get from other samples/the overall population?
- Can you infer someone else’s results from those you have?
If it has external validity (if it’s externally valid), then it has _________________
Generalizability.
- How ‘big picture’ are your results?
- How related/applicable are your results to other people/settings?
Can you generalize the findings?
Summarize study validity, internal validity, external validity, confounding variable, and generalizability.
Study validity: Ensures that HOW you measure something is valid, not just WHAT you measure.
Internal validity: Is your result “real” (valid) or is it due to a methodological error or an artifact of how the study was designed/carried out?
External validity: Is your study broadly applicable (valid) or specific to the sample you studied (because of a methodological issue, like an unrepresentative sample)?
Confounding variable: An unmeasured third factor that could explain a result, something you didn’t account for but confounds the result
Generalizability: Can you generalize your findings? How applicable are they to the ‘big picture’?
