Chapter 3: Three Claims, Four Validities Flashcards
Claim
An argument someone is trying to make.
What do claims base their focus on?
empirical research
What do claims follow?
The data in the theory-data cycle
What are the 2 challenges of claims?
- Claims based on psychological variables often depend on indirect measurements of concepts
- Different types of claims require different kinds of empirical evidence.
What is an example of a psychological variable that can be hard to measure?
Self control; need to be specifically defined
Operational Definition
Defines how a concept is explicitly measured.
How is an operational definition created?
Takes a concept that is not directly measured and is changed so it can be directly measured.
How many operational definitions can be made for one conceptual definition?
Multiple
What is the checklist for Operational Definitions?
- Has clear rules
- Does not over depend on interpretation of another concept that is not explained = observable
- Results in a variable with multiple values for each person in your sample study.
- Results in a valid measurement of construct.
What are the 3 types of claims?
- Frequency
- Association
- Causal
Frequency claim
Statements of how something occurs
How many variables are covered with frequency claims?
1
What is the intent of frequency claims not focused on?
Explaining what causes the level of variable
What are frequency claims often used to draw attention to?
Prevalence
What should the first question when analyzing a frequency claim be?
“What is the sample from which this prevalence estimate has been made?”
Association Claim
Asserts that two variables are related to each other.
What is measured in association claims?
Multiple variables + their connection
What can you not assert about association claims?
Why the relationship between the two variables exists
What should you ask yourself when analyzing an association claim?
“Could another variable plausibly cause both?”
Positive Correlation
As one variable increases, so does the other
What is an example of a positive correlation?
Ice cream sales and accidental drownings
What is the 3rd variable viewed as in correlations?
The problem; no connection between variables if the third variable is not present.
Negative Correlation
As one variable increases, the other decreases.
What is an example of negative correlation?
The more it rains, the less you water the garden.
No correlation
There is no pattern between the 2 variables
Causal Claim
Argue that two variables are related because one variable causes another variable.
How is a causal claim usually asserted?
“Variable X causes a change in variable Y”
What does manipulating an independent variable help us do?
Make causal claims by measuring the change in the dependent variable.
Independent Variable
Manipulated during an experiment
Dependent Variable
Measured during an experiment
What happens when the operational definition is invalid?
Makes it impossible to evaluate claims.
What 2 questions should you ask to evaluate the operational definition?
- Is the operational definition appropriate?
- Does the measure provide an accurate estimate of the theoretical construct?
Validity
How accurate the claim is it is given the data at hand.
What are the 4 types of validity?
- Construct
- External
- Internal
- Statistical
Construct Validity
The extent to which a measure provides an accurate estimate of the concept.
What needs to be addressed to evaluate the construct validity for an association claim?
Each variable included
What question needs to be asked when evaluating the construct validity of a study?
- Does the independent variable and its manipulation reflect a change in the construct represented by the independent variable?
What needs to be evaluated and measured in construct validity?
Evaluate: Operational definition
Measure: Dependent Variable
External Validity
The extent to which the results of a study generalizes to a larger population and/or other situations.
What question should you ask yourself about the people to evaluate the external validity?
Who does the result creating the claim apply most to?
What question should you ask yourself about the situation to evaluate the external validity?
What is the situation and would results apply to other contexts, times, or places?
Internal Validity
The extent to which confounds in the relationship between two variables have been minimized.
What question should you ask yourself when analyzing internal validity?
Is there an alternative explanation for a change in the dependent variable?
What happens if you answer yes to there being an alternative explanation about the internal validity?
A confound in the data is present
Confound
Something other than the intended manipulated variable also varies between conditions or levels of the independent variable.
What effect does the confound have on internal validity?
Weakens
How can you improve internal validity?
Add more control on third variables and directionality
How do you test internal validity?
Analyze any potential confounds in the studies and determine if they were controlled.
What is the relationship between internal and external validity?
Decisions that help maximize internal validity often hurt the external validity
- limit population and situations studied
Statistical Validity
The extent to which the study analyses were accurate and the strength of the effect.
What should you ask yourself when analyzing statistical validity?
Did they do the right statistical test?
What 3 questions help assess statistical validity?
- How big is the effect?
- What is the margin for error in the outcome variable?
- What is the likelihood of these results occurring by chance; can the effect be replicated?
What is the order of importance for types of validity?
- Construct
- Internal
- External
What type of validity is this scenario questioning:
Clarissa questions the study, saying, “I don’t trust their measure of sleep, fit bits cannot tell the difference between watching tv and going to bed.”
Construct
What type of validity is this scenario questioning:
Justin questions, “Do you think the results would generalize to students at another University?” Justin’s concern is addressing which of the following?
External
What type of validity is this scenario questioning:
Sara questions whether the researcher’s claim really fits with their data, “How can they claim that sleep consistency improves memory, couldn’t self-control be related to both sleep consistency and consistency of studying for tests?”
Internal
What type of validity is this scenario questioning:
Suppose you want to get into the discussion because you have an issue with the researcher’s operational definition of memory as grades in the class. Your concern is addressing which of the following?
Construct
In a study focused on how amount of sleep impacted a students grades in their class, what variable would needs to be manipulated to create a causal claim?
Sleep
Reliability
Obtaining consistent scores for each study.
What is the relationship with reliability and validity?
A measure must be reliable to be valid
What is the center of the target known as?
Concept Truth
How is reliability displayed in the target?
All points clustered together in a certain area
How is validity displayed in the target?
The marks closeness to the center of the target.
Explain how the target that is reliable and valid looks?
All points clustered together in the center.
Explain how the target that is reliable but not valid looks?
All points are clustered outside of the center
Explain how the target that is not reliable and not valid looks?
Al points are spread out across the whole target
What are the 3 kinds of reliability?
- Test-retest
- Internal
- Interrater
Test-Retest Reliability
Consistent results with a person each time they take a test/survey
What is the given assumption in test-retest reliability?
There is no true change occurring to cause different results
What should all tests of stable variables have?
Good test-retest reliability
What question should be asked to verify test-retest reliability?
Does the test give similar results when given multiple times to the same person?
What is an example of test retest reliability?
A person gets consistent scores each time she retakes a test
How can test-retest reliability be verified?
Plotting points on a graph to determine the closeness to the identity line
What does each dot on the graphs represent?
A person
Identity Line
Where measurements fall if they are exactly the same at each measurement time.
What would perfect reliability look like on a graph?
Having the dots directly on the line
What are the values of the correlation score?
0-1
What score reflects a higher correlation between variables?
Closer to 1
What correlation score is acceptable and ideal for test-retest reliability?
Acceptable: >0.5
Ideal: > 0.7
Internal Reliability
Quantifies how similar responses are to the questions designed to measure the same construct.
What is internal reliability applied to?
- Surveys (BIG ONE)!
- Achievement tests
- Skill tests
How can internal reliability be measured?
Cronbach’s Alpha
Cronbach’s Alpha
Quantifies consistency of responses to questions that are supposed to measure the same thing.
What scores for Cronbach’s Alpha is acceptable and ideal?
Acceptable: >.7
Ideal: >.8
What is an example of internal reliability?
People give consistent responses on every item of a questionnaire due to questions being similar.
Interrater Reliability
Measure of how consistent multiple raters are when evaluating the same thing.
When is interrater reliability important?
Turning observations of behavior into a variable
What question should be asked when testing interrater reliability?
How similar are two different people’s recorded observations of a behavior?
Rater
A person recording observations about behavior and turning it into a score.
Inter Rater
Similarity between the raters.
What is an example of interrater reliability?
Hypothetical ratings for “self-talk” strategy in the marshmallow test, operationalized as the number of times the child talks themselves through resisting an impulse.
What needs to be assumed for each rater in order to get similar ratings?
The variable being rated is operationally defined the same way.
What are the 3 types of validity tests?
- Criterion
- Convergent
- Discriminant
Criterion Validity
Asks whether the measure correlates with a real-world outcome of the concept.
What needs to be assumed about the construct and outcome in criterion vaidity?
Assume construct has a causal influence on the outcome
What influence does the measure have on construct and outcome in criterion validity?
-If the measure is highly related to construct, then the measure should also be highly related to the outcome.
Convergent Validity
How strongly scores on your measure are related to other measures of the same concept.
What do other measures not need to be in convergent validity?
Real world or concrete behavioral outcome
What correlation score are we looking for in convergent validity?
Higher than .70
What is an example of testing convergent validity?
Compare the marshmallow test scores with another measure of self-regulation in children.
How should the graph of convergent validity look?
All values are correlated making a straight line.
Discriminant Validity
How different are the scores between your measure and a measure of a different concept.
What does convergent validity reflect?
Whether the measure is only measuring what it intends to measure.
What is expected when testing discriminant validity?
A weaker relationship with a measure of a different concept
What is an example of testing discriminant validity?
Compare marshmallow test with score of extraversion and long-term memory.
How should the graph look for discriminant validity?
Dots all over the place; no correlaction
Face Validity
Subjectively the measure seems plausible.
Which validities can we use to maximize frequency claims?
- External
- Construct
- Statistical
What validities can we use to maximize ALL types of claims?
- Statistical
- Construct
Content Validity
Subjectively the measure contains all the parts that the theory says it should contain.
