Problems of Measurement Flashcards
How can indirect experience contribute to empirical statments?
Indirect experiences can be based on other experiences through inferences we make
Know the definitions of the five types of statements (empirical, analytic, value, attitude, metaphysical) and be able to identify them
Empirical: information is obtained through the experience of our senses, through observations
Analytical: assert something about the meaning of words, not the observable world. They are true or false.
Value: express some positive or negative evaluation of something or someone
Attitude: express feeling or what they are thinking about something, but little to no observation about that something
Metaphysical: asserting that something cannot be observed with out senses, lack empirical meaning.
What is the verifiability principle and how does it relate to operational definitions in research?
Verifiability principle: empirical statement tells us what sense experiences people would have if the statements were TRUE
Operational definitions: defining your measures and what they mean, so people can understand what you are doing
Words used in the statement have to have the same meaning about experience for everyone who wants to verify
What is the role of falsifiability in empirical statements? How can a belief system become unfalsifiable?
Falsifiability principle: an empirical statement should also tell us what sensory experience we should have if the statement were FALSE
Belief systems can undermine falsifiability by relying on excuses so they cannot be falsified
Understand how empirical statements are not necessarily true or immediately verifiable
Truth or falseness depends on observations made and the observed reality may or may not confirm these
immediately verifiable: may not be possible to make the necessary observations, but we should still know what observations would be necessary if it were possible. We do not need to prove statements for them to be empirical.
Be aware how slight changes in wording can re-classify one kind of statement to another
Can change non empirical, measurable, statements into measurable statements. e.g. going from “I love you” to “I spend more time with you than anybody else”.
Be able to define, give examples of, and identify the follow properties of measurement, random error/noise
Error values fluctuate randomly around the underlying true value of the variable. Repeated measure can cancel out the mean random error
Be able to define, give examples of, and identify the follow properties of measurement, systematic error/bias
An error that distorts measurements consistency by a fixed amount from the underlying true value/ If experimenter bias impacts one group more than this can be critiqued
Be able to define, give examples of, and identify the follow properties of measurement, reliability/precision
An index which measures how well random noise has been controlled
Be able to define, give examples of, and identify the follow properties of measurement, internal consistency
The more reliable the measure the better the statistical analysis
Be able to define, give examples of, and identify the follow properties of measurement, test-retest reliability
Test at one timepoint then again at another and compare
Be able to define, give examples of, and identify the follow properties of measurement, expectancy effects
Research process by the experimenter unconsciously influencing responses
Be able to define, give examples of, and identify the follow properties of measurement, selection bias
The kind of participants selected may exaggerate or diminish results
Be able to define, give examples of, and identify the follow properties of measurement, testing effects
The process of testing may change people and give artificial results
Be able to define, give examples of, and identify the follow properties of measurement, demand characteristics
Participants figure out what the study is testing and play along
Be able to define, give examples of, and identify the follow properties of measurement, response bias
Participants may give biased responses in the yes direction, avoid extremes, or try to make themselves look good
Be able to define, give examples of, and identify the follow properties of measurement, internal validity/accuracy
How well you measure what you want to measure and not something else, depends on how well systematic error and confounds have been controlled for
Be able to define, give examples of, and identify the follow properties of measurement, convergent validity
The used scale should correlate better with things related to it. e.g. depression should correlate with mood swings
Be able to define, give examples of, and identify the follow properties of measurement, discriminate validity
your scale should correlate better with what you are actually measuring than something else. e.g. depression and not anxiety
Be able to define, give examples of, and identify the follow properties of measurement, external validity
How well the measure generalises to a variety if real world situations
Be able to define, give examples of, and identify the follow properties of measurement, measurement invariances
Same relationship among items in different populations
Know how measurement and manipulations in psychology can be improved beyond the current standards and the roles of manipulation checks and testing
Measurement in psychology: done on the spot, making up items, with no attempt to test their reliability or internal validity beforehand
Manipulations in psychology: are often not tested, either with a manipulation check in the study or with a separate pre-test
To improve these we should do the opposite
Be familiar with the evidence and argument for widespread poor measurement practices in psychology
Evidence: most measurements used in experiments are lacking evidence, according to Barry et al., (2014), they reported 40% to 93% or measures lacked validity evidence
Weidman et al., (2017) reported that among 356 measurement instances coded in the reviews of emotions research, 69% included no reference to prior research.
Argument: this occurs because of a lack of transparency
What is the argument as to why there is widespread poor measurement practices in psychology?
Due to a lack of transparency
What connections are there between questionable measurement practices and other bad practices such as “forking paths”?
QRP: practices which exploit ambiguities in what is acceptable for the purpose of obtaining a desired result
Flexibility is inherent in the research process and exists regardless of the researchers conscious intent, the flexibility is called “Forking Paths”. As each decision can take you down a different path.
QRP and Forking Paths are connected as both can be done on accident and they always exist, while they can also be done with malintent
Understand Shadish’s additional properties of statistical-conclusion validity and how bad measurement practices threaten it
Four types of validity: internal, external, statistical conclusion, and contract.
Additional properties of statistical-conclusion validity:
Are conclusions from the statistical analysis correct?
Bad measurements threaten it as without detailed information this can lead to faulty conclusions, poor validity and replications. Causing bad research.
Understand how each of Flake and Fried’s six questions help to improve use of measurement and/or reporting of measurement in psychology research
- What is your construct?
Allowing the reader can agree or disagree with its theoretical underpinnings. Reporting this allows the reader to understand and navigate all the measurements spoken about - Why and how did you select your measure?
Important as there is usually lots of methods and potential instruments to choose from. Protects against the jingle and jangle fallacies (jingle, two instruments, same names, but don’t do the same thing. Jangle, two instruments measure different things because they have different names). Not doing this contributes to the jingle-jangle. - What measure did you use to operationalise the construct?
Reporting how you structured everything, e.g. the number of stimuli, where the instrument or task came from, response format, what version used, etc. Not doing this impacts validity as it cannot be replicated. - How did you quantify your measure?
Establishing prior rules, how you score an instrument. Allows readers to rule out threats to validity and to be more confident and to know the measurement flexibility was not exploited to obtain the results - Did you modify the scale? And if so, how and why?
Using an existing scale reduces researchers degrees of freedoms. If changes are not declared than this can impact validity - Did you create a measure on the fly?
When doing this, five main questions need to be addressed to justify it. If they are not answered then it can lead to invalid interpretations of the score meant to measure the construct. Disclosing as much evidence as to why it was created can protect against threat to validity
