Problems of Measurement

Question 1

How can indirect experience contribute to empirical statments?

Answer 1

Indirect experiences can be based on other experiences through inferences we make

Question 1

Know the definitions of the five types of statements (empirical, analytic, value, attitude, metaphysical) and be able to identify them

Answer 1

Empirical: information is obtained through the experience of our senses, through observations

Analytical: assert something about the meaning of words, not the observable world. They are true or false.

Value: express some positive or negative evaluation of something or someone

Attitude: express feeling or what they are thinking about something, but little to no observation about that something

Metaphysical: asserting that something cannot be observed with out senses, lack empirical meaning.

Question 2

What is the verifiability principle and how does it relate to operational definitions in research?

Answer 2

Verifiability principle: empirical statement tells us what sense experiences people would have if the statements were TRUE

Operational definitions: defining your measures and what they mean, so people can understand what you are doing

Words used in the statement have to have the same meaning about experience for everyone who wants to verify

Question 3

What is the role of falsifiability in empirical statements? How can a belief system become unfalsifiable?

Answer 3

Falsifiability principle: an empirical statement should also tell us what sensory experience we should have if the statement were FALSE

Belief systems can undermine falsifiability by relying on excuses so they cannot be falsified

Question 4

Understand how empirical statements are not necessarily true or immediately verifiable

Answer 4

Truth or falseness depends on observations made and the observed reality may or may not confirm these

immediately verifiable: may not be possible to make the necessary observations, but we should still know what observations would be necessary if it were possible. We do not need to prove statements for them to be empirical.

Question 5

Be aware how slight changes in wording can re-classify one kind of statement to another

Answer 5

Can change non empirical, measurable, statements into measurable statements. e.g. going from “I love you” to “I spend more time with you than anybody else”.

Question 6

Be able to define, give examples of, and identify the follow properties of measurement, random error/noise

Answer 6

Error values fluctuate randomly around the underlying true value of the variable. Repeated measure can cancel out the mean random error

Question 7

Be able to define, give examples of, and identify the follow properties of measurement, systematic error/bias

Answer 7

An error that distorts measurements consistency by a fixed amount from the underlying true value/ If experimenter bias impacts one group more than this can be critiqued

Question 8

Be able to define, give examples of, and identify the follow properties of measurement, reliability/precision

Answer 8

An index which measures how well random noise has been controlled

Question 9

Be able to define, give examples of, and identify the follow properties of measurement, internal consistency

Answer 9

The more reliable the measure the better the statistical analysis

Question 10

Be able to define, give examples of, and identify the follow properties of measurement, test-retest reliability

Answer 10

Test at one timepoint then again at another and compare

Question 11

Be able to define, give examples of, and identify the follow properties of measurement, expectancy effects

Answer 11

Research process by the experimenter unconsciously influencing responses

Question 12

Be able to define, give examples of, and identify the follow properties of measurement, selection bias

Answer 12

The kind of participants selected may exaggerate or diminish results

Question 13

Be able to define, give examples of, and identify the follow properties of measurement, testing effects

Answer 13

The process of testing may change people and give artificial results

Question 14

Be able to define, give examples of, and identify the follow properties of measurement, demand characteristics

Answer 14

Participants figure out what the study is testing and play along

Question 15

Be able to define, give examples of, and identify the follow properties of measurement, response bias

Answer 15

Participants may give biased responses in the yes direction, avoid extremes, or try to make themselves look good

Question 16

Be able to define, give examples of, and identify the follow properties of measurement, internal validity/accuracy

Answer 16

How well you measure what you want to measure and not something else, depends on how well systematic error and confounds have been controlled for

Question 17

Be able to define, give examples of, and identify the follow properties of measurement, convergent validity

Answer 17

The used scale should correlate better with things related to it. e.g. depression should correlate with mood swings

Question 18

Be able to define, give examples of, and identify the follow properties of measurement, discriminate validity

Answer 18

your scale should correlate better with what you are actually measuring than something else. e.g. depression and not anxiety

Question 19

Be able to define, give examples of, and identify the follow properties of measurement, external validity

Answer 19

How well the measure generalises to a variety if real world situations

Question 20

Be able to define, give examples of, and identify the follow properties of measurement, measurement invariances

Answer 20

Same relationship among items in different populations

Question 21

Know how measurement and manipulations in psychology can be improved beyond the current standards and the roles of manipulation checks and testing

Answer 21

Measurement in psychology: done on the spot, making up items, with no attempt to test their reliability or internal validity beforehand

Manipulations in psychology: are often not tested, either with a manipulation check in the study or with a separate pre-test

To improve these we should do the opposite

Question 22

Be familiar with the evidence and argument for widespread poor measurement practices in psychology

Answer 22

Evidence: most measurements used in experiments are lacking evidence, according to Barry et al., (2014), they reported 40% to 93% or measures lacked validity evidence

Weidman et al., (2017) reported that among 356 measurement instances coded in the reviews of emotions research, 69% included no reference to prior research.

Argument: this occurs because of a lack of transparency

Question 23

What is the argument as to why there is widespread poor measurement practices in psychology?

Answer 23

Due to a lack of transparency

Question 24

What connections are there between questionable measurement practices and other bad practices such as “forking paths”?

Answer 24

QRP: practices which exploit ambiguities in what is acceptable for the purpose of obtaining a desired result

Flexibility is inherent in the research process and exists regardless of the researchers conscious intent, the flexibility is called “Forking Paths”. As each decision can take you down a different path.

QRP and Forking Paths are connected as both can be done on accident and they always exist, while they can also be done with malintent

Question 25

Understand Shadish’s additional properties of statistical-conclusion validity and how bad measurement practices threaten it

Answer 25

Four types of validity: internal, external, statistical conclusion, and contract.

Additional properties of statistical-conclusion validity:
Are conclusions from the statistical analysis correct?
Bad measurements threaten it as without detailed information this can lead to faulty conclusions, poor validity and replications. Causing bad research.

Question 26

Understand how each of Flake and Fried’s six questions help to improve use of measurement and/or reporting of measurement in psychology research

Answer 26

1. What is your construct?
   Allowing the reader can agree or disagree with its theoretical underpinnings. Reporting this allows the reader to understand and navigate all the measurements spoken about
2. Why and how did you select your measure?
   Important as there is usually lots of methods and potential instruments to choose from. Protects against the jingle and jangle fallacies (jingle, two instruments, same names, but don’t do the same thing. Jangle, two instruments measure different things because they have different names). Not doing this contributes to the jingle-jangle.
3. What measure did you use to operationalise the construct?
   Reporting how you structured everything, e.g. the number of stimuli, where the instrument or task came from, response format, what version used, etc. Not doing this impacts validity as it cannot be replicated.
4. How did you quantify your measure?
   Establishing prior rules, how you score an instrument. Allows readers to rule out threats to validity and to be more confident and to know the measurement flexibility was not exploited to obtain the results
5. Did you modify the scale? And if so, how and why?
   Using an existing scale reduces researchers degrees of freedoms. If changes are not declared than this can impact validity
6. Did you create a measure on the fly?
   When doing this, five main questions need to be addressed to justify it. If they are not answered then it can lead to invalid interpretations of the score meant to measure the construct. Disclosing as much evidence as to why it was created can protect against threat to validity