BRINGMANN & ERONEN, 2016: WHAT PSYCHOLOGISTS CAN LEARN FROM THEORY OF PHYSICS

Question 1

Current theories of psychological measurements are largely disconnected from discussions of measurement in the natural sciences. There are substantial differences between psychological and physical measurement. Name three of these substantial differences

Answer 1

• Human participants are capable of learning and may react differently at different times
• Psychological measurements are usually sum scores made up of item responses
• The results may have social and ethical implications, etc.

Question 2

What is ‘the most fundamental and controversial issue is psychological measurement’? Why is it controversial?

Answer 2

Validity is the most fundamental and controversial issue is psychological measurement. The classical definition of validity is the extent to which the test or instrument measures what it is intended to measure. However, contemporary literature on validity different greatly and no agreement is reached on how validity should be defined.

Question 3

Is validity an issue in sciences such as physics?

Answer 3

The natural sciences do not use the same terminology as psychologists or psychometricians and do not talk about the validity of measurements. This does not mean that validity is not a related issue in physics

Question 4

Draw a link between an issue of validity within physics and the issues psychology face now

Answer 4

In the 19th century, the focus on measuring temperature was to make it increasingly precise, consistent, and compatible without exploring theoretical work on the nature of temperature. Similarly, current psychological ideas and practice of measurement focus on criteria such as reliability, generalisability and correlation with other measures.

Question 5

How does psychology differ from physics on where they place their focus in establishing measurement? Why is this relevant?

Answer 5

The focus is far less on theories on psychological attributes or causal processes underlying the measurements. The atheoretical approach of early temperature measurement was insufficient and progress was only made when measurements were linked to theory. Hence, psychology should also emphasise theory of measurement and validity

Question 6

What ‘three concrete conclusions’ do the authors propose to be drawn from the temerature case and relevant to psychology?

Answer 6

1. Studying the causal mechanisms underlying the measurements can be important for evaluating whether a measurement is valid.
2. Psychologists would benefit from focusing more on the robustness of measurements.
3. It is possible to make good science based on (relatively) bad measurements, and explanatory success of science can contribute to justifying the validity of measurements.

Question 7

What is meant by the robustness of measurements?

Answer 7

Robustness is the idea that if there are several independent ways of measuring something, this increases the confidence in the measurements.

Question 8

What is meant by the term causal mechanisms?

Answer 8

Mechanism is a set of components that are organised together to perform a function (covers physical, cognitive, and biological mechanisms). Causal and causation are in terms of difference-making and potential manipulation and control.

Question 9

What did the early atheoretical approach to measurement result in?

Answer 9

The early atheoretical approach to measurement resulted in very precise and comparable temperature measurement, but it not result in increased understanding of what temperature is or in assessing what happens when the validity of measurements is unclear.

Question 10

What can provide evidence for the validity of instruments?

Answer 10

High degree of precision, consistency, and comparability is not necessary for making valid measurements and theoretical progress can provide indirect evidence for the validity of measurements.

Question 11

What was the measurement of temperature originally based on?

Answer 11

The measurement of temperature was not originally based on physical theory but from subjective experiences and simple empirical regularity.

Question 12

What did Regnault contribute to temperature?

Answer 12

The improvement of precisions and consistency of temperature measurements was not influenced by theoretical developments. This approach of improving thermometers based on empirical criteria (e.g. precision, consistency & reliability, robustness, and comparibility) was the work of French scientist Regnault.

Question 13

What was Regnault’s attitude towards developing measurements for temperature and how effective was it?

Answer 13

Regnault shunned all theoretical speculations about the nature of temperature and emphasized the importance of rigorous testing with minimal assumptions. Although his atheoretical approach can guarantee that measurements are consistent and comparable in controlled conditions, it falls short when the conditions are new or unknown.

Question 14

What was required in order to improve measurement of theory in novel situations?

Answer 14

It is not possible to resolve this issue of reliability based on empirical data. In order to solve the problem, theoretical knowledge on how the measurement instrument actually worked was required.

Question 15

Where else did Regnault’s approach fall short?

Answer 15

Regnault’s approach also did not result in increased understanding of what temperature is or in new connections with other areas of physics. Hence, the temperature scale remained arbitrary despite his efforts. Only after theoretical connections were made did it become possible to formulate an objective definition for what a change of one degree temperature means to calculate absolute zero.

Question 16

What other positive effect did the advancement of theory have on measurement of temperature once it had allowed for formulating an objective definition for what a change of one degree temperature means to calculate absolute zero?

Answer 16

This also allowed for increasing the robustness of temperature measurements.

Question 17

What claim do the authors make about bad measurements?

Answer 17

In order to achieve valid measurements and scientific progress, the high degree of precision, consistency, and comparability is not necessary. Hence, it is possible to do good science based on bad measurements. Additionally, theoretical progress can contribute to the validity of measurements retroactively.

Question 18

What can psychology learn from the history of temperature?

Answer 18

The general moral that psychology can draw from the history of temperature measurement is that theory is crucially important for measurement and validity. While an atheoretical approach will result in measurements that are consistent and comparable under a limited range of conditions, it will not guarantee the validity of measurements or lead to scientific progress. Psychology is facing the same issue as physicists faced when measurement temperature. The focus is on criteria such as reliability and invariance at the expense of theory-building or theoretical speculations.

The standard approach in psychometric modelling is to find statistical models that fit the data, which can be done without theoretical assumptions of the construct that is measured. Psychological scales and units lack clear theoretical foundations and a lack of understanding of the nature of the attributes are being measured. The atheoretical approach of current psychological research can result in consistent and precise measured under a limited range of conditions, but will not guarantee the validity of the measurements.

Question 19

According to the authors, there are three important insights for validity (and psychological measurement in general) that can be drawn from physical cases (e.g. temperature case). What are these?

Answer 19

1. Causal mechanisms
   Theoretical understanding of the causal mechanisms of measurement is crucial for assessing the validity of measurements, especially in novel circumstances.
2. Robustness
   If there are several independent ways of achieving the same result, this increases the confidence in the results.
3. Bad measurement can result in good science
   Relatively bad measurements can result in good science, and scientific progress can justify the validity of measurements in hindsight. What increases confidence in the validity of measurements is the success of the theories that are based on them, and what justifies the success of those theories is the explanatory and predictive power.

Question 20

In regards to the first insight regarding causal mechanisms, what questions can arise?

Answer 20

In regards to first insight, the questions arises whether in psychology the relevant mechanisms can actually be studied/measured.

Question 21

What does to Embretson’s (1983) account of validity state?

Answer 21

According to Embretson’s (1983) process-oriented account of validity, traditional assessment of construct validity needs to be supplemented with studies of the cognitive processes and strategies that participants use to response to test items, based on cognitive psychology. The items selected for the test are thus based on cognitive theory and the models include parameters representing the cognitive demands of the item. Hence, cognitive theory influences both test construction and the measurement models.

Question 22

To what extent does Embretson’s (1983) process-oriented account of validity satisfy the point of explaining causal mechanisms?

Answer 22

Including cognitive parameters in measurement models is not the same as describing the causal mechanisms underlying the measurement process. Model should be used to describe the steps in the causal process that start with the attribute intended to be measured and end with the measurement outcome

Question 23

What reason might there be that explaining the causal mechanisms is often not attempted in psychology? What message does the author have to say about this

Answer 23

It may be that the reason why this is not generally attempted, or not even seen
as a goal, is that the causal mechanisms in psychology are thought to be so complex that
figuring out the causally relevant components is practically impossible.

They acknowledge that the challenges may seem daunting at present, but do not believe
that the situation is hopeless with regard to the future: great progress has been made in
recent decades in discovering mental mechanisms

Question 24

What is essential for the second insight of robustness?

Answer 24

The idea of independence is essential for the second insight of robustness. Statistical independence is not required but it is necessary that any problematic or unconfirmed assumptions should be shared by the different ways of measurement.

Question 25

When are different ways of measurement fully independent of each other?

Answer 25

Different ways of measuring are fully independent when they rely on different assumptions and different parts of theory. . Independence and robustness is a matter of degree not none-or-all

Question 26

How do we determine robustness?

Answer 26

Determining the degree of robustness depends on theories and models about the experimental setups, measuring instruments, and things being measured.

Question 27

In contemporary validity theory, where did the idea of robustness come up?

Answer 27

In contemporary validity theory, the idea of robustness comes up in the form of convergent validity. Convergent validity is evidence from other measures that are intended to assess the same or a similar attribute.

Question 28

Is convergent validity a good metric for robustness?

Answer 28

No special attention is paid to independence of measured or deriving the results from theory. Hence, convergent validity is a weak form of robustness. Furthermore, psychology does not mention convergent validity often whereas in the natural sciences robustness is central to the validity of measurements.

Question 29

What do these guys say about that loser Michell?

Answer 29

Mitchell (1997) has argued that psychologists are treating attributes as quantitative without exploring whether they meet the requirements for being a quantitative structure. Hence, psychological measurements lacks foundation and as long as psychology does not explore quantitative structures then it is a “pathological” science.

Question 30

Following this, what shortcomings have psychological theories faced?

Answer 30

Psychology so far has failed to produce theories that significant or have explanatory power, and current theories are not rich and detailed enough to provide serious tests for the hypothesis that psychological measurements are valid.

Question 31

Overarching theories like those developed in the case of temperature are not foreseeable in psychology according to the authors. Why is this? What can be done to aid this?

Answer 31

Overarching theories like those developed in the case of temperature are not foreseeable in psychology. This is partly due to the fact that psychology and the life sciences tend to be more local than in physics. It is more likely that instead of one unified theory of human psychology, there will be increasingly precise theories or models of psychological constructs. Theoretical development in psychology might be hampered by the assumption that theories are better the more general they are. However, it might be better for psychology to focus on local mechanisms and their specific and limited roles and functions. Then these local theories and explanations can be elaborated upon and made as plausible as possible (e.g. short-term memory).

Question 32

What main differences between physical attributes and psychological attributes did the authors conclude on and what conclusion did they draw from this?

Answer 32

The main differences between physical attributes and psychological attributes are that (1) most psychological attributes have not been embedded into any successful and widely accepted theory and (2) there is not solid theoretical foundation for the units, ratios, and scales for psychological attributes. These differences do not rule out the possibility of measuring psychological attributes.