Lec. 27 - Measurement Theory and Statistics

Question 1

What does this lecture deal with?

Answer 1

1- mini-history of measurement theory
2- scale levels and transformations
3- implications for statistical practice
4- debate between measurement and statistics

Question 2

1. Mini-history of measurement - general first steps

Answer 2

• 500 years ago there were no sciences
• only few people were studying science
• the basis became “empirical research” (check ideas against observations)
• Galileo Galilei→ you can represent relations in nature through mathematical equations, in a systematic way
  > people started measuring nature (e.g. trying to distinguish white and black pieces and how far they had to go to be able to distinguish it)
  = arose idea that subjective qualities (e.g. sight) can be measured with units

Question 3

Francis Galton
- what did he discover?

Answer 3

• measure intelligence, regression coefficient, fingerprint, weather map, eugenics (to make better race)
• “anthropometric laboratory” → he stated that he could measure human intelligence

> the important part is that he tried to measure human intelligence (not in the same way as we do now of course); the other discoveries are not as important to know

Question 4

Charles Spearman
- what did he discover?

Answer 4

• positive manifold of intelligence
  > on average, if you’re good at one intelligence task, you are good at all of them (e.g. verbal intelligence, spatial skills, …)
  > all correlations are positive between the subparts of intelligence
  ! it goes for many things in life, not just intelligence
• coined the concept of “general intelligence”, which can be measured in degrees, and explains positive manifold

Question 5

what are the consequences of Spearman’s discoveries?

Answer 5

• psychologists start talking about “measurements”
  > e.g. total score of IQ-test is interpreted as general intelligence
• some psychollogists hold that there is no essential difference between physics and psychology
• psychophysics researchers measurements to assess observable differences to infer subjective qualities (e.g. sight)
  → physicists disagree with psychologists using measurements without meeting necessary scientific requirements

Question 6

Campbell
- what was he arguing against?

Answer 6

• one of the more vocal physicists against measurements used in psychology
  > “there is a very specific way in which real measurements are formed and it is not met in psychology”

Question 7

Campbell
- what are measurements a result of ?
- what is this known as?

Answer 7

• measurement is the result of a process of assigning numbers such that:
  > each object is represented by a single number
  > the sum of two assigned numbers represents the results of an empirical combination of objects (e.g. putting two stones on a weighting scale)
  → concatenation operation

Question 8

Fundamental Measurement - what is it?

Answer 8

• measurement with no prior quantification
  > balance scale: can be used to decide whether one object is heavier than the other (qualitative scale; no numbers)

Question 9

How can we use a qualitative system to construct a numerical representation (that has the properties that we normally associate with measurement)?

Answer 9

• e.g. with balance scale
  > we decide which object is heavier without assessing any number to them
  → qualitative assessment
  > we can also concatenate massess (put them in same arm of balance)
  → here we empirically add things, without the use of mathematics
  > by putting two units on one side and putting a mass that is equal in weight on the other side, we are rcreating a standard sequence
  → now we can assign numbers to sequence, and predict what would happen when we manipulate reality based on the mathematical manipulation of the numbers
  (see picture 1)

Question 10

Standard sequence

Answer 10

• sequence of weights where difference between each weight is one unit
• units are arbitrary (we could choose any unit)
  (see picture 2)

Question 11

Representational measurement theory

Answer 11

• numbers are assigned such that relations between numbers mirror relations between objects
• numerical structures are mirroring empirical structures
  (see picture 3)

Question 12

2. Scale levels and transformations
   - what happens when we transform the numbers and not account for it?

Answer 12

• e.g. we square the numbers (1, 2, 3, 4 → 1, 4, 9, 16)
  > we can’t do that because the difference between 1&2 and 2&3 … is always 1, while the difference between 1&4 and 4&9 … is not the same
  → prediction is that it will not balance out = we lose the direct representation of the empirical relation (break the mirror between empirical and numeric)
  (see picture 4&5)

Question 13

What was Campbell’s claim on measurement?
- How did the British Association respond?

Answer 13

• fundamental measurement always requires concatenation (must have additive structure)
  > this means that measurement is impossible in psychology (no concatenation operations)
• the British Association for the Advancement of Science failed to reach an agreement on whether psychological measurement is possible

Question 14

How can concatenation be explained in everyday life?

Answer 14

• you can concatenate length and mass
  > 1km+5km=6km
• not true for temperature
  > 20°+40°=30° (average)
  → “temperature is not measurable”

Question 15

look at picture 6
- is this claim right or wrong?

Answer 15

• wrong: measurement is not overlooked by physicists
  > they can study and manipulate measurements precisely, as they are assessing events and objects
  > in psychology instead, measurement is more complicated because we assess humans
• right: in psychology, we don’t actually know that our measurements are precise
  > e.g. we don’t know for sure how to measure intelligence, or depression, ….

Question 16

S. S. Stevens - what did he invent?

Answer 16

• one of the most influential psychologists of all times
• he invented the scale levels
  > nominal, interval, ordinal, …

Question 17

How did Stevens define measurement?

Answer 17

• measurement is the assignement of numerals according to rule
  > any systematic rule would work
  > the rule determines what kind of measurement we have

Question 18

How should a rule be structured?

Answer 18

1. determine equality
2. determine order
3. determine scale levels
   = one is always measuring, the question is at which level
   - some things are inflexible in measurement, like mass; other things’ measurements can be manipulated more easily
   > therefore, important is what “breaks the mirror”
   > e.g. you can’t just square the numbers because mathematical relationship does no longer reflect empirical relationship

Question 19

The representational solution

Answer 19

• measurement involves an act of representation
• measurement attepts to capture the structure of an attribute in symbolic form
• numeric relations are isomorphic to empirical relations
• scale levels are defined by the set of transformations that leave the isomorphism intact
  (this is what was in the slide; he didn’t explain further)

Question 20

Steves’ measurement levels

Answer 20

• nominal
• ordinal
• interval
• ratio
  > we can choose scale that we want, as long as we stick to it

Question 21

Nominal scale

Answer 21

• numbers only represent equivalence
  > the objects that get the same number have the same property
  > e.g. postal codes

Question 22

Ordinal scale

Answer 22

• number represent order
  > objects that are assigned the higher numbers have more property represented
  > e.g. rankings
  > e.g. hardness scale of metals (metals in order for which can scratch other metals and not be scratched, …)

Question 23

Interval scale

Answer 23

• numbers represent order, but distances between assigned numbers also have a definite meaning that is the same at all levels of the scale
  > the zero-point is arbitrary (you assign zero to anything)
  > e.g. temperature in degrees celsius (zero does not mean absence of temperature), IQ

Question 24

Ratio scale

Answer 24

• numbers represent order, distances between assigned numbers have meaning, and the ratio of numbers have meaning as well
  > zero-point is not arbitrary, but indicates absence of property
  > e.g. mass in kg, time, length

Question 25

So, is measurement possible after all?
What did Mitchell debate?

Answer 25

• still debatable
  > Mitchell wrote many books saying that psychology is overstepping the boundaries of measurement
  > he tried to measure many things that are not measurable, and made sense out of them (proving that psychology might be doing the same)

Question 26

what are admissible transformations in the nominal scale?

Answer 26

• all one-to-one transformations are admissible
  > e.g. divide females and males as 1&2, or red&blue, or 35&40, …

Question 27

what are admissible transformations in the ordinal scale?

Answer 27

• all monotonic transformations are admissible
  > = all transformations that leave the order of numbers intact
  > e.g. multiply all numbers by 2

Question 28

what are admissible transformations in the interval scale?

Answer 28

• all transformations that preserve order and distances between numbers are admissible
  > all linear transformations X’-a+bX, where b is a positive constant
• the stronger the scale level, the less you can do to the assigned numbers without breaking the mirror
  (see picture 7)

Question 29

what are admissible transformations in the ratio scale?

Answer 29

• all transformatinos that preserve the ratio between numbers are admissible
  > = tranformations of the form X’=bX, where b is a positive constant
  > the fact that there is no translation as in interval level above implies an absolute zero

Question 30

3. Statistical implications
   what are the statistical implications of changing the scale?

Answer 30

• parametric tests such as t-test and ANOVA are sensitive to transformations that change distances between scale points
• as a result, nonlinear transformations will change one’s conclusions

Question 31

Why can’t we do a t-test on nominal numbers?

Answer 31

• e.g. we have an IQ scale, and we want to square the results
• it seems at first that the order of the results would remain the same, because they are just squared
• if we square the results though the distribution changes, and a t-test on the changed distribution would give different results
• therefore, if we do a non-linear transformation, we would get different results
  (see picture 8)

Question 32

Important takeaway !

Answer 32

• if we have nominal or ordinal scale, it implies that if we transform the data according to admissible transformations (which is ok given said scale level), we will get different results
  → results are not robust under scale transformations

= we are allowed to transform every scale (look at rules in flashcards 27-29), but when we transform nominal and ordinal scale, the distance between points change. Because of this, the distributions are also different, and when you do a statistical analysis the results will change. With interval and ratio scales however, the rules follow linear models, so we can statistically analyse the results

Question 33

Logical process of Q&As about statistical transformation

Answer 33

• Q: when does the t-test have a definite interpretation?
  A: only if you don’t transform the data such that the distances between scale points change
• Q: so why wouldn’t I be allowed to do that?
  A: because the data are at the interval level of measurement, so only linear transformations are admissible

= we can’t square numbers when it’s an interval square, or the distance between points will change and so will the distribution
= ! no t-test if the data aren’t at the interval level of measurement !

Question 34

Methods police vs Statisticians

Answer 34

• Methods police: no t-test on non-quantitative data! → you can use nonparametric statistics instead
• Statisticians: “there is no assumption in t-test that says that there should be a certain level of measurement, people can do t-test whenever”

Question 35

The statistical treatment of football numbers
- paper by Frederic Lord
(it’s a long flashcard, but it’s just a story, so don’t worry!)

Answer 35

Professor X works in a university, and he is a hardcore measurement theorist. He was in charge of giving out numbers to football players in the uni team, and he created a vending machine to get assigned a number. Younger students complained that they were getting lower numbers than the older students, and the professor didn’t know how to analyse data (those numbers identified players, and were nominal data). The professor went to a statistician, and statistician calculated mean and standard deviation of numbers as if they were not nominal. When attacked, the statistician replied “the numbers don’t know that! They behave the same way regardless”, and carried out with his statistical analysis.
- that line is super famous, used in conferences etc.
(see picture 9, and look at lecture for theatrical representation of dialogue)

Question 36

What is the conclusion of this story?

Answer 36

• Lord’s professor draws a sensible conclusion on the bases of illegal statistics
• thus, prohibitions like “don’t run ANOVAs on nominal data” are not always justified
  = no clear connection between the measurement level and statistical level

Question 37

Moral of the story

Answer 37

• the reasoning behind Steven’s rules is strong, and makes a lot of sense
• also is a good argument for non-parametric statistics, because these are not sensitive to nonlinear transformations
• however, as Lord shows, clever people who know what they are soing can still obtain sensible results
  (from slides)

Question 38

4. Dialogue between measurement theory and statistics
   - what are the main take-aways?

Answer 38

• dialogue is ongoing (very slow discussion, new argument every many years)
  1. reasoning behind the rules is not trivial, it’s quite good
  > e.g. good reason why you should be careful about doing t-test on ordinal data
  2. you should understand that there are many cases where we can work with data even though it might not be “legal”
  = so, how do we deal with this situation?

Question 39

The pragmatic solution

Answer 39

• concept of robustness
  1. transform your data according to transformations you think should not matter
  2. if we don’t know what the right analysis is, we can run them all
  > not p-hacking (we don’t just pick the one with the results that we want)

Question 40

What happens when we run many analysis and the results are the same/different?

Answer 40

• same results: robustness proved
• different results: we must figure out why and reconsider scale levels

Question 41

Discussion

Answer 41

• methodological rules are not written in stone, but are based on assumptions and arguments
• rules do exist, but were proposed and argued for by people (on basis of ideas, arguments and analysis)
  > therefore rules contain errors and assumptions can be questioned
• no methodological rules are safe from debate

Question 42

Chapter 2 - Scaling
- The property of identity

Answer 42

• objects or events can be sorted into categories that are based on similarity features
• it informs about whether two individuals are similar or different (no more info)

Question 43

what are the rules to follow when sorting people into categories?

Answer 43

1- the people within a category must satisfy the property of identity
2- the categories must be mutually exclusive
> items can fit only in one of the two categories
3- the categories must be exhaustive
> the number of categories must be the same as the number of possibilities of those features

Question 44

What is an example to explain the property of identity? How does it apply to the aforementioned characteristics?

Answer 44

• teachers want to differentiate students into a group with behavioral probems, and a group without
  1> each student in each group has or doesn’t have behavioural problems, depending on the group they are in
  2> the categories are mutually exclusive because a student can’t both have and not have behavioural problems
  3> the categories are exhaustive because there are no other possibilities outside of having or not having behavioral problems (no third option)

Question 45

The property of Order

Answer 45

• it conveys information about the relative amount of an attribute that people possess
• numerals serve as labels (we could also use letters instead)
  ! it does not tell us the degree of difference

Question 46

The property of Quantity

Answer 46

• provides information about the magnitude of differences between people
• numerals reflect real numbers
  > e.g. number 1 reflects the size of the basic unit on any particular scale, while other numbers are multiples or fractions of one
• real numbers: scalar, metric, cardinal, quantitative values (these are just synonyms of real numbers)

Question 47

The number 0
- what are the two meanings of zero?

Answer 47

a) the attribute of an object or event has no existence
> called absolute zero
> e.g. 0cm long → no length
b) arbitrary quantity of an attribute
> called arbitrary zero
> e.g. 0°C → arbitrary choice of that temperature as zero degrees; does not mean that there is zero temperature

Question 48

What is the dilemma about the interpretation of zeros?

Answer 48

• when someone scores zero on a test, is it absolute or abitrary zero?
  ! properties and meanings of zero are fundamental to assess the answer
  > property of identity→ most fundamental one
  > property of order→ after identity, and fundamental for quantity
  > property of quantity

Question 49

Units of Measurement
- how can measurement units be arbitrary?

Answer 49

1. the unit size can be arbitrary
2. some units of measurement are not tied to any one type of object
   > e.g. cm can measure pencils, tables, pizzas, …
3. when taking a physical form, some units of measurement can be used to measure different features of objects
   > e.g. using a ruler to measure the weight of a pineapple, by putting rulers on a weight so that they weight like a pineapple

Question 50

do units of measurement apply all characteristics?

Answer 50

• usually, units of measurement are only arbitrary in size, but they are tied to a particular object
  > e.g. unit of measurement for IQ is specific only to IQ
• however, standard measures can be used to measure psychological attributes (e.g. reaction time)

Question 51

Addivity - what does it require?

Answer 51

• assumption that unit size does not change
• the size of measurement unit should not change as the conditions of measurement change
  > conjoint measurement
  > e.g. size of a meter should remain constant no matter the day and time

Question 52

Does counting count as measurement?

Answer 52

• some say that counting qualifies as measurement only when one is counting to reflect the amount of some feature or attribute of an object
  > e.g. when counting the amount of correct responses, you are measuring the proportion of knowledge

Question 53

Scaling - what is it?

Answer 53

• particular way in which numbers are linked to behavioral observations to create a measure
  > some people mistake measurement and scaling

Question 54

Interval scales

Answer 54

• have arbitrary zero
  > e.g. temperature
  > in psychology, vast majority of intelligence tests, personality tests, achievement tests, developmental tests are treated as interval scales
• the size of unit is constant and additive, but the scale does not allow multiplicative interpretations
  → e.g. you can add two degrees to 40°, and you would get 42°, however by multiplying 40x2=80°, it’s wrong to interpret that 80° is twice as warm as 40

Question 55

Ratio scales

Answer 55

• absolute zero point
  > e.g. physical distance
• allow additivity as well as multiplicity
  > e.g. right to interpret 8km as twice as far as 40km
• there are no ratio scales in psycology
  → this is because e.g. when measuring reaction time, it’s impossible for the individual to have 0 reaction time. Therefore, we must distinguish the instrument’s vs the people’s absolute zero