Chapter 5: Identifying good measurement

Question 1

All statistical analysis can be divided into two branches or categories:

Answer 1

Descriptive or inferential

Question 2

Descriptive

Answer 2

Literally just describing the data in your sample,
with no attempted conclusions about the
entire population. Means, medians, standard
deviations, etc., and some other stuff like effect sizes.

Question 3

Inferential

Answer 3

Using your data as the basis to make inferences about
how things work in the overall population (i.e.,
beyond just your data). Assessing statistical
significance.

Question 4

The overall goal of descriptive statistics is

Answer 4

to summarize data in a way that is efficient and accurate.

Question 5

Measures of central tendency tends to

Answer 5

One aspect of that is providing single a value that represents the many numbers that make up a data set

Question 6

The most common measures of central tendency are

Answer 6

mean, median, mode

Question 7

variability

Answer 7

How dispersed or spread out the values in a data set are. Examples of such measures include standard deviation, variance, and range.

Question 8

To do a good job describing data

Answer 8

you need to provide information on central tendency AND variability. Neither one alone provides enough information.

Question 9

Bar graph

Answer 9

Top of bar=central tendency, error=measure of variability

Question 10

Z scores

Answer 10

(x-μ)/σ
x= Individual’s score
μ=Mean score from sample
σ=Standard deviation of the sample’s scores.
Result of standardizing a score (value) with reference to the rest of the scores in the sample (a course section, in this case). Thus, it represents a score’s position relative to the mean in units of standard deviation. Where the score falls relative to the mean.

Question 11

The correlation coefficient (r-value) is usually considered

Answer 11

a descriptive statistic, not surprising given its description of an association.

Question 12

This inference relates to whether the conclusions drawn from the sample data can be applied to the population. But how?

Answer 12

A statistical test e.g., a t-test
A test statistic e.g., t
A p-value
Is it statistically significant?

Question 13

null hypothesis testing

Answer 13

It relates explicitly to inferential statistics and is a bit different from the theory-data cycle’s concept of a hypothesis.

Question 14

Null hypothesis

Answer 14

is always that the result is NOT statistically significant,

and statistical significance tells the researcher to reject the null hypothesis in favour of the alternative hypothesis.

Question 15

Rejecting the null hypothesis

Answer 15

The result is statistically significant

Question 16

Retaining the null hypothesis

Answer 16

A result of this magnitude is not statistically significant+

Question 17

As mentioned before, the threshold for significance (or alpha, α) is usually

Answer 17

p < 0.05, meaning that the likelihood that the observed results are due to random chance and not “real” (applicable to the whole population) is < 5%.

Question 18

t-value is to t-test as

Answer 18

r-value is to correlation

Question 19

t-test

Answer 19

common whenever you are comparing the mean and variability of two groups. The result is again a p-value

Question 20

an analysis of variance (ANOVA)

Answer 20

a statistical test that works like a t-test but compares more than two groups. After they show that overall difference, they compare between each pair
of groups (usually called pairwise or post-hoc comparisons or tests).

Question 21

Alternative hypothesis

Answer 21

The result is statistically significant.

Question 22

Overall, there are tons of different statistical tests that can be use depending on the situation (and that you don’t have to know), but they all lead to

Answer 22

a p-value that can be used to assess statistical significance

Question 23

There are four possible outcomes of the act of making an inference about statistical significance

Answer 23

Two correct conclusions and two types of errors.
Error 1: Reject the null hypothesis (conclude there is an effect) when there is no effect.(“false positive”)
Error 2: Retain the null hypothesis (conclude there is not enough evidence for an effect) when there is one. (“miss”)

Question 24

Type I error can be avoided by

Answer 24

lowering the alpha level (e.g. α = 0.01), making the threshold for significance harder to reach.

Question 25

Conversely, type II error can be avoided by

Answer 25

raising α (e.g., α = 0.10).

Question 26

Decisions about α depend on

Answer 26

the research topic, hypothesis, statistical analysis, etc. E.g., a researcher might use a higher alpha when expecting a new medicine to have a small but important effect.

Question 27

What’s great about p-values?

Answer 27

They can be applied to any type of statistical test,

and p < 0.05 always means the same thing!

Question 28

Weakness of p-value and inferential tests

Answer 28

The main potential weakness is that statistical significance (p-values) depend heavily on sample size. The relationship shown above may be statistically significant or not depending on sample size. In theory, with large enough sample size, any relationship could be statistically significant. the descriptive statistics stay the same, but p-value changes dramatically with sample size. With very large sample sizes, it’s increasingly likely that you will detect a difference or effect where there really isn’t one (Type I error). In the same spirit, a study with a small sample size might “miss” an effect that really exists (Type II error).

Question 29

The solution is to not base your conclusions only on statistical significance, but also look back to and use another type of measure within descriptive statistics:

Answer 29

Effect size: a quantitative measure of the size
or magnitude of a relationship or effect.Like other descriptive statistics (e.g., central tendency), it can be measured in several different ways. Unlike them, it describes not just some sample of data but some relationship within the data.

Question 30

Cohen’s d

Answer 30

d = M1 – M2 / SD (spooled)

In other words, Difference between two sets
of data’s means divided by The overall
variability in the data.
Can be used to determine which sample size is necessary.

Most common measure of effect size. One strength of this and other effect sizes is that it’s not
dramatically influenced by sample size.

Question 31

Increasingly in recent years, papers usually include both

Answer 31

inferential statistics (i.e., p-value) and a measure of effect size.

Question 32

Effect sizes typically have associated guidelines for use in interpreting the data:

Answer 32

Cohen’s d goes up from 0 and can go beyond 1, with the value indicating the strength or magnitude of the relationship
0.2=weak
0.5=moderate
0.8=strong
This means that if two groups’ means don’t differ by 0.2 standard deviations or more, the difference is trivial, even if it is statistically significant.

Question 33

There are other measures of effect size, including

Answer 33

including eta squared (ηp2) and others. Note the inclusion of r, which highlights that looking at effect size is not so much adding a wild new way of assessing things, more so just putting stock in the descriptive statistics themselves and not focusing entirely on inferential.

Question 34

In terms of statistical validity, the results are statistically significant, but the effect sizes are tiny. What’s the story?

Answer 34

Big sample size

Question 35

construct validity

Answer 35

“How well did they operationalize the variable?”

AKA how did they take a construct, with its conceptual definition, and manipulate or measure it in a specific way?

Question 36

Operationalizing variables

Answer 36

1. Self-report measures
2. Observational measures
3. Physiological measures

Question 37

Categorical (or nominal) variables are used when

Answer 37

the different levels of a variable are categories (e.g., gender, presence/absence of disease, etc.). In keeping with “nominal”, the numbers signify categories and don’t mean anything in a quantitative sense. An independent variable manipulated in an experiment is likely to be a categorical variable.

Question 38

One type of quantitative variable is an ordinal scale

Answer 38

where the numbers represent an ranked order

Question 39

Another quantitative variable is interval scale

Answer 39

where the numbers represent intervals between levels but there is no “true zero”. Temperature and angle are simple examples: numbers can be related to each
other (e.g., 45° vs 90°) but zero doesn’t mean a complete absence. Zero degrees Celsius is not “no temperature.”

Question 40

Lastly, ratio scale works much like interval scale with the difference that

Answer 40

zero really means “nothing”. For example, in this memory task where you remember and repeat back sequences, getting a zero really means you remembered zero things. For example, in this memory task where you remember and repeat back sequences, getting a zero really means you remembered zero things.

Question 41

In terms of assessing construct validity for a

variable, there are two main aspects:

Answer 41

Reliability: How consistent are the results?
Validity: Is it measuring what it’s supposed to measure?
Where’s “valid but not reliable”? No such thing…reliability is a prerequisite for validity.

Question 42

Test-retest reliability

Answer 42

For constructs that you expect to be stable (e.g., IQ), are the scores for an individual consistent across time (aka from test to retest)? If not, how could it be a reliable measure? Information related to the reliabilities can be found in either the
Methods or Results section, depending on the paper. r = 0.50 or so is good test-retest reliability…The STAI is a good example as it measures both trait anxiety (consistent
over time) and state anxiety (may vary over time) via separate questions. As one would expect, test-retest reliability is high for one but not the other. Relevance depends on whether or not it’s a construct that you would expect to be stable over time.

Question 43

Test-retest reliability is usually seen

Answer 43

with questionnaires, but can apply to
behavioural measures as well. For example, the above disease rating scale has test-retest reliability > 0.90 (over a finite time period, at least).

Question 44

Interrater reliability

Answer 44

How similar are the scores or measures of two or more different observers? E.g. smiles as a measure of happiness Usually Measured via the correlation between the scores from one observe with those of another, often via r-value. Here, a good rule of thumb is r > 0.70 or so. Sometimes, researchers use the intraclass correlation coefficient (ICC), which is a bit more complicated than r-values but works similarly… for example, it still runs from 0 to 1. Relevance depends on whether there is a subjective element to the operationalization (e.g., self-report vs behavioural).

Question 45

For categorical variables, Cohen’s kappa (κ, another fancy type of correlation) is used

Answer 45

to measure interrater reliability. It works like r, but statistically factors in the possibility of raters agreeing by chance.

Question 46

Internal reliability (or “internal consistency”)

Answer 46

Do the various aspects of an operationalization meant to measure a
certain construct change in association with each other? Put another way, subjects’ answers should generally “point in the same direction.” Measured via Cronbach’s alpha (α) (“Coefficient alpha” above), another
fancy sort of correlation; it measures the covariance among the items. Again, >0.70 is a general rule of thumb. Needs to be addressed with most every scale/questionnaire, since they typically feature multiple items. As with test-retest, internal reliability is mostly seen with questionnaires but may also be used with other types of measures. Relevance depends on whether the operationalization
features multiple components or not.

Question 47

Assessing the validity of a measure

Answer 47

does it measure what it is supposed to

measure? – involves looking at five different components.

Question 48

Two of them are subjective in nature

Answer 48

face and content validity. Both of these validities bring to mind the conceptual definition, and whether the operationalization reflects it

Question 49

Face validity

Answer 49

Does it look like it’s a good operationalization (like it measures what it should)? Since this is a judgement call, researchers might consult experts in the field.

Question 50

Content validity

Answer 50

Does it cover all parts or aspects of the construct or variable? Another judgement call for which we may get expert help.

Question 51

Assessment of content validity depends on

Answer 51

your perspective / definition.

Question 52

construct validity

Answer 52

1) Criterion validity
2) Convergent
3) Discriminant

Question 53

criterion validity

Answer 53

Does the measure correlate with some
“real world” or behavioural outcome associated with the conceptual definition? If your measure is a test of sales aptitude, a sensible criterion validity would be actual sales performance. If these don’t match up, there’s probably a problem with your
measure.

Question 54

One special approach to assessing criterion

validity is the known-groups paradigm

Answer 54

whether your measure differs between
groups that are otherwise established to be
different in some regard. A depression scale with criterion validity should generate higher scores for those diagnosed with depression (a “known group”) than for others. This approach works the same regardless of how the groups have
been previously divided up.

Question 55

Criterion validity could be assessed with

Answer 55

both or either approach, depending on
the variable and what is available for
comparison.

Question 56

Convergent (sometimes called concurrent) validity

Answer 56

Does the measure correlate with an
established measure of the same
(or very similar/related) construct? Does it correlate with something that you would expect it to correlate with? E.g., I just designed a new depression scale… does it produce results similar to that famous depression scale that everyone uses? Note that we are looking for a correlation, but not absolutely necessarily a positive one! One important part is that the scale being used for comparison should itself have been rigorously assessed for validity in the past

Question 57

Discriminant (sometimes called divergent) validity:

Answer 57

Does the measure NOT correlate with
measures of constructs that are
different/unrelated? Evidence in the form of a lack of correlation with very different/unrelated constructs. A strong association would be a red flag