Chapter 5- Identifying good measurement

Question 1

Conceptual definition

Answer 1

The researcher’s definition of the variable in question at a theoretical level.

Question 2

Operational definition

Answer 2

A researcher’s decision about how to measure or manipulate the conceptual variable

Question 3

How are conceptual variables operationalized?

Answer 3

Researchers start by stating a definition of their construct (the conceptual variable) and then create an operational definition. Ex- measuring gratitude toward a partner by asking people how often they thank their partner for something they did. Even a simple variable like gender needs to be operationalized.

Question 4

3 common types of variables

Answer 4

1. Self report measures
2. Observational measures
3. Physiological measures

Question 5

Self report measures

Answer 5

Operationalize a variable by recording people’s answers to questions about themselves in a questionnaire or interview. Diener’s five-item scale is an example, as is asking someone to report their gender identity. For children, self reports can be replaced with parent or teacher reports

Question 6

Observational measures

Answer 6

Operationalize a variable by recording observable behaviors or physical traces of behaviors. For example, operationalizing happiness by observing how many times a person smiles. Intelligence tests are also observational measures, since an individual’s intelligent behaviors are being observed

Question 7

Physiological measures

Answer 7

Operationalizes a variable by recording biological data, such as brain activity, hormone levels, or heart rate. Usually requires equipment to amplify, record, or analyze. One way to operationalize stress could be to measure the amount of cortisol released in the saliva (stress hormone).

Question 8

Which operationalization is best?

Answer 8

One construct can be operationalized many different ways. Physiological measures aren’t necessarily the most accurate, and they must also be validated by other measures. For example, fMRI tests can be used to learn that the brain works more efficiently relative to level of intelligence. However, in this case participant intelligence was determined prior to the scans using an IQ test- an observational measure.

Question 9

How many levels must each variable have?

Answer 9

All variables must have at least 2 levels, but the levels of operational variables can be coded using different scales of measurement.

Question 10

Categorical/nominal variables

Answer 10

Variables that are categories- sex, species, and others. The researcher might assign numbers to each category, but the numbers don’t have numerical meaning or quantify the difference between categories.

Question 11

Quantitative/continuous variables

Answer 11

Variables that are coded with meaningful numbers, like height, weight, level of brain activity, or scales that produce quantitative scores (Diener’s scale of well being).

Question 12

3 types of quantitative variables

Answer 12

1. Ordinal scale
2. Interval scale
3. Ratio scale

Question 13

Ordinal scale

Answer 13

Applies when the numbers of a quantitative variable represent a ranked order- these rankings could be unequal. For example, a bookstore might rank their top 10 best selling books, but we don’t know how many more copies of book 1 were sold than book 2.

Question 14

Interval scale

Answer 14

Applies to the numerals of a quantitative variable that represent equal intervals (distances) between levels. Also, numerals must not have a “true zero”. For example, the distance between each degree on the celsius scale is equal. There is also no true zero, because 0 degrees celsius (freezing point) does not mean that something has “no temperature”.

Question 15

Ratio scale

Answer 15

Applies when the numerals of a quantitative variable have equal intervals, and a value of zero actually means “none”. For example, a score of zero on a knowledge test when measuring how many questions people get right does actually mean zero- the individual got 0 questions correct.

Question 16

Reliability

Answer 16

Refers to how consistent the results of a measure are

Question 17

Validity

Answer 17

Refers to whether the operationalization is measuring what it’s supposed to measure

Question 18

3 types of reliability

Answer 18

1. Test-retest reliability
2. Interrater reliability
3. Internal reliability

Question 19

Test-retest reliability

Answer 19

A study participant will get pretty much the same score each time they are measured with it. Applies whether the operationalization is self report, observational, or physiological, but it’s most relevant when researchers are measuring constructs. If participants take an IQ test one day and then take it again a month later, the pattern of scores should be consistent.

Question 20

Interrater reliability

Answer 20

Consistent scores are obtained no matter who measures the variable.

Question 21

Internal reliability

Answer 21

A study participant gives a consistent pattern of answers, no matter how the researchers phrase the question. People who agree with the first question on the well being scale should also agree with the next few questions

Question 22

Statistical devices that researchers can use for data analysis (2)

Answer 22

Scatterplots and the correlation coefficient

Question 23

How can scatterplots indicate interrater reliability?

Answer 23

Scatterplots can show interrater agreement or disagreement. Example- two observers rate how happy children seem while playing. With high interrater reliability, each observer would give each child a similar ranking, and the data points would be clustered close to the line of best fit on a scatter plot. With low interrater reliability, rankings will differ more significantly, and the points are not clustered close to the line.

Question 24

What could cause low interrater reliability?

Answer 24

This could be due to the observers not having a clear enough operational definition of happiness. Also, the coders might not have been trained well enough yet.

Question 25

Correlation coefficient (R)

Answer 25

A single number that indicates how close the dots are to the line on the scatterplot. The R value can be positive or negative, which indicates the slope direction. The R value is always between -1 and 1. A strong relationship means the R value is close to -1 or 1. If there is no relationship, r will be .00 or close to it

Question 26

Slope direction

Answer 26

The direction of the slope of the line of best fit. It can be positive, negative, or zero

Question 27

Strength of the relationship

Answer 27

The relationship between variables is considered to be strong when the dots in a scatterplot are close to the line

Question 28

How is test-retest reliability assessed using r?

Answer 28

To assess this, we would assess the same set of participants on that measure at least twice. We would record each person’s score at time 1 and time 2 (around 2 months apart) and calculate R. If R is positive and strong (.5 or higher) the test-retest reliability is good. If positive but weak, we know that the scores have changed.

Question 29

When would a low r value indicate poor test-retest reliability?

Answer 29

A low R is a sign of poor reliability if we are measuring something that should stay the same over time. If measuring IQ, it should stay the same over the span of two months. If measuring something like seasonal stress, R will be low because this is a construct that changes over time

Question 30

How is interrater reliability assessed using r?

Answer 30

To test this, we would ask two observers to rate the same participants at the same time, and then compute R. If R is positive and strong (.70 or higher), then reliability is good. If positive and weak, reliability is low. A negative correlation is rare but would indicate a problem with the observers

Question 31

R is best used to evaluate interrater reliability when observers are rating which type of variable?

Answer 31

R can be used to evaluate interrater reliability when the observers are rating a quantitative variable. A statistic called kappa is more appropriate when observers are rating a categorical variable. A kappa close to 1 means that the raters agree.

Question 32

When is internal reliability relevant?

Answer 32

Internal reliability is relevant for measures that use multiple items or observations to get at the same construct. A scale with 5 items that say roughly the same things worded differently should mean that a participant should answer all items consistently

Question 33

How are responses quantified to determine internal reliability?

Answer 33

Researchers ask the participants to answer all of the items. Then, they compute the correlations between every item and every other item. They compute the average inter-item correlation (AIC)- the average of all of these correlations. AIC from .15-.50 means that the items go well together. They compute Cronbach’s alpha- mathematically combines the AIC and the number of items in the scale. The closer it is to 1, the better the scale’s reliability

Question 34

Construct validity

Answer 34

How well a measure measures the conceptual variables it was intended for

Question 35

How are validity and reliability different?

Answer 35

Validity and reliability are separate concepts. For example, an adult’s scale might say they weigh 50 pounds every time they step on it. It’s reliable (consistent), but not valid (the measurement isn’t accurate). Reliability is necessary for validity- a measure can be less valid than it is reliable, but it can’t be more valid than it is reliable. If a measure doesn’t correlate with itself, then how can it be more strongly associated with some other variable?

Question 36

2 subjective ways to assess validity

Answer 36

Face and content validity

Question 37

3 empirical ways to assess validity

Answer 37

Criterion, convergent, and discriminant validity

Question 38

How do we measure validity of abstract concepts?

Answer 38

Abstract concepts would include happiness, intelligence, stress, and self-esteem. There is no way of directly measuring how happy someone is, although we can estimate it in multiple ways. We can know if operationalizations are measuring our construct by collecting a variety of data and evaluating it in light of our theory about the construct

Question 39

Face validity

Answer 39

A measure has face validity if it is subjectively considered to be a plausible operationalization of the conceptual variable in question. Measures with this validity align well with the conceptual definition. Example- head circumference would have a high face validity for hat size but low face validity for intelligence

Question 40

Content validity

Answer 40

A measure must capture all parts of a defined construct. Ex- a conceptual definition of intelligence could be the ability to “reason, plan, solve problems, think abstractly, comprehend complex ideas, learn quickly, and learn from experience”. To have adequate content validity, an operationalization of intelligence should include questions or items to assess each of these 7 components.

Question 41

Criterion validity

Answer 41

Evaluates whether the measure under consideration is associated with a concrete behavioral outcome that it should be associated with, according to the conceptual definition. Criterion validity is important for self report measures because the correlation can indicate how well people’s self reports predict their actual behavior.

Question 42

Types of evidence for criterion validity (2)

Answer 42

1. Correlational evidence for criterion validity | 2. Known-groups evidence for criterion validity

Question 43

Correlational evidence for criterion validity

Answer 43

For example, a sales company is choosing between aptitude test A and aptitude test B - they have face and content validity, but do they correlate with the key behavior- work success? The company can collect data to tell them how well aptitude tests are correlated with success with sales. Both sales tests are given to all current sales representatives and then their number of sales is determined- two scatter plots are made to determine the correlation between aptitude test A and sales and aptitude test B and sales. Aptitude test A has a stronger correlation- we can conclude that test A has better criterion ability as a measure of selling ability

Question 44

Known groups paradigm

Answer 44

Another way to gather evidence for criterion validity in which researchers see whether scores on the measure can discriminate among two or more groups whose behavior is already confirmed. For example, to validate salivary cortisol as a measure of stress, a researcher could compare the salivary cortisol levels in two groups of people- those who are about to give a speech in front of a classroom and those who are in the audience. If salivary cortisol is a valid measure of stress, people in the stress group (public speaking) should have higher cortisol levels than those in the audience.

Question 45

How is the known groups method used to validate self report measures?

Answer 45

An example is the Beck Depression Inventory. This is a 21 item self report scale where participants circle one of 4 choices. The scores are added to get a total from 0-63. Participants answered the inventory. Then, psychiatrists conducted clinical interviews to diagnose each person with depression (if they were depressed), as well as their level of depression. The average BDI score of the known group of depressed people was higher than the average score of the known people who were not depressed. The level of the BDI inventory also correlated with the level of depression

Question 46

How was convergent validity determined for the BDI?

Answer 46

If the BDI really quantifies depression, it should be correlated with other self report measures of depression. A strong positive correlation between the 2 scores provides evidence for the convergent validity of the BDI. Convergent validity evidence also includes similar constructs, not just the same one. BDI scores were also strongly correlated with a score quantifying psychological well being. The strong negative correlation makes sense because people who are depressed are also expected to have lower levels of well being

Question 47

How was discriminant validity determined for the BDI?

Answer 47

The BDI should not correlate strongly with measures of constructs that are very different from depression- it should show discriminant validity with them. We would not expect the BDI to be strongly correlated with a measure of perceived physical health problems, for example. We would expect the BDI to be much more strongly correlated with similar constructs than with constructs that aren’t similar. Example- many developmental disorders have similar symptoms. We wouldn’t want a screening instrument to diagnose a child with autism when they actually have a speech delay. It’s not necessary to establish discriminant validity with random other variables- we want to focus on other variables that are “near neighbors” of the one being evaluated.

Question 48

Convergent and discriminant validity

Answer 48

Convergent validity and discriminant validity- the patterns of correlations with measures of theoretically similar and dissimilar constructs. Convergent and discriminant validity are usually evaluated together, as a pattern of correlations among self report measures. A measurement should have higher correlations (higher r values) with similar traits (convergent validity) than it does with dissimilar traits (discriminant validity).