Exam 2 - Chapter 5 (Measurement Issues)

Question 1

True Score vs. Measurement Error

Answer 1

True Score: Someone’s real value on a give variable
* (ie: true intelligence, true reaction time, true happiness, etc)
* a true score cannot be directly measured (measurement error will always impact the score)

Measurement Error: When measuring a “true” score, something will cause deviation from the real value.

Question 2

Reliability

Answer 2

Reliability = Consistency

Reliability assessment helps us figure out if our measure is consistent/stable

A measurement that is as close to its real score as possible.

Question 3

Validity

Answer 3

Validity = Accuracy

Validity helps us figure out if we’re truly studying what we intended to study.

Question 4

Differentiating: Reliability & Validity

Reliability vs. Validity

Answer 4

A good study should be BOTH Reliable & Validity

Consistency and Accuracy are important and work together.

Question 5

Types of Reliability

Answer 5

1. Internal Consistency Reliability – Reliability of Items across people

• Item Total Correlations
• Split-Half Reliability
• Cronbach’s Alpha

2. Reliability across Time – Reliability of scales over time (or versions)

• Test-retest and alternate form

Reliability across People – Reliability of ratings across raters

• Inter-rater Agreement

Question 6

Internal Consistency Reliability: Item Total Correlation

Answer 6

Item Total Correlation: How well a specific item tracks responses to the rest of the scale.

• Useful for creating & refining questionnaires

Question 6

Internal Consistency Reliability: Split-Half Correlation:

Answer 6

Split-Half Correlation: Split data in half, and find the correlation between the halfs.

Issue: Room for malpractice because “half” can be defined differently.

• Someone could try picking different “halfs” until they get a correlation coefficient that they like

Question 6

Internal Consistency Reliability

Answer 6

How much individual items in a scale/survey relate to each other and how consistent they are in measuring the same concept or trait in their results.

The more they overlap/are related, the greater the Internal Consistency

Question 7

Internal Consistency Reliability: Cronback’s Alpha:

Answer 7

Cronback’s Alpha Statistical solution to estimating reliability across infinite number of sets (ie: the average of every possible combination of ‘halfs’)

• alpha > 0.8

Question 8

Reliability Across Time: Test-Retest Reliability

Answer 8

Test-Retest: How well does one agree with himself at multiple time points.

• Give the same test at two points in time.

Question 9

Reliability Across Time: Alternate Forms Reliability

Answer 9

Alternate Forms Reliability: Correlations between multiple measures of the same category.

*Give two different forms of the same test at two points in time. (Used when given closer together)

Question 10

Reliability Across People:
Inter-Rater Reliability

Answer 10

Inter-Rater Reliability: Degree of agreement or consistency between multiple people (raters) assessing or scoring the same thing.

• It shows how much raters produce similar results when evaluating the same subject.
• Cohen’s Kappa: To calculate the consistency use Cohen’s kappa

Question 11

Types of Validity

Answer 11

• Construct validity

What are we measuring?

• Face validity
• Content validity
• Concurrent validity

How do our measures relate to other measures?

• Convergent validity
• Divergent (i.e., discriminant) validity
• Predictive validity

Question 12

Construct Validity

Answer 12

How well an operational definition of a variable accurately reflects the variable being measured or manipulated.

• Other types of Validity fall under Construct Validity

Question 13

Face Validity

Answer 13

How well a measurement device appears to accurately measure a variable. (not that useful)

• Face Validity is not sufficient to conclude that a measure is valid.

Question 14

Content Validity

Answer 14

How well a test or survey relates to/covers all parts of the concept it aims to measure.

(pretty useful)

Question 15

Convergent Validity

Answer 15

Checks if a measure is strongly related to other measures of the same construct.

• Our measure should be related to other measures that assess a similar construct
• Asks: Is my new scale of measurement as good as the old scale that measures the same thing?

Question 15

Concurrent Validity

Answer 15

Tells us if the measurement used successfully differentiates people who are theoretically supposed to be different

• Test this by giving the measure to a different demographic and see if they score much lower than the target demographic

Question 16

Divergent/ Discriminant Validity

Answer 16

Tests whether a measure is not strongly related to different or unrelated constructs

• This confirms that the measure accurately distinguishes between distinct concepts.
• Our measure should not be related to other measures that assess different constructs.

Question 17

Predictive Validity

Answer 17

How well does our measure predict future scores on another test related to the concept it is assessing.

• Ie: a measure of depression should predict loneliness at a future time