Validity and intercoder reliability (3.3) + (4.1)

Question 1

Validity and Reliability

Answer 1

•Variables need to be operationalized (their metrics need to be
defined)
•Are they valid?
- Do they measure what they are supposed to measure?
•Are they reliable?
-Would you get the same value if you remeasured the same thing?

Question 2

Types of validity

Answer 2

• Face validity
• Content Validity
• Construct Validity
• External Validity (Ecological Validity)
• Internal validity

Question 3

Face Validity

Answer 3

Does the operationalization superficially seem like it measures the construct of interest?

Question 4

Content Validity

Answer 4

Does your operationalization measure all

aspects/dimensions/contents of a construct?

Question 5

Construct Validity

Answer 5

The extent to which your operationalization is actually measuring the
construct you claim it is measuring.

Homogeneity – does it measure / manipulate only one construct?

Convergent – Does your operational definition correlate strongly with
other measures of the same construct

Divergent – Does your operational definition correlate only weakly
with similar but distinct constructs

Predictive – does your operational definition predict behaviors
associated with the construct?

Question 6

External Validity

Answer 6

Can the conclusions of the research be generalized beyond the
context of the original research?

• Dif. participants, time, situations

Question 7

Ecological Validity

Answer 7

Is the research similar to what naturally occurs in the real world?

Question 8

Internal Validity

Answer 8

• Methodological mechanics
• Cause and effect relationships does x really cause y?
• Is the experiment free of confounds or alternative explanations?

Question 9

Methods for assessing reliability:

Answer 9

How similar are the values when you remeasure the same thing?
•Methods for assessing reliability
•Split-half reliability
•Parallel-forms reliability
•Intercoder reliability

Question 10

Intercoder reliability

Answer 10

• How reliable are your measurements?
• Resolve them using a predetermined method to deal with disagreements
• Interobserver reliability for two coders
• Propor

Question 11

Split-half reliability

Answer 11

Split the items into two subsets
•Does performance on one half correlate strongly with those from the other
half?

Question 12

Parallel Forms Reliability

Answer 12

•Create two versions of the same items but in different orders
•Does performance on version A have a strong positive correlation with that
from version B?