Research Methods/Psychometrics

Question 1

ii. Nature of Reliability
1. Consistency in measurement

2. As reliability increases:
   a. Error decreases
   b. The number of mistakes we make when measuring variables goes down
   c. Measurement precision increases
3. Reliability and validity
   a. Reliability is the upper bound of validity
   i. Reliability is the variable correlated perfectly with itself, and you can’t correlate something else more than perfectly
   b. Reliability of a heterogeneous sample should be higher than a homogenous sample
   i. Homogenous sample is likely to have a range restriction issue
   ii. Heterogeneous sample will have more variance to play around with (will result in a more reliable measure)
4. Sources of error inform reliability
   a. Blue stones (in curling) are reliable because they cluster together, but they are not valid because the bulls eye would be validity

Question 2

Sources of Error
In CTT: Error is assumed random variance only

2. Sources of random variance

a. Random variance due to the passage of time
i. Test-retest reliability
1. To what extent is this variance ok – how long do you wait to test and retest the theory?
2. A stable attribute – wait longer before retest
3. Attributes that change frequently – test sooner
4. Is the reliability coefficient appropriate for what I am trying to index?

b. Random variance due to different forms
i. Parallel forms reliability
1. Random variance because they are different forms
2. Will be an underestimate if not truly parallel

c. Random variance due to different raters
i. Interrater reliability
1. Interrater Agreement- Raters agree perfectly
2. Interrater Consistency- Consistency in the rank ordering
3. People want agreement, but they accept consistency
4. If the raters don’t agree, why? Are they looking at performance differently?

d. Random variance due to different items
i. Internal consistency
1. Are my views changing, or am I just being inconsistent with errors in my judgment?
ii. Split-half reliability
1. Correlation of two halves of a test
2. Halves should be parallel – wouldn’t be parallel if the first half of the test was about one sub-dimension and the second half was about a different on

Question 3

Definition of Validity—notion of the accuracy of the conclusions we draw about…. either from measurement, selection system, training, manipulation…. the accuracy of the inferences we draw based on the assumed level of the construct, however we change that level

Ex. Manipulation in lab (control vs. experiment), the accuracy of inferences we draw on construct standing on self report measurement, performance as rated by supervisor, level of cultural awareness we increased by having ppl go through this training

Question 4

Validity- What is Validity?

Evaluation of how sure your inferences from test scores
Evaluation of inferences from test or reasoning
Existence of construct- changes in construct causes change in measurement
Extent of match between scale to construct
In psychology, you don’t know if it’s the construct of the measurement that changed
 Progression
Purports to measure: Cronbach-we have the scale and construct how do we measure?
Trinitarian view
Unitarian view Guion (2002) Messick (1995) Is this scale construct valid?
Purport to measure Borsboom (2004)
Popular Views: Trinitarian view, Validity arguments- refers to a set of logical proposition that we would expect to see if the scores on our test is actually indicative of our construct that is testable and falsifiable
 Importance of Validity
Assumptions about human behavior from the measurements we use
Not just about testing!
- Important for all methods of indexing constructs (no matter what kind of scale you use it has to valid)
- Important for manipulating construct (the manipulating of construct has to be construct valid)
We have to make sure we are actually measuring/manipulating what we think we are
A collection of items a valid scale does not make! (Can’t just slap together a set of items and use it as a measurement)
 Validity Ingredients
- Theory of the construct (Starting point of everything-construct definition)
What do you expect the construct to do? Eg. (as my tolerance score goes up so does my tolerance for failure)
- Internal Structure (two factor or one factor scale)
Eg. Tolerance of failure Achieving a goal or standard
- Content of the Domain
Domain Sampling Theory
Infinite observations, items, content areas, etc.
- Variations in the Construct Causes Variation in the Measurement
Experiment is the strongest test of validity of your scale (Guion)
Weakest MTMM, nomological network correlations of constructs (correlation is not causation)
- Validity Argument (3 quarter theory 1 quarter data)
Stems from theory of construct
- Latent Variable Model (Borsboom)
If the construct exists it should cause variations in the measurement and you can use the model to test it