Measurement issues

Question 1

Conceptualization

Answer 1

The process of specifying what we mean by a term.

• Working definition in deductive research
• The dev. of a new concept - used to make sense of related observations.

Question 2

Operationalization

Answer 2

The process of connecting concepts to observations.

• Goal is to devise operations that measure the concepts we intend to measure - achieve validity.
• What and how to measure.
• Univariate operational def - one single test.
• Multivariate def - two or more indicators are used - the common variance !

Question 3

Development of a scale

Answer 3

Phase 1 - Initial Pool of Items.
Phase 2 - Refine the item pool.
Phase 3 - Psychometric properties.

Question 4

Phase 1 - Initial pool of items

Answer 4

• Review literature
• In depth interview
• Focus groups
• Content validity
• Cognitive test

Question 5

Phase 2 - Refine pool of items

Answer 5

Exploratory MIRT.

Question 6

Phase 3 - Psychometric properties

Answer 6

• Confirmatory MIRT
• Internal consistency reliability
• Discriminant validity

Question 7

Where do you look for scales?

Answer 7

• Literature
• HealthMeasures
• PhenoX Toolkit

Question 8

Consideration selecting instruments:

Answer 8

• Cost of a test
• Permission and responsibility for using instruments
• Time and length
• Psychometric issues are essential

Question 9

Reliability

Answer 9

• A prerequisite for validity
• Infers “low error”, hence the observed score will be closer to the true score.
• A valid test is a reliable test.
• A reliable test is not necessary a valid test.
• A reliable measure is optimally free from random error.
• A valid measure is optimally free from random error and systematic error.

Question 10

Forms of Reliability:

Answer 10

• Internal Consistency (Calculate correlation of each item with every other item on the test (NOT total correlations). Intercorrelations depend only on the true components. Hence reliability can be deduced from intercorrelations. Resulting measures is called Cronbach’s Alpha. Use Kuder-Richardson (K-R-20) for dichotomous items. Calculated by SPSS.
• Split-half reliability (test divided, often odd vs even, separately administered)
• Test-retest reliability - temporal stability
• Inter-rater reliability

Question 11

Cronbach’s Alpha ( Cronbach, 1951)

Answer 11

• The square root of Cronbachs Alpha is an estimate of the correlation between observed and true scores.
• CA is the proportion of the variance of the scale that can be attributed to a common source.
• CA is the average of all possible split-half reliabilities from the set of items.

Question 12

What does Cronbach’s Alpha show?

Answer 12

• “Internal consistency”: refers to the interrelatedness of a set of items, whereas “homogeneity” refers to the unidimensionality of the set of items.

Question 13

Generalized Spearman-Brown Formula:

Answer 13

• Measures reliability, indicating increase in reliability with increase in test length.

Question 14

Assumption of Cronbach’s Alpha:

Answer 14

1 - The scale adheres to “tau” equivalence.
2 - Scale items are on a continuous scale and normally distributed.
3 - The errors of the items do not covary.
4 - The scale is unidimensional.

Question 15

“Tau Equivalence”

Answer 15

• is the statistically precise way to state that each item on a scale contributes equally to the total scale score.
• the standardized factor loading’s for each item would need to be nearly identical to all other items on the scale.

Question 16

Item Response Theory:

Answer 16

• An alternative to classic test theory
• IRT refers to a set of mathematical models that describe, in probabilistic terms, the relationship between a person’s response to a survey question/test item and his or her level of the latent variable(hypothetical construct, cant be measured by a single variable/item) being measured by the scale.
• IRT provides a clear statement (picture) of the performance of each item in the scale/test and how the scale/test functions, overall, for measuring the construct of interest in the study population.

Question 17

Validity:

Answer 17

• A measure is valid if it does what it claims to do.
• Not as simple as reporting one number as with reliability.
• Instead we must consider:
  1 - Face Validity
  2 - Concurrent Validity
  3 - Predictive Validity
  4 - Content Validity
  5 - Construct Calidity

Question 18

Face Validity:

Answer 18

• Refers to the extent to which a measure “appears” to measure what it is supposed to measure.
• Not statistical - involves the judgement of researcher (and participant).
• Measure has face validity of people think it does.
• Just because it has face validity, does not mean it is a valid measure.

Question 19

Two types of Criterion-related validity:

Answer 19

1 - Concurrent validity - Measure and criterion are assessed at the same time.
2 - Predictive validity - elapsed time between the administration of the measure to be validated and the criterion is a relative long period (months, years).
- Predictive validity refers to a measure’s ability to distinguish participants on a relevant behavioral criterion at some point in the future.

Question 20

Construct Validity:

Answer 20

• Refers to the question of whether a test measures what (an unobservable construct) it purports to measure.
• Is the process testing these hypotheses through the use of empirical data.
• Involves accumulation of evidence about a network of hypothesized relationships stemming from a construct, which is measured by a test.
• To have construct validity a measure should both:
  1) Correlate with other measures that it should be related to (convergent validity).
  2) And, NOT correlate with measures that it should not correlate with (discriminant validity).

Question 21

Convergent Validity:

Answer 21

The alternative measures of the same construct should be highly intercorrelated.

Question 22

Discriminant Validity:

Answer 22

The measures of different constructs should be, at most, moderately intercorrelated.

Question 23

Multitrait-Multimethod matrix (MTMM):

Answer 23

At least two constructs using at least two methods.

Question 24

Sensitivity:

Answer 24

• The sensitivity of a test in the ability of the test to identify correctly affected individuals.
• Proportion of individuals testing positive among affected individuals.

Question 25

Specificity:

Answer 25

• The specificity of a test in the ability of the test to identify correctly non-affected individuals.
• Proportion of individuals testing negative among non-affected individuals.

Question 26

Predictive value of a positive test:

Answer 26

• Is the probability that an individuals testing positive is truly affected.
• Proportion of affected people among thoose testing positive.

Question 27

Predictive value of a negative test:

Answer 27

• Is the probability that an individual testing negative truly is non-affected.
• Proportion of non-affected people among those testing negative.

Question 28

The likelihood ratio:

Answer 28

• Combines information from sensitivity and specificity, gives an indication of how much the odds of disease change based on a positive or negative result.

Question 29

ROC - Curves:

Answer 29

• Roc curve is a graphical representation of the trade off between the false negative and false positive rates for every possible cut off.
• The ROC curve is the representation of the trade-offs between sensitivity (Sn) and specificity (Sp).
• By tradition, the plot shows 1-Sp on the X axis and Sn on the Y axis.