Scale development and psychometric properties

Question 1

What types of variables to we measure in psychological research?

Answer 1

Behavior: observable and measurable, objective
Construct: concept of interest, can measure related variables, subjective

Question 2

Can we trust data from an inconsistent measure?

Answer 2

No

Question 3

Pearson’s r

Answer 3

Correlation coefficient: Used to observe reliability of a measure. Should be .8+ (.9 is gold standard, which indicates 10% error)

Question 4

test-retest

Answer 4

A type of reliability test in which you have the same person complete the same measure twice and then use r to evaluate the relationship. The higher the coefficient, the more reliable the measure.

Question 5

Alternate forms

Answer 5

A type of reliability test in which you create two versions of same measure, and administer each version to same participants, then use r to evaluate the relationship

Question 6

Split-half

Answer 6

A type of reliability test in which you randomly separate one measure into two parts- administer both parts to same participants and then use r to evaluate the relationship

Question 7

Cronbach’s alpha

Answer 7

Evaluates internal consistency of a measure

• calculates all possible split-half configurations
• .7 acceptable, .9 gold standard
• scores range from 0-1, like r

Question 8

Factors that affect reliability

Answer 8

test length
homogeneity of items
test-retest interval
variability of scores
variation in test situation
sample size

Question 9

interrater reliability

Answer 9

Used to measure the evaluation of a subjective measure. Two independent raters use a measure for the same observation, then evaluate consistency with r
Especially common in SCD and academic intervention research

Question 10

reliability=

Answer 10

consistency

Question 11

What are the two issues in validity?

Answer 11

What a test measures, and how well it measures it

Question 12

Is validity all or nothing?

Answer 12

No, there are degrees of validity and it requires ongoing consideration because what we know about a construct changes and “normal” functioning changes

Question 13

How does Pearson’s r relate to validity?

Answer 13

We can use r to evaluate how valid a measure is compared to other measures.

Question 14

Convergent construct validity

Answer 14

Compares measure to other similar measures

Question 15

Discriminant Construct validity

Answer 15

does not correlate with a measure of logically unrelated construct

Question 16

Predictive validity

Answer 16

The measure correlates with related future outcomes (Ex: GRE- compare GRE score to measure of grad school success)

Question 17

Factors that affect validity

Answer 17

reliability
range of attributes being measured
length of interval between administration of test and criterion measure
range of variability in criterion measure
individual validity

Question 18

Why is it important to be critical when analyzing validity of measures?

Answer 18

Data you collect are interpreted to make high stakes decisions for individuals you serve. The data are only as good as the measures used to collect them.

Question 19

What is the difference between reliability and validity?

Answer 19

Consistency v. measuring what you intend to measure

Question 20

Considerations of survey development

Answer 20

the effects of how you word questions
the order of your questions
question format: qualitative or quantitative

Question 21

Guidelines for survey questions

Answer 21

• each question should be short and clearly worded
• as few questions as possible
• control questions
• keep it simple, specific, exhaustive, individual (make sure questions ask only one thing), optional (allow skips), neutral and balanced (response options should be weighted around neutrality if using a scale)

Question 22

What types of survey response types should you use?

Answer 22

Stay away from open response; use forced choice as multiple choice or rating items

Question 23

Considerations for rating scales

Answer 23

• consider validity of items
• items should be short and clearly worded
• some control items
• no open response
• scale should be the same for all items (Likert type is most common, can also be descriptive items across a similar scale, like the BDI)

Question 24

What is the research finding connecting autism with vaccines an example of?

Answer 24

Type I error

Question 25

What is the difference between reliability and validity?

Answer 25

Validity indicates what a measure measures and how ell it measure it. Reliability indicates the consistency of a measure.

Question 26

What is an acceptable coefficient when evaluating the psychometric properties of a measure?

Answer 26

.8