Lesson 9 - Measurement In Evaluation

Question 1

What is the core essence of validity?

Answer 1

Accuracy

Is the scale accurate each time you step on it?

Question 2

What are the 4 major types of validity?

Answer 2

1. Face Validity
2. Content Validity
3. Criterion Validity
4. Construct Validity

Question 3

Define Face Validity

Answer 3

Extent to which the instrument APPEARS to be measuring what it is supposed to measure.
- Face validity assesses whether the test “looks valid” to the examinees who take it, the administrative personnel who decide on its use, and other technically untrained observers.

Question 4

What is one way it can be determined if the instrument has face validity?

Answer 4

Ask a colleague to look over an instrument and see if s/he agrees that it appears to measure what you intend to measure.

Question 5

What is a problem with a colleague determining face validity?

Answer 5

It is a SUBJECTIVE opinion

- Depends on who you ask and what they know.

Question 6

How does Face Validity differ from the other forms of validity?

Answer 6

Face validity lacks some form of systematic logical analysis of the content.

Question 7

Define Content Validity

Answer 7

The degree to which the items within a research instrument represent the universe of content for the concept being measured or the domain of a given behavior.

• A type of face validity
• Subjective
• Cost effective

Question 8

In what type of validity do you ask multiple ‘experts’ in the field to evaluate the instrument?

Answer 8

Content Validity

- Problem with this is that it is subjective

Question 9

Who should Content Validity be conducted by?

Answer 9

• persons with technical expertise to make sound judgements
• persons with a willingness to provide, if necessary, negative feedback
• professionals with diverse experience

Question 10

Since neither content or face validity are sufficient for establishing that a scale has ‘true’ validity, why use them?

Answer 10

• no other way

- most cost effective

Question 11

Define Criterion Validity

Answer 11

Criterion validity is the extent to which the measures are demonstrably related to concrete criteria in the “real” world.

Question 12

What are the two subtypes of criterion validity?

Answer 12

Predictive and concurrent validity

Question 13

Define Concurrent Validity

Answer 13

Concurrent validity is demonstrated where a test correlates well with a measure that has previously been validated. The two measures are taken at the same time.
(current status)

Question 14

What is the main difference b/w concurrent and predictive validity?

Answer 14

The difference between concurrent validity and predictive validity rests solely on the time at which the two measures are administered.

Question 15

Define Predictive Validity

Answer 15

Predictive validity is where one measure occurs earlier and is meant to predict some later measure.
(Future performance)

Question 16

In cases when there is not an existing criterion from which to compare, or the phenomenon that planners want to measure is more abstract than concrete, how can validity be established?

Answer 16

Using construct validity

Question 17

Define construct validity

Answer 17

Construct validity is the degree to which a measure correlates with other measures it is THEORETICALLY expected to correlate with.

Question 18

What is the core essence of reliability?

Answer 18

Consistency

The scale be be consistently wrong or right

Question 19

What are the two major approaches to reliability?

Answer 19

1. Internal consistency reliability

2. Test-retest reliability

Question 20

How is internal consistency reliability established?

Answer 20

Each item that makes up a scale is examined for how well it correlates with the scale as a whole.
- Consistency across all parts of an instrument

Question 21

How can test-retest reliability be established?

Answer 21

By measuring the same group at two different points in time times (day, year, etc)

Question 22

Although both must be considered, which is more important validity or reliability and why?

Answer 22

Validity because if an instrument doesn’t accurately measure what it is suppose to measure (validity), then reliability is an irrelevant issue. a
(Consistently supplying a wrong answer)

Question 23

Often must reliability and validity be assessed?

Answer 23

EACH time the instrument is administered to a sample.

Question 24

Why may you NOT cite past experiences or existing data on psychometric properties?

Answer 24

Cannot simply cite past experience or existing data on psychometric properties because each group will respond differently.

Question 25

This procedure in itself doesn’t give much confidence in the measure’s validity. This procedure is simply an “eyeball” test.
- What is this statement referring to?

Answer 25

Face Validity

Question 26

The measure must cover the full range of the concepts meaning. Each dimension of the concept should be represented by items in the measure.
- What are these statements describing?

Answer 26

Content Validity

Question 27

Is the measure related to those other measures that theory and prior research indicate that it should be related to?
- What is this question indicative of?

Answer 27

Construct Validity

Question 28

Is the measure related to a more direct measure of the same phenomenon?
- What is this question indicative of?

Answer 28

Criterion Validity

Question 29

What are validity and reliability characteristic of?

Answer 29

Validity and reliability are NOT a characteristic of the measuring instrument itself. They are characteristic of the scores/data produced.
(An example of this is using a ‘valid instrument’ to collect data on people too young or too old)

Question 30

What is considered “sloppy speak” according to Dr. Barry about validity and reliability?

Answer 30

Saying that this survey of valid and reliable.

Question 31

Torgen et al., set out to examine the reliability of a self-administered questionnaire about past and present physical activities both at work and during subjects’ free time. 44 respondents completed the questionnaire a first time, and then again after two weeks.
- By doing this, the researchers obtained information on what type of reliability?

Answer 31

Test-retest reliability

Question 32

A recent study by Clark et al., raised questions about the accuracy of adolescents’ self-reports of pregnancies and STIs. After checking subjects’ answers to questionnaire items against their clinical charts, the authors concluded that patients’ histories of visits for STIs and pregnancies were often not substantiated by their medical records.
- Do adolescents’ self-reports suffer from validity or reliability problem?

Answer 32

Validity

Question 33

Between 1979 and 1981, the San Antonio Heart Study collected on ethnic differences in cardiovascular risk factors. The following items were used to measure subjects’ knowledge related to prevention of heart attacks: Eat a low-cholesterol/low fat diet; Do not smoke; Control high blood pressure; Exercise. As the theory had suggested, all items correlated well with each other and could be used as indicators of general knowledge about preventing heart disease.
- This means that each of those items possessed what type of validity?

Answer 33

Construct validity
(theoretically driven)

Question 34

Imagine a study in which you investigate the instruments social workers use in order to diagnose their clients’ condition and match them with the most appropriate treatment. You suspect, in particular, that answers aimed at assessing a client’s depressive state could be interpreted by one therapist as symptoms of mild depression, and by another as signs of a more serious condition.
- Are you testing those instruments’ validity or reliability?

Answer 34

Validity

Question 35

A team of marriage therapists comes up with a questionnaire to be administered to married and cohabiting couples in order to assess their risk factors for domestic violence. Their answers are then compared with those of known abusive relationships.
- What type of validity are the therapists attempting to establish?

Answer 35

Predictive Validity