Lesson 9 - Measurement In Evaluation Flashcards
What is the core essence of validity?
Accuracy
Is the scale accurate each time you step on it?
What are the 4 major types of validity?
- Face Validity
- Content Validity
- Criterion Validity
- Construct Validity
Define Face Validity
Extent to which the instrument APPEARS to be measuring what it is supposed to measure.
- Face validity assesses whether the test “looks valid” to the examinees who take it, the administrative personnel who decide on its use, and other technically untrained observers.
What is one way it can be determined if the instrument has face validity?
Ask a colleague to look over an instrument and see if s/he agrees that it appears to measure what you intend to measure.
What is a problem with a colleague determining face validity?
It is a SUBJECTIVE opinion
- Depends on who you ask and what they know.
How does Face Validity differ from the other forms of validity?
Face validity lacks some form of systematic logical analysis of the content.
Define Content Validity
The degree to which the items within a research instrument represent the universe of content for the concept being measured or the domain of a given behavior.
- A type of face validity
- Subjective
- Cost effective
In what type of validity do you ask multiple ‘experts’ in the field to evaluate the instrument?
Content Validity
- Problem with this is that it is subjective
Who should Content Validity be conducted by?
- persons with technical expertise to make sound judgements
- persons with a willingness to provide, if necessary, negative feedback
- professionals with diverse experience
Since neither content or face validity are sufficient for establishing that a scale has ‘true’ validity, why use them?
- no other way
- most cost effective
Define Criterion Validity
Criterion validity is the extent to which the measures are demonstrably related to concrete criteria in the “real” world.
What are the two subtypes of criterion validity?
Predictive and concurrent validity
Define Concurrent Validity
Concurrent validity is demonstrated where a test correlates well with a measure that has previously been validated. The two measures are taken at the same time.
(current status)
What is the main difference b/w concurrent and predictive validity?
The difference between concurrent validity and predictive validity rests solely on the time at which the two measures are administered.
Define Predictive Validity
Predictive validity is where one measure occurs earlier and is meant to predict some later measure.
(Future performance)
In cases when there is not an existing criterion from which to compare, or the phenomenon that planners want to measure is more abstract than concrete, how can validity be established?
Using construct validity
Define construct validity
Construct validity is the degree to which a measure correlates with other measures it is THEORETICALLY expected to correlate with.
What is the core essence of reliability?
Consistency
The scale be be consistently wrong or right
What are the two major approaches to reliability?
- Internal consistency reliability
2. Test-retest reliability
How is internal consistency reliability established?
Each item that makes up a scale is examined for how well it correlates with the scale as a whole.
- Consistency across all parts of an instrument
How can test-retest reliability be established?
By measuring the same group at two different points in time times (day, year, etc)
Although both must be considered, which is more important validity or reliability and why?
Validity because if an instrument doesn’t accurately measure what it is suppose to measure (validity), then reliability is an irrelevant issue. a
(Consistently supplying a wrong answer)
Often must reliability and validity be assessed?
EACH time the instrument is administered to a sample.
Why may you NOT cite past experiences or existing data on psychometric properties?
Cannot simply cite past experience or existing data on psychometric properties because each group will respond differently.
This procedure in itself doesn’t give much confidence in the measure’s validity. This procedure is simply an “eyeball” test.
- What is this statement referring to?
Face Validity
The measure must cover the full range of the concepts meaning. Each dimension of the concept should be represented by items in the measure.
- What are these statements describing?
Content Validity
Is the measure related to those other measures that theory and prior research indicate that it should be related to?
- What is this question indicative of?
Construct Validity
Is the measure related to a more direct measure of the same phenomenon?
- What is this question indicative of?
Criterion Validity
What are validity and reliability characteristic of?
Validity and reliability are NOT a characteristic of the measuring instrument itself. They are characteristic of the scores/data produced.
(An example of this is using a ‘valid instrument’ to collect data on people too young or too old)
What is considered “sloppy speak” according to Dr. Barry about validity and reliability?
Saying that this survey of valid and reliable.
Torgen et al., set out to examine the reliability of a self-administered questionnaire about past and present physical activities both at work and during subjects’ free time. 44 respondents completed the questionnaire a first time, and then again after two weeks.
- By doing this, the researchers obtained information on what type of reliability?
Test-retest reliability
A recent study by Clark et al., raised questions about the accuracy of adolescents’ self-reports of pregnancies and STIs. After checking subjects’ answers to questionnaire items against their clinical charts, the authors concluded that patients’ histories of visits for STIs and pregnancies were often not substantiated by their medical records.
- Do adolescents’ self-reports suffer from validity or reliability problem?
Validity
Between 1979 and 1981, the San Antonio Heart Study collected on ethnic differences in cardiovascular risk factors. The following items were used to measure subjects’ knowledge related to prevention of heart attacks: Eat a low-cholesterol/low fat diet; Do not smoke; Control high blood pressure; Exercise. As the theory had suggested, all items correlated well with each other and could be used as indicators of general knowledge about preventing heart disease.
- This means that each of those items possessed what type of validity?
Construct validity (theoretically driven)
Imagine a study in which you investigate the instruments social workers use in order to diagnose their clients’ condition and match them with the most appropriate treatment. You suspect, in particular, that answers aimed at assessing a client’s depressive state could be interpreted by one therapist as symptoms of mild depression, and by another as signs of a more serious condition.
- Are you testing those instruments’ validity or reliability?
Validity
A team of marriage therapists comes up with a questionnaire to be administered to married and cohabiting couples in order to assess their risk factors for domestic violence. Their answers are then compared with those of known abusive relationships.
- What type of validity are the therapists attempting to establish?
Predictive Validity
