Measure Quality Flashcards
What are the 2 things psychologists need to consider about the quality of their measurements?
Reliability/Precision = exactness (consistency)
Validity/Accuracy = correctness (truthfulness)
What is precision?
Exactness (consistency)
What is accuracy?
Correctness (truthfulness)
What is Reliability?
Precision
- The extent to which our measure would provide the same results under the same conditions
What is Validity?
Accuracy
- The extent to which it is measuring the construct we are interested in
What is Validity?
Accuracy
- The extent to which it is measuring the construct we are interested in
Simply = Whether the results really do represent what they are supposed to measure
What does it mean when our measures have high validity and high reliability?
Our measures are precise and accurate
What does it mean when our measures have high validity and low reliability?
Our measures are precise but not accurate
What does it mean when our measures have low validity and high reliability?
Our measures are not precise but are accurate
What does it mean when our measures have low validity and low reliability?
Our measures are neither precise or accurate
We want to investigate the relationship between head size and intelligence
Is head size a …?
1) Reliable measure
2) Valid measure
1) Reliable measure = Yes. If you were to measure the head again with measuring tape it would be the same size
2) Valid measure = No. Because the head size is not actually measuring intelligence
What are the 2 forms of reliability?
1) Temporal consistency
2) Internal consistency
What is temporal consistency?
Assumes that there is no substantial change in construct being measured between 2 occasions
Simply= If we test our instrument twice, with a time interval between the two tests, there will be no change in results
What is internal consistency?
The extent to which items within an instrument measure various aspects of some characteristics
Simply = How well does an instrument actually measure what you want it to measure?
What is the Test-Retest Reliability?
It measures fluctuations from one time to another
We administer the same test/measurement twice over a period of time to a group of individuals and see if we still get the same results
Important for constructs which we expect to be stable (e.g. personality type)
What is one limitation of the Test-Retest Reliability?
Order effects
What is Inter-Rater Reliability?
It measures fluctuations between observers
It evaluates the extent to which different judges agree in their assessment decisions (to assess the reliability of answers produced by different items on a test)
What is Parallel Forms Reliability?
Measures reliability by using different versions of the assessment tool/test to the same group of participants
Different versions of a test can be useful to help eliminate memory effects as the questions are different
List one advantage and one disadvantage of Parallel Forms Reliability?
Pro = Different versions of a test can be useful to help eliminate memory effects as the questions are different
Con = Order effects
What is internal consistency (reliability)?
Measures the degree of homogeneity among the items on a test, such that they are consistent with one another and measuring the same thing
e.g. all items in a questionnaire are measuring
the same construct
What is one disadvantage of internal consistency (reliability)?
Order effects
What are the 4 forms of reliability?
1) Test-retest reliability
2) Inter-rater reliability
3) Parallel forms
4) Internal consistency
What are the 4 forms of validity?
1) Face validity
2) Content validity
3) Criterion validity
4) Construct validity
What is face validity?
Evaluates whether a test appears to measure what it’s supposed to measure (is it relevant to what it’s assessing?)
Does it look like a good test?
e.g. do the questions in the RM exam reflect the RM knowledge students should have learnt?
What is content validity?
The extent to which a test/measurement tool evaluates all aspects of the topic, construct or behaviour it is designed to measure
Does our test measure the construct fully?
e.g. the RM exam should cover knowledge of quantitative and qualitative methods
What is criterion validity?
The extent of how well/accurately a test measures the outcome (and predict future outcome) it was designed to measure
Does the measure give results which are in agreement with other measures of the same thing?
e.g. do RM exam quiz scores relate to final exam grades?
What is the difference between Concurrent and Predictive Criterion validity?
Concurrent = Comparison of new test with established test
Predictive = Does the test predict the outcome of another variable?
What is construct validity?
The extent of how well a test measure the concept/trait it was designed to evaluate?
Is the construct we are trying to measure valid?
i.e. does the construct itself exist?
The validity of a construct is supported by cumulative research evidence collected over time
What are the 2 ways construct validity can be assessed?
1) Convergent validity = Whether our measurement correlates with other tests of related constructs
e.g. Our test for general happiness should correlate with studies of extreme happiness, moderate happiness and other similar studies
2) Discriminant validity = When our measurement doesn’t correlate with tests of different or unrelated constructs
e.g. Our test for general happiness should not correlate with studies of depression, low happiness and other unrelated studies
What is convergent validity?
Whether our measurement correlates with other tests of related constructs
e.g. Our test for general happiness should correlate with studies of extreme happiness, moderate happiness and other similar studies
What is discriminant validity?
When our measurement doesn’t correlate with tests of different or unrelated constructs
e.g. Our test for general happiness should not correlate with studies of depression, sadness and other unrelated studies
What validity is this?
Does our test measure the construct fully?
Content validity
What validity is this?
The comparison of new tests with established tests
Concurrent Validity
What validity is this?
Does the test correlate with measures of the same and related constructs?
Convergent Validity
What validity is this?
Does it look like a good test?
Face validity
What validity is this?
Is there a lack of correlation with measures of different and unrelated constructs?
Discriminant validity
What validity is this?
Does the measure give results which are in agreement with other measures of the same thing?
Criterion validity
What validity is this?
Does the test predict the outcome on another variable/some other variable measuring a different construct?
Predictive validity
What validity is this?
Is there evidence that the construct exists?
Is the construct we are trying to measure valid?
Construct validity
If a test gives the same result at two different points in time, it has demonstrated good (Inter-rater/Test-retest/Parallel form) reliability.
Test-retest reliability
If the items of is scale a correlated with one another, the test has demonstrated good (Inter-rater/Internal consistency/Parallel forms) reliability
Internal consistency
One way to assess internal consistency is by calculating (Inter-rater/Split half/Test-retest) reliability
Split half reliability
If two different people administer the same test to the same people and their results are different, the test has demonstrated poor (Inter-rater/Test-retest/Parallel form) reliability
Inter-rater reliability
If an individual is given two different versions of the same test and does well on one but badly on the other, the tests have demonstrated poor (Inter-rater/Test-retest/Parallel form) reliability
Parallel forms reliability
What type of reliability measures fluctuations from one-time point to another?
a. Inter-rater reliability b. Internal consistency c. Parallel forms reliability d. Test-retest reliability
D
Parallel forms reliability is evidence that…
a. A construct is stable b. A construct is valid c. Researchers agree on their ratings d. Two tests measure the same thing
D
The validity of a measure refers to the:
a. Comprehensiveness of the measurement b. Consistency of the measurement c. Particular type of construct specification d. Accuracy with which it measures the construct
D
Order effects are NOT problematic for which of the following types of reliability…
a. Test-retest b. Inter-rater c. Parallel forms d. Internal consistency
B
