Reliability and Validity

Question 1

Assumes that each person has a true score that would be obtained if there were no errors in measurement.

Answer 1

Classical Test Score Theory

Question 2

assumes that the items that have been selected for any one test are just a sample of items from an infinite domain of potential items

Answer 2

Domain Sampling Theory

Question 3

the process of choosing test items that are appropriate to the content domain of the test

Answer 3

domain sampling

Question 4

this model considers the problems created by using a limited number of items to represent a larger and more complicated construct

Answer 4

the domain sampling model

Question 5

using this, the computer is used to focus on the range of item difficulty that helps assess an individual’s ability level.

Answer 5

item response theory

Question 6

refers to the degree which scores from a test are stable and results are consistent

Answer 6

reliability

Question 7

test reliability is usually estimated in one of three ways

Answer 7

• test-retest method
• method of parallel forms
• method of internal consistency

Question 8

in the ____, we consider the consistency of the test results when the test is administered on different occasions

Answer 8

test-retest method

Question 9

using the ____, we evaluate the test across different forms of the test

Answer 9

method of parallel forms

Question 10

we examine how people perform on similar subsets of items selected from the same form of the measure with the _____

Answer 10

method of internal consistency

Question 11

this effect occurs when the first testing session influences scores from second session

Answer 11

carryover effect

Question 12

compares two equivalent forms of a test that measure the same attribute

Answer 12

parallel forms / equivalent forms reliability

Question 13

• a test given and divided into halves that are scored separately
• the results of one half of the test are then compared with the results of the other

Answer 13

split half method

Question 14

Source of Error and Method for:
same test given at two points in time

Answer 14

Source of Error: Time sampling
Method: Test-Retest Method

Question 15

correlation between scores obtained on the two occasions
* source of error
* method

Answer 15

Source of Error: Time Sampling

Method: Test-Retest

Question 16

different items used to assess the same attribute

Answer 16

item sampling

Question 17

different items used to assess the same attribute

Answer 17

item sampling

Question 18

correlation between equivalent forms of the test that have different items

Answer 18

item sampling

Question 19

determined by dividing the total set of items relating to a construct of interest into halves and comparing the results obtained from the two subsets of items thus created

Answer 19

split half reliability

Question 20

also known as cronbach’s alpha

Answer 20

coefficient alpha

Question 21

• a measure of internal consistency, that is, how closely related a set of items are as a group
• it is considered to be a measure of scale reliability

Answer 21

coefficient alpha

Question 22

used to estimate the reliability of binary measurements

Answer 22

KR20 (Kuder and Richardson Formula 20)

Question 23

• takes into account chance agreement
• defined as (observed agreement - expected agreement)/(1-expected agreement)

Answer 23

Kappa Statistics

Question 24

best method for assessing the level of agreement among several observers

Answer 24

kappa statistics

Question 25

value of kappa when two measurements agree **only at the chance level**

Answer 25

0

Question 26

value of kappa when two measurements agree **perfectly**

Answer 26

1.0

Question 27

range in which reliability estimates are **good enough** for most purposes in basic research

Answer 27

.70 and .80

Question 28

what to do about low reliability?

Answer 28

increase the number of items according to the domain sampling model

Question 29

the ____ the sample, the more likely that the test will represent the true characteristic

Answer 29

larger

Question 30

- can be applied **to correct for half-length** - allows to estimate what the correlation between the two halves would have been if each half had been the length of the whole test

Answer 30

spearman-brown formula

Question 31

can be defined as the **agreement between a test score** or measure **and the quality it is believed to measure**

Answer 31

validity

Question 32

answers the question, **"does the test measure what is supposed to measure?"**

Answer 32

validity

Question 33

3 types of evidence in validity

Answer 33

1. construct-related 2. criterion-related 3. content-related

Question 34

is the **mere appearance** that a measure has validity

Answer 34

face validity

Question 35

the only type of evidence besides face validity that is **logical rather than statistical**

Answer 35

content validity

Question 36

describes the **failure to capture** important components of a **construct**

Answer 36

construct underrepresentation

Question 37

occurs when scores are influenced by factors **irrelevant** to the construct

Answer 37

construct irrelevant variance

Question 38

tells us just **how well a test corresponds with a particular criterion**

Answer 38

criterion validity evidence

Question 39

**standard** against which the test is compared

Answer 39

criterion

Question 40

**forecasting function of tests** is usually a type or form of criterion validity evidence known as ______

Answer 40

predictive validity evidence

Question 41

the **relationship between a test and a criterion** is usually expressed as a correlation called _________ __________

Answer 41

validity coefficient

Question 42

established through a series of activities in which a researcher **simultaneously defines some construct and develops the instrumentation to measure it**

Answer 42

construct validity evidence

Question 43

involves **assembling** evidence about what a test means

Answer 43

construct validation

Question 44

when a measure **correlates well** with other tests believed to measure the same construct, ___________ ___________ for validity is obtained

Answer 44

convergent evidence

Question 45

- also called divergent validation - demonstration of uniqueness - to demonstrate for validity, a test should have **low correlations with measures of unrelated constructs**, or evidence for what the test does not measure

Answer 45

discriminant evidence

Question 46

refers to the standardized test that are designed to **compare and rank test takers** in relation to one another

Answer 46

norm-referenced test

Question 47

the process of evaluating (or grading) the learning of students against a set of pre-specified qualities or criteria, **without the reference to the achievement of others**

Answer 47

criterion-referenced test

Question 48

indicates that the measure **does not represent a construct** other than the one for which it was derived.

Answer 48

discriminant evidence

Question 49

simple guidelines for item writing

Answer 49

- **define clearly** what you want to measure - **generate an item pool** - avoid** exceptionally long items** - keep the level of **reading difficulty appropriate for those who will complete the scale** - **avoid double-barreled items** that convey two or more ideas at the same time - **consider mixing positively and negatively worded items**