Psyc Testing and Assessment-Chapter 4 (Reliability)

Question 1

What are two sources of Errors in test construction?

Answer 1

• Random Error

- Systematic Error

Question 2

What is Random Error?

Answer 2

Situational factors:

• ->Noise(ie. room should be queit and confidential if oral testing is being conducted.
• -> Lighting
• ->Fire drill
• ->sense of urgency (bathroom)

Question 3

What is Systematic Error?

Answer 3

Measure was inaccurate.
–>for example, the ruler was off by one-tenth of an inch so all the measurements previously taken with that ruler were systematically off.

Question 4

What are two sources of Errors in Testing

Answer 4

1) Test Construction

2) Test Administration

Question 5

Sources of Errors in Test Construction

Answer 5

Domain Sampling Model-small test as an estimate of true ability may not represent the writer’s true knowledge/ability

Question 6

Sources of Errors in Test Administration (3)

Answer 6

1) Test environment-room temperature, lighting, ventilation, pencil or desk surface
2) Test Takers-sleep, effects of drugs or medication, emotional problems, physical discomfort, answered “b” instead of “d”
3) Examiner/Test Administrator-physical appearance of test administrator/examiner (maybe they are intimidating), presence or absence, any clues given

Question 7

Sources of Errors in Test Scoring and Interpretation (3)

Answer 7

1) growing reliance on objective, computer scorable items (instead of testing critical thinking ability)
2) Intelligence test (lacks creativity aspect)
3) Personality test-open ended questions (hard to interpret)

Question 8

Methodological Error

Answer 8

Interviewers may not have been trained properly

Question 9

Test-Retest Method

Answer 9

Test is given once to a participant, and then they are given the same test at a later date to compare results.

• Easy correlation b/w scores from T1 and T2
• MEASURES TRAITS WELL (as traits do not change)
• DOES NOT measure constantly changing characteristics

Higher the reliability with a correlation the better
-correlation is maximum +1 (for positive correlation) and -1 (for negative correlation)

Question 10

Issues of Test-retest Method (2)

Answer 10

1) Carryover effect
2) Practice effects

Interval between testing sessions must be selected and evaluated carefully

Question 11

Carryover Effects

Answer 11

Participant remembers answers from the first test (issue of test-retest method)

Question 12

Practice Effects

Answer 12

Participants may score better with practice (issue of test-retest method)

Question 13

Trial Test

Answer 13

Some researchers will conduct a trial test to ensure that someone has less anxiety for the test that will be measured.
Trial test ensures that the participant understands the test and what is expected (instructions) to ensure reliability

Question 14

Parallel Forms Method

Answer 14

• Two forms of the test (given to everyone)
• BOTH TESTS have to measure the same level of construct
  - ->can divide the questions into two sets = many items that reflect the same construct

-This can be burdensome as two versions of the test must be created.

Question 15

Split-Half Method

Answer 15

Test is given and divided into halves that are scored separately and results are compared (correlation)

Question 16

Issue of the Split-Half Method

Answer 16

• Test scores gain reliability as the number of items increases;
  
  • ->Therefore two half tests would be less reliable than the whole test (not the best test unless you have enough items that can be reliably added/divided)
• The first and second half of the test may not be divided evenly when it comes to difficulty/intensity

Question 17

What is Sampling error (4)

Answer 17

1) Item Sampling
2) Time sampling
3) Interater/interobserver
4) Internal consistency

Question 18

Item Sampling:

Answer 18

the same construct or attribute may be assessed using a wide pool of items.
–>this sort of error is assessed using alternate forms, or parallel forms reliability; the correlation between two forms of a test is created by randomly sampling a large pool of items believed to assess a particular construct

Question 19

Time sampling:

Answer 19

behavior is only watched during a certain time-may not be accurate display
–>Same test given at different points in time may produce different scores, even if given to the same test takers.

Question 20

Interater/interobserver

Answer 20

Agreement Reliability-do these observations match?

• ->80-85% is agreement is best to ensure reliability
• ->Kappa statistic can adjust the index of agreement

Question 21

Reliability

Answer 21

refers to several methods that are used to assess different sources of error.
–>All potential sources of error should be taken into consideration when evaluating reliability

Question 22

Internal Consistency

Answer 22

Refers to the intercorrelations among items within the same test.
–>internal consistency is evaluated using split-half reliability, the KR 20 method, or coefficient alpha.

Question 23

How reliable is reliable (measure)?

Answer 23

• You want to see a score of .70%-80% (or higher) for most purposes.
• Reiko suggests that 80-85% is best
• Cronbach’s alpha on SPSS will let us know which items should be removed to make the other items/test more reliable.
• Clinical setting should have 95%

Question 24

How to correct low reliability?

Answer 24

• Increase number of items
• Factor and item analysis (helps to determine if some of those items belong to the category they are in or if they belong elsewhere)
  
  • ->items in the wrong category may be affecting your reliability.
  • ->Valid = Reliable
  • ->Reliable does not equal Valid

Question 25

Kudar Richardson 20

Answer 25

Forumla for calculating the reliability of a test in which the items are dichotomous (scored usually as 0 or 1 for right or wrong)

Question 26

Cronbach’s Coefficient Alpha

Answer 26

• can confirm that a test has substantial reliability; cannot determine if a test is unreliable
• marks only the lower bound for reliability (actual reliability may be high)
• used when two halves of a test have UNEQUAL VARIANCES
• high alpha: can assume reliability of the test is acceptable b/c the lowest boundary of reliability is still high; reliability will not drop below alpha