Reliability & Validity

Question 1

Reliability

Answer 1

• Are the results consistent?

- Provides an estimate of the proportion of unsystematic error <—need to know the degree of to determine reliability

Question 2

Validity

Answer 2

• Does it measure what it says it measures?
• Overall eval of evidence and degree of trustworthiness
• Determine if enough support exists to use the test in a certain way

Question 3

Classical Test Theory

Answer 3

• Observed score = T + E
• T is the true score if the test is completely free from error
• E is the error

Question 4

Unsystematic Error

Answer 4

• Random errors: mood, health, fatigue
• Administration differences
• Scoring differences
• Random guessing

Question 5

Systematic Error

Answer 5

Constant errors that occur every time tested, like a typo

Question 6

Reliability Related to Validity

Answer 6

• High validity can occur if high reliability exists
• High validity cannot occur if low reliability
• High reliability does not suggest high validity

Question 7

Correlation Related to Reliability

Answer 7

• Correlation: Statistical technique used to examine consistency
• Reliability is often based on consistency between two sets of scores

Question 8

Positive Correlation

Answer 8

As one increases, so does the other

Question 9

Negative Correlation

Answer 9

As one increases, the other decreases

Question 10

Correlation Coefficient (Pearson-Product Moment)

Answer 10

• Correlation coefficient: numerical indicator of the relationship between two sets of data
• PPM correlation coefficient - most common
• -1 to +1: closer to absolute value 1=stronger relationship

Question 11

Test-Retest

Answer 11

• Give same test twice to same group
• Correlation between first and second administration (2-6 weeks away)
• Possible influences: shorter gap, high correlation, changes in administration, interventions, practice test
• Ex: skills-based test

Question 12

Alternate Forms

Answer 12

• Very difficult
• Correlation off scores from two equivalent forms of a test
• Measures stability (over time) and equivalence (construct similarity)
• Use sample of different times from same domain

Question 13

Internal Consistency

Answer 13

• One administration
• One form of instrument
• Divides instrument and correlates the scores from the different portions

Question 14

Split-Half Reliability

Answer 14

• Given once then split in half to determine reliability
• Need to divide instrument into equivalent halves, like even and odd
• Problem: dividing instrument in half makes number of items smaller —> smaller correlation

Doesn’t work if test increases in difficulty and doesn’t quick fix problem

Question 15

Kinder-Richardson

Answer 15

• KR-20: heterogeneous items
• KR-21: homogenous items - single construct (cannot be used if items are from the same domain or differ in difficulty)
• Lower reliability coefficient then split-half
• Purpose: Estimate the average of all split-half reliabilities from all ways of splitting the instrument

Question 16

Pearson-Product Coefficient Alpha

Answer 16

• Used for non-dichotomous scoring
• Ex: Likert scales
• Cronbach’s alpha
• Takes into account variance of each item
• Conservation estimate of reliability
• Most common

Question 17

Standard Error of Measurement (SEM)

Answer 17

• Provides estimate of range of scores if someone were to take instrument repeatedly
• Based on idea that if someone takes test multiple times, scores would fall into a normal distribution

Question 18

SEM v. SD

Answer 18

• SD is spread of scores between students
• SEM is spread of scores for one student
• Uses same estimations

Question 19

Content-Related Validity

Answer 19

• Test items measure the objectives they are supposed to measure
• Focus on how content was determined
• May be based on test creator’s own analysis of topic or expert analysis
• How well do test items reflect the domain of material being tested

Question 20

Criterion-Related Validity

Answer 20

• Test scores related to specific criterion/variable
• Sources of criterion scores: academic achievement, level of education, performance in specialized training, job performance, psychiatric diagnosis, ratings by supervisors, correlations with previously available tests

Question 21

Concurrent Validity (Criterion-Related)

Answer 21

• Scores on test and criterion measure are collected at same point
• Ex: achievement, certification
• Scorer typically higher than predictive
• Require reliable and bias-free measures

Question 22

Predictive Validity (Criterion-Related)

Answer 22

• Test is administered first and scores on criterion measure are collected at a later time
• Ex: SAT, college GPA
• Require reliable and bias-free measures

Question 23

Construct Validity

Answer 23

• What do scores on this test mean or signify
• Construct: Grouping of variables that make up observed behavior patterns
• Ex: Self-efficacy, personality
• Measured by correlation of 2 scores or factor analysis
• Often seen in psych tests

Question 24

Convergent v. Discriminant (Construct Validity)

Answer 24

-Covergent: Positive correlation with other tests measuring the same/similar construct

Question 25

Threats to Construct Validity

Answer 25

• Too many variables
• Under-represented: missing measuring parts of construct
• Extra questions
• Items are too similar

Question 26

Overall Threats to Validity

Answer 26

• History: outside events during course of test
• Maturation: natural development with age
• Testing: repeat testing; changes due to practice
• Instrumentation: changes in measurement procedures
• Statistical regression: regression to mean after extreme score first time
• Interaction: any combo of 2
• Mortality: drop out
• Collection of subjects: bias of collecting subjects and assigning to groups

Question 27

Face Validity

Answer 27

• Not legitimate

- Based on appearance of the measure and its test items

Question 28

Types of Evidences

Answer 28

• Test content
• Response processes
• Internet structures
• Relations to other variables
• Consequences of testing

Question 29

Item Analysis

Answer 29

• Examine and eval each item in the test —> get rid of items that don’t work
• Done during instrument development or revision

Question 30

Item Difficulty

Answer 30

• Index reflecting proportion of people getting item correct
• 0.0= no one got it correct
• 1.0= everyone got it correct
• 0.5= ideal for differentiation

Question 31

Item Discrimination

Answer 31

• Degree to which item correctly differentiates among test takers
• Extreme group method: 2 groups - high scores, low scores (works with normal distribution)
• Correlational method: performance of test v. item

Question 32

Item Response Theory (IRT)

Answer 32

• Focus on each item -considers mathematical relationship between abilities
• 2 major assumptions: unidimensionality, local independence
• Most common in testing where there is a right/wrong answer v. preference
• Models student ability using each question instead of aggregate score

Question 33

Unidimensionality

Answer 33

Each item measures one ability or trait

Question 34

Local Independence

Answer 34

Unrelated to responses on other items

Question 35

Selecting Tasks

Answer 35

• Determine what info is needed
• Consider what info is needed
• Search assessment resources
• Eval possible instruments

Question 36

Administering Tests

Answer 36

• Pre-testing procedures
• Administration
• Scoring: by hand, computer, Internet

Question 37

Communicating Results

Answer 37

• Simple language
• Individual v. Group
• Written v. Oral
• Communicate test’s strengths and limitations
• Know the manual
• Describe v. Just report cases
• Use various results
• Involve client
• Encourage asking questions
• Relate test to a goal

Question 38

Problems with Reporting Result

Answer 38

• Acceptance
• Readiness of client
• Negative results
• Flat profiles and doesn’t show anything
• Motivation and attitude

Question 39

Communicating Test Results for Parents

Answer 39

• Identifying information
• Reason for referral
• Background info
• Test results and interpretation
• Diagnostic impressions and summary