Testing Concepts- Validity and Reliability

Question 1

What are test selection considerations?

Answer 1

Population, ease and feasibility of test administration, ease of normative data comparison, ethics and fairness and validy

Question 2

What is Valdity?

Answer 2

The ability of a test to measure accurately , the degree to which it measures what it purports to measure, and is dependant on reliability

Question 3

What are common types of validity evidence?

Answer 3

Construct validity
Face validity
Criterion Validity
Convergent validity

Question 4

How is validity estimated?

Answer 4

logically or statistically

Question 5

What is evidence of Face Validity?

Answer 5

The mesure involves the performance being measured, there is no statistical evidence required
eg-the test looks like the actual job being performed

Question 6

What is Criterion validity evidence?

Answer 6

Also called statistical or correlation validity

• how closely scores on a test relate to the scores of a standard or criterion that has been developed by a gold standard test.
  eg. metabolic cart for VO2, firefighters who are already working in the field became the criterion

ie- how well the beep test reflects scores from a metabolic cart test

Question 7

What R scores are we looking for in criterion validity?>

Answer 7

R>0.9

Question 8

What are the two types of validity within criterion related evidence?

Answer 8

Concurrent Validity
- The criterion is measured at approximately the same time as the alternate measure and the scores are compared
Predictive Validity
- the criterion is measured in the future

Question 9

What is Convergent Validity Evidence?

Answer 9

2 or more measurements are used to establish that the test battery is measuring what it purports to measure
eg. Not only is the physiology of the test tasks representative of the job task, but the people taking the task also indicate that it feels close

Question 10

What is Reliability?

Answer 10

• the consistency or repeatability of an observation

- the ability for someone to do a test over and over and get the same result

Question 11

How is reliability measured? (stats)

Answer 11

we use test-retest scores to calculate a stability reliability coeff:
when r>0.90 there is high reliability

Question 12

What are the types of reliability?

Answer 12

1. Stability Reliability
2. Internal-consistency reliability
3. Objectivity

Question 13

What is internal consistency reliability?

Answer 13

• The person administering the test should give at least two trials within a single day
• Changes to the score between trials indicates poor internal consistency
• it’s basically concerned with how consistent results are within a testing session

Question 14

What is the difference between the IC (internal consistency) coeff and the stability reliability coeff ?

Answer 14

the IC coeff is almost always higher

Question 15

What is objectivity reliability equal to in a test battery?

Answer 15

the judge reliability, and inter-tester reliability

Question 16

What factors affect objectivity?

Answer 16

1. the clarity of the scoring system (the amount of instruction helps reduce lack of objectivity)
2. the degree to which a judge can assign a score accurately

Question 17

What is the relationship between validity, relevance and reliability?

Answer 17

• see diagram

Question 18

What are some considerations for reducing measurement error?

Answer 18

Valid and reliable tests
Instructions
Test complexity
warm up and test trials
Equipment quality & calibration
Testing environment
Scoring Accuracy
Experience and state of mind of the person testing
State of mind of the person being tested

Question 19

How do we calibrate things? When do we measure them?

Answer 19

Requires comparison between two measurements (one of known magnitude and one that needs to be confirmed (the test instrument)
We calibrate at least every 6 months

Question 20

What should we do for reliability when there is a learning effect?
What is more reliable complex or discrete tasks?

Answer 20

test for many days!!
discrete tasks are more reliable- the more complex a task is, the more likely it is that there will be a learning effect!

Question 21

Stability reliability

Answer 21

scores do not change across days

- practice tests, pre-test guidelines, calibration