Standardized Assessment Tools: Reliability and Validity

Question 1

What is a standardized assessment tool?

Answer 1

A tool used for the purpose of assessment that has been developed following a careful standardisation procedure
- Specific instructions for how to administer, how to apply and how to interpret.

Question 2

What is reliability?

Answer 2

Related to the consistency or repeatability of your measures; degree to which observed scores are free of measurement error

Question 3

What is validity?

Answer 3

Related to the soundness of the relevance of the interpretation of the measures
- the appropriateness, meaningfulness and usefulness of the specific inferences made from the test scores

• do the scores actually inform us about what we want to know

Question 4

What is the classical true score theory of measurement?

Answer 4

Observed score (measurement) = true score + error
X = T +e

Every measurement has error in it
You cannot measure either the true score or error
Smaller error = more certain of true score

Question 5

What are assumptions about the classical true score theory of measurement?

Answer 5

The true score is real and the observed score in itself cannot be trusted ( must consider the error present)
If the error is small, a large proportion of the observed scores is accounted for by the true score
Need to estimate both true score and error

Question 6

What is the true score?

Answer 6

considered the average of all the observed scores that you collect

Question 7

What is error?

Answer 7

Synonymous with variability or variance

Computed as the difference between the observed and the true score

Question 8

What is variance?

Answer 8

Sum of (x-x(average)^2/(n-1)

Question 9

What is random error?

Answer 9

Caused by any factor that randomly affects measurement of a variable across a sample.

• adds variability to the data but does not affect the average score
• amount of sleep prior to midterm

Question 10

What is systematic error?

Answer 10

Caused by factors that systematically affect measurement of the variable across the sample

• do not change the variability of the data but does affect the average score
• party the night before the exam that everyone went to

Question 11

What is test-retest reliability?

Answer 11

The same test is administered to the same sample (or a very similar) on two different occasions
Investigating the consistency of the repeated measures over time
Assumption: that there is no substantial change ing the construct being measured between the two occasions

Question 12

What is inter-rater reliability?

Answer 12

Interested in the consistency of measurements taken by different people(rates)
Interested in the consistency of the interpretation of the measurement done by different people

Question 13

What is intra-rate reliability?

Answer 13

Interested in the consistency of measurements taken by one person at different times
How consistent you are as a rater

Question 14

What is alternate form of reliability?

Answer 14

Parallel forms
Allows you to determine if the scores from two different versions of the same test are equivalent
Strong relationship between the scores from the different instruments = increases the reliability of the scores from each instrument

Question 15

What is internal consistency reliability?

Answer 15

Reliability is assessed by estimating how well the items that represent the same construct yields similar results
Consistency of results for different items representing the same construct

Question 16

What are common problems with reporting reliability?

Answer 16

Reliability is based on previous studies (not actually measured in the study you are reading)
Time between test and retest sometimes not mentioned
Reliability often based on a small sample

Question 17

What are the basic elements of a theory?

Answer 17

Construct - things that exist but are not directly measurable or observable
Hypothesis - specific ways in which variation in one construct is assumed to be related
Operational definitions

Question 18

What is construct validity?

Answer 18

The degree to which the measures actually reflect or represent the construct they were designed to measure
-How things actually work corresponds with how you think they work

Question 19

What is content validity?

Answer 19

The degree to which a measurement is judged to reflect the meaningful elements of a construct and not to reflect any extraneous elements
All about determining if you have a good definition of your construct and if your operational measures align well with your definition of the construct

Question 20

What are the requirements for content validity?

Answer 20

A thorough description of your construct
Ability to distinguish your construct from others
The items included in the measurement should be representative of many different elements of your construct

Question 21

What is criterion validity?

Answer 21

You check the performance of your operationalization/measure against some criteria
You make predictions about relationships between measures based on your theory of the construct
The degree to which your measures behave the way they should, given the theory of the construct

Question 22

What is predictive validity?

Answer 22

The extent to which one measure can correctly predict another measure that you theoretically think should be able to predict

Question 23

What is convergent validity?

Answer 23

The degree to which two tests measuring the same construct are highly correlated

Question 24

What is discriminant validity?

Answer 24

The degree to which two tests measuring different constructs should be uncorrelated

Question 25

What is face validity?

Answer 25

The degree to which the instrument appears to or looks like it measures the construct
Main function: increases acceptability of a test
If evident does not mean other forms of validity are present (vice versa)