Validity and Reliability

Question 1

What is a theory?

Answer 1

A body of interrelated principles that present a systematic of phenomena

Question 2

What is a hypothesis?

Answer 2

A statement of the expected relationship between variables

Question 3

What 4 things are generally referred to as variables?

Answer 3

1) tests
2) diagnoses
3) treatments
4) effects of interest in a research question

Question 4

What are the 2 reference standards?

Answer 4

1) norm-referenced

2) criterion-referenced

Question 5

What are norm-referenced standards?

Answer 5

Standard that judge’s an individual’s performance off of the group’s performance
*Growth curves for children are examples of clinical measurements that are norm-referenced

Question 6

What are criterion-referenced standards?

Answer 6

Standard that an individual’s performance off of an absolute standard
*Discharge criteria such as “transfers independently” or “ambulates 300 ft” are examples of clinical situations that are criterion-referenced

Question 7

What is measurement reliability?

Answer 7

The amount of variability in a measure (consistency)

Question 8

Measurement reliability have different forms primarily classified by what two things?

Answer 8

instrument and rater (person or people taking the measurements)

Question 9

3 Measurement Reliability for Instruments

Answer 9

1) Internal Consistency
2) Parallel Forms
3) Spilt-Half

Question 10

What is internal consistency reliability?

Answer 10

Refers to the degree to which tests or procedures measure the same concept or construct
The instrument should demonstrate internal consistency for each construct
*Health-related quality of life questionnaires for example

Question 11

What is parallel forms reliability?

Answer 11

Used to assess consistency of the results of two tests constructed in the same way from the same content domain

Question 12

What is split-half reliability?

Answer 12

The reliability of an instrument established by testing two versions of the same tool that are combined into one survey administered at one time
In other words administer the entire instrument to a sample of people and calculate the total score for each randomly divided half and determine the degree to which scores agree

Question 13

2 Measurement Reliability for Raters

Answer 13

1) Inter-rater reliability

2) Intra-rater reliability

Question 14

What is inter-rater reliability?

Answer 14

Refers to the consistency of measurement values between different individual raters
*If physical therapists from the same clinic take turns collecting data for a study it is considered inter-rater reliability

Question 15

What is intra-rater reliability?

Answer 15

Refers to the consistency of measurement values of one individual
*A single PT is responsible for measuring joint ROM following an experimental stretching technique should be able to obtain nearly the same score for the same position each time the measurement is taken

Question 16

What is Measurement Validity?

Answer 16

The ability of a measure to capture what it is intended to capture
For example: a goniometer that measures joint position in degrees is a valid instrument for ROM, whereas a thermometer is not

Question 17

How is validity strengthened?

Answer 17

by the decisions made in a design or methodology

Question 18

How is validity threatened?

Answer 18

by decisions made in design or methodology

Question 19

4 Forms of Measurement Validity

Answer 19

1) Face validity
2) Content Validity
3) Construct Validity
4) Criterion validity

Question 20

Describe Face Validity

Answer 20

It is the simplest and weakest form of measurement validity. It is typically tested with a yes or no question such as “Does this instrument appear to be the appropriate choice to measure this variable?”

Question 21

Describe Content Validity

Answer 21

An instrument has content validity if it represents all of the relevant facets of the variable it intends to measure
*Fitness variables for example

Question 22

Why is content validity challenging to determine?

Answer 22

Because there is no external standard or statistical criterion against which an instrument can be judged

Question 23

What is construct validity?

Answer 23

This type of instrument validity is determined based on the degree to which the measure reflects the definition of the concept or construct that it represents
For example: patient satisfaction surveys

Question 24

Why is construct validity difficult to achieve?

Answer 24

Because definitions evolve over time through repeated testing that changes people’s understanding of the theoretical basis underlying the concept or construct of interest

Question 25

What is Criterion Validity?

Answer 25

This type of instrument validity reflects the degree to which its scores are related to scores obtained with a reference standard instrument
For example: clinician examination techniques will have high criterion validity if they produce the same results of the MRI

Question 26

What are the two methods of evaluating criterion validity?

Answer 26

1) Concurrent Validity

2) Predictive Validity

Question 27

What is concurrent validity?

Answer 27

a method of evaluating criterion validity that involves administering the test of interest and reference standard test at the same time.

Question 28

What is predictive validity?

Answer 28

a method of evaluating criterion validity that reflects the degree to which the results form the test of interest can predict a future outcome, preferably measured by a reference standard

Question 29

What is responsiveness?

Answer 29

The ability of a measure to detect change in the phenomenon of interest

Question 30

What 3 things does responsiveness to change depend on?

Answer 30

1) the fit between the instrument and the operational definition of the phenomenon interest
2) the number of values on the instrument scale
3) the standard error of measurement associated with the instrument

Question 31

What is the first requirement for a responsive instrument?

Answer 31

construct validity: the measure should match the operational definition of the phenomenon of interest

Question 32

What is the second requirement for a responsive instrument?

Answer 32

The more values on a scale the more responsiveness to change

Question 33

What is the third requirement for a responsive instrument?

Answer 33

Standard error of measurement (SEM) is extent to which observed scores are disbursed around “true score”

Question 34

An instrument with a large standard error of measurement (SEM) will be ____ responsive to change.

Answer 34

Less

Question 35

When do floor and ceiling effects occur?

Answer 35

When the scale of the measure foes not register a further decrease or increase in scores for the lowest or highest scoring individuals

Question 36

Describe floor effect

Answer 36

There is no change is seen in scores even with lower performance
For Example: Pain is increasing but the therapist cannot see it

Question 37

Describe Ceiling effect

Answer 37

There is no change is seen in scores even with better performance
For Example: Patient feels better but there is no increase in the scores

Question 38

What is interpreting change mean?

Answer 38

The ability of an instrument to detect meaningful change is an essential criterion for its selection

Question 39

Internal vs. External Validity

Answer 39

Internal validity occurs when a researcher controls all extraneous variables and the only variable influencing the results of a study is the one being manipulated by the researcher. (MOST IMPORTANT)

External Validity refers to the extent to which the results of a study can be generalized or extended to others outside the experimental situation

Question 40

When does the internal validity of a study decrease?

Answer 40

If there is no control group, not enough subjects, you use faulty instruments, etc.