Evaluating Quantitative Research

Question 1

What are the types of validity evidence?

Answer 1

Logical validity, construct validity, internal validity, external validity

Question 2

What is logical validity?

Answer 2

The quality of researchers’ arguments, their application of theory to support their study, and appropriate interpretation of results.
Examine the logical flow of researchers’ arguments and decisions from beginning to the end of their research.
Answers the questions “does this logically make sense, and has it been sufficiently justified by the researchers?”
Must consider:
-logical flow from introduction to purpose statement to hypothesis
-research methods make sense
-overall logical flow from beginning of the study to the final conclusions

Question 3

How do we apply logical validity?

Answer 3

Always ensure previous research is being cited appropriately.
Read and critically evaluate all literature to be cited.
Clearly note the kind of evidence claimed for each citation.
Clearly summarize experimental effects.
Critically present all sides and consensus in research.
Emphasize reliable sources and original evidence.
Strive for perfection in referencing details.

Question 4

What is construct validity?

Answer 4

Sometimes referred to as measure validity or test validity.
Do the measures used by researchers actually measure what they intend to measure?
Construct is the operations used to measure it.
Measuring variables is a fundamental part of research.
A measure should have validity evidence or evidence of construct validity.
Extent to which a variable has been adequately measured (soundness/appropriateness of an instrument in measuring what it is designed to measure).
Validity is a matter of degree (not all or nothing).

Question 5

How can you apply/evaluate construct validity?

Answer 5

Have the tool reviewed by experts in the field.
Examine consistency in the scores across multiple measurements.
Compare groups with known differences to detect differences on the tool.
Correlate the tool with other measures that already have well-established construct validity evidence.
Test how sensitive the tool is.
NOTE:
-the evidence for and against construct validity will range on a continuum
-the task for researchers is to carefully examine construct validity evidence and make a decision as to whether or not a measure is sufficient
-should also describe the measure and report construct validity evidence

Question 6

What is the goal for researchers with construct validity?

Answer 6

The goal for researchers is to develop a nomological network.

Question 7

Nomological Network

Answer 7

A web of evidence that researchers collect to justify their choices for using particular measures and support that the measured indeed measure what they intend to measure.
Evidence used to justify researchers’ choices to use particular measures, and supports that the measures of different constructs and behaviours.

Question 8

Why do we care so much about construct validity?

Answer 8

Construct validity helps identify whether an instrument might be a good one to use.
A valid instrument means you can draw meaningful and useful inferences from the scores.

Question 9

What is internal validity?

Answer 9

The researchers’ abilities to claim that any change in an outcome is the result of a treatment or intervention and not a result of other factors.
One of the types of validity that is relevant to experimental designs.

Question 10

What are some threats to internal validity?

Answer 10

Factors that may alter the DV separate from the effects of IV.
-may be related to experimental procedures, treatment/manipulation of the IV, and/or participant characteristics.

Question 11

What are some strategies for controlling threats to internal validity?

Answer 11

Control group: many of the threats to internal validity can be controlled for by having a control group as similar to the experimental group as possible.
Random assignment: increases the likelihood of groups being similar prior to intervention; treatment should be the only systematic effect.
Placebos, blind setups, and double-blind setups: reduce hawthorne effect and halo effects.

Question 12

Hawthorne Effect

Answer 12

Occurs when participants modify their behaviour because they know they’re being observed.

Question 13

Halo Effect

Answer 13

Occurs when researchers expectations influence how they interact with participants.

Question 14

Placebos

Answer 14

Participants don’t know if they got drug/treatment that is being tested.
Still giving “treatment” but don’t know if it is the true treatment or not.

Question 15

Blind setups

Answer 15

Only participant’s don’t know if they got treatment or not, but researchers know who got what.

Question 16

Double-blind setups

Answer 16

Researchers and participants don’t know what groups got what (treatment or not).

Question 17

What is external validity?

Answer 17

The researchers’ abilities to see similar success of the treatment or intervention with other populations, in other contexts, and across time.
A judgement of external validity requires a treatment or intervention, os it is standard of evaluation that we can only adequately make in the context of experimental designs.

Question 18

What are some threats to external validity?

Answer 18

Arise when experimenters draw incorrect inferences form the sample data to other persons, other settings, and past or future situations.
One key is to have representative samples from populations.

Question 19

What is WEIRD research?

Answer 19

Is associated with external validity. Relying on data from WEIRD samples limits the generalizability of research findings.
Western
Educated
Industrialized
Rich
Democratic
Kinesiology research often relies on non-random selection/volunteers for studies, and their interests/preferences for participating may impact findings and conclusions of research.

Question 20

What are some applications for external validity?

Answer 20

The key to generalizing study findings to other people, contexts, or time is ensuring that the sample chosen for a study is representative of the population that researchers wish to generalize their result to.
Another factor is the extent to which the conditions of the study reflect the real world (ie. natural setting vs lab).
One way to enhance external validity is to clearly identify inclusion criteria specifying who can and cannot be part of sample studied (ie. specific sample is used).

Question 21

Can we evaluate internal and external validity for all studies?

Answer 21

No
-can evaluate them for experimental studies, but cannot evaluate them for non-experimental studies.

Question 22

Measurement Validity

Answer 22

The extent to which a variable has been adequately measured.

Question 23

Measurement Reliability

Answer 23

The consistency (or repeatability) of a measure.

Question 24

What is reliability?

Answer 24

An integral part of measurement validity is measurement reliability, which pertains to the consistency and dependability of a measure.
If you cannot depend on successive trials to yield the same results, then a measure cannot be trusted.
Reflects the degree to which a measure is free from error variance.
Any measurement is comprised of a true score and measurement error.

Question 25

What are some common methods for testing reliability?

Answer 25

Internal consistency
Test-retest reliability

Question 26

Internal consistency

Answer 26

How closely aspects (variables) in the study are related.
Want the variables to be similar, but not exactly the same so that we are assessing everything.
The consistency of scores within a test, or the degree of interrelatedness among items.
Assessed by Cronbach’s coefficient alpha, ranging from 0-1.
-based on inter-item correlation
-the higher the items covary the greater internal consistency
-a limitation is that it is affected by the number of items
-alpha score greater than 0.7 = good internal consistency, but want it be greater than 0.85 for it to be great.

Question 27

Test-retest reliability

Answer 27

Do test, wait a bit, then bring them back to retest to see likelihood of same results.
Coefficient between scores on repetitions of the same test.
-are scores stable over time?
Not appropriate for measures affected by repetition or dynamic constructs.

Question 28

A measure can be reliable…

Answer 28

But not valid.