Research Principles Flashcards
Construct validity
Construct validity evaluates whether a measurement tool really represents the thing we are interested in measuring. It’s central to establishing the overall validity of a method.
If you develop a questionnaire to diagnose depression, you need to know: does the questionnaire really measure the construct of depression? Or is it actually measuring the respondent’s mood, self-esteem, or some other construct?
Content validity
Content validity assesses whether a test is representative of all aspects of the construct.
To produce valid results, the content of a test, survey or measurement method must cover all relevant parts of the subject it aims to measure. If some aspects are missing from the measurement (or if irrelevant aspects are included), the validity is threatened.
A mathematics teacher develops an end-of-semester algebra test for her class. The test should cover every form of algebra that was taught in the class. If some types of algebra are left out, then the results may not be an accurate indication of students’ understanding of the subject.
Face Validity
Does the research design pass a smell test?
Face validity considers how suitable the content of a test seems to be on the surface. It’s similar to content validity, but face validity is a more informal and subjective assessment.
Criterion Validity
Criterion validity evaluates how closely the results of your test correspond to the results of a different test.
To evaluate criterion validity, you calculate the correlation between the results of your measurement and the results of the criterion measurement. If there is a high correlation, this gives a good indication that your test is measuring what it intends to measure.
Internal Validity
Internal validity refers to the extent to which the independent variable can accurately be stated to produce the observed effect.
If the effect of the dependent variable is only due to the independent variable(s) then internal validity is achieved. This is the degree to which a result can be manipulated.
Put another way, internal validity is how you can tell that your research “works” in a research setting. Within a given study, does the variable you change affect the variable you’re studying?
External Validity
External validity refers to the extent to which the results of a study can be generalized beyond the sample. Which is to say that you can apply your findings to other people and settings.
Think of this as the degree to which a result can be generalized. How well do the research results apply to the rest of the world?
A laboratory setting (or other research setting) is a controlled environment with fewer variables. External validity refers to how well the results hold, even in the presence of all those other variables.
Ecological Validity
Ecological validity is the extent to which research results can be applied to real-life situations outside of research settings. This issue is closely related to external validity but covers the question of to what degree experimental findings mirror what can be observed in the real world (ecology = the science of interaction between organism and its environment). To be ecologically valid, the methods, materials and setting of a study must approximate the real-life situation that is under investigation.
Inter-Rater or Inter-Observer Reliability
Used to assess the degree to which different raters/observers give consistent estimates of the same phenomenon.
Test-Retest Reliability
An assessment or test of a person should give the same results whenever you apply the test. Test-retest reliability evaluates reliability across time.
Administering a test to a group of individuals
Re-administering the same test to the same group at some later time
Correlating the first set of scores with the second
Parallel Forms Reliability
One problem with questions or assessments is knowing what questions are the best ones to ask. A way of discovering this is do two tests in parallel, using different questions.
Parallel-forms reliability evaluates different questions and question sets that seek to assess the same construct.
Internal Consistency Reliability
Used to assess the consistency of results across items within a test.
What user-centered measures should we prioritize?
- Internal UX goals
- Correlation with business metrics (e.g., what predicts engagement best? Is it happiness? Ease of use?)
