validity and reliability Flashcards
Quant validity definition and how ensure:
Concept is accurately measured
Internal: extent to which results represent the truth in the population studied
Once established, can assess external: whether results apply to similar populations
ASCM increase validity by more appropriate control units
Threats: available units and covariates to fit pretreatment fit, bias in BPD data
Biased police reporting of crime and arrest data (see above)
Qual validity definition and how ensure
Prefer terms like authenticity and trustworthiness
Assessing whether qual data is credible (to sources you got it from), dependable (heard from multiple sources), confirmable (when ask, can confirm)
will conduct member checking with SSB staff after findings have been indexed and preliminarily summarized to assess whether the findings reflect their experiences.
identity of the researcher may influence qualitative findings both at the data collection and member checking phases which would reduce the internal validity of the findings.
Awareness and acknowledgement of the role of the researcher identity as well as efforts to create an authentic interaction will attempt to reduce the impacts of this threat
Switch between emic (insider perspective) and etic (outsider perspective)
Investigate disagreements among participants- why? Welcome negative evidence- normal variation? My lack of knowledge? Genuinely unusual case
Continue to look for alternative explanations for phenomenon
Acknowledgement: fear of being judged, cynicism about whether anything can really change, skepticism about the worth of evaluation, concern about the time and money costs of evaluation, frustration from previous bad evaluation experiences, especially lack of use
Get knowledge about organizational context to increase understanding and reasons for resistance, resistance not just a burden but also important source of informaiton related to development and hcange.
Mixed validity definition and how ensure
Referred to as legitimation:
Focus on quality of the design and rigor of interpretation
To increase the mixed methods internal validity, data will be triangulated across data sources and among individuals from the same SSB site/case. (collect data throgh several sources)
Additionally, disconfirming evidence and extreme cases will be included to provide perspectives that are contrary to primary findings.
Specific mixed methods analytic techniques such as joint displays and cross case comparisons will be used to increase the mixed methods internal validity, but due to resource and time constraints it is possible that inferences from the integration of the data may not be meaningful, reducing the internal validity.
It is also possible that using the SSB site as a case will not be a sufficiently bounded case to allow for holistic cross case comparison
Quant reliability define and ensure
To assess the reliability of the quantitative findings, sensitivity analyses will be conducted to assess whether modeling decisions influenced the main quantitative findings.
Any sensitivity analyses that demonstrate meaningfully different findings will be presented in addition to the primary analyses
Use different modeling approaches- eg log binomial regression models. Ideally models show similar patterns of effect
Qual reliability define and ensure
two people will independently code a subset of transcripts after the codebook has been established. Intercoder agreement will be calculated based on the percentage of codes that the two individuals applied in the same way. A minimum threshold of 80% will be required to consider the qualitative coding reliable
Mixed reliability define and ensure
Consistency, dependability of results
the reliability of the mixed results will be improved by detailed methodological accounts of how the data was collected e.g.
This will include information on the researcher identity, participants and interview setting, as well as detailed accounts of data collection and analysis for the separate strands of data and their integration
Triangulation can also help reliability
Mixed Methods Evaluation Rubric
no agreed upon standards for rigor/quality
depends on paradigm
Transparency
Amount of mixing
Interpretive comprehensiveness : Different ways that multiple viewpoints are considered. downplay triangulation, uplift meta inference
Methodological foundation: demonstrate expertise, construct validity
social desirability bias- define, impact, how reduce
tendency to underreport socially undesirable attitudes and behaviors and to over report more desirable attributes
Social desirability bias may result in findings that are more positive or more aligned with program expectations than the individual’s reality.
Using indirect questions about the behaviors of others and prefacing the question with context that, for example, acknowledges the enormity of the challenge of their job may help to reduce this bias (Bergen and Labonte, 2020).
Additionally, awareness and acknowledgement of the role of the researcher identity as well as efforts to build rapport will attempt to reduce the impacts of these threats (Bernard et al., 2016).
