Research Final Flashcards
What is the older EBP model?
3 circles
1- Clinical state & circumstances
2- Best research evidence
3- Patient preferences & actions
Overlapping section is clinical expertise.
What is the more traditional EBP model?
4 circles
1- Health care system & service organization
2 - Relevant scientific evidence
3 - Clinical judgement
4 - Patients’ values, goals & preferences
What are the 5 stages of EBP?
- Ask
- Acquire
- Appraise & Interpret
- Apply
- Evaluate
What are the levels in the 6S pyramid?
Part of acquiring information in EBP.
Bottom to top:
Single studies
Synposes of single studies
Syntheses
Synopses of syntheses
Summaries
Systems
What is the proposed new EBP pyramid?
- Systematic reviews become the handle/lens in which we understand the other studies.
Explain EBP for researchers
Data collection is in Acquire stage
- As researchers and research teams considerable background search and literature searching will be done in Ask to make sure we have a good question.
- Appraise & interpret – based on the evidence you found you appraise it and then apply it. For researchers this stage is analyzing what you’ve collected and then thinking about how it applies in practice.
What is a background question?
- Seek general knowledge
- About a single concept
- Who, what, where, when, how & why
E.g., terminology, general pathology, general info
What is a foreground question?
- Seek specific knowledge
- Bring together multiple concepts
- Inform clinical decisions
- Foreground q’s compare, often target a certain group, qualitative or quantitative, can be peer reviewed
- Do foreground questions always have to have a comparator? NO. Some might just look at 1 intervention and whether it causes a particular effect or not.
What is the framework for foreground questions overall?
Feasible
Interesting
Novel
Ethical
Relevant
What are foreground question frameworks for quantitative?
PICO - patient/population, intervention, comparison, outcome
PICO-T - adds in time frame
What are foreground question frameworks for mixed methods?
SPIDER - sample, phenomenon of interest, design, evaluation, research type
What are foreground question frameworks for qualitative?
PEO - population/problem, exposure, outcomes/themes
SPICE - setting, perspective, intervention, comparison, evaluation
PS - patient/population, situation
What are foreground question frameworks for measurement?
COSMIN - Consensus-based standards for the selection of health measurement instruments.
What are the PICO and PICO-T used for?
Driven to investigate intervention effectiveness
What is the COSMIN designed for?
COSMIN is designed to look at the development of instruments
- Measurement – look at the development of an outcome measure tools – psychometric type studies, reliability and validity questions.
What is FINER used for?
Determining the quality of a question, rather than the structure of it.
Applicable to the researcher who is going out to collect data
Where will you find background research and what are the considerations?
Textbooks, websites, clinical experts
Consider authorship and recency
Where will you find foreground research and what are the considerations?
Research evidence 6s pyramid
Consider database coverage, question frameworks, search structure (let the question drive)
Types of quantitative research, some common research designs and evidence synthesis?
PICO - therapy, diagnosis, prognosis, etiology/harm
RCT, Cohort studies, case studies, measurement studies
Evidence - Systematic reviews/meta-analyses
What is an example of mixed methods research and evidence synthesis?
Program evaluation
Evidence - Mixed methods review, scoping reviews, clinical practice guidelines
Types of qualitative research, some common research designs and evidence synthesis
Lived experiences
Grounded theory, phenomenology, ethnography, case studies
Evidence - meta syntheses
What is a platform?
Platforms allow you to structure your searches and dig into the databases that are like pots of research they pull from
Key word vs Subject heading
Key word is an abstract or title
Subject heading is a tag that the data base has sorted for you.
Complimentary strategies
What are some ways qualitative research can support evidence for practice?
- When it presents new info
- When it discusses complex or nuanced situations
- Who it can provide into that is context specific
What is the purpose of qualitative research?
- Development of theory
- Study subjective experience, meaning and contextual aspects of human action and interaction
- Exploration and discovery
- Study individuals in their natural settings
- Not interested in generalizability
What are the origins or qualitative research?
Basis in anthropology, philosophy and sociology
What assumptions are made in qualitative research?
- Multiple realities
- Social reality is dynamic and contextual
How are qualitative research questions generally organized?
- Exploratory not explanatory
- Open not closed
- Focus on meaning
- How? What? Why?
What are the 5 traditions of qualitative research?
- Phenomenology
- Ethnography
- Grounded theory
- Critical theory
- Participatory action research
What is phenomenology and how is it conducted?
- Focus on lived experiences of a phenomenon
- We can only understand a phenomenon through the experiences of those living with it/experiencing it
- Researchers focus on understanding/describing the experiences of those living it
- Interviews a common approach to data collection (but not the only one)
- Do see focus groups
- Don’t know in advance how many people you’ll need – can try to achieve data saturation (hard to do pragmatically and is done after the fact so hard to put in research proposal)
Inductive vs Deductive reasoning
Inductive is from specific to general conclusions - see in phenomenology - specific experiences to general themes.
Deductive is general premises to specific conclusion.
What is ethnography and how is it conducted?
- Roots in anthropology
- Description & interpretation of a cultural or social group or system
- Fieldwork: prolonged observation of a group within a setting
- Immersion – researcher being immersed
- Fieldwork – prolonged observation in the natural setting (wherever that is – e.g., classroom). Prolonged is historically year or years. Observations can look different but are typically very detailed.
- If you are researching and participating, you become so much part of the group that it might taint your understanding. Difference between observation and participant observation.
What is grounded theory and how is it conducted?
- Theory generation/construction as the focus
- Researchers typically do NOT immerse themselves in published literature in advance
- Common data collection: interviews, focus groups, observation
Construct theory from qual data
What is critical theory and how is it conducted?
- Focus on social realities that create barriers through dominant structures/processes. i.e., Challenging hegemony (dominant view of the world)
- Common data collection: Discourse analysis of policy documents; case study; interviews; observation
- About critique – understanding common understandings of issues, challenges and then putting it in the context of challenging some of those. Understanding political and social dominant structures that might be influencing our understanding of those.
What is participatory action research and how is it done?
- Engaging with those who are the focus of the research as research partners
- Action and understanding are inextricably linked
- Participatory approach from project formulation through to knowledge translation
- Mixed methods of data collection are common
-Assumes that the people who are meant to benefit from the research are active participants in the research from beginning to end. Design research question with you and are involved in every step.
- PAR is in line with the idea of multiple truths and realities
- Typically, the action is part of the research and you’re examining as a result of that action what changed, and you prepare a report based on that
What 2 principles guide participant selection in qualitative research and how do you describe them?
Appropriateness & Adequacy
- Appropriateness: select participants with exposure to the focus of the research
- Adequacy: who can be engaged with a goal of saturation
Often need to consider the pragmatics
What is purposive sampling?
Purposefully sought out ppl – e.g., ppl in first year of the program and aimed to get a range in terms of students ages, academic background and gender. Describe key characteristics as part of your samplings. Ask ppl what categories they fit in. Aren’t turning ppl away but are we getting a sample that ticks the important boxes in terms of your purpose.
Intentionally picking someone based of a characteristic/criteria.
What is snowball sampling?
Research participants are asked to assist researchers in identifying other potential subjects.
What is maximum variation?
Intent on trying to get as much variation from the group as you can – is a kind of purposive sampling
What is the DEJA Model and what is it used for?
Used for sample size determination
- Define – the sampling strategy (e.g., non-probability)
- Explain – how you are implementing the strategy (e.g., purposive)
- Justify – consider justification based on qualitative tradition, method of data collection, expected depth of data
- Apply – consider pragmatic issues that might affect number of participants
- Developed in response to critiques of “data saturation” as the determinant for sample size
What is the goal of qual research?
Description and often interpretation
What are codes?
Small chunks of data that you can label in some way
Generally, codes emerge from the data.
What comes after codes?
- Categories/Sub-themes
o Once you have a set of codes identified, and perhaps a short description of each, consider how the codes link to each other? Can they be clustered together in some way that makes sense? - Themes
o Can codes or categories be clustered together in a “bigger picture” idea or theme?
What are the 4 ways to look at trustworthiness/rigour in qualitative data?
- Rigor/trustworthiness is the overarching theme of critical appraisal
- Credibility
- Transferability
- Dependability
- Confirmability
What is credibility and some strategies used?
- Establishing believability
- Strategies:
o Prolonged engagement
o Reflexivity
o Member checking
o Triangulation of data
What is prolonged engagement?
Researcher immerses themselves as an observer or participant observer for prolonged amounts of time (ethnography) need a sense they were with participants long enough to believe them.
What is reflexivity?
Critically examining your own beliefs and positionality – critical reflection – want a sense that they set themselves up for critical reflection. Did more than 1 person code/collect the data (more so triangulation) and have those people had critical discussion related to what they’re thinking/learning
What is member checking?
Send the transcript back to the person you interviewed and ask them to review it to make sure it reflects what they were meant to say (can be controversial b/c they may want to change the data).
What is triangulation?
Collecting diff types of data, maybe from different types of people (e.g., profs and students).
What is transferability and some strategies used?
- Can study findings be transferred to other contexts?
- Strategies
o Dense description of participants/sampling strategies - who was recruited and how
o Findings described clearly and coherently with adequate supporting data to understand context and findings
What is dependability and some strategies used?
- Consistency between data and findings - Do the quotes illustrate the points that they’re trying to make in the description of the findings
- Procedural rigour
- Strategies:
o Audit trails
o Detailed description of methods - Want a sense of how they did the coding, who was involved, did they have training, how often did they meet, how many ppl were involved.
o Multiple researchers conducting analyses
o Triangulation of analyses (not super common)
What is an audit trail?
Record of all the transition/decision points you make once you have the data and are trying to interpret it.
What is confirmability and some strategies used?
- Analytic rigour/reducing bias in managing data
- Strategies:
o Rich description of findings (quotes support themes) - do they share quotes that support the themes/categories and do those quotes speak to how they describe
o Audit trail re analysis
o Reflexive analyses - have they talked about in the methods section they went through this sort of process
o External audit - doesn’t have to be someone not on the research team – but maybe haven’t been involved in the coding part. That person might ask can I see how you developed these themes from this data
Confirmability questions how the study findings are supported by the data. It identifies any bias that may have been present. It’s the level to which the findings can be confirmed or corroborated. Confirmability is concerned with determining that data & interpretations of the findings are not made up by the researcher’s imagination, but clearly derived from the data.
Generalizability vs transferability
- Transferability is what we use in qual and generalizability we use in quant. Both talk about applicability. How can I apply this research to my setting/practice. Transferability has a real focus on the findings and the context. can I apply this to my situation? Can we transfer the findings to a different context?
- Generalizing from the sample in the quant study are they generalizing to the sample I am working with. What are the characteristics of that sample?
What are the 4 parts of critical appraisal in qual research?
Research question
Design
Methods
Rigour
What are 4 types of foreground questions in quant research?
Treatment/Therapy - Does this intervention work? With whom? How well? For how long?
Diagnosis - Will this measurement help me understand the condition?
Prognosis - What is the likely outcome for this person/group of people?
Etiology/Harm - How does this intervention/exposure affect the development of the condition?
What are the 2 question styles in quant research?
Descriptive - how many? what factors?
Relational - Degree and/or direction of association? Causation?
What is an independent variable?
- Investigated as a potential agent of change in the dependent variable.
- Intervention/exposure
- Manipulated
What is a dependent variable?
- Thought to vary based on exposure to the independent variable
- Outcome of interest
- Measured NOT manipulated
What is an extraneous variable? Explain the 2 types.
- Not investigated but could influence the dependent variable
- Covariate - characteristic separate from the IV that isn’t of direct interest to the researcher but sometimes it might be controlled for (balance the groups based on it) or can use statistics
- Confounder - can influence IV and DV so the results of the experiment don’t represent the true relationship between the 2.
Quant study designs: Did the investigators decide receipt of the IV? What happens if you say yes?
Yes = experimental study
Did investigators randomly assign participants to receipt of the IV?
Yes = RCT
No = Non-randomized control trial (quasi-experimental study)
Quant study designs: Did the investigators decide receipt of the IV? What happens if you say no?
No = observational study
Do investigators compare participants on presence of the IV?
No = descriptive study
Yes = Analytical study
Do investigators look forwards or backwards in time from the intervention/exposure?
Leads to cohort, case-control study or cross-sectional study.
What is an experimental study?
Investigators decide who receives the intervention/exposure.
What is a randomized control trial?
Each participant has the SAME chance of allocation to intervention/exposure
What is a non-randomized control trial?
Aka quasi-experimental study.
Each participant allocated to intervention/exposure based on non-random factors.
E.g., Might have different sites where we recruit participants from and different sites might be allocated different levels of the IV.
What is an observational study?
Investigators do not decide who receives the intervention/exposure.
More of a natural presence - it’s there naturally in certain people.
What is a descriptive study?
Demonstrates possible association between the intervention/exposure and the outcome in a single group.
E.g., here are some heavy smokers and many have lung cancer
Here are some highly active older adults and relatively few of them have falls.
1 group, 2 variables, hints at a relationship so is often a starting point for more research
What is an analytical study?
Explores reasons for possible association between the intervention/exposure and the outcome.
Do control who they select to be the comparison group - still manipulated but not assigned.
Can match between groups to eliminate some of the confounders.
What is a cohort study?
Participants who do/do not receive are analyzed prospectively
Look at ppl who do or do not receive a treatment and if they develop certain types of outcomes.
What is a case-control study?
Participants who do/do not receive are analyzed retrospectively.
Outcome is the starting point and then you look backwards for exposure.
What is a cross-sectional study?
Participants who do/do not receive are analyzed at a single point in time.
Looking for relationship between exposure and outcome.
What do measurement studies look at?
Look at reliability/validity of certain tools
What do feasibility studies look at?
Looking at if an intervention is worth developing further
What are similarities between experimental & observational studies?
Both have independent variables
Both have comparison/control groups
Both have dependent variables
What are differences between experimental & observational studies?
Experimental
- Assigned
- Determined randomly or non-randomly
- Prospective
Observational
- Present/observed
- Exist/selected
- Prospective/retrospective/
simultaneous
What is bias?
A systematic error that can distort measurement and/or affect investigations and their results.
Uncontrolled phenomenon in a study that could potentially invalidate the results.
Consider: direction & magnitude of bias
What is a source of bias in the sampling or group/allocation/selection stage?
Selection Bias
What is a source of bias in the intervention/exposure stage?
Performance bias
What are sources of bias in the outcome measurement stage?
Detection bias or attrition bias
What is a source of bias in the interpretation stage?
Reporting bias
What is selection bias?
Participants or groups in a study sample differ systematically from the population at baseline.
- How are participants recruited?
- Who is enrolled in the study?
- How are participants allocated to groups?
- How is the allocation sequence concealed?
What is performance bias?
Inadvertent systematic differences between groups in intervention/exposure to other salient factors.
- Are participants blinded?
- Are interventionists blinded?
- How are interventionists trained?
- Is treatment fidelity assessed?
Might be an inadvertent systematic difference between the amount of attention one group gets compared to the other - then maybe it’s just the interaction impacting the DV.
What is fidelity?
Can we track the consistency of treatment between participants of 1 intervention.
The extent to which delivery of an intervention adheres to the protocol or program model originally developed.
What is detection bias?
Systematic differences between groups in how outcomes are measured.
- Are outcome measures objective/subjective?
- Are outcome assessors blinded?
A type of selection bias that results when one population is more likely to have the disease or condition detected than another because of increased testing, screening or surveillance in general.
What is attrition bias?
Systematic differences between groups in withdrawals
- Are outcome data for all participants reported?
- Are reasons for attrition examined?
Is there a trend that maybe the intervention is so hard to take that ppl can’t handle it. Is outcome data for all reported? Have they told you what happened with the missing data? Was it analyzed?
A type of selection bias due to systematic differences between study groups in the number and the way participants are lost from a study.
What is reporting bias?
Systematic differences in reported vs unreported findings.
- Are all identified outcomes reported?
- Are statistically significant and non-significant effects reported and linked to analyses?
What is random sampling?
Randomly choose participants for the study. Important for generalization.
Choosing participants for the study randomly from the population of interest. Helpful if you want to apply it to larger group.
What is random allocation?
Randomly choose group/receipt of the IV. Important to detect true effects of intervention on outcome.
Only happens in experimental studies.
When you do an RCT what is random?
Which treatment people get. Random allocation is what they talk about when talking about RCTs.
Often random sampling isn’t possible in health research.
Categorical vs continuous data
Categorical - Data has a discrete value in a distinct group. E.g., female and male or eye colour
Continuous - Data can have any value in a defined range. Something that can go onto more of a number line.
What are the 2 types of categorical data?
Nominal - Categories are not ordered in a numerical way (e.g., hair colour)
Ordinal - Categories have a natural numerical order but distance between categories is unknown (e.g., likert scale)
What are the 2 types of continuous data?
Interval - Values are equally spaced on a scale with no true zero (Celsius, IQ)
Ratio - Values are equally spaced on a scale with a true zero (grip strength)
What is validity?
How well does a measure capture what it intends to measure?
What is reliability?
Can results from a measure be reproduced under the same conditions?
If I administer a test at diff times of day or someone else administering it do you get the same results?
What is responsiveness?
To what degree does a measure capture change?
What is clinical utility?
How useful is a measure for informing practice for different types of clients?
Are measures culturally useful and appropriate
Why do we use statistics?
Statistics allow us to use the sample to make, justify and qualify inferences about the population.
Also allow us to identify when patterns in the sample are likely due to real relationships/differences between variables.
What 3 parts of a study do statistics help researchers with?
Study design
Study execution
Study results
How does statistics help evidence based practitioners?
Understanding research evidence
Evaluating research evidence
Applying research evidence
What is inferential statistics?
Use the sample data to make inferences about the population.
Using the subset to learn about the whole.
- Prove or disprove a theory/hypothesis
- Determine relationships or association between variables.
What are descriptive statistics?
Summarize the main features of the sample data.
Describe the state of the independent, dependent & extraneous variable(s) in the sample and/or sample groups at particular time points in a study.
Helps you understand who the participants are.
What is another name for categorical data?
Discrete
How is frequency represented and what is it?
n = frequency count
% = proportion
Frequency = total # of occurrences
What is relative frequency?
RF = total # of occurrences/total # of possible occurrences
Get a decimal which you can then multiply into a percent if needed.
Variable levels are MUTUALLY EXCLUSIVE
What is a bar chart used for?
Used for frequency
Explain the set up of a bar chart
Categorical labels on x axis
Numerical labels on y axis
Bars typically separated by space
Bar charts are for categorical data!
What is binning?
In continuous data the bins need to be the same size with no gaps in between.
Data is binned into discrete categories.
bins are mutually exclusive. Total frequency = total # of participants
bins are of equal numerical size with no gaps in between.
E.g., age might have categories of 40-44, 45-49, 50-54, 55-59 etc.
What are histograms used for and how are they set up?
Used to summarize continuous data (frequency distribution)
Numerical label on x axis
Numerical label on y axis
Bars typically not separated by space
What is the best measure of central tendency for ordinal data?
Median
Median is best b/c an assumption is made when calculating mean that it’s continuous and space between points is equal. And that’s not the case.
When is mode not used?
Not often used with continuous data, but is useful in summarizing categorical data.
What do outliers do?
Skew the frequency distribution
Why is mean useful with continuous data?
Most precise
Most sensitive to change
Point estimate for population mean in a normal distribution
Sample mean in normal distribution is the best option of what the mean would look like in the population.
Where is mean, median, mode in a negative skew?
Tail is to the left (negative side). Mode is highest peak but mean is more to the left than median b/c mean gets pulled.
Where is mean, median, mode in a positive skew?
Tail is to the right (positive side). Mode is highest peak but mean is more to the right than median b/c mean gets pulled.
What is range and how is it calculated?
Measure of dispersion
Length of the interval that contains all the data.
Range = highest data point - lowest data point
- simple to calculate
- misleading when there are outliers or when data sets are very small
What is dispersion?
Summarizes data set by describing how widely spread out it is.
What is interquartile range?
Measure of dispersion
Range of the middle 50% of data set sorted by ascending numerical order.
= interval from 25th percentile to 75th percentile.
order data set, find median, find centre of upper and lower halves of data set.
More robust to outliers and skewness than range
Shown usually as IQR = Q1-Q3 (not a minus just the range).
if shown as Q3-Q1 will see just one value.
Explain a box and whisker plot
Measure of dispersion
Box covers middle 50%. Edges are at Q1 and Q3. length of the box is IQR. The dotted lines (whiskers), extend out from the box to indicate how far the data extends out past that IQR. Don’t typically expand further than 1.5 x IQR then beyond that is an outlier.
In the box is a line representing the median
What is variance and how is it calculated?
Measure of dispersion
Average of the squared difference of each data point from the sample mean.
Variance = Sum of differences between the data points and sample mean, squared divided by total # of data points
SUM the squares
Squaring makes the differences positive to avoid canceling
Final number isn’t meaningful to us. We use it to get to SD.
What is standard deviation and how is it calculated?
Measure of dispersion
Average of the absolute differences of each data point from the sample mean.
Standard deviation = square root of variance
- Most robust and accurate measure of dispersion because it uses all the data points
- Expressed in the same units as the data
What percentages fall in 1-3 SD’s?
approx 68% fall within 1 SD of the population mean
approx 95% fall within 2 SD of the population mean
approx 99% fall within 3 SD of the population mean
What is z-score?
= number of population standard deviations away from population mean.
useful as a standard score that applies across variables.
What 2 ways is categorical data usually summarized?
Frequency and proportion
What 3 ways is continuous data usually summarized?
Frequency
Measures of central tendency
Measures of dispersion
What is ready-reference searching?
Is typically very specific and usually resolved through a closed-ended search process.
o E.g., an OT wanting to know the prevalence, etiology, prognosis and functional implications of a disease
o The information retrieval task is to identify an appropriate info source and to access the info
What is subject searching?
An iterative and ongoing information-seeking process that involves successive steps to arrive at info that is as close as possible match to the info being sought.
What are information queries?
Words or phrases that re entered into the interface of online information-retrieval tools.
What is a peer reviewed journal and a peer reviewed journal article?
Journal in which submitted articles are rigorously evaluated by members of an editorial board who have expertise in research methods and in the topic area under study
Articles are published research studies, review articles, and other commentaries that have met with sufficient approval from the members of the editorial review board of a journal to be included in the journal publication.
Structured vs Serendipitous Browsing
o Structured browsing involves a preplanned organization of a list of topics. This approach follows a directed path to the information sought
o Serendipitous browsing is a more random process. Nonlinear path of info.
What is querying?
Involves seeking matches to a particular text or keyword
What are the 4 steps to searching for information?
- What (am I looking for)?
- Where (can I find it)?
- How (do I access and retrieve it)?
- How well (does it satisfy my info requirements)?
What is gray literature?
Refers to info that is not published or available in usual formats such as journals. E.g., abstracts of conference papers, dissertation or unpublished thesis reports.
What is the goal of quantitative research?
To discover cause and effect relationships by comparing 2 or more individuals or groups based on differing outcomes associated with exposure or interventions.
What are the study designs in the pyramid?
Bottom to top:
Animal and lab studies
Case series and case reports
Case control studies
Cohort studies
RCT’s
Systematic reviews
Meta-analysis
What is a meta-analysis?
A statistical technique that summarizes the results of several studies in a single weighted estimate, in which more weight is given to results of studies with more events and sometimes to studies of higher quality
What is a systematic review?
A review in which specified and appropriate methods have been used to identify, appraise, and summaries studies addressing a defined question. (It can, but need not, involve meta-analysis). In clinical Evidence, the term systematic review refers to a systematic review of RCTs unless specified otherwise
What is a controlled clinical trial?
A trial in which participants are assigned to 2 or more different treatment groups. In Clinical Evidence, we use the term to refer to controlled trials in which treatment is assigned by a method other than random allocation. When the method of allocation is by random selection, the study is referred to as a RCT. Non-randomized controlled trials are more likely to suffer from bias than RCTs.
Which type of cohort studies are more reliable?
Prospective
What is a case control study?
A study design that examines a group of people who have experienced an event (usually an adverse event) and a group of people who have not experienced the same event, and looks at how exposure to suspect (usually noxious) agents differed between the 2 groups. This type of study design is most useful for trying to ascertain the cause of rare events, such as rare cancers.
What is a case series?
Analysis of series of people with the disease (there is no comparison group in case series).
Deductive processes
When investigators make use of existing concepts and knowledge, including wisdom received from others while growing up as human beings and from professional training.
Inductive processes
Investigators keep their eyes open to see new ways and build up new ideas
What is a limitation to critical theory?
Individual is not conceptualized as a willful, autonomous person. Individuals are viewed as bound by social rules or norms; they are defined simply as the sum of family, work and community roles.
What 4 aspects of human experience need to be explored in a phenomenological study?
- The lived space (spatiality)
- The lived body (corporeal embodied experiences)
- Lived social relationships (relationality)
- Lived time (temporality)
What are the 4 main methods for gathering qualitative data?
- Participation in the setting
- Direct observation
- In-depth interviewing
- Analyzing written documents and material objects
What is another name for social desirability bias?
Hawthorne effect
Threatens validity
What is a key-informant interview?
- In one form, “elite” interviews are conducted with individuals to be considered to be influential, prominent, and/or well-informed people in an organization or community.
- Key informants (elites) are specifically selected for interview on the basis of their expertise in areas relevant to the research
What is triangulation and what does it prove?
- When multiple methods of data collection are used in the same study, this is called triangulation
- This is one way of increasing the rigor of qualitative study
- Refers to the use of 2 or more strategies to collect and/or interpret or analyze information
- Purpose is to validate a particular finding
When triangulation between observers is done might see peer debriefing
What are project-based methods?
Project-based methods include daily operational procedures that help to minimize errors in all processes related to data collection, data storage, and data management.
What do audit serve to do?
Serves as a means of reliability checking for both the procedures and conclusions of a study
What is qualitative analysis?
Qualitative analysis is about understanding meanings, process, people, and their thoughts and actions through the interpretation of people’s words
What is inscribing social disclosure?
Inscribing social disclosure is a deliberate act to ensure that interactional details (e.g., in a semi-structured interview) are fully captured
What are the 4 steps in qualitative data analytic process?
- Perform a formal analysis or interpretation of one data source.
- Select a particular code for further analysis
- Compare and contrast data sources using a particular code (same code from step 2)
- Draw some general conclusions about what a coding strategy or arising data pattern means.
What is rigour?
This is the systematic approach and techniques used to ensure reliability and validity of the study.
What is trustworthiness?
Trustworthiness is concerned with how the researchers establish the study findings as credible, transferable and confirmable.
What is a two-group post-test only randomized experiment?
- Two-group post-test only randomized experiment is the most basic experimental design in which participants are randomly assigned to either a treatment group or the no-treatment control group
o Both groups are assessed following the treatment to see if they differ on the desired outcome.
o There is no pretest assessment in this design b/c the random assignment to groups renders the groups probabilistically equivalent (i.e., although the 2 groups are not equal in terms of all variables, any differences between the 2 groups are likely based on chance rather than on some systematic pattern of variation).
What is a two-group pretest post test randomized experiment?
- The two-group pretest-post-test randomized experiment is nearly identical to the other method, but there is the addition of a pretest before the treatment is offered
o Addition of a pretest introduces greater control within the experiment in that it allows for the measurement of change within the 2 groups of participants
o Further demonstrates that any observed difference between groups did not occur independently without the treatment and was not attributable to error, such as unanticipated difference between participants in the 2 conditions.
What are the 4 types of control groups? Explain
- Placebo - led to believe they are receiving a treatment of significant benefit
- Usual-care - continue to receive what is considered the usual standard of care for that impairment.
- Delayed-treatment - receive the treatment after a period of time after the treatment group has already received it.
- No-treatment - participants are naive to any treatment for their impairment or have independently withdrawn from or have deliberately been withdrawn from treatment for purposes of the study
When are Usual-care, delayed-treatment, and no-treatment control conditions used?
Often utilized in quasi-experimental designs, in which control groups still exist , but participants are not randomly assigned to a condition and so are more likely to be aware that they are not receiving the experimental treatment.
What is a quasi-experimental design?
- Quasi-experimental designs utilize random selection or other methods of non-random assignment to groups based on convenience sampling, in which participants may be fully aware of the group to which they are being assigned, and researchers estimate that groups are equivalent in terms of key features, but they cannot be as sure as if the groups were determined by random assignment.
- Quasi-experimental designs allow for ethical concerns about feasibility of implementation to be addressed
Explain nonexperimental designs
- Include studies that are purely descriptive or observational in which variables that are thought to be associated with one another are analyzed statistically
o Aka correlational b/c the central question concerns an association between variables that may not be causal or ordered in a specific way - Typically lacks manipulation of an independent variable and only relies on measurement of information
What is fidelity?
A study is said to have fidelity if there is documentation that the IV is administered just as planned.
What is bleeding?
A participant assigned to one condition actually experiences some or all the comparison condition
What is responsiveness?
(aka sensitivity) is the degree to which a dependent variable shows appropriately small but meaningful increments of change over time.
o A 3-point scale might not show progress to show on the measure
What are the 2 non-responsive measures?
- A ceiling effect occurs where the sample’s scores are already so high on the scale that little improvement is possible.
o A floor effect occurs when the opposite is the case
What is a nondirectional hypothesis?
Does not predict which version of the IV is better, just that one is better (superior).
What type of data is a categorical scale referring to?
Nominal - 2 or more categories
Dichotomous is 2
What is sampling error?
The extent to which an observed difference (or improvement) occurs by chance is referred to as sampling error
What is a p value?
In stats, the probability of a true difference or improvement is represented as a p value. The p value is used as a cutoff point to determine the extent to which a conclusion about a sample reflects a true difference within the larger population, versus the conclusion being due to error or chance
P-value is the probability that the results you have could have happened by chance.
What is hypothesis testing?
The process by which outcomes or significant differences found in a study reflect true outcome or are due to chance or error.
Is there enough strength of evidence to be able to reject the conclusion that the observed outcomes/differences are due to error/chance.
What is a null hypothesis?
States that an observed difference or outcome is due to chance. Point is to reject the null.
What is the alternative/research hypothesis?
Specifies a relationship between variables that is not due to chance. Relationship may be directional or nondirectional.
What is parametric statistics?
Parametric statistics is a branch of stats that assumes that sample data come from a population that follows a probability distribution based on a fixed set of parameters.
- Parametric statistical tests may be used provided that the following criteria are met:
o The dependent variable(s) being measured are continuous (i.e., use an interval or ratio scale); and
o The sample size is large enough (N>30) so that assumptions about normality and variance are likely to be met
What is a related-samples test?
When a statistical test is being performed on variables from the same sample group, it’s referred to as a related-samples test.
What is an independent-samples test?
When a statistical test is being performed on variables from 2 or more different sample groups, it’s classified as an independent-samples test.
AKA between-subjects test
Negative vs positive correlation
- A positive relationship means that the 2 variables increase or decrease in tandem.
- A perfect negative relationship means that as one variable increases, the other decreases
Explain the values of correlation (Pearson correlation) and their meaning
o R = 0 to +/- 0.20 suggests a negligible relationship between the two variables
o R = +/ 0.20 to +/- 0.40 suggests a low correlation
o R = +/- 0.40 to +/- 0.60 suggests a moderate correlation
o R = +/- 0.60 to +/- 0.80 suggests a high correlation
o R = +/- 0.80 to +/- 1.00 suggests a very strong relationship
What is an independent samples t test?
Independent-samples t-tests compare the means from 2 independent groups or samples
What is a paired-samples t test?
- Paired-samples t-tests compare the means from a single group of clients, typically at 2 points in time
- Also denoted as a t
Can be the same individual but at 2 points in time
What is a one way ANOVA?
A one-way analysis of variance (ANOVA) allows you to test differences between 3 or more means (or levels) of a single independent variable.
- In a one-way ANOVA, scores for each level of the independent variable on some anticipated outcome are reflected on values for the dependent variable
- The term one-way refers to the fact that this test allows you to only test a single independent variable.
- The statistic that denotes whether the means from different levels are equal or different is denoted as F.
What are nonparametric statistics?
- Nonparametric statistics and statistical tests are used when assumptions about normality and homogeneity of variance are not met.
- Generally this is the case when:
o The dependent variable(s) is nominal or ordinal and
o The sample size is small (N<30)
What is confidence interval telling you?
An x% confidence interval is interpreted as indicating a range within we can be x% certain that the true effect lies. Larger studies tend to give more precise estimates of effects (hence have narrower Cis) than larger studies. With precision see narrower CIs.
What is power telling you?
Power is the ability of a study to demonstrate that there is a difference if one exists
Determined by several factors: magnitude of observed differences (effect size), variance - power increases as variance with data is reduced and sample size (n).
Need a large sample size to get greater power.
What is the difference between a research question and a hypothesis?
RQ:
- Specific and explicit
- Describe sample, design, intervention, how the outcome will be measured, over what period of time.
Hypothesis:
- State null hypothesis based on formulated research question.
What is a hypothesis and its 3 types?
A hypothesis predicts the direction of the relationship between 2+ variables of interest.
Directional - predicts the path and direction of the relationship
Nondirectional - predicts the path but not the direction.
Null (statistical) - predicts there is no relationship between or among the variables under study
We seek to disprove the null
Path = association/relationship
What are the 4 steps to hypothesis testing?A
- State the hypothesis
- Set the criteria for a decision
- Specify and compute the statistic
- Make a decision
What do we do with the null?
Assume it’s true unless we can prove otherwise
What is the alpha criterion?
Level of significance
Usually set at 5%, lower makes it harder to reject the null.
This is chosen/stated before hand. This helps you determine sample size needed for study. If alpha criterion is set at 1% will probably need a larger sample. Most researchers set it at 5%. P value tells us if we’ve met that %. Can report the p value to show where your data actually lies so might set alpha a little higher, results can be better than that.
What is a test statistic?
Test statistics allow us to determine how likely the sample outcome is at accepting (i.e., no difference) or rejecting (i.e., a difference exists and in what direction) the null
It’s the method used to give you a result and whether or not that result is significant or not.
- Begin with dataset (sample)
- Test data for (normal) distribution
- Distribution –> statistical test –> test statistic
- Reject the null if the value of the test statistic is less than the alpha criterion
If normal run parametric tests
What are the 2 possible decisions with a hypothesis?
- Fail to reject H0, fail to accept Ha
- Done if insufficient evidence to reject the null - Reject H0, accept HA.
- Done when there is sufficient evidence to reject the null
What is a type 1 error?
Reject H0 when H0 is true
- We conclude that an intervention is of benefit when it’s not
What is type 2 error?
Fail to reject H0 when HA is true
- We conclude that an intervention did not show benefit when it really does work
- Could be a result of a small sample size
How is power calculated?
1 minus beta
What are the 3 things statistical tests look at?
- Differences between groups
- Intervention group mean is different than control group mean - Differences within a group
- Association between variables
What is a chi-square test used for?
Non continuous data
Like a t test but for nominal or ordinal data.
What is an independent samples t-test (2 sample t test)?
- Analysis of 2 means
- Data are continuous
- Data re normally distributed
- 2 samples are independent
What is an analysis of variance (ANOVA)?
- Overall comparison between at least 3 independent sample (group) means
- Inferences about means are made by examining the variance of the samples
- Will tell you if there is a difference but won’t tell you which groups are different from each other. Gives you more of a superficial level of info.
What is a confidence interval?
- Range of values from your sample that has a high probability of containing the true population value
- Tells you precision and location
- Length changes with changing SD and sample size
- Typically the critical value to determine level of convince is 95%
What is correlation?
Measures the degree of association between 2 continuous variables (including direction and strength).
Can’t predict one from another
What is regression?
Helps us describe the relationship between variables more accurately
How is independent variable (x) associated with dependent variable (y)
With regression you can make the predictive analysis.
Dependent variable on one side of equation
Independent + other variables on the other side of the equation
Allows you to consider the association between multiple variables to understand their influence together on the dependent variable
Regression for continuous dependent
Logistic regression for nominal data (get an odds ratio out of it)
Can use regression to predict things. Correlation only looks at 1 comparison, regression is multiple.
What is an odds ration?
Outcome variable is dichotomous or categorical
Put ppl in a category and determine an odds ratio for that. E.g., % of ppl employed or unemployed. Then you can try to determine factors associated with that
What is effect size?
Standardized mean difference and effect size are used interchangeably
Looking at difference between experimental mean and control mean and normalized/standardized to the variant.
P-value tells us a difference exists that didn’t occur by chance; Effect size tells us more about how big the difference actually is.
A way to standardize the way that we measure differences between groups in inferential statistics
Effect size tells you how meaningful the relationship between variables or the difference between groups is. It indicates the practical significance of a research outcome. A large effect size means that a research finding has practical significance, while a small effect size indicates limited practical applications.
What do effect size values mean and how is it calculated?
0.2 = small
0.5 = medium
0.8 = large
How large is the difference between means, taking into account the variance?
What are the y and x axis on a normal distribution curve?
Like the histogram, the X axis is in theory in the same unit as the variable but is not binned so that we have the nice smooth curve that you see here
The Y axis captures Probability Density = likelihood of observing a value on the x axis in the population
What does alpha predict?
Probability of type 1 error (reject the null but the null is true - false positive)
What is the relationship between p value and sample size?
P values get smaller as sample gets bigger
What are t tests always about?
Comparing means
What usually dictates causation?
Causation is controlled by study design, not statistics.
What are the 3 types of mixed methods research questions?
Questions that focus on
- How much and how
- Quantity and meaning
- Perceptions and change
Mixed method vs multimethod
Mixed methods is quant and qual combined. Multimethod could be two different forms of quant or qual.
Mixed method is intentional combination of qual or quant
Typology of mixed methods designs
The tradition in CAPS is the one taking priority
E.g. QUAL + quan is giving qualitative research priority
Sequential design is indicated by –>
First one is used to guide the second method.
Concurrent design is indicated by +
Transformative – implies the goal is to change something (not just understand)
Generally, the typology is related to data collection.
What is nested design?
Nested – always concurrent. E.g., nested quantitative inside qualitative – one circle is inside the other, taking account of the quant data within a larger qual approach.
Nested is about sampling – if you did a big RCT on something you might sample from within the sample a subgroup of ppl who experienced the intervention on how they experience it. Can still be sequential even though it’s nested.
How can you limit mixed method data sets?
- Consider methodological integrity
- Converting Data to allow integration
Somewhere in the paper, they should be integrating the results across qual and quant.
What is the MMAT?
Mixed methods appraisal tool.
What are the screening questions for MMAT?
- Are there clear research questions?
- Do the collected data allow to address the research question?
What is the critical appraisal tool for qualitative?
CASP - Critical appraisal skills programme
What is an example of a quantitative critical appraisal tool and some categories?
Used the Dartmouth in class
- Patient follow-up
- Randomization
- Intention to treat analysis
- Similar baseline characteristics of patients
- Blinding
- Equal treatment
- Conflict of interest
What are the 4 principles of mixed methods designs?
- Recognize the theoretical drive of the project
- Adhere to the methodological assumptions of the core
- Recognize the role of the imported models to the project
- Work with as few data sets as possible
What are the 2 theoretical drives of a mixed methods project?
Discovery or testing
Explanatory vs Exploratory
In mixed methods
Exploratory = primarily qualitative
Explanatory = core method is quantitative
What’s the core method in sequential transformative design?
Either one can be
Sample size and how narrow the CI is?
Larger studies provide more accurate estimates effects which means they are narrower.
What does the coefficient of variation do?
Helps us compare variation between measures with different units.
