EXITS Flashcards
External validity
External validity is the extent the results can be generalized to the target population or clinical settings
Internal validity
Internal validity is the extent conclusions from the studies can be made, based on the studies design settings and measures.
Construct Validity
The extent to which a test measures the theoretical construct the test is supposed to measure.
Content Validity
The extent to which a test fully measures all the domains of the construct of interest.
Criterion Validity
That determines the relationship of the scores on a test to a specific criterion, which is usually another validated test
Inter-rater Reliability
The extent results are consistent among different raters who are rating the same information.
How to measure inter-rater reliability
Cohen Kappa coefficient (for categorical items) Intraclass correlation coefficient (for continuous items) 0.2-0.4=slight to fair 0.4-0.6=moderate 0.6-0.8= substantial >0.8=almost perfect
Test-retest Reliability
The extent results are consistent when the test is conducted at two different times with no intervention in between.
Bias
Systematic error that arises from the from the design, conduct and analysis of a study, resulting in observed result which deviate from the truth.
What are the common selection bias for case control study
referral, non-responder, volunteer bias, incidence prevalence bias
What is Hawthorne effect
The Hawthorne effect refers to the increase in performance of individuals who are noticed, watched, and paid attention to by researchers or supervisors
What is Case Control study
Type of analytical observation study used to investigate the relationship between risk factor and outcome
The study is created by having a group of subject with the outcome and matched with a group without the outcome and the differences in previous exposure to risk factors were being investigated.
(3)Pros and (4)cons of case control study
pros: efficient, useful to study rare disease/outcome, use to study a wide range of exposure
Cans: RECALL BIAS, cannot estimate prevalence or incidence, temporal relationship can be uncertain, not efficient for rare exposure, relative risk is measured indirectly (since we cannot estimate prevalence)
Cohort study
is a form of longitudinal study that samples a group sharing a exposure to match a group that is not exposed to a risk factor. Both groups are followed up to investigate the diff likelihood of the outcome on the two groups
Pros and cons of cohort study
Pros
no selection bias
Well suited for rare exposure
Can study multiple diseases from a single exposure
Cons
Very expensive, Long time to complete, if prospective will have loss of followup
Randomised controlled trial
An intervention study in which a group of subjects with similar characteristics are randomised to received one of the several defined intervention
Factorial trial
Studying more than 1 intervention and also the effect of their interactions
Cross over trial
Study where each subject received two or more study treatment in a specific order. Each subject become his own control
Systematic Review
A type of comprehensive literature review which assess and review all the pertinent articles in the field using explicit criteria and pool the results together to answer a clinical question
Meta analysis
Quantitative assessment of a systemic review
Which involves pooling the results of independent studies together to produce an overall estimate of effect
Confidence Interval
A range of values within which the true value lies with a certain level of assurance.
Standard Error
Standard error is used to calculate the confidence interval of the result.
Standard deviation over square root sample size
Alpha level
Is the threshold at which we will accept or reject the null hypothesis.
Typically sets at 5%.
A alpha level of 5% means that the observed results had occurred by chance at most 5% of the time.
P value
It represents the likelihood the observed result has occurred by chance.
Null Hypothesis
States that any observed difference has occurred due to chance, and there is no real difference.
Effect Size
Measure of the magnitude of treatment effect.
How to measure effect size
Typically calculated using Cohen’s D, which is the (difference between the experimental and control means) divided by (standard deviation).
- 2: small
- 5: moderate
- 8: large
Or calculated using correlation coefficient
What is another name of effect size
standardized mean difference
Central Limit Theorem
States that no matter what the probability distribution of a sample is, in a huge sample size, the distribution follows a normal distribution
In a normal distribution, the mean=median=mode and the skew is zero
Odds
Number of times an event is likely to occur over number of times an event is likely not to occur
Odds Ratio
Odds of an event happening in the experimental group over
Odds of an event happening in the control group
Relative Risk
Risk of outcome in the experimental group over
Risk of outcome in the control group
Absolute Risk
Incidence rate of outcome in the group
Absolute Risk Reduction
Actual reduction in risk moving from control group to the experimental group
CER-EER
Numbers Needed to Treat
Numbers of subjects who must be treated with the intervention, compared with the control, to produce 1 beneficial outcome
1/ARR
Clinical Trial, what are the phases
Study design which aim to evaluate a treatment effect
Phase 0: Micro-dosing to assess PK
Phase 1: Testing on healthy subject, dosing regime and range, ascertain side effects
Phase 2: Testing on subjects (20-80 ppl), estimate treatment effect and tolerability
Phase 3: Clinical trial phase, test safety and effectiveness compared to alternative treatment
Phase 4: Post-market survey
Confounder
A variable that is associated with both the exposure and outcome of interest but does not lie in the causal pathway between the exposure and outcome.
How to control confounders
Restriction (exclusion criteria) Matching Stratification Randomization Statistical adjustment (stratification, multivariate regression analysis, direct standardization)
Matching
Technique used to distribute confounding factors evenly
Subjects are chosen in a way that identified confounders are evenly distributed
Stratification
i) as a form of matching
ii) as a form of statistical adjustment
i) Technique used to distribute confounding factors evenly
Subjects are chosen to ensure that identified confounding in strata are evenly distributed
ii) Statistical method that stratified the comparison groups according to the confounding variable. It then combines the effect measures in each strata to yield a summary effect measure.
Randomization
Systematic process of allocating subjects to the study groups in such a way that each subject has an equal chance of being allocated to either group
Types of randomisation
1) Simple (no constraint in allocation sequence)
2) Restricted (A sequence is generated to ensure equal ratio among the groups)
3) Block (Ensure that each group has equal numbers of participants in blocks)
4) Stratified
5) Adaptive (The chance of allocation to study group is adjusted according to the existing imbalance in baseline characteristic of the group)
Multivariate regression Analysis
A statistical method that describe the relationship between 1 dependent outcome and 2 or more independent variables.
Power
Probability of detecting a difference when a true difference exist.
Typically set at 1-beta=0.8
There is 80% probability that a true difference will be detected
What increases power
Increase sample size, effect size, alpha
dec standard deviation
Definition Correlation
Spearman Rho: Assessment of the magnitude and direction of the association between 2 variables on an interval or ratio scale which is not normally distributed
Pearson R: Assessment of the magnitude and direction of the association between 2 variables which are normally distirbuted
Ranges from +1 to -1 (zero= no relationship)
>0.8: Strong
<0.5: Weak
Represented on scatter plot
Coefficient of Determination
Measure of how much of the variation of the dependent variable can be explained by the independent variable.
(Linear and normally distributed)
r square = 0.9 x 0.9 = 0.81
81% of the variation in the dependent variable can be explained by the independent variables in the equation.
Regression
Express the relationship between 2 or more variables in the form of an equation
Assumption: There is a 1-way causal effect between explanatory variables and dependent variable.
Simple linear (1 and 1) Multiple linear (1 dependent, 2 or more independent – categorical/continuous) Simple logistic (1 categorical dependent variable with 2 values) multiple logistic(1 categorical dependent variable with 2 values)
Prevalence
Number of individuals with the disease in a population
- at a point of time (point)
- over a period of time (period)
Incidence
No. of new cases/disease over a period of time out of the population at risk
Type 1 Error
False positive
Null hypothesis is rejected when it is actually true
Type 2 Error
False negative
Null hypothesis is accepted when it is not true
Designated by beta, which is typically set at 0.2
Sensitivity
How to inc sensitivity
True positive
Number of people with the condition who will be tested positive on the test
Increased with doing other tests (for ruling out)
Specificity
True negative
Number of people without the condition who will be tested negative on the test
Increased with serial test (for ruling in)
Positive Predictive Value
Number of people tested positive on the test who actually has the condition
Negative Predictive Value
Number of people tested negative on the test who actually do not have the condition
ROC: Receiver’s operating characteristic
Sensitivity (true positive) as y-axis (vert)
1-Specificity (false positive) as x-axis (horizon)
Best option is top left hand corner of the curve
Accuracy of a test is area under the curve, ideally is 1
Likelihood Ratio Positive Test
How much more likely is a positive test to be found in a person with the condition compared to a person without the condition
Likelihood Ratio Negative Test
How much more likely is a negative test to be found in a person with the condition compared to a person without the condition
Bradford hills criteria
Group of principles in determining causal relationship
SSPECTAC + Biological gradient
Strength (strength if association)
Specificity
Plausibility (possibly with explanation)
Experiment evidence (can be done in an experiment)
Coherance (Coherence between epidemiological and laboratory findings increases the likelihood of an effect)
Temporality
Analogy (This criterion uses previous evidence of an association between a similar exposure and disease outcome to strengthen the current argument for causation.)
Consistency
biogical gradiant (dose dependant)
Allocation Concealment
Method used to prevent selection bias during intervention assignment
It involves protecting the allocation sequence before and until assignment as knowledge of the next assignment may cause selective enrollment
Blinding
Process whereby participants, accessors/health providers and even data analyst are kept unaware of the intervention allocation after the subjects have been assigned. Preventing performance and ascertainment bias.
Lack of blinding can account for 9% of observed improvement.
Placebo Effect/Nocebo Effect Placebo
Improvement experienced by subject when they are on placebo treatment, otherwise known as effect of expectation
Negative effect experienced by subject because of the negative beliefs about their treatment
Intention To Treat Protocol
Statistical analysis whereby all subjects who were randomized at the start will be included in the analysis in the group they were initially assigned to, regardless of the intervention they truly receive or whether they complete the study.
Per-Protocol Analysis
Statistical analysis whereby only subjects who complied to the protocol through to completion will be included in the final analysis.
Imputation Methods (4)
Used in intention-to-treat analysis protocol
Method used to replace missing data in order for data analysis to proceed
1.Worst/best case:
All subjects with missing data are assumed to be best/worst case
2. Hot deck:
Outcome from a similar subject will be used to replace the missing data
3. LOCF
4. Sensitivity analysis
Issues with Last observation carried forward
What is LOCF
The most recently recorded outcome measure is taken to hold for the subsequent outcome assessment.
Can lead to attrition bias esp if attrition due to side effects.
Can also underestimate treatment effect.
Sensitivity Analysis
Statistical method whereby outcomes are being recalculated under alternative assumptions to determine the impact of various factors on the outcome.
Heterogeneity
Presence of variability among studies that is more than expected by chance alone
How to measure heterogeneity
I2, Galbraith plot (type of scatter plot), forest plot, Cochran’s Q
Publication Bias
Occur when strength and direction differs between published and unpublished studies
Typically happen when small or negative studies not being published, resulting in over-representation of positive result
Method to measure publication bias
Funnel plot, trim and fill method, Fail-safe number:
How many unpublished data with zero effect size, is required to eliminate a significant overall effect size
Random Effect and Fixed Effect Models
Type of linear model used in data analysis which assume there is heterogeneity among studies and hence allow for between studies variation
Model used in data analysis whereby the assumption is that there is no heterogeneity among the studies
PLacebo
similar to specified treatment but has no treatment effect
How to account for heterogeneity (3)
use of random effect size model, subgroup analysis and sensitive analysis
What is transitivity
it is a measure of validity of making indirect comparison
There is no systemic difference between included studies
What is systemic review
a study design where the authors collect a group of study in a systemic fashion. Then it will be statistically review in a meta-analysis
What is the advantage of meta-analysis (3)
1) with a bigger sample size among different results, the results can be more precise
2) can decrease type 2 error
3) Able to see if effects are consistent across diff clinical population and to investigate whether difference in how the intervention was administered affects outcomes
4) to able to provide a conclusive evidence where smaller studies cannot
What are the weakness of metaanalysis (4)
1) sources of bias as not controlled by the method
2) A good meta analysis of a bad designed studies will still results in bad statistics
3) High cost and high effort
4) may have chance of being perceptible to publication bias
What is a network metaanalyiss
a technique for comparing 3 or more intervention simultaneously in a single analysis y combining both direct and indirect evidence across the network of studies
What is fidelity score
the term fidelity denotes how closely a set of procedures were implemented as they were supposed to have been.
What is economy analysis
Comparative analysis of alternate course of actions in terms of both their cost and consequences/effects, in an attempt to evaluate choices in resource allocation
What is QALY and DALY
measure of disease burden
QALY: Arithmetic product of number of extra life years and the measure of the quality of the life years
DALY: Expressed as number of years lost to disability = YLL + YLD
ICER
Statistic used in cost effectiveness analysis
Defined as:
Difference in cost between 2 intervention
divided by difference in their effect
Average incremental cost associated with 1 additional unit of measure of effect
Ie. 1 QALY about 20-40K (quality adjusted life years)
ICER: Incremental cost-effectiveness ratio
John Henry effect
bias introduced to an experiment when members of the control group are aware that they are being compared to the experimental group and behave differently than they typically would to compensate for their perceived disadvantage.
What are the usual level of bias (5)
1) Selection bias: Systematic differences between baseline characteristics of the groups that are compared. can be due to: Sequence generation/randomisation. Allocation concealment.
2) Performance bias. Systematic differences between groups in the care that is provided, or in exposure to factors other than the interventions of interest. can be due to: Blinding of participants and personnel. Other potential threats to validity.
3) Detection bias. Systematic differences between groups in how outcomes are determined. can be due to: Blinding of outcome assessment. Other potential threats to validity.
4) Attrition bias.
Systematic differences between groups in withdrawals from a study.
can be due to:
Incomplete outcome data
5) Reporting bias.
Systematic differences between reported and unreported findings.
can be due to:
Selective outcome reporting
objectively evaluating a RCT
objectively evaluating a RCT via Jadads method
range of 0-5
Question Yes No
- Was the study described as random? 1 0
- Was the randomization scheme described and appropriate? 1 0
- Was the study described as double-blind? 1 0
- Was the method of double blinding appropriate? (Were both the patient and the assessor appropriately blinded?) 1 0
- Was there a description of dropouts and withdrawals? 1 0
How to objective evaluate a systemic review
” A measurement tool for the ‘assessment of multiple systematic reviews’” (AMSTAR)
How to approach positive results that is not expected ?
- due to chance
- not adequately power, eg not enough sample size.
- confounding factors: eg low adherence, john henry effect
What are the instrument that evaluate quality of
1) RCT
2) Cohort
3) Case Control
4) Cross sectional
5) systemic review and meta-analysis
1) Cochrane risk of bias tool, jadad scale, Joanna Briggs institute checklist for randomised control trial
2) Cohort: Newcastle-Ottawa Scale (NOS)for cohort study, joanna Briggs institute checklist for cohort study
3) Case control: Newcastle-Ottawa Scale for case control study, joanna Briggs institute checklist for cohort study for case control
4) Cross sectional, JBI critical apprasal checklist for analytical cross sectional
5) A Measurement Tool to Assess Systematic Reviews (AMSTAR) 2, Risk of Bias in Systematic Review (ROBIS)
What is power
Power is the probability to detect the deviation of null hypothesis, should the deviation exist
What is cross sectional study
cross sectional study is a study design that analyse data collected from a sample population at a specific moment/period
Pros and cons of cross sectional study
Pros Quick and easy to conduct can identify associations betwene variables, can expand research questions Less expensive No loss to follow up
Cons
Relation between cause and effect cannot be established
Unable to provide an explanation for findings
Cannot study rare disease
Recall bias, non response bias
What is Cronbach alpha
measure of internal consistency, that is, how closely related a set of items are as a group.
what is internal consistency
Internal consistency is a measure based on the correlations between different items on the same test (or the same subscale on a larger test). It measures whether several items that propose to measure the same general construct produce similar scores.
is an assessment of how reliably survey or test items that are designed to measure the same construct actually do so.
Nested Case Control
Case-control study done in the population
of an ongoing cohort study, thus “nested”
inside the cohort study.
what is the pros and cons of nest case control
Advantages
● preserves advantages of cohort studies : The study design minimised selection bias compared with a case-control study, Recall bias was minimised compared with a case-control study
● most useful for costly analyses of specimens
● can potentially include fatal cases (as part of the cohort study), which cannot be done in case-control studies
● case and controls are taken from the same cohort
Disadvantages
● reduced precision and power due to sample of controls
● possibility of flaws in the design or implementation
What is a scoping review
a scoping review is defined as a type of research synthesis that aims to ‘map the literature on a particular topic or research area and provide an opportunity to identify key concepts; gaps in the research; and types and sources of evidence to inform practice, policymaking, and research
Rapid review
“Rapid reviews are a form of evidence synthesis that may provide more timely information for decision making compared with standard systematic reviews.” (AHRQ) The methods of conducting rapid reviews varies widely, and are typically done in less than 5 weeks
Sampling for quanlitative research
Sampling
Convenience sampling
A convenience sample simply includes the individuals who happen to be most accessible to the researcher.
Voluntary response sampling
Similar to a convenience sample, a voluntary response sample is mainly based on ease of access. Instead of the researcher choosing participants and directly contacting them, people volunteer themselves (e.g. by responding to a public online survey).
Purposive sampling
This type of sampling, also known as judgement sampling, involves the researcher using their expertise to select a sample that is most useful to the purposes of the research.
Snowball sampling
If the population is hard to access, snowball sampling can be used to recruit participants via other participants. The number of people you have access to “snowballs” as you get in contact with more people.
Pros and cons for Intention to treat analysis
Pros
preserve sample size
pragmatic more representative of real life, useful for effectiveness trial
cons
Dilution due to underestimate of treatment effect
Overestimation of side effect
Not good for biological efficacy
How to do sensitive analysis for qualitative analysis
Triangulation, through different method member checks (participant views differs from interview view) respondent validation flexibility deviant case analysis
triangulation definition in qualitative study
a qualitative research strategy to test validity through the convergence of information from different sources
Study type in qualitative research (4)
Study types ethnography phenomenology grounded theory -> data saturation then theoretical sampling Case study
What is indication bias
occurs when the risk of an adverse event is related to the indication for medication use but not the use of the medication itself
Issues with suicide study
1) ability to consent
2) confidentiality may need to be break
3) ?keeping and treating of results
4) survival bias
5) definition of suicide and self harm are hard to ascertain
when we can waise consent for study
- no more than minimal risk to patient
- if individual’s privacy is assured
- if waive will not affect the welfare of the research participant
- consent is not practical
ASSENT
“Assent” is a term used to express willingness to participate in research by persons who are by definition too young to give informed consent but are old enough to understand the proposed research in general, its expected risks and possible benefits, and the activities expected of them as subjects.
pros and cons of cluster randomisation
pro:
large sample size
less time and cost
no contamination between interventions
cons
cluster will know which allocation which introduce bias, recruitment, issues with allocation concealment
Similar social demographic within each cluster
Power is reduced
use IntraCluster Correlation between cluster, if too high, need more variation
need multistage randomization
what is a pilot study also known as
what is the most crucial thing about pilot study
feasibility study
-to make sure they can replicate the methods in the subsequent study, so that the results can be replicated
Memory test immediately after intervention should not be done because
we need to to test the sustainability of the results
what is expectancy effect
What is cointervention bias
When someone expects a given result, that expectation unconsciously affects the outcome or report of the expected result.
May need to over estimation of treatment effect
=====================
Co-interventions are additional treatments, advice or other interventions that a patient may receive, and which may affect the outcome of interest.
Whats the most important for non inferiority trial
must be powered, so that if there is any difference in effect size should there be any sig, inferiority
