Research design

Question 1

Qualities of a research question

Answer 1

Specific: with respect to time/place/subjects/condition as appropriate

Answerable: such that the relevant data are available or able to be collected

Novel: in some sense so that the study either makes a contribution to knowledge or extends existing knowledge

Relevant: to current medicine

Question 2

Types of studies

Answer 2

• Interventional vs observational
• Time-course: prospective; retrospective; cross-sectional
• Source of data: new data; routine data; patient notes; existing data, e.g. secondary data analysis, meta-analysis

Question 3

Historical controls

Answer 3

• The comparison of the treatment group and the control group is not concurrent and may be problematic as other factors change over time, such as hospital staff and patient mix
• Interpretation is difficult – it is impossible to be sure that any differences observed between the new treatment group and the control group are solely due to the treatments received

Question 4

Approach for observational studies

Answer 4

• Collect as much data as possible on the subjects’ key characteristics.
• Use statistical analysis to adjust for these differences.
• Note that, even with statistical adjustment, there may still be differences between the groups that are unknown and so comparisons may still be biased. We probably won’t know.
• Interpretation of non-randomized trials is difficult and firm conclusions are hard to draw.

Question 5

Describe an RCT

Answer 5

A randomized controlled trial (RCT) is an intervention study in which subjects are randomly allocated to treatment options.

Randomized controlled trials (RCTs) are the accepted ‘gold standard’ of individual research studies. They provide sound evidence about treatment efficacy which is only bettered when several RCTs are pooled in a meta-analysis.

Question 6

Choice of the control group

Answer 6

When a best therapy currently exists this should be used as the control

It is unethical to randomise to placebo when a current treatment exists (Declaration of Helsinki, item 32)

When comparing an agent to placebo you are more likely to find a beneficial effect versus compared to best current therapy

Question 7

Declaration of Helsinki Item 32

Answer 7

‘The benefits, risks, burdens and effectiveness of a new intervention must be tested against those of the best current proven intervention, except in the following circumstances:
• The use of placebo, or no treatment, is acceptable in studies where no current proven intervention exists; or

• Where for compelling and scientifically sound methodological reasons the use of placebo is necessary to determine the efficacy or safety of an intervention and the patients who receive placebo or no treatment will not be subject to any risk of serious or irreversible harm’

Question 8

Declration of Helsinki

Answer 8

Developed in 1964 by World Medical Association (WMA)

Outlines principles for:
• Duties of those conducting research involving humans

• Importance of a research protocol
• Research involving disadvantaged or vulnerable persons
• Considering risks and benefits
• Importance of informed consent
• Maintaining confidentiality
• Informing participants of the research findings

Question 9

Why randomise participants?

Answer 9

Randomization ensures that the subjects’ characteristics do not affect which treatment they receive. The allocation to treatment is unbiased

In this way, the treatment groups are balanced by subject characteristics in the long run and differences between the groups in the trial outcome can be attributed as being caused by the treatments alone

This provides a fair test of efficacy for the treatments, which is not confounded by patient characteristics

Randomization makes blindness possible

Question 10

Minimization

Answer 10

Minimization is another method of allocating subjects to treatment groups while allowing for important prognostic factors.

The allocation takes place in a way that best maintains balance in these factors. At all stages of recruitment, the next patient is allocated to that treatment which minimizes the overall imbalance in prognostic factors

Question 11

Blocking

Answer 11

Blocking is used to ensure that the number of subjects in each group is very similar at any time during the trial.

The random allocation is determined in discrete groups or blocks so that within each block there are equal numbers of subjects allocated to each treatment.

Question 12

Consenting in research

Answer 12

Declaration of Helsinki item 24

Adequately informed of the aims, methods, sources of funding, any possible conflicts of interest, institutional affiliations of the researcher, the anticipated benefits and potential risks of the study and the discomfort it may entail

Question 13

Steps in informed consent

Answer 13

• This requires giving patients detailed description of the study aims, what participation is required, and any risks they may be exposed to
• Consent must be voluntary
• Consent is confirmed in writing and a cooling off period is provided to allow subjects to change their minds
• Consent must be obtained for all patients recruited to an RCT
• Giving or withholding consent must not affect patient treatment or access to services
• For questionnaire surveys, consent is often implicit if the subject returns the questionnaire where it is clear in the accompanying information that participation is voluntary
• Consent may not be required if the study involves anonymised analyses of patient data only

Question 14

Qualities of a placebo

Answer 14

• An inert treatment that is indistinguishable from the active treatment
• In drug trials it is often possible to use a placebo drug for the control which looks and tastes exactly like the active drug
• The use of a placebo makes it possible for both the subject and assessor to be blinded

Question 15

Qualities of a parallel group RCT

Answer 15

• This is a trial with a head-to-head comparison of two or more treatments
• Subjects are allocated at random to a single treatment or a single treatment programme for the duration of the trial
• Usually, the aim is to allocate equal numbers to each trial, although unequal allocation is possible
• The groups are independent of each other, UNLIKE a crossover trial

Question 16

Crossover trials

Answer 16

• This involves a single group study where each patient receives two or more treatments in turn
• Each patient therefore acts as their own control and comparisons of treatments are made within patients
• The two or more treatments are given to each patient in random order
• Crossover trials are useful for chronic conditions such as pain relief in long-term illness or the control of high blood pressure where the outcome can be assessed relatively quickly
• They may not be feasible for treatments for short-term illnesses or acute conditions that once treated are cured, for example antibiotics for infections
• It is important to avoid the carry-over effect of one treatment into the period in which the next treatment is allocated. This is usually achieved by having a gap or washout period between treatments to prevent there being any carry-over effects of the first treatment when the next treatment starts
• The simplest design is a two treatment comparison in which each patient receives each of the two treatments in random order with a washout period of non-treatment in between
• There are some particular statistical issues that may arise in crossover trials which are related to the washout period and carry-over effects, and how and whether to include patients who do not complete both periods.

Question 17

Advantages of parallel group designs

Answer 17

• The comparison of the treatments takes place concurrently
• Can be used for any condition, especially an acute condition which is cured or self-limiting such as an infection
• No problem of carry-over effects

Question 18

Disadvantages of parallel group designs

Answer 18

• The comparison is between patients and so usually needs a bigger sample size than the equivalent cross-over trial

Question 19

Advantages of crossover designs

Answer 19

• Treatments are compared within patients and so differences between patients are accounted for explicitly
• Usually need fewer subjects than the equivalent parallel group trials
• Can be used to test treatments for chronic conditions

Question 20

Disadvantages of crossover designs

Answer 20

• Cannot be used for many acute illnesses
• Carry-over effects need to be controlled
• Likely to take longer than the equivalent parallel designs
• Statistical analysis is more complicated if subjects do not complete all periods

Question 21

Advantages of Zelen’s single randomized design

Answer 21

• It avoids patient refusal at the outset due to the possibility of their being allocated to control
• It avoids later withdrawal in subjects who initially consent but then withdraw when they are allocated to the control group
• It allows a new and potentially desirable programme to be evaluated rigorously in a randomized trial

Question 22

Disadvantages of Zelen’s single randomized design

Answer 22

• Patients in the control group do not know they are in a trial, which has ethical implications
• The design leads to three groups and will lead to bias if subjects are not analysed in the group to which they were allocated irrespective off the treatment they chose or received
• Will only work if the data required are routinely collected, otherwise no data will be available for the control group
• It is less efficient statistically than a straightforward two-group design since, when subjects choose not to accept the allocated treatment, the true treatment effect is diluted

Question 23

Advantages of Zelen’s double randomized design

Answer 23

• It randomizes patients but allows them to choose which treatment they prefer
• It avoids the ethical problems of not seeking consent for patients allocated to control
• It thus allows a new and potentially desirable programme to be evaluated rigorously in a randomized trial

Question 24

Disadvantages of Zelen’s double randomized design

Answer 24

It almost inevitably leads to severe contamination of the groups since some patients will choose the opposite treatment to which they have been allocated

• It is less efficient statistically than a straightforward two-group design since, when subjects choose not to accept the allocated treatment, the true treatment effect is diluted

Question 25

Equivalence trials

Answer 25

• Seek to test if a new treatment is similar in effectiveness to an existing treatment
• Appropriate if the new treatment has certain benefits such as fewer side effects, being easier to use, or being cheaper
• Trial is designed to be able to demonstrate that, within given acceptable limits, the two treatments are equally effective
• Equivalence is a pre-set maximum difference between treatments such that, if the observed difference is less than this, the two treatments are regarded as equivalent
• The limits of equivalence need to be set to be appropriate clinically
• The tighter the limits of equivalence are set, the larger the sample size that will be required
• If the condition under investigation is serious then tighter limits for equivalence are likely to be needed than if the condition is less serious
• The calculated sample size tends to be bigger for equivalence trials than superiority trials

Question 26

Superiority trials

Answer 26

• Seek to establish that one treatment is better than another
• When the trial is designed the sample size is set so that there is high statistical power to detect a clinically meaningful difference between the two treatments
• For such a trial a statistically significant result is interpreted as showing that one treatment is more effective than the other

Question 27

Importance in deciding superiority vs equivalence

Answer 27

A trial designed to test superiority is unlikely to be able to draw the firm conclusion that two treatments which are not significantly different can be regarded as equivalent

Question 28

Intention to treat analysis

Answer 28

• Analyse subjects in the groups they were originally allocated to even if they don’t comply or change treatment
• This provides an unbiased comparison of the treatments i.e. balanced patient characteristics post randomisation remain intact
• Per protocol analysis may be useful but only in addition to ITT and not as the primary analysis
• Keep an record of all subjects to be able to account for their treatment and for any subjects who withdraw

Question 29

Sample size for case-control studies

Answer 29

It is common to choose the sample size so that there is the same number of cases as controls. For a given total sample size this gives the greatest statistical power, i.e. the greatest possibility of detecting a true effect.

If the number of available cases is limited, then it is possible to increase the power by choosing more controls than cases However, the gain in power diminishes quickly so that it is rarely worth choosing more than 3 controls per case

Question 30

Limitations of case-control design

Answer 30

• The choice of control group affects the comparisons between cases and controls
• Exposure to risk factor data is usually collected retrospectively and may be incomplete, inaccurate, or biased
• If the process that leads to the identification of cases is related to a possible risk factor, interpretation of results will be difficult (ascertainment bias) e.g. suppose the cases are young women with high blood pressure recruited from a contraception clinic. In this situation a possible risk factor, the oral contraceptive (OC) pill, is linked to the recruitment of cases and so OC use may be more common among cases than population controls for this reason alone.

Question 31

Outcome limitations from case-control studies

Answer 31

• Time-course relationships need careful interpretation since changes in biological quantities may precede the disease or be a result of the disease itself. For example a raised serum troponin level is associated with myocardial infarction, but is only raised after the event. Therefore a case–control study may find that high troponin levels are associated with myocardial infarction but this cannot in fact be a risk factor
• Risk estimates for exposures cannot be estimated directly because the case and control groups are not representative samples of their respective target populations and so estimates of risks are biased. This has implications for the statistical analysis and the interpretation of results. Risks are usually estimated using odds and ratios of odds, and these only approximate to risks and ratios of risks when the disease under investigation is rare
• This limitation can be overcome with certain designs, for example where a case–control study is nested in a cohort study where all cases and controls are identified prospectively and a truly random sample of controls is available (Research design Cohort studies, p. [link]). In this situation, the relative risk can be calculated directly

Question 32

Requirement for retrospective cohort studies

Answer 32

Cohorts can be retrospective but requires that full risk factor data are obtained on all individuals with and without the disease of interest using data that were recorded prospectively

Question 33

Difficulties with cohort studies

Answer 33

• A large number subjects is needed to obtain enough individuals who get the disease or condition, particularly if it is uncommon
• The length of follow up may be substantial to get enough diseased individuals and so the cohort study is not feasible for rare diseases
• There is difficulty in maintaining contact with subjects, particularly if the follow-up is lengthy
• The resources required may be very high

Question 34

Advantages of case-control

Answer 34

Useful for establishing associations in:

• Rare disease
• Acute outbreaks in which there is not time for a cohort study
• Diseases that take a very long time to develop

Question 35

Nested case-control studies

Answer 35

In a cohort study it may be worthwhile to identify all individuals with a disease and then retrospectively select a sample of the non-diseased individuals for comparison. This design may be desirable if:

• The resource implications of collecting data on all non-diseased individuals is too high
• All information was available but unprocessed
• Biological samples were collected but not analysed

This study is known as a nested case–control study and provides an efficient way of investigating particular factors once the outcomes from the cohort have been established.

Bias in risk factor data

• In a nested case–control study such as this, the risk factor data should not be as biased as it may be in a conventional case–control study, since it was collected prospectively
• There is a potential problem if there is differential loss to follow-up as this would reduce the availability of true controls and bias the comparisons

Question 36

Cross-sectional studies

Answer 36

Assess outcome at one-point in time

Use when:
• Surveys of prevalence, such as a survey to ascertain the prevalence of asthma

• Surveys of attitudes or views, such as: studies of patient satisfaction, patient/professional knowledge; studies of behaviour, such as alcohol use and sexual behaviour
• When inter-relationships between variables are of interest, for example a study to determine the characteristics of heavy drinkers, a cross-sectional study allows comparisons by sex, age, and so on

CANNOT assess:
-Temporal trends/causality e.g. did a disease cause HTN or did the HTN cause a disease
Cross-sectional studies that appear to be longitudinal

-Cross-sectional studies can be misinterpreted as if they were longitudinal studies. For example, a cross-sectional study in a sample of fetuses where the gestational age of the fetuses spans a range, say 22–28 weeks. Some researchers have used data such as these to estimate growth trends. This is dubious because each fetus is measured just once and so the trend is being estimated from different fetuses. Thus differences between fetuses are likely to contribute to some of the differences observed by gestational age.

Question 37

Bradford-Hill Criteria

Answer 37

Allows causality to be more confidently inferred from observational studies:

• Strength of association
• Consistency in different studies, settings, etc.
• Specificity of association of risk factor with a particular disease
• Temporal relationship – exposure precedes disease
• Dose–response relationship
• Biological plausibility for causality
• Coherence – association is consistent with current knowledge
• Experimental evidence for causality
• Existence of analogous evidence between a similar exposure and disease

Question 38

Clinical audit

Answer 38

‘Clinical audit is a quality improvement process that seeks to improve the patient care and outcomes through systematic review of care against explicit criteria and the implementation of change. Aspects of the structures, processes and outcomes of care are selected and systematically evaluated against explicit criteria. Where indicated, changes are implemented at an individual team, or service level and further monitoring is used to confirm improvement in healthcare delivery.’1

Question 39

Secondary data

Answer 39

These are data collected and recorded for another research study, and which are available for use.

Advantages

• Relatively quick to obtain
• Usually already processed so that minimal checking and data cleaning is required
• Usually much lower cost than primary data collection

Disadvantages

• No control over data available
• Limited control over missing data and ability to fill gaps and resolve queries
• Data may not be in required or desirable format
• May be out of date

Question 40

Advantages of composite outcomes

Answer 40

• They allow several outcomes to be combined in settings where different outcomes are of similar importance but reflect different clinical events, for example in a trial of treatment for gestational diabetes, the primary outcome was a composite measure of serious perinatal complications, defined as one or more of: fetal death, shoulder dystocia, bone fracture, and nerve palsy
• Main advantage of using a composite outcome is the gain in statistical power – where individual events are uncommon, a large sample will be required to demonstrate conclusive differences. Using a composite will increase the event rate and allows trials to recruit a lower sample size.

Question 41

Disadvantages of composite outcomes

Answer 41

• It may be hard to determine the minimum clinical difference for the composite, this requires an estimate of the incidence of the composite itself and not just the incidence of the individual components as well as clinical judgement about what constitutes an important change in rate
• The interpretation of results may be difficult – it is important that the separate component effect sizes are each reported as well as the combined effect size, to allow clinical interpretation
• If the effect sizes (e.g. relative risks) vary among the components then overall interpretation of the findings is difficult, for example if a new treatment reduces subsequent adverse events but increases death rates

Question 42

Surrogate marker

Answer 42

• A surrogate outcome should be closely related to the clinical outcome of interest such as a biomarker or process variable

Question 43

Advantages and disadvantages in converting a continuous variable into categorical variable

Answer 43

e.g. cholesterol in mmol into normal vs high

Pros:

• Allows clinical applicability and aids clinical decision making
• Able to summarise data

Cons:
- leads to a loss of information which in turn has statistical consequences.
=loss of power

Question 44

Estimating a mean with a specified precision

Answer 44

The following information is required:

• The standard deviation (SD) of the measure being estimated
• The desired width of the confidence interval (d)
• The confidence level

The standard deviation is needed because the sample size depends partly on the variability of the measure being estimated. The greater the variability of a measure, the greater the number of subjects needed in the sample to estimate it precisely.

The standard deviation can be estimated from previously published studies on the same topic, from contact with another worker in the field or from a small pilot study.

The desired width of the confidence interval, d, indicates the precision of the mean and is decided by the researcher.

The confidence level is usually set at 95%, giving a sample confidence interval that contains the true population mean with probability 95%. Other values such as 90% or 99% can be used, but are unusual in practice.

n=1.962×4SD2/d2

Question 45

Estimating a proportion with a specified precision

Answer 45

The following information is required:

• The expected population proportion, p
• The desired width of the confidence interval (d)
• The confidence level

Question 46

Type 1 error:

Answer 46

We conclude that there is a difference between the groups in the target populations when in fact there is not. This is actually the significance level of the test and so when we use 0.05 or 5% as the cut-off for statistical significance, then the probability of a type 1 error is 5%. This is often denoted by ‘α’.

Question 47

Type 2 error:

Answer 47

We conclude that there is no difference between the groups in the target population when in fact a real difference of a given size does exist. The type 2 error is often denoted by ‘β’ and 1–β is the power of the study.

Question 48

Calculating sample size for power in intervention studies comparing means

Answer 48

The following information is required:

• The standard deviation (SD) of the measure being compared
• The minimum difference (d) that is clinically important
• The significance level (α)
• The power of the test (1–β)

Question 49

Calculating sample size for power in intervention studies comparing proportions

Answer 49

• The expected population proportion in group 1, P1
• The expected population proportion in group 2, P2
• The significance level (α)
• The power of the test (1–β)

The expected population proportion in group 1 and the expected population proportion in group 2 are the best estimates of what these values will be. The difference therefore reflects the anticipated change in the proportion which would be regarded as clinically important.

The significance level, α is the type 1 error and is usually set at 5%.

The power of the test, 1–β, is the probability of getting a significant result when the true difference between the proportions is d and is set at 80% or more, preferably 90%.

Question 50

Assumptions of sample size formulae for means and proportions

Answer 50

• There is no attrition, i.e. the total number of patients successfully recruited and who complete the study is equal to the number required
• For comparative studies, there are equal numbers of subjects in each group
• Samples are simple random samples; any randomization is at the individual level. Sample size calculations are different for cluster samples or cluster randomization and the usual calculations will give too few subjects (see below)
• For comparative studies, a simple comparison of two groups only will be made. Multiple regression or logistic regression (Research design Chapter 12, p. [link]) is not planned
• The samples are large enough to use large sample methods for the analysis

Question 51

When are sample size calculations not required?

Answer 51

• Qualitative (descriptive) studies

- Pilot studies

Question 52

Use of log scale in ratios

Answer 52

Odds ratio and risk ratios are usually based on log-transformed data which makes the data SYMMETRICAL!

Common feature is that ratio statistics all have an absolute lowest value of 0, which can extend up to infinity

Without log-transformed data, the number scale is not symmetric
e.g. OR of 0.5 is half risk
OR of 2 is double the risk
In the middle means equal risk but the average of 0.5 and 2 is not 1!

However, log to the base e of the above:
OR 0.5 –> -0.69
OR 2 –> 0.69
Average to 0 i.e. previous value of 1

Question 53

Log-based meta analysis

Answer 53

Graphical displays for meta-analyses performed on ratio scales usually use a log scale.

This has the effect of making the confidence intervals appear symmetric, for the same reasons as log-based odds ratios.

Question 54

Using odds ratios for effect measure

Answer 54

For interventions that increase the chances of events, the odds ratio will be larger than the risk ratio, so the misinterpretation will tend to overestimate the intervention effect, especially when events are common (with, say, risks of events more than 20%).

For interventions that reduce the chances of events, the odds ratio will be smaller than the risk ratio, so that, again, misinterpretation overestimates the effect of the intervention. This error in interpretation is unfortunately quite common in published reports of individual studies and systematic reviews.

Question 55

Problems with using risk ratios for intervention effects

Answer 55

The risk difference is naturally constrained (like the risk ratio), which may create difficulties when applying results to other patient groups and settings. For example, if a study or meta-analysis estimates a risk difference of –0.1 (or –10%), then for a group with an initial risk of, say, 7% the outcome will have an impossible estimated negative probability of –3%. Similar scenarios for increases in risk occur at the other end of the scale. Such problems can arise only when the results are applied to populations with different risks from those observed in the studies.

Question 56

Standardized mean difference

Answer 56

Used when outcome was measured in different ways

Necessary to standardize the results of the studies to a uniform scale before they can be combined. The SMD expresses the size of the intervention effect in each study relative to the between-participant variability in outcome measurements observed in that study.

Question 57

Time to event: censored

Answer 57

Patients who contribute to some period of time but did not results in an event are said to be censored

Question 58

Time to event analysis

Answer 58

For each participant, two factors are important:

1. Time that event did not happen
2. Whether the end-point was due to an event occurring or just end of observation

It is not appropriate to analyse time-to-event data using methods for continuous outcomes (e.g. using mean times-to-event), as the relevant times are only known for the subset of participants who have had the event.

Question 59

Hazard ratio

Answer 59

Hazard ratio describes how many times more (or less) likely a participant is to suffer the event at a particular point in time if they receive the experimental rather than the comparator intervention.

When comparing interventions in a study or meta-analysis, a simplifying assumption is often made that the hazard ratio is constant across the follow-up period, even though hazards themselves may vary continuously. This is known as the proportional hazards assumption.

Question 60

Cluster-randomised controlled trials

Answer 60

In cluster-randomized trials, groups of individuals rather than individuals are randomized to different interventions. We say the ‘unit of allocation’ is the cluster, or the group. The groups may be, for example, schools, villages, medical practices or families. Cluster-randomized trials may be done for one of several reasons. It may be to evaluate the group effect of an intervention, for example herd-immunity of a vaccine. It may be to avoid ‘contamination’ across interventions when trial participants are managed within the same setting, for example in a trial evaluating training of clinicians in a clinic. A cluster-randomized design may be used simply for convenience.

One of the main consequences of a cluster design is that participants within any one cluster often tend to respond in a similar manner, and thus their data can no longer be assumed to be independent. It is important that the analysis of a cluster-randomized trial takes this issue into account. Unfortunately, many studies have in the past been incorrectly analysed as though the unit of allocation had been the individual participants (Eldridge et al 2008). This is often referred to as a ‘unit-of-analysis error’ (Whiting-O’Keefe et al 1984) because the unit of analysis is different from the unit of allocation. If the clustering is ignored and cluster-randomized trials are analysed as if individuals had been randomized, resulting confidence intervals will be artificially narrow and P values will be artificially small. This can result in false-positive conclusions that the intervention had an effect. In the context of a meta-analysis, studies in which clustering has been ignored will receive more weight than is appropriate.