RCTs

Question 1

Randomised Controlled Trials (RCTs): definition

Answer 1

Questions RCTs might answer:
o	Efficacy (in controlled clinical settings)
o	Effectiveness (in the ‘real world’)
o	Harm
o	Mediators
o	Moderators

Definition of Randomized Controlled Trials (RCTs):
o A planned intervention study in which each member of a study population has the same chance of receiving one or more experimental or control treatments
o Randomisation is the only unique feature of RCTs
o Randomisation can be achieved by any procedure that assigns people (or units) to conditions based on chance alone

Key features of RCTs: Equipoise
o Genuine uncertainty about the relative merits of the treatments being compared (Freedman, 1987); unethical otherwise
o Practically easier under certain conditions, e.g. when:
- Demand outstrips supply
- Many participants express no preference
- Lottery is expected

Conducting an RCT

1. Plan, get ethical approval, pre-register trial
2. Identify the study population
3. Take baseline measures
4. Randomly assign participants to the intervention or control group
5. Provide the intervention (or not)
6. Measure outcomes
7. Analyse, then write up, using CONSORT guidelines

Levels and types of randomization
o	Individual
o	Clusters (e.g., household, classroom, clinic, village, etc.) 
o	Weighted (e.g., 60% / 40%) 
o	Multi-arm

Question 2

Why are RCTs commonly considered the “gold standard” of study designs?

Answer 2

“Because the randomised trial, and especially the systematic review of several randomised trials, is so much more likely to inform us and so much less likely to mislead us, it has become the ‘gold standard’ for judging whether a treatment does more good than harm” (Sackett et al., 1996, p. 72)

Designers of social interventions aspire to the “gold standard” of randomized controlled trials (RCTs) largely because, through their inherently random allocation of subjects to either a treatment or control group, RCTs maximize the likelihood that any observed effects can be attributed to the intervention in question, the assumption being that the process of randomization distributes potentially confounding variables equally to both the treatment and the control group

General consensus places RCTs below only systematic reviews and meta-analyses in the hierarchy of scientific evidence employed to guide policy and practice, with effectiveness in a randomized controlled trial thought a promising gauge of effectiveness in a broader setting (Greenhalgh, 1997)

Advantages of RCTs
o Efficient for investigating causality because ‘cause’ precedes the ‘effect’
o Possible confounding factors balanced – known and unknown
o Randomisation facilitates simple statistical analysis
o Practical way to minimise several sources of bias (e.g., selection bias; ascertainment bias)

Randomisation helps ensure internal validity
o Allocation to the intervention and comparison groups should be unbiased with respect to prognosis and responsiveness to treatment; it is not determined by the investigators, the clinicians/practitioners, or the study participants

Randomisation tends to produce comparable groups
o The measured and unmeasured, known and unknown prognostic factors and other characteristics of the participants at the time of randomisation will be, on average, evenly balanced
o Thus, any threats to internal validity are evenly distributed across conditions; alternative explanations for the outcome are implausible, as these cannot be correlated with treatment condition

Statistical theories for analysing trials are based on the premise of random assignment

Randomisation provides best foundation by which a treatment effect can be estimated, and a hypothesis tested without the use of covariate information

Question 3

Criticisms of RCTs as an approach for evaluating social interventions

Answer 3

Disadvantages of RCTs
o Requires rigorous control of the allocation process
o Can be long and/or expensive
o May not be ideal for rare conditions or problems with a long latency
o Generalisability – depends who is recruited and how
o Beware the volunteer – very often not representative of the wider population
o But consider, what is the alternative? Other methods may suffer from same drawbacks (or not)

Question 4

Cluster Randomised Controlled Trials (cRCTs): purpose and challenges

Answer 4

Cluster randomised controlled trials (cRCTs) are:
o ‘[…] experiments in which intact social units, rather than independent individuals, are randomly assigned to intervention groups’ (Donner, 1998, p. 95)

Rationales for randomizing at the cluster level

1. Theory of change
2. Mode of delivery
3. Outcomes under evaluation

Bronfenbrenner’s Ecological System

1. Individual choice (References to personal choices or behaviors; e.g., personal decisions to smoke or not smoke)
   - ->
2. Interpersonal and household environments (References to social interactions, including those that occur within households, families, or close peer groups)
   - ->
3. School and work environments (References to environments and policies in school or work settings)
   - ->
4. Neighborhood/community physical and social environments (References to physical and social environments, including commercial or retail establishments, parks, playgrounds, and streets, and the social exchanges that occur in those settings)
   - ->
5. Policy inputs (Commentaries on government programs or agencies, regulations, laws, and taxes)

Clustering
o At what level do you assign participants? (e.g., individual, group, area)
o At what level do you measure outcomes?

Theory of change: Individual-level interventions
o Individual therapy
o Mentoring/tutoring
o Pharmaceutical treatment (e.g., DOLAB)

Theory of change: Environment-level interventions
o Media campaign
o Whole-school interventions
o Community organizing
o Organisational behaviour change interventions
o Physical environment or infrastructure changes

Media campaign to reduce tobacco use
o	Who delivers it?
- Celebrities
- Government agencies
- NGOs
o	Where/how?
- Internet ads
- Television and radio stations
- Billboards, advertisements on buses

Reducing social isolation among the elderly
o	Who delivers it?
- Community volunteers
o	Where/how?
- Assisted living facilities
- Community centres

Obesity screening and education on healthy dietary choices
o	Who delivers it?
- Medical staff
o	Where/how?
- GP surgeries
- Emergency departments

Reading support programme
o	Who delivers it?
- Classroom teachers
- Teaching assistants
o	Where/how?
- Schools

Evaluation of cRCTs
o What outcomes are we trying to evaluate?
- Need to think about level of intervention versus level of inference
o Risk of contamination
- Will the comparison group be compromised without cluster randomization?

Ethical issues in cRCTs
o Informed consent:
- Who consents?
- Who can ethically act as an informed decision maker?
- Is it realistic to obtain informed consent from all individuals in a cluster?
- What about assent? (for interventions that involve children; i.e., those who have not yet reached the age of majority)

Study design and statistical analysis in cRCTs: Various considerations

o Political

• Local governments (e.g., state/province, county)
• Devolved powers and local-level laws

o Geographical

• Different infrastructure
• Rural/urban

o Cultural
- Unique customs, heritage

o Demographic characteristics

• Ratio of young to old people
• Do people tend to immigrate to this region or emigrate from it?
• Education levels

o Occupational influences/environments

• Types of industries employing people (e.g., technology, agriculture, skilled/unskilled labour)
• Rate of unemployment
• Schools, hospitals, businesses

Ways to randomize
o Simple
- Generate random numbers on the computer, using clusters rather than individuals
o Matched
- Pairing clusters based on similar characteristics and allocating one cluster of the pair to the intervention and the other to control
o Stratification
- Clusters are divided into different strata (separate groups) and a random sample is selected from each one
o Multi-staged
- Two or more stages of sampling occur, sampling first from the higher level of clustering and then at the lower level

Stratified, multi-staged randomization: England

1. Stratify country into geographical REGIONS
2. Randomly sample COUNTIES within each REGION
3. Randomly sample CITIES within each COUNTY

Considerations
o Degree to which individuals within a cluster are alike
o Number of clusters (total sample size)
o Sample size of each cluster (effective sample size)

What is the effective sample size (ESS)?
o When individuals within a cluster are alike, we must adjust the total sample size (TSS) to account for their relatedness
–> TSS = m*k, where: m = the sample size per cluster, and k = the number of clusters
o So, to adjust for relatedness, we divide the TSS by the design effect (DE)
o The more clusters you have, the better for your power
o If you are limited in the number of clusters available, then the more people per cluster the better

Beware – attrition possible at multiple levels
o Just as we want to keep all (or as many as possible) individuals in a trial, we also want to keep all clusters in a trial
- Risk of individuals dropping out within clusters
- Risk of clusters dropping out of study
o Need to consider why they dropped out
- Systematic or random?
- What if doing a matched pair design?

Question 5

What can randomized trials tell us about taking evidence to scale?

Answer 5

Taking evidence ‘to scale’: what does this mean?
o Widespread implementation (across services…high uptake, reaching much of target population…)
o Dissemination – broad
o Diffusion of an evidence-based intervention
o Efficacy, Effectiveness, Diffusion, Population-level approaches (aim to influence/change rate of problem in whole population)

Going to scale – why important?
o Growing number of evidence-based interventions
o Evidence-based interventions often have firm science base – studies of prevalence, theory + risk factors, mediators, effectiveness
o But what use if effective interventions don’t reach large parts of the population?
o Need for impact at level of public health / wellbeing – potential for prevention of social problems
o Growing demand from donor organizations (e.g., USAID) and governments for effective programming

What is needed before scale-up? (Society for Prevention Research (SPR) Standards of Evidence (New Generation) – Gottfredson et al., 2015)
o Intervention must meet criteria for “effectiveness” AND have:
- Clear cost information (set-up costs and ongoing costs)
- Materials with information on activities and optimal methods of delivery (manuals)
- Training and supervision processes (from facilitator – coach – trainer)
- Technical assistance (project preparation, implementation, Monitoring and Evaluation – M/E)
- Fidelity tools
- Systems for documenting adaptations
- Monitoring systems with feedback loops
- System to support planning and monitoring of client recruitment
- [Scale up efforts should be rigorously evaluated]

Key considerations
o Macro-level:
- Buy-in from governments etc. for sustainability (understand constraints; current programming)
- Delivery systems (social services, education)
- Stability of targeted site (conflict zone?)
- Capacity of training partners to support scale-up

o Micro-level

• Achieving and maintaining fidelity
• -> Addressing a diversity of needs (context, severity, delivery method) – surface/substantial adaptations – are core components still intact?
• -> Ongoing supervision to maintain fidelity
• Recruitment and engagement (provision of incentives – vouchers, meals, transport)

Four challenges in meeting the demand for tested and scalable programmes to reduce violence against children

1. Evidence is expensive:
   a. Establishing evidence requires substantial time and resources to meet international standards of effectiveness
2. Thresholds for evidence differ:
   a. Implementing agencies often require different thresholds for evidence before scaling up that are less scientifically rigorous than WHO-Inspire or Blueprints
3. Adaptation may be required:
   a. Implementation across contexts may require adaptation to fit local cultures and to integrate within existing delivery systems
4. Considerations beyond delivery:
   a. Beyond delivery there are other components organizations need in order to deliver a program, including budget, fidelity, M/E and accreditation

Cartwright and Munro introduce some concerns about how to interpret RCTs, arguing that they are by themselves typically “insufficient to meet the needs of policy or practice decision makers” (Cartwright and Munro, 2010, p. 265, as cited in Cowen et al., 2017)

“The primary result of an RCT is a judgment about the effectiveness of a treatment in producing an outcome in the study population, although the contribution of other factors to the outcome is unknown. The purpose of RCTs in the context of EBP is to help estimate the effectiveness of a treatment in a target population or populations different from the one on which the experiment was conducted, which raises questions about whether and when such inferences are warranted” (Cowen et al., 2017)

Cartwright and Munro, 2010 (p. 262) trace three kinds of causal claims (as cited in Cowen et al., 2017):
1. It-works-somewhere claims: treatment T causes outcome O somewhere, under some conditions (e.g., in study population X, administered by method M)

2. Capacity claims: T has a (relatively) stable capacity to promote O, so that it can be expected to do so widely
3. It-will-work-for-us claims: T would cause O in population Q if administered as directed by policy P (i.e., effectiveness claims)

“RCTs are immediately relevant for estimating the first type of causal claim, since they tell us whether (or the degree to which) the intervention influenced the targeted outcome in the population enrolled in the experiment”

“RCTs’ relevance for the other two types of causal claims is indirect and incomplete. Whether a treatment can be assumed to be relevant to a given untreated population depends on a fabric of other knowledge”
o “The second type of claim requires enough theoretical, empirical, and conceptual knowledge to support the claim that what happens in one or a handful of study settings will happen widely”
o “For the third type of claim, there must be good reason, both theoretical and empirical, to warrant the assumption that the RCT population and the target are alike in just the right ways to support the same causal pathways from intervention to outcome”

“Moving from the first to the second or third types of claim, then, requires a great deal of knowledge that cannot be warranted by the RCT itself. The problem is that ‘this kind of complicated causal reasoning is hard, even if we are prepared to be rough in our approximations and figure out ways to tolerate uncertainties’ (Cartwright and Munro, 2010, p. 263)” (Cowen et al., 2017)

“Some interventions will work only because of very special circumstances; they can work in some places but don’t have a widespread potential to succeed. Even those that have widespread potential do not operate on their own; they will work only when the requisite support factors are in place, or some suitable substitute for them” (Cowen et al., 2017)

“Supposing that a particular intervention has relatively wide potential to improve targeted outcomes, we have identified two specific kinds of information that need to be acquired if we are to predict how successful the intervention would be in a new context: [1] what the support factors are and how they are distributed in the new context and [2] how to ‘de-abstract’ or ‘contextualize’ generalizations”

1. “Support factors are those factors required for the intervention to work in the context in question. We need to know that they (or an appropriate substitute) will be in place at the right time and to the right degree”
2. “General truths – claims that apply consistently in new and different contexts – tend to use fairly abstract concepts…Knowledge formulated in abstract concepts is only of use in practice if we know, for the situation at hand, how to ‘de-abstract’ or ‘contextualize’ it”
   - -> “…it is helpful to be able to lump together interventions that differ in a variety of ways but all satisfy the same abstract description, so long as that description is relevant”
   - -> “…human beings are affected by heterogeneous motivations, which is one commonly cited reason for the difficulty the social sciences have in coming up with general theories that hold reliably across individuals. Yet, lumping under more abstract descriptions does improve strength of warrant, so long as the descriptions are relevant (Simpson 2017), so it is not surprising that researchers and EBP sites try to do so”

Question 6

Does the evidence-based practice movement attach too much importance to evidence generated from RCTs?

Answer 6

What is Evidence-Based Practice (EBP)?
o “…the conscientious [ethical, effective, honest], explicit [transparent, auditable], and judicious [considered, prudent] use of current best evidence [as rigorous as possible, subject to updating] in making decisions about the care of individual [people]…integrating individual clinical expertise with the best available external clinical evidence from systematic research” (Sackett et al., 1996, p. 71)

o “Without clinical expertise, practice risks becoming tyrannized by evidence, for even excellent external evidence may be inapplicable to or inappropriate for an individual [person]. Without current best evidence, practice risks becoming rapidly out of date, to the detriment of [people]” (Sackett et al., 1996, p. 72)

Common objections to EBP:
o Ignores clinical expertise
o Ignores client values and preferences
o Is a cookbook approach
o Is simply a cost-cutting tool
o Is limited to clinical research
o Cannot be done; is an ivory-tower concept
o Results in ‘therapeutic nihilism’
- i.e., a contention that it is impossible to cure people or societies of their ills through treatment…In medicine, it was connected to the idea that many “cures” do more harm than good, and that one should instead encourage the body to heal itself (Michel de Montaigne; Jacob Stegenga)
o Assumes that professionals are rational agents
o Can always find evidence for a favored point of view

Cartwright and Munro introduce some concerns about how to interpret RCTs, arguing that they are by themselves typically “insufficient to meet the needs of policy or practice decision makers” (Cartwright and Munro, 2010, p. 265, as cited in Cowen et al., 2017)

“The primary result of an RCT is a judgment about the effectiveness of a treatment in producing an outcome in the study population, although the contribution of other factors to the outcome is unknown. The purpose of RCTs in the context of EBP is to help estimate the effectiveness of a treatment in a target population or populations different from the one on which the experiment was conducted, which raises questions about whether and when such inferences are warranted” (Cowen et al., 2017)

“It is widely acknowledged that we generally don’t know all the important causes for a factor, let alone knowing [their] distribution…in the study and the target populations” (Cartwright and Munro, 2010, p. 261)

There are two categories of causal factors, besides the intervention itself, that might affect the outcome (Cowen et al., 2017):

1. First, there are factors that operate independently of the intervention. These will affect the overall size of the outcome but have no effect on what the intervention itself contributes to the outcome
2. Second, there are factors that moderate how much effect the intervention can produce, factors that must be in place lest the intervention fail to produce its expected contribution. These are called support factors (sometimes also “interactive factors” or “moderator factors”)
   - -> Support factors…bear on the central question of EBP: whether the intervention or policy will “work” in a targeted setting, i.e., whether it will produce some positive contribution to the desired outcome there

“Evidence based medicine is not restricted to randomised trials and meta-analyses. It involves tracking down the best external evidence with which to answer our clinical questions. To find out about the accuracy of a diagnostic test, we need to find proper cross sectional studies of patients clinically suspected of harbouring the relevant disorder, not a randomised trial. For a question about prognosis, we need proper follow up studies of patients assembled at a uniform, early point in the clinical course of their disease. And sometimes the evidence we need will come from the basic sciences such as genetics or immunology. It is when asking questions about therapy that we should try to avoid the non-experimental approaches, since these routinely lead to false positive conclusions about efficacy”

“Because the randomised trial, and especially the systematic review of several randomised trials, is so much more likely to inform us and so much less likely to mislead us, it has become the ‘gold standard’ for judging whether a treatment does more good than harm. However, some questions about therapy do not require randomised trials (successful interventions for otherwise fatal conditions) or cannot wait for the trials to be conducted. And if no randomised trial has been carried out for our patient’s predicament, we must follow the trail to the next best external evidence and work from there” (Sackett et al., 1996, p. 72)