Practical Tips Flashcards
When conducting our power calculation, we did not consider covariates (control variables) in the formula, even though we plan to use baseline outcomes and covariates when conducting the final analysis. As such, compared to the estimates from our power calculation, in our final analysis
The inclusion of baseline covariates reduces residual variance; in doing so, it increases the power of the experiment and allowing for the detection of a smaller effect size (i.e., the minimum detectable effect size becomes smaller.)
One major lesson from observing ICCs?
ICC is context-dependent and hard to predict
We would sometimes want to pool different treatment arms (i.e. Groups A and B vs Control rather than either Group A vs. Control or Group B vs. Control) when conducting our analysis because (select all that apply):
It gives us a larger sample size than if we were to only compare either A or B to control & Whether “any treatment” (A or B) is effective is an interesting question to answer
In order to identify interaction effects, one would need to provide both treatment A and treatment B to one group; merely pooling the two separate groups would not allow for the identification of interaction effects. Pooling the groups also does not have a bearing on the allocation ratio between A and B since individuals will still maintain their original treatment assignment. What a pooled treatment arm does is to increase the sample size and power for a comparison of “any treatment” versus a comparison group. This is useful when comparing “any treatment” to the control is a comparison that addresses a question that one is interested in
What effect size should you use when designing your experiment?
Smallest effect size that is still cost effective
Which statement is NOT correct?
- MDE has a big influence on power, for a given sample size
- When calculating MDE we need to think about likely take up rates
- A smaller MDE means the effect will be estimated more precisely (all else equal)
- In general, cheaper programs can be tested with smaller sample sizes
In general, cheaper programs can be tested with smaller sample sizes
Cheaper programs can have a smaller impact and still be cost-effective. Therefore we would likely want to choose a smaller MDE when evaluating these programs, which would require a LARGER sample size. Take-up rates impact the detectable effect of one’s experiment, while a smaller MDE means that the effect will be more precisely estimated, all other things being equal. For a given sample size, one will have less power to measure smaller MDEs than larger ones.
(For individual-level randomization) Increasing the sample size by a factor of 4 results in
Making the standard error half the size
Recall that the standard error is a function of the population standard deviation and the sample size. Specifically, for a (non-cluster) randomized trial the standard error is given by the standard deviation divided by the square root of the sample size. Increasing the sample size from N to 4N would thus decrease the standard error by 1 divided by the square root of 4 i.e. 1/2. Quadrupling the sample size would thus halve the standard error.
Taking a random sample from the entire target population..
means that one’s sample is more representative of the broader target population. This means that any inferences about impact that one draws from the sample are likely to apply to the target population at large as well (external validity.
drawing a random sample from within each cluster…
allows us to draw inferences about the cluster without surveying everyone in it; it is akin to thinking of the cluster as the target population.
In the maternal literacy project, we did not want to select clusters that were too big because we were afraid that
If Pratham only held one class in that cluster, take-up of mothers in our sample might be too small
In the maternal literacy project, we did not want to select clusters that were too small because:
Our sample size calculations were based on 20 households per cluster
If clusters that were too small were selected (with fewer households in the cluster than were assumed while doing the power calculations), this would have effectively reduced the sample size and lowered the power of the study to detect an impact.
We conducted an Rural Resource Assessment primarily because we needed to identify
Appropriately-sized clusters for our unit of randomization
As Marc explains in the Mother Literacy example, the Rural Resource Assessment came after the study location (in villages in a district of Bihar) was determined. Habitations (clusters of approximately 20 households) were selected as the unit of randomization. What the Rural Resource Assessment served to do was to help identify which habitations/clusters would be of a sufficient size for the study.
We conducted a census for the maternal literacy project primarily because we needed to identify
Households to randomly select as our unit of analysis
Once the Rural Resource Assessment helped determine which clusters would be of a sufficient size for the study, a census helped identify which households had eligible mothers, which would be randomly selected as the unit of analysis and to collect data on.
Within the maternal literacy program, within one stratum of 10 villages, if the combined intervention (ML + CHAMP) was administered in 2 villages, which other arm(s) would also be administered in exactly 2 villages.
Control
As Marc illustrates in the previous video, within a stratum of 20 villages there were 5 villages each that received ML, CHAMP, ML + CHAMP, and neither ML nor CHAMP. However, in a stratum of 10 villages, 3 villages each received either ML or CHAMP, with 2 villages receiving both ML and CHAMP and 2 villages receiving neither, serving as the pure control.
Steps to calculate power
- Set desired power (80%, 90%) and significance (95%)
- Calculate residual variance (& ICC) using pilot data, national data sources, or data from other studies
- Decide number of treatments
- Set MDE size for T vs C and between treatments
- Decide allocation ratio
- Calculate sample size
- Estimate resulting budget
- Adjust parameters above (e.g. cut number of arms)
- Repeat
Residual variance & power relationship
Variance reduces power because of the risk that we might pick “successful” people for treatment
• Some variation can be explained by observables
• Using controls in analysis soaks up variance, impact more precisely estimated, more power
• Calculate residual variance by regressing outcome on controls in existing data
