2.RCT Flashcards
Vad menas med intern validitet
Zero conditional assumption holds
Internal validity is the extent to which a study establishes a trustworthy cause-and-effect relationship between a treatment and an outcome. 1 Internal validity also reflects that a given study makes it possible to eliminate alternative explanations for a finding.
Wiki:
Internal validity is the extent to which a piece of evidence supports a claim about cause and effect, within the context of a particular study. It is one of the most important properties of scientific studies, and is an important concept in reasoning about evidence more generally. Internal validity is determined by how well a study can rule out alternative explanations for its findings (usually, sources of systematic error or ‘bias’).
Vilka hot finns det mot intern validitet vid RCT?
Failure to randomize
Failure to follow treatment producol
Attrition
Experimental effect - Hawtorn.
Vad menar man med failure to randomize som ett hot mot den interna validiteten i en RCT?
Random assignment to treatment and control groups is the fundamental feature of a randomized experiment that makes it possible to estimate the causal effect. If treatment is not assigned randomly, but instead is based in part on the characteristics or the preferences of the subjects, the experimental outcomes will reflect both the effect of the treatment and the effect of nonrandom assignment. In general, nonrandom assignment leads to bias in the OLS estimator.
Vad menar man med “failure to follow treatment protocopl” som ett hot mot den interna validiteten i en RCT?
In actual experiments, subjects do not always do what they are told. Therefore, even if the treatment assigned is random, the treatment actually received might not be random. The failure of subjects to follow completely the randomized treatment protocol is called partial compliance with the treatment protocol. With partial compliance, the treatment and control groups no longer are random samples from the larger population from which the subjects were originally drawn: instead the treatment and control groups have an element of self- selection. Failure to follow the treatment protocol leads to bias in the OLS estimator
Vad menar man med “attrition” som ett hot mot den interna validiteten i en RCT?
People may move off during the experiment. If people who leave have particular characteristics systematically related to the outcome then there is attrition bias (Non response bias)
Vad menar man med “eperimental effects” som ett hot mot den interna validiteten i en RCT?
In experiments with human subjects, the mere fact that the subjects are in an experiment can change their behavior, a phenomenon sometimes called the Hawthorne effect.
Vad menas med extern validitet?
External validity is the validity of applying the conclusions of a scientific study outside the context of that study. In other words, it is the extent to which the results of a study can be generalized to and across other situations, people, stimuli, and times
Vilka hot finns det mot den externa validiteten i en RCT?
Non-representive sample
Non representive program or policy
General equlibrium effects
Treatment vs. Eligibility effects
Vad menar man med “non-representative sample” som ett hot mot den externa validiteten i en RCT?
The population studied and the population of interest must be sufficiently similar to justify generalizing the experimental results.
An example of a when a non-representative sample might arise is when the experimental participants are volunteers. Even if the volunteers are randomly assigned to treatment and control groups, these volunteers might be more motivated than the overall population and, for them; the treatment could have a greater effect. More generally, selecting the sample non-randomly from the greater population of interest can compromise the ability to generalize the results from the population studied to the population of interest.
Vad menar man med “non-representative program or policy” som ett hot mot den externa validiteten i en RCT?
The policy program of interest also must be sufficiently similar to the program studied to permit generalizing the results.
One important feature is that the program in a small-scale, tightly monitored experiment could be quite different than the program actually implemented. Another difference between an experimental program and an actual program is its duration: the experimental program only lasts for the length of the experiment, while the actual program under consideration might be available for longer periods of time.
Vad menar man med “generall equlibrium effects” som ett hot mot den externa validiteten i en RCT?
An issue related to scale and duration concerns what economists call “general equilibrium” effects.
Turning a small, temporary experimental program into a widespread, permanent program might change the economic environment sufficiently that the results from the experiment cannot be generalized. Phrased in econometric terms, an internally valid small experiment might correctly measure a causal effect, holding constant the market or policy environment. General equilibrium effects means that these other factors are not, in fact held constant when the program is implemented broadly.
Vad menar man med “Treatment vs. eligibility effects ” som ett hot mot den externa validiteten i en RCT?
Another potential threat to external validity arises because, in economics and social programs more generally, participation in actual (non-experimental) program is voluntary. Thus, an experimental study that measures the effect of the program on randomly selected members of the population will not, in general, provide an unbiased estimator of the program effect when the recipients of the actual implemented program are permitted to decide whether or not to participate
Beskriv utförligt de två hoten mot en studies ”power” vid RCT.
Small samples:
Because experiments are difficult to administer, samples are often small, which makes it difficult to obtain significant results. It is important to compute power calculation before starting an experiment (what is the sample size required to be able to discriminate from 0 an effect of a given size?)
Experiment design and power of the experiment:
When the unit of randomization is a group (e.g. a school), we may need to collect data on a very large number of individuals to get significant results, if outcomes are strongly correlated within groups
Berätta om bra och dåliga kontroller:
Bra kontroll variabel: påverkar bara effekten i Y och är i sig själv inte påverkad av behandlingen.
Dålig kontroll variabel: Påverkar Y men blir också påverkad av behandlingen. Detta är dåligt då man stänger ned en kanal av hur behandlingen påverkar Y. Man plockar alltså bort en del av effekten.
Irrelevant kontroll variabel: påverkar inte Y.
Dålig kontroll:
Exempel med Y = testpoäng, X1 = lärare som genomgår träningsprogram eller ej. X2 = studentnärvaro
Om man kontrollerar för närvaro kommer man minska en del av effekten av träningen, då träningen förmodligen gör att elever går mer på lektionenerna Bra kontroll variabel: påverkar bara effekten i Y och är i sig själv inte påverkad av behandlingen.
Dålig kontroll variabel: Påverkar Y men blir också påverkad av behandlingen. Detta är dåligt då man stänger ned en kanal av hur behandlingen påverkar Y. Man plockar alltså bort en del av effekten.
Irrelevant kontroll variabel: påverkar inte Y. läraren är bättre. Så effekten på Y av X1 blir mindre när man kontrollerar för X2 i det här fallet.
Bra kontroll:
Bättre vore att kontrollera för IQ hos eleven eller nivån av utbildning i dennes familj, då detta är variabler som inte påverkas av X1 men som kommer påverka Y.
Hur påverkas en RCT av kontrollvariabler.
Effekten är otydlig, men vid riktiga experiment kommer man alltid öka precisionen då variansen i β1^ minskar då variansen i u minskar.
Man minskar alltså inte bias vid randomiserade experiment genom kontrollvariabler, utan man minskar bruset/ökar precisionen!
DOCK kan man minska effekten på av β1 om det är en dålig kontrollvariabel. Men, det blir mer precist.
