Causal analysis

Question 1

When does an OLS estimator capture a causal relationship? When does it not and what type of relationship is this instead?

Answer 1

An OLS estimator captures a causal relationship when the exogeneity assumptions holds, which states that the independent variable/regressor is uncorrelated with the error terms.
If the exogeneity assumption is violated then we are only able to interpret a relationship as an association or correlation.

Question 2

Define an exogenous regressor and write this out mathematically.

Answer 2

An exogenous regressor is one which which is not correlated with the error terms i.e. Xi and ui are not correlated.
E(ui | Xi) = 0. Alternatively, exogeneity can be written as Cov(Xi, ui) = 0*

Question 3

Define an endogenous regressor.

Answer 3

An endogenous regressor is one where Xi and ui are correlated. For example, if higher values of Xi are systematically associated with higher values of ui.

Question 4

What are four possible reasons for endogeneity?

Answer 4

Reasons for endogeneity:

1. Omitted variable bias
2. Measurement error
3. Simultaneity bias
4. Selection bias

Question 5

Define omitted variable bias.

Give an example of when omitted variable bias may occur.

Answer 5

Omitted variable bias is where a relevant variable is left out of the regression model which effects the dependent variable and is correlated with one of the regressors (is second part of this necessary?*)
For example, regressing wages on education may lead to endogeneity due to omitted variable bias. Ability is likely to affect wages and is correlated with education. The estimated coefficient for education will therefore be overestimated because ability and education are positively correlated, and the coefficient will therefore partially capture the effect of ability on wages.
Positive correlation because those who have higher ability are more likely to choose to stay on in education.

Question 6

Define measurement error.

Give an example of when measurement error may occur.

Answer 6

Measurement error is where there is substantial error involved in measuring Xi, the independent variable.
For example, if testing the permanent income hypothesis measurement error may occur if using current income to estimate permanent income.

However, only use this answer if substantial measurement error is necessary.

Question 7

Define simultaneity bias.

Give an example of simultaneity bias.

Answer 7

Simultaneity bias is where the dependent variable effects the independent variable, as well as the other way around. In other words, we are unsure which variable causes which, or both. They may be determined simultaneously.
For example, supply and demand; or police expenditure and crime rate.

Question 8

Define selection bias.

What may cause selection bias to occur?

Answer 8

Selection bias is where the sample used in estimating a regression bias is not representative of the population, leading to the estimates being positively or negatively bias.
Selection bias may occur because the observations we observe may be self-selecting or skewed in some way.
The missing observations mean that we are unable to conclude whether it is a causal relationship, and estimates we derive are misleading.

Question 9

What are three possible solutions to the problem of endogeneity?

Answer 9

1. Randomisation
2. Instrumental variables
3. Regression discontinuity design.

Question 10

What is randomisation and how does this help to alleviate the problem of endogeneity?

Answer 10

Randomisation is when the variable of interest, Xi, is randomly assigned using, for example, a random number generator, rolling a dice, flipping a coin.
This makes it more likely for regressors to be exogenous because they will not be correlated with unobserved background characteristics i.e. randomisation smooths out disparate background characteristics.

Question 11

What is the terminology used for randomisation in medical trials or by economists?

Answer 11

If Xi is the randomly assigned binary variable then:
Xi = 1 represents the treatment occurring
Xi = 0 represents the control group
The coefficient of Xi captures the treatment effect, which is the average change in outcome due to the treatment compared to the control group.

Question 12

What are four problems or limitations that may be associated with randomised experiments?

Answer 12

1. Unethical to force participants to participate in some experiments, for example giving health care to some and not others.
2. Groups will often be self-selecting and therefore will have certain background characteristics, meaning they are no longer random
3. True and complete randomisation is often impossible, for example you cannot randomly assign gender, ethnicity, or having health insurance. The results often reflect having the CHOICE of receiving treatment rather than actual effect of the treatment.
4. Experiments are often time-limited and short, making it difficult to find long-term effects.

Question 13

Define an instrumental variable.

Explain how instrumental variables help with endogeneity problems.

Answer 13

An instrumental variable is a substitute or proxy for a variable which is suspected to be endogenous or stochastic*?.
The instrument helps with endogeneity problems because it allows us to generate exogenous variation in Xi which is uncorrelated with the error terms, ui.

Question 14

What three assumptions must the instrument(s) satisfy?

Answer 14

An instrument(s) must satisfy three assumptions:
1. Relevance assumption - the Z(s) must be correlated with X after accounting for Wi, the other exogenous variables in the regression
i.e. the relationship must be non-zero.
cov(Xi, Zi) /= 0
2. Exogeneity assumption/exclusion restriction/validity. This says that Z must not be correlated with the error terms, ui.
cov(Zi, ui) = 0
3. Z cannot be another regressor in the model, it must be an additional variable

Question 15

What method is used to estimate a regression when using instrumental variables?
Describe briefly (in a sentence) how to carry out this method.

Answer 15

Two Stage Least Squares (2SLS) is used to estimate a regression when using instrumental variables. This involves two consecutive OLS regressions.

Question 16

Describe the two stages of 2SLS.

Answer 16

2SLS
Stage 1: Regress Xi, the independent endogenous variable, on the instrument, Zi, to obtain predictions for Xi, denoted X-hati. This captures the exogenous variation in X driven by the instrument Z.
Xi = A1 + A2Zi + A3Wi + vi
X-hati = a1 + a2Zi
where Wi are other exogenous variable(s) in the model.
Stage 2: Regress Yi, the dependent variable, on the predicted values for X. In other words replace Xi with X-hati in the original regression.
Yi = B1 + B2X-hati + ui
The estimates b1 and b2 are the IV 2SLS estimates.

Question 17

Can we test the relevance assumption for an instrumental variable?

Answer 17

We can test the relevance assumption for an instrumental variable, simply by looking at the correlation between X and Z.

Question 18

Can we test the exogeneity assumption for an instrumental variable?

Answer 18

We cannot directly test the exogeneity assumption for the instrumental variable because the error terms are unobserved.

However, we can indirectly test the validity/exogeneity assumption when the model is OVER-identified.
This uses the ‘over-identification’ or SARGAN test:
H0: All instruments are valid and exogenous
H1: At least one instrument is not valid

In Stata, the command is estat overid which is performed AFTER the two stages of 2SLS. We look at the p-value associated with the Sargan score of the chi-square distribution.

Question 19

What is a problem with manually calculating 2SLS?

What is a solution for this?

Answer 19

A problem with manually calculating 2SLS is that the stand errors produced are wrong.
The standard errors are wrong because they do not take into account the fact that X-hati is itself an estimate from the first stage OLS regression.
This means we cannot perform hypothesis testing with these standard errors.

A solution for incorrect standard errors produced in 2SLS is to use the Stata command ivregress 2sls y (x = z1 z2) w
Stata corrects for this and produces correct standard errors.

Question 20

2SLS and the coefficient of determination?

Answer 20

R^2, the coefficient of determination, has a different interpretation when doing 2SLS; it can even be negative.

Question 21

How can we measure the strength of an instrument?

Answer 21

We can measure the strength of the instrument by using the F-statistic for joint significance AFTER running the first-stage of 2SLS (NOT the F-statistic of the actual regression obtained after both stages of 2SLS since this also tests the significance of the other exogenous variables, Wi).
In Stata, the F-statistic can be computed using the command test z1=0 z2=0*

If the F-statistic is LESS than 10, this implies it is a WEAK instrument.
If the F-statistic is GREATER than 10, this indicates a STRONG instrument.

Question 22

What are two methods for testing whether a regressor is endogenous (one informal, one formal)?

Answer 22

1. Intuition and common sense - always consider the possibility of endogeneity. In particular, if CHOICE is involved then the variable is bound to be endogenous since it is likely to be systematically related to background characteristics which drive the individual to making a particular choice.
2. We can formally test whether a regressor is endogenous by performing a variant of the Hausman test which involves two stages:
   First: Regress the regressor, Xi, on the instrument and the other exogenous regressors, Wi.
   Xi = A1 + A2Zi + A3Wi + vi
   and predict the residuals, v-hati
   Second: Regress the outcome variable, Yi, on Xi (NOT X-hati), all other exogenous variables Wi, and the predicted residuals from the first stage, v-hati.
   Yi = B1 + B2Xi + B3Wi + C1v-hati + ui

We then test the null hypothesis:
H0: C1 = 0
(i.e. the endogenous, problematic part of X is not related to Y, so X is not endogenous and no further instruments are needed)
H1: C1 /= 0 (X is endogenous)

If we do not reject the null this implies we do not have significant evidence to show that X is endogenous.

Question 23

What types of instrumental variables are more likely to be valid and exogenous?

Answer 23

Instrumental variables that are randomly assigned are more likely to be valid and exogenous because they are less likely to be correlated with background characteristics.

Question 24

If complete randomisation of an instrumental variable is not possible, what type of effect might we be able to capture instead?

Answer 24

If complete assignment under randomisation is not possible for an instrument, we can instead try to capture the Intention To Treat (ITT) effect rather than the actual treatment effect.

Question 25

What does the ITT effect capture?

Give an example of an instrument that may be able to capture an ITT effect?

Answer 25

The ITT effect captures a CORRELATION effect rather than a causal effect since it ignores non-compliance, and therefore may over- or under-estimate the true causal effect of the treatment.
In other words, the ITT effect capture the effect of having the OPTION of being treated on the outcome variable.

For example, we may want to test the effect of watching a certain video on the test results of a set of students.
However, we cannot completely randomly assign watching the video to students.
Instead, we send around an email with the video link to a random selection of students, and use this to capture the ITT effect.
In this case, the ITT effect is the effect of receiving the email on the test scores, regardless of whether the student then went on to watch the video.
Yi = D1 + D2Zi + ei
Where Zi is the instrument, the email containing the video link, and Yi is the outcome variable, the test score.
The ITT is the coefficient of Zi, which is D2.

Question 26

How can we obtain a causal effect from the ITT effect?

Answer 26

We can obtain a causal effect from the ITT effect (which only captures a correlation) by using the ITT effect to calculate the Local Average Treatment Effect (LATE).

Question 27

What is LATE, what does it capture, and how do we calculate it?

Answer 27

LATE stands for Local Average Treatment Effect and captures the causal effect of a treatment by using the ITT effect.
However, LATE only captures the causal effect of a treatment on an outcome variable among those who took the treatment as a result of being given the option of treatment within the trial conditions.
To calculate LATE, we divide the ITT (D2 from card 25) by the difference in compliance rates of taking the treatment (this is the coefficient on the instrument from the first stage of 2SLS, where the endogenous variable is regressed on the instrument).

Question 28

When instrumental variables are used to estimate the causal effect of an endogenous TREATMENT variable, what is the limit of what this can capture?

Answer 28

When instrumental variables are used to estimate the causal effect of an endogenous treatment variable, the estimation strategy only capture the Local Average Treatment Effect (LATE). This is the average treatment effect for one subset of the population - these are the ‘units’ which receive treatment if and only if they are induced by an exogenous instrumental variable.

Question 29

What is the next best alternative if a Randomised Control Test is not possible?
Give examples.

Answer 29

If a Randomised Control Test is not possible to estimate the causal effect of a treatment variable, the next best alternatives are natural or quasi-experiments.
Natural or quasi-experiments exploit ‘natural events’ which are treated as sources of randomness.
For example, rainfall, changes in regulations/policies, natural disasters, unforeseen economic crises.

Question 30

What is one technique that exploits ‘natural’ events to make causal inferences?

Answer 30

Regression Discontinuity (RD) experiments exploit specific rules and change in policy to make causal inferences, using these as sources of randomness.

Question 31

What are RD experiments? What is the main assumption made when performing an RD experiment?
What is the variable introduced into the RD model which the treatment depends on?
Give an example of an RD model.

Answer 31

RD experiment exploit random/unexpected changes in specific rules or policies in order to make causal inferences.
The main assumption behind RD is that differences in the outcome between those just above and below the cut-off point are only due to the treatment - in other words the background characteristics of individuals near to either side of this threshold are assumed to be the same.

A running variable, denoted Q, is introduced into the model which the treatment depends on. Q is an observed continuous variable, such as age, weight, birth date.

For example, an RD model may estimate the causal effect of receiving neonatal care on educational attainment by exploiting the medical rule that babies weighing under 2500g receive additional neonatal care.

Another example: causal effect of drinking on mortality rate, RD could exploit the policy that individuals under 21 are not allowed to purchase or consume alcohol in US.

Question 32

What are two types of RD design?

Answer 32

Two types of RD design:
Sharp RD
Fuzzy RD

Question 33

Define what a sharp RD is.

Give an example of a sharp RD.

Answer 33

A sharp RD is one where the treatment status is a DETERMINISTIC function of the running variable - in other words the rule/policy is applied strictly.
For example, alcohol restrictions are applied rigorously in US to those under the age of 21. This would constitute a sharp RD design.

Question 34

Define what a fuzzy RD is.

Give an example of a fuzzy RD.

Answer 34

A fuzzy RD is one where the treatment status is NOT A DETERMINISTIC function of the running variable. Instead, the probability or intensity of the treatment jumps at the cut-off. This may be because the rule/policy is not applied strictly, or the treatment is only affected indirectly.

For example, raising the school leaving age in England, Wales and Scotland in 1947 from 14 to 15 - however there were still some 14-year olds who left school after the policy was enacted.

Question 35

What are the limitations of RD designs and its validity?

What must be true around the cut-off point for the RD design to be valid?

Answer 35

RD can only tell us about the causal effect of a treatment for a specific section of the population. The further we extrapolate from the cut-off point, the more speculative our estimates are.

There must be no other discontinuities at the cut-off point, such as other rules or policies that are applied using this threshold too.
If this happens, it’s impossible to distinguish the effect of one policy from another on the outcome variable.