Linear regression-2 Flashcards
P-values, linear regression
What is the problem with R squared
R squared is sensitive to amount of data. When we see a pattern in small dataset, we don’t have confidence to say that the pattern is not due to random chance.
What is p-value?
p-value gives us a measure of confidence in the results from statistical analysis.
Set the intuition of p-value with example
Let us consider two drugs. Drug A and Drug B. p-value helps us determine if both these drugs are equally efficient or one is more efficient than other. If p-value allows us to establish a difference, then we worry about if A is good/worse than B
p-value interpretation when A and B are different.
We did one experiment giving drugs A and B to people. A cured 37% and B cured 31%. From the overall picture, A worked better than B. But how confidently can we say that A is better than B. This is where p-values help. p-values are numbers between 0 and 1 and they quantify how CONFIDENT WE SHOULD BE THAT DRUG a IS DIFFERENT THAN DRUG B.
What does p-value 0 denote
The closer a p-value is to 0, the more confidence we have that drugs A and B are different.
The closer a p-value is to 0, the more confidence we have that drugs A and B are different. - How close should a p-value should be to 0?
In practice, commonly used threshold is 0.05.
What does p-value threshold of 0.05 mean?
It means that if theres no difference between drugs A and B, and if we did the exact same experiment a bunch of times, then ONLY 5% OF THOSE EXPERIMENTS WOULD RESULT IN WRONG DECISION
How to calculate p-value
Using a statistical test like - Fischer’s test.
Explain the definition of p-value threshold when the value is 0.05 with example
Example:
1. Let’s give same drug, drug A to two groups of people. The first test gave p-value 0.09 - failed to see a difference between the two groups.
2. Repeat the same experiment and the p-value will be really high and we will fail to see the difference.
3. Next, once in a while there might be a chance that group of people allergic to the drug might end up in the same group. In this case drug A might fail to work on them and will get a small p-value - suggesting that there is a difference in the drug used by the two groups, although they are using the same drug. This is a FALSE POSITIVE.
A threshold of 0.05 means that 5% of the experiments, where the only differences come from weird random things will generate a p-value smaller than 0.05.
How to set threshold for p-value
For extremely important test, like the effectiveness of drug, we need high confidence when make a statement that the drugs are either different or not difference. In such cases we can use extremely small thresholds like 0.01 or 0.001 (1 in 1000 experiments can lead to False Positive)
Likewise, for not so important tests, we can use a bit higher threshold like - 0.2 (2 in 10 can lead to FP)
p = 0.24 denotes that drugs …..
We are not confident that drugs A and B are different.
p = 0.02 denotes that drugs ….
We are confident that the drugs A and B are different.
What is Hypothesis testing?
The idea of trying to determine if the drugs are same or not is called Hypothesis testing.
What is Null Hypothesis?
The Null Hypothesis is the drugs are same and the p-values helps us to decide if we should reject the Null Hypothesis.
What does p-value DOESN’T tell us
Although p-value helps us decide if drugs A and B are different, they don’t tell us HOW DIFFERENT they are.