Hypothesis Testing

Question 1

What does the p value tell us?

Answer 1

The p value tells us the likelihood of obtaining our results by chance (i.e. When our defined null hypothesis is true).

Question 2

What does a p value of 0.025 mean?

Answer 2

It would tell us that the chance of obtaining our results by chance was 0.025. This is very unlikely and we can use this result to reject the null hypothesis.

Question 3

What does a small p value tell us?

Answer 3

Indicates that there is a difference (or association) and we can reject the null hypothesis.

Question 4

What does a large p value indicate?

Answer 4

It indicates that there is no evidence of a difference (or association) and we fail to reject the null hypothesis.

Question 5

True or false - the p value, the size of the effect and the number of observations (sample size) are all interrelated.

Answer 5

True. If you carry out a small study, you can get a p value which is not significant even when the effect is large. If you carry out a large study, even small differences which are clinically and epidemiologically irrelevant, may achieve statistical significance.

Question 6

Summarise how to go about hypothesis testing.

Answer 6

1) . Start by specifying the study hypothesis and the null hypothesis (which is usually that there is no difference between the groups).
2) . We assume the null hypothesis is true. That there is no difference between our two groups.
3) . We calculate the chance that we would get the difference that we observed if the null hypothesis were true. This chance is called the p value.
4) . We then accept or reject the null hypothesis on the basis of the size of the p value. If the p value is small, we reject the null hypothesis. If the p value is big, we accept the null hypothesis.

Question 7

How do we choose what hypothesis test to use in order to obtain the p value?

Answer 7

We use a different test depending on the design of the study (unpaired or paired) and on what sort of outcome variable we are dealing with, continuous or categorical, whether it is normally distributed or not. Each test may have some additional assumptions associated with it, and you should always check to make sure these are valid before carrying out and reporting the test.

Question 8

What is the most important question to answer when deciding how to analyse continuous variables?

Answer 8

One of the most important questions to answer when deciding how to analyse continuous variables is whether they are following a normal distribution or not, so this should be the first thing you look at. You need to assess the histogram and summary statistics.

Question 9

What is another name for the Student t-test?

Answer 9

The independent samples t-test.

Question 10

What is the independent sample t-test used for?

Answer 10

The independent sample t-test (or student t-test) is used when we want to compare two groups and the outcome variable is a continuous normally distributed variable, such as birth weight. The standard t-test also assumes that the variation (scatter) in the two groups is approximately the same.

The t-test provides a p value which is interpreted as previously described. The probability that we are after is the probability of getting the difference in the sample means that we observed when the true difference is zero. Now the probability will depend on how big the difference is between our two sample means, and it will depend on how much (sampling) error there might be in our estimate of the difference in means. We already know that we have a measure of the amount of error in our estimate: this is called the standard error or the mean difference. We used this when we calculated the confidence interval for the difference between two means, but all you need to realise is that, just as for a single mean, this standard error is a formula which depends on the sample sizes and the standard deviation of the thing we are measuring.

The t-test uses the observed difference in the sample means, and the standard error (sampling error) for the difference in means to calculate the p value.

Question 11

What is a t-statistic?

Answer 11

A t-statistic is the difference in sample means divided by the standard error of the difference in means.

A large difference in sample means and a small standard error will lead to a large t-statistic, indicating that the probability that the observed difference happened by chance is small and producing a small p value.

A small difference in means or a large standard error will produce a small t statistic meaning that the probability that the observed difference happened by chance is large and the p value will be large.

The t statistics can be converted into a p value using statistical tables and the t-distribution, but more commonly using statistical software.

Question 12

When is it appropriate to use the t-test?

Answer 12

When comparing observations on a continuous variable between two groups the t-test is valid only when the data are normally distributed and when the two populations have equal variances. Because of these assumptions about the underlying distribution, it is known as a parametric test.

Question 13

What do we do if the assumptions for using a t-test (parametric test) do not hold?

Answer 13

If your data do not appear to be normally distributed, it will be necessary to use a non-parametric test, and to present medians (and difference in medians). Non-parametric tests make no assumptions about the underlying distribution of the data.

Question 14

What are the advantages and disadvantages of using non-parametric tests?

Answer 14

Non-parametric tests make no underlying assumptions of the data. However, they do have their disadvantages:

• they are less powerful than parametric tests, ie they are less likely to detect a true effect as significant.
• it is not easy to obtain confidence intervals using the non-parametric approach (SPSS does not calculate these; programs such as Minitab will give you an estimated confidence interval for some non-parametric tests.)

Question 15

Describe the Wilcoxon rank sum tests.

Answer 15

The Wilcoxon rank sum test is the non-parametric equivalent of the independent samples t-test. The Mann-Whitney U test is a. Alternative to the Wixcoxon rank sum test that uses a different formula but results in the same p-value. The theory behind this test is given in the notes for information but you will only be expected to carry out this test using SPSS.

Question 16

What hypothesis test would you use to assess the differences between categorical variables statistically?

Answer 16

A chi-squared test.

Question 17

What is the particular hypothesis test used for assessing e association between two categorical variables called?

Answer 17

The chi-squared test or the chi-squared test for independence.

Question 18

Describe how you would go about carrying out a chi-squared test.

Answer 18

1) . State the null hypothesis (e.g. There is no association between smoking and disease).
2) . Compute the test statistic

The chi-squared test statistic tells us how close the actual values seen on the table (observed values) are to the values we would have expected (expected values) were there no association between the two variables. Large values of the test statistic suggest they are not close, so the data are inconsistent with the null hypothesis. Small values suggest the observed and expected values are similar, which is consistent with the null hypothesis. If you are sing SPSS it will compute this statistic for you. However, you should also understand how to compute it by hand.

First compute the expected values. From the contingency table, we know the overall proportion with disease for the smoking example was 20/200=0.1 or 10%. Therefore if the null hypothesis were true, we would expect 10% of smokers to have disease and 10% of the non-smokers to have disease. So the expected number of smokers with the disease would be 10% of 78 = 10/100 x 78 = 7.8.

As there are are 78 smoker in the study and we would expect 7.8 of them to have disease, it follows that 78-7.8 = 70.2 would be expected to not have disease.

Expected numbers of non-smokers with and without disease can be calculated similarly. An easy way to compute the expected numbers is too use the general formula:

Expected number = row total x column total / overall total

We can then construct a table of the numbers of each expected value (smokers with disease, non-smokers with disease, smokers with no disease, non-smokers with no disease).

The test statistic simply compares the observed values (O) in the cells of the table with the expected values (E). It is computed by taking each cell of the table in turn:

• subtract the expected value (E) from the observed value (O)
• square this value
• divide this by the expected value
• having done this for all cells, sum the numbers obtained together

X^2 = (the sum of) (O-E)^2 / E

To decide how big this test statistic has to be before we conclude it to be inconsistent with the null hypothesis we refer the test statistic to the chi-squared distribution and obtain the corresponding p-value.

3). Obtain a p-value

Again, using the computer the p-value will be given to you automatically. However, if doing a chi-squared test by hand,you will need to look up your value of the test statistic on the table of the chi-squared distribution and read off the p-value. To read the table, you need to know the degrees of freedom which are given by ‘the number of rows minus one’ multiplied by ‘the number of columns minus one’:

Degrees of freedom = (r-1)x(c-1) where r = number of rows and c = number of columns.

Fin the row on the table that corresponds to the degrees of freedom that you have. Read across the table to find where your test statistic falls. For the two numbers between which your statistic falls, read the top of the columns and see what p-values they correspond to. So if your test statistic falls between the two values under P=0.05 and P=0.025, then the p-value is between these values.

4). Interpret the p-value

Remember the p-value tells us how likely differences as large as those seen in our sample would have arisen were the null hypothesis true, ie if there truly was no association. Standard practice is to use the cut-off of p0.05, we fail to reject the null hypothesis and conclude that we have no evidence for a significant association.

Question 19

What kind of testing do we use to obtain the p value?

Answer 19

Hypothesis testing

Question 20

The chi squared test can only be used on 2x2 tables. True or false?

Answer 20

False. The chi-squared test can be used on any sized tables. Larger tables are sometimes called r x c tables where r is the number of rows and c the number of columns.

Question 21

What is continuity correction or Fishers exact test used for?

Answer 21

If you have a small sample, the chi-squared test may not be valid. If we have a 2x2 table, we can use a continuity correction or Fishers exact test when we have small numbers.

If the expected numbers in the cells of a 2x2 table are small, we can improve the chi-squared test by applying a continuity correction (known as Yates’ continuity correction). Instead of the previous chi-squared formula we use a slightly different formula. SPSS will also do this for you. This test statistic is then used to obtain a p-value in the manner as previously described.

There is no set rule as to when to apply this correction, but one suggested guide is to use it if any of the expected numbers in the table are less than 10. If you look at any statistical literature you will find some people that recommend this correction and some people who claim it is too conservative and may lead to a type 2 error (failure to reject the null hypothesis when you should).

Question 22

Describe Fishers exact test for 2x2 tables.

Answer 22

If the expected numbers are very small, the chi-squared test (even with the continuity correction) is not valid and a special test called Fishers exact test must be used. As a guide, use the exact test rather than Chi-squared when either of the following is true:

• the overall total is

Question 23

Describe a Chi-squared test for trend.

Answer 23

A special case arises when the outcome variable is binary and the exposure variable is an ordered categorical variable. The usual chi-squared test assesses whether the proportion with the outcome differs between the different levels of the exposure. However when the exposure variable is an ordered categorical variable, a more sensitive test is to look for an increasing or decreasing trend in the proportions across the exposure categories. This test is called the chi-squared test for trend.

The null hypothesis is the same as for an ordinary chi-squared test, namely that there is no association between the two variables. The test is conducted by assigning the numerical scores 1,2,3 etc to the columns of the table, and then calculating the mean scores in those who have and in those who don’t have the outcome/disease and comparing them.

We will not delve further into the details of this test, but simply recognise that this test exists and is provided by SPSS as we shall see later where either the row or column variable is binary. Remember, it only makes sense to look at this test when the other variable is ordered in some way e.g. age group.

Question 24

Describe significance and magnitude.

Answer 24

We have seen how to assess the significance of an association and we have seen previously how to assess the magnitude using the appropriate measure of effect such as an odds ratio. It is important to understand that these are different and tell us different things so both should be presented. For example we could get an odds ratio if 1.2 (small magnitude) but if the sample was very large, this could be statistically significant (ie unlikely to have arisen by chance). Conversely, a study could yield an odds ratio of 3 (large magnitude), but if based on a small sample, statistical significance may not be reached (eg p=0.1) suggesting such a difference could have arisen by chance alone.

Question 25

Describe how you would go about analysing the relationship between continuous variables.

Answer 25

In this section we will examine methods to analyse the relationship between two continuous variables. The two methods that will be used are correlation and linear regression. Though these two methods are closely related, each method is distinctive and should be used appropriately.

A first step for analysing the relationship between two continuous variables is to plot them out in a scatter diagram, in which the y axis is the outcome (or response/dependent) variable and the horizontal or x axis is the explanatory (or exposure/independent) variable. In some situations, there is no clear outcome and explanatory variables and there the choice of axis is arbitrary.

Question 26

What is correlation?

Answer 26

Correlation measures the closeness or degree of association between two continuous variables. The standard method used is called Pearson’s correlation and this measures the degree of linear association. Pearson’s correlation should only be used if both variables are normally distributed.

Pearson’s correlation coefficient is denoted by r. The value obtained for r will always lie somewhere between -1 and +1. A value of +1 means perfect positive correlation, that is the points all lie exactly on a straight line and as one variable increases, so does the other. A value of -1 means perfect negative correlation, so again the points will all lie exactly on a straight line but as one variable increases the other decreases. A value of 0 means no association.

A scatter diagram is always the first step because r=0 does not mean necessarily that there is no relationship between x and y.

A 95% confidence interval can be computed for Pearson’s correlation coefficient which tells us where we are 95% confident the true value lies. The square of the correlation coefficient (r2) is an estimate of the proportion of the total variation of our outcome variable that is explained by the other variable. So if we are investigating the association between blood pressure and level of stress and found that the correlation coefficient r is 0.80, and therefore r^2=0.64, we can say stress levels account for 64% of the total variation in blood pressure.

As well as giving the correlation coefficient and confidence interval which tell us about the strength of association, we also need to present a p-value which tells us about the significance of the correlation, ie could the observed association have arisen by chance?

Question 27

Describe the use of non-parametric or rank correlation.

Answer 27

If one or both of the variables are not normal then a non-parametric correlation coefficient needs to be used. The most commonly used on is Spearman’s rank correlation coefficient; Kendall’s coefficient is an alternative but is less widely used. These rank the values of each of the two variables and examine how closely the ranks are correlated. Like Pearson’s, Spearman’s will like between -1 (perfect negative correlation) and +1 (perfect positive correlation).

Question 28

Describe some of the common misuses of correlation.

Answer 28

• correlation should not be used when there are multiple,e measurements on each individual in the sample, eg if you measure oestrogen levels and blood pressure weekly for a month in the same individuals. The observations that make up a variable should be independent of one another, ie there should only be one observation for each individual in the study.
• another important consideration is data dredging. Collecting information on 10 variables for example, and looking at every possible pairing would result in 45 different correlations and by chance we would expect one or two to have p

Question 29

Does correlation equate to causation?

Answer 29

It should be noted that just because and observed correlation between some exposure variable x and outcome variable y is strong and statistically significant, this does not necessarily mean the exposure caused the outcome. It could be that:

• x influences or causes y
• y influences x
• both x and y are influenced by one or more other variables

When there is no background knowledge on the relationship then correlations should be interpreted cautiously.

Question 30

When interpreting correlations what should you always mention?

Answer 30

When interpreting correlations, always talk about the direction (positive or negative r), the strength (size of r) and the significance (p value).

Remember the p value is affected by sample size. You can get a very low p value when you have a very large sample size even when the relation between two variable is weak.

Question 31

Describe linear regression.

Answer 31

Linear regression. Summarises or describes the relationship between some exposure variable and a continuous outcome variable. It does this by estimating the best fitting straight line through the data, in other words, the straight line that best describes how the outcome increases (or decreases) as the exposure variable increases. The equation of this line also allows us to predict one variable (outcome) from the other (exposure variable).

The first step in understanding linear regression is by going back to some basic algebra and the equation of a line:

y=a+bx

Where y=the outcome variable
x=the exposure variable
a= the intercept, ie the point at which the line crosses the y axis or the value of y when x=0
b=the slope of the line, ie the increase or decrease in y per unit increase in x, or the rise of the line over the run of the line

The values a and b are known as the regression coefficients. They are sometimes alternatively called beta coefficients and denoted by beta0 and beta1 instead of a and b.

This best fit line is determined by the method of least squares. This method finds the values of the regression coefficients a and b which minimises the vertical distances (the residuals) between the points and the line.

In linear regression we are primarily interested in the slope, b, as this tells us by how much the outcome y increases or decreases per unit increase in exposure x. The average of the intercept, a, is of less interest and is often not a meaningful number as it represents the value of y when x=0. In reality x can never actually take the value 0, eg you can’t have a weight or height of 0. However, the intercept can be of use when used to make predictions.

As with any estimate, the regression coefficient b is subject to sampling variation. In other words it is simply an estimate of the population or ‘true’ value. It is therefore important to compute a 95% confidence interval around b. We can also get a p value to test the null hypothesis of no association between the exposure x and outcome y.

Question 32

What assumptions must hold for a linear regression to be valid!

Answer 32

1) . The relationship between the two variable x and y should be lienar. To assess this we look at the scatter diagram and check the association is approximately linear.
2) . For any value of x, the values of y are normally distributed. To determine this we can use a histogram.
3) . The variability of y should be similar for each value of x. The scatter diagram can give you a feel for this.

Question 33

Describe how you would go about writing a regression equation.

Answer 33

Having obtained our regression coefficients a and b, we can write out the whole regression equation and use this equation to predict the value of y for any value of x. Note however that predictions of y should only be made in the range of x for which you have data available.

Suppose we want to describe the relation between weight and systolic blood pressure. If we carry out a linear regression analysis, we get the following values for the regression coefficients.

Intercept a = 98.5 and slope b = 0.43

As b is positive, this tells us that as weight increases, so does blood pressure (ie positive relation). The value of b is an estimate of how much blood pressure changes as weight increases, so here, we estimate that for each unit (kg) increase in weight, blood pressure increases by 0.43mmHg.

The value of a tells us about the estimated value of blood pressure for someone with a weight of 0kg, something that isn’t actually biologically plausibly and therefore not very useful. However, knowing the value of a does allow us to write down the complete equation of the line and hence make predictions:

y=a+bx

Systolic blood pressure (mmHg) = 98.5 + 0.43 x weight (kg)

So to predict the blood pressure of someone of weight 80kg, for example:

Systolic blood pressure (mmHg) = 98.5 + 0.43 x 80 = 132.9mmHg

Question 34

Describe multiple linear regression.

Answer 34

What we have looked at so far is simple linear regression where we describe the simple, unadjusted relation between one exposure and the outcome of interest:

y=a+bx

However, what we often want is a multivariate analysis where we can allow for other exposure variables that are potential confounders. For continuous outcomes we use multiple linear regression, which is just an extension to the simple linear regression model:

y=a+b1x1+b2x2+b3x3+……….

For example, suppose we want to estimate the effect of weight on systolic blood pressure, but allowing for, or adjusting for, age since age may be a confounder. The equation would be:

Systolic blood pressure (mmHg) = a + b1xweight +b2xage

And the value of b1 would be the age-adjusted regression coefficient for weight. In other words, it is an estimate of how much blood pressure changes per kg increase in weight, but adjusted for any confounding effect of age. We could also look at b2 which would tell us about the relation between age and blood pressure, adjusted for (or independent of) weight.

Question 35

What purposes do regression and correlation serve?

Answer 35

It is important to remember that correlation and regression serve different purposes and they answer different questions.

• correlation tells us about how close the relationship between x and y is (ie how close the the line are the points), but not about the nature of it, ie how much does y change with x.
• linear regression describes the nature of the relation between x and y by giving us the equation of the line of best fit, but does not tell us how well that line fits the data, ie about the degrees of closeness in association.

Question 36

What three things should we always mention when describing an association or relationship?

Answer 36

When describing an association or relationship we should always mention direction, size and significance.

Question 37

Under what conditions is chi squared generally valid?

Answer 37

Chi-squared is generally valid if less than 20% of the values are less than 5 and none of the expected values are less than 1.

Question 38

When is it recommended that we use fishers exact test?

Answer 38

We should use fishers exact test when the overall total is less than 20 or the overall total is 20-40 but the smallest value expected is less than 5.