Simple Linear Correlation and Regression

Question 1

Regression Analysis

Answer 1

• Regression Analysis is a way of estimating the relationship between different variables by examining the behavior of the system
• There are many techniques for modeling and analyzing the dependent and independent variables
• You are basically trying to derive an equation from the graph of your data.

Question 2

Linear Regression Analysis

Answer 2

The easiest kind of regression is linear regression. Imagine that all of your data lined up in a neat row. You could draw a straight line connecting all points and would be able to create a simple equation Y = mx + b that we talked about earlier. That way you would have a model that would faithfully predict what your system would do given any input of x.

But what if your data only “kinda-sorta” looks like a line?

Multiple linear regression is an extension to methodology of simple linear regression

Question 3

Linear Regression

Answer 3

• Statistical technique to estimate the mathematical relationship between a dependent variable (usually denoted as Y) and an independent variable (usually denoted as X).
  
  • In other words, predict the change in the dependent variable according to the change in the independent variable.
  • Dependent Variable or Criterion Variable - is the variable for which we wish to make a prediction
  • Independent Variable or Predictor Variable - the variable used to explain the dependent variable

Question 4

When to Use Linear Regression

Answer 4

• In simple linear regression, there is only one independent variable used to predict a single dependent variable.
• In multiple linear regression more than one independent variables used to predict a single dependent variable.
  
  • The basic difference between simple and multiple regression is in terms of explanatory variables.
    
    • E.g. compare the crop yield rate against the rain fall rate in a season

Question 5

Notes about Linear Regression

Answer 5

• The first step of linear regression is to test the linearity assumption, this can be performed by plotting the values in a graph known as a scatter plot, to observe the relationship between dependent and independent variable, because if the data is exponentially scattered then there is no meaning to create the regression equation.
• Draw the line which covers the majority of the points
  
  • this line is considered best fit line or line of best fit
• The mathematical equation of the line is
  
  • y=a+bx+ε
    
    • Where:
    • b – Slope of the line
    • a – y intercept when x=0
    • Random error (ε-Epsilon) – The difference between an observed value of y and the mean value of y for a given value of x.

Question 6

Assumption of Linear Regression

Answer 6

• Linear relationship between dependent and independent variable
• All variables of regression to be multivariate normal
• Particularly there is no or little multicollinearity in the data
• Response variable is continuous and also residuals are almost same throughout the regression line

Question 7

The Method of Least Squares

Answer 7

• The method of least squares is a standard approach in regression analysis to determine the best line for a given data
  
  • It basically provides a visual relationship between the given data points
• In general, the dependent variables are demonstrated on the y-axis
• The independent variables are demonstrated on the x-axis
• The least square method determines the position of a straight line or also called trend line and the equation of the line.
  
  • This straight line is also known as best for line

The least square method means that the overall solution minimizes the sum of squares of the errors made in the results of every single equation. For instance, Least Squares Equation can be used to find the values of the coefficients a and b

The normal rules of Standard Deviation apply here; 68% of the points should be within +/- 1 Standard Error of the line, 95.5% of the points within +/- 2 Standard Error.

Question 8

Least Squares

a and b computed

Answer 8

Question 9

Linear Regression example in DMAIC

Answer 9

• Linear Regression is specifically used in Analyze phase of DAMIC to estimate the mathematical relationship between a dependent variable and an independent variable.
• Example: A passenger vehicle manufacturer reviewing the 10 salespersons training records. In fact, their main aim to compare the salespersons achieved target (in %) with the number of sales module training completed.
  
  • a^ = y^ - b^xbar
    
    • where y^ =10% of sales target achieved total = 10% of 822=82.2

Question 10

Estimate the Variability of Random Errors

Answer 10

Question 11

Estimate the Variability of Random Errors

Example

Answer 11

o^e=square root of o^ 2 e

E.g. o^2 e = 28.95

o e = 5.38

Question 12

Test of Slope Coefficient

Answer 12

• The existence of a signification relationship between dependent and independent variable can be tested by whether b is equal to 0. If b is not equal to 0 there is a linear relationship.
• The null hypotheses and alternative hypotheses are:
  
  • The null hypothesis H₀ : b=0
  • The alternative hypothesis H₁: b≠0
• Degrees of freedom = n-2

Question 13

Test of Slope Coefficient

Example

Answer 13

t-table critical value chart

OR

Refer to Appendix Q

Values of the t-Distribution

Located in Handbook 2nd Edition Green Tab

Question 14

Confidence Interval Estimate for the Slop b

Answer 14

Question 15

Correlation Coefficient

Notes and Formula

Answer 15

Question 16

Correlation Coefficient

Example

Answer 16

Question 17

How is correlation analysis used to compare bivariate data?

Answer 17

• Measure of central tendency, variance, or spread summarizes a single variable by providing important information about its distribution.
• Often, more than one variable is collected in a study or experiment,
• When two variables are measured on a single experiment unit, the resulting data are called bivariate data.
  
  • Ex. job satisfaction stratified by income
• In most instances, in bivariate data, it determines that one variable influences the other variable.
  
  • The quantities from these two variables often represented using scatter plots to explore the relation between two variables
• Depends on the type of data, bivariate data can be described with graphs and numerical measures.
  
  • If one or both variables are qualitative, then yes a pie chart or bar chart
    
    • For example, compare the relationship between opinion and gender. If the two variables are quantitative, use the scatter plot.
  • The Correlation Coefficient is often used in comparing bivariate data

Question 18

Correlation Coefficient Example

Answer 18

• The correlation coefficient varies between -1 and +1.
  
  • Values approaching -1 or +1 indicate strong correlation (negative or positive) and values close to 0 indicate little or no correlation between x and y
• Correlation does NOT mean causation
• A positive correlation can be either good or bad news
• A negative correlation is not necessarily bad news.
  
  • It merely means that as the independent variable goes more negative, the dependent variable goes negative as well
• r = 0; does not indicate the absence of a relationship, a curvilinear pattern may exist; r=-0.76 has the same predictive power as r=+0.76

Question 19

Coefficient of determination (R²)

Answer 19

• The coefficient of determination is the proportion of the explained variation divided by the total variation, when a linear regression is performed.

Question 20

Coefficient of determination (R²)

Example

Answer 20

Question 21

Coefficient of Correlation is r

Answer 21

Just take the square roof of the coefficient of determination. Sqrt(R Squared)

Question 22

Measuring the validity of the model

Answer 22

• Use the F-statistic to find a p value of the system
• The degrees of freedom for the regression is equal to the number of Xs in the equation (in linear regression, this is 1 because there is only 1 x in the equation y=mx+b)
• The smaller the p value, the better.
  
  • But you really judge this by finding the acceptable level of alpha risk and seeing if that percent is greater than the p value.
    
    • Example: If your alpha risk level is 5% and the p value is 0.014, then you have to reject the hypothesis - in this case you’d reject that the line that was created is a suitable model as it was not able to create significant results

Question 23

Steps to Calculate the Correlation Coefficient

Answer 23

1. Calculate the mean for all x values(x bar) and the mean for all y values (y bar)
2. Calculate the standard deviation of all x values (S_x) and the standard deviation for y values (S_y)
3. Calculate (x - x bar) and y - y bar) for each pair (x, y) and then multiply these differences together
4. Get the sum by adding all these products together
5. Divide the sum by S_x X S_y
6. Divide the results of step 5 by n - 1, where n is the number of (x, y) pairs