Lecture 5: Principle Component analysis

Question 1

What does a PCA attempt to do?

Answer 1

Attempts to explain variables in as few new ‘components’ as possible

Question 2

How does a PCA attempt to explain variables in as few new ‘components’ as possible? What are these components

Answer 2

Transforms variables by rotating axes, these are the new
components.
•Axes are chosen to maximise explained variance.
•Total variance of new components is the same as total variance of the original variables.

Question 3

Explain the process of axis rotation

Answer 3

First we rotate the axis on which the data points are plotted on to draw a line through the dimensions (variables/ data points) which accounts for the most variance. You could describe this as the line that describes the data the best or costs to the data points in a p-dimensional space. For our second component we find a line that is perpendicular to the first one in one of the p-1 dimensions (if 2D then simply perpendicular.) If there are more than 2 dimensions then we repeat step 2 for each of the remaining p dimensions, finding a line through the data which is perpendicular to all previous components and explains the most possible variance in the data.

In other words:
1. Maximises amount of variance accounted for by the 1st (principal) component.
2. For each subsequent component: maximises variance accounted for as long as:
• component is orthogonal (perpendicular, uncorrelated) to all previous components

Question 4

How is the maximum number of components decided?

Answer 4

The number of dimensions/ variables

Question 5

How are the different principle components related to each other?

Answer 5

They’re not; Each principal component loading vector is unique (excluding sign flip)

Question 6

How does total variance differ following a PCA?

Answer 6

It stays the same

Question 7

What is meant by the eigen vectors in terms of PCA?

Answer 7

The orientation of the components (lines through the data)

Question 8

Finish this from the first lecture:

All p x p matrices A have _______ for which __ = __

Answer 8

All p x p matrices A have associated scalers λj and vectors xj for which Axj = λjxj

Question 9

In regards to PCA, what do the variables represent in the following equation?

Axj = λjxj

Answer 9

A represents an R (correlation) or S (covariance) matrix. λj represents the eigen value component, how much variance the component explains. xj represents the eigenvectors; how much each variable in the dataset loads onto each component (e.g 40% of petal width and 90% of petal length.)

Question 10

How can we project data onto the new component axis?

Answer 10

We can project the data onto the new component axis using a weighted sum of the original variables with the weights being the eigen vectors.

Question 11

Describe two important constraints when determining these component vectors

Answer 11

1. We choose the eigen vector that maximises the accounted variance.
2. The squared elements elements of an eigen vector add up to one

Question 12

Should you centre your variables in a PCA? What happens if you do or don’t?

Answer 12

You should always centre your variables and if variances differ greatly, you can also scale them with their standard deviation. If you don’t centre your variables then you’ll have to centre rotate the data after the PCA, so the origin of the new components at the new zero point.

Question 13

The eigen vectors are also known as the _____ and tell us how _____

Answer 13

The eigen vectors are also known as the rotation matrix and tell us how much to rotate the original axis.

Question 14

What does this matrix look like and how is it used?

Answer 14

the vector matrix may look like this in R;
(0.38772  -0.92178)
(0.92178    0.38772)
The rotation matrix template looks like:
(cosθ  -sinθ)
(sinθ    cosθ)

Since 0.388 represents the cosine of an angle, we take the arcosine to solve for the angle theta. arccoss .388 equals about 67 degrees so we rotate our component counterclockwise by 67 degrees.

Question 15

What is the next step after we have rotated the axis?

Answer 15

Plug the values into the components and we can have a new formula to draw the new axis for z1 and z2.9. Each component consists of the column values.
Component 1: z1 = 0.388y1 + 0.922y2
Component 2: z2 = 0.922y1 + 0.388y2

Question 16

If you have a point in the original axis with coordinates 1 for petal width and 1 for petal length, what would its coordinates be on the new axis?

Answer 16

You would take the first eigen vector element (.388) times the petal length (1) to get .388, then the second eigen vector element (.922) times the petal width (1). Component 1: .388 + .922 = 1.31. Component 1: .388 - .922 = -.534.

Question 17

What assumptions should we check before carrying out a PCA to check whether it is useful or not?

Answer 17

1. Check that R or S is not an identity matrix using functions from psych
   package.
2. Check if means and variances are similar. If not, then usually best to scale variables or analyse correlation matrix.

Question 18

How do we check that R or S is not an identity matrix using functions from psych package?

Answer 18

Using the KMO measure of sampling adequacy (>.7):
psych::KMO(cor(iris4D))

Or Bartlett’s test of spericity (p < .05):
psych::cortest.bartlett(cor(iris4D), nrow = nrow(iris4D))

Question 19

WHy use the psych packagesrather than the ones specified in the book?

Answer 19

Not are checked for their accuracy, we know the psych package is thoroughly checked

Question 20

How do we check if means and variances are similar?

Answer 20

apply(iris4D, 2, mean)
apply(iris4D, 2, var)

If one has a higher variance it will inevitably load on the 1st component so you should choos to conduct an alalysis with scaling.

Question 21

How do you carry out scaling?

Answer 21

In the line of code for the PCA, specify scale. = TRUE

Question 22

How do you centre your data in R?

Answer 22

Specify center = TRUE

Question 23

Write out a line of code to carry out a pca on the iris data

Answer 23

iris4Dpca

Question 24

how does the prcomp function produce the eigen composition different?

Answer 24

It gives the square roots of the eigenvalues which are standard deviations of components

Question 25

Give 4 commands to get 4 different types of data from the comp function

Answer 25

iris4Dpca$center #gets the original means
iris4Dpca$scale #equals original standard deviations
iris4Dpca$rotation #rotation matrix
iris4Dpca$x #transformed data

Question 26

How do you decide how many components you’re gonna use?

Answer 26

Create a scree plot and loot for the ‘elbow’. proportion explained (pve) = component variance explained / total variance

Question 27

How would you plot this in R?

Answer 27

pve

Question 28

What other plot is often used with a PCA? How do you plot this in R?

Answer 28

Biplot:
biplot(iris4Dpca,
xlabs=rep(“·”, nrow(iris4D)),
scale=0)

Question 29

How should you interpret a biplot?

Answer 29

You should look to see which dimensions load the most on which components by inspecting how far they are along each axis.

Question 30

Give four reasons on why we would use PCA

Answer 30

Dimension reduction: reduces  data 
down to its essential  components, 
dropping  unnecessary  ones.
• Essential when # predictors > # 
observations.

Solve multicollinearity: transform correlated variables to uncorrelated components
• Principal Components regression

Clustering alternative: Discover groups of items or people

Principal Axis Factoring (PAF):
• PCA way of conducting Factor Analysis
• conducted on correlation matrix where 1’s along diagonal are replaced by communality estimates (the % of variance explained by the underlying factors)

Question 31

What is PCAs main application in machine learning? Why is this helpful?

Answer 31

Dimension reduction

This is helpful for :
•Data visualisation
•Natural Language Processing (NLP)
•Recommender systems

Question 32

Contrast PCA and LDA/ MANOVA on transformations

Answer 32

PCA involves orthagonal linear transformations to rotate axes while the manova involves oblique linear transformations to rotate axes

Question 33

Contrast PCA and LDA/ MANOVA on their learning methods

Answer 33

PCA involves unsupervised learning, LDA involves supervised learning.

Question 34

Contrast PCA and LDA in regards to their relationship with variance

Answer 34

PCA maximises the explained variance for all observations while LDA/ MANOVA maximises explained variance for between group means

Question 35

Contrast PCA and LDA in regards to their assumptions

Answer 35

PCA has no assumptions about distributions as it’s a data transformation, not a statistical model. LDA assumes homoscadesticity and multivariate normality (although ignored for LDA in ML
applications).

Question 36

Contrast PCA with an exploratory factor analysis in regards to their transformations, learning, assumptions about variance, relationship with variance and the type of model

Answer 36

Both PCA and EFA use orthogonal linear transformations to rotate axes, use unsupervised learning and the variables are assumed not to contain errors. PCA maximises the explained variance for all observations while an EFA maximises explained variance common to all variables. PCA is not a statistical model while EFA is a statistical model.