12 - Dimension Reduction

Question 1

What is high dimensionality in data science?

Answer 1

High dimensionality refers to a data set with a large number of predictors, for example, 100 predictors describe a 100-dimensional space.

Question 2

What is multicollinearity?

Answer 2

Multicollinearity occurs when there is substantial correlation among predictor variables, leading to unstable regression models.

Question 3

What is double-counting in the context of regression models?

Answer 3

Double-counting occurs when highly correlated predictors overemphasize a particular aspect of the model.

Question 4

What is the curse of dimensionality?

Answer 4

As dimensionality increases, the volume of the predictor space grows exponentially, making the high-dimensional space sparse.

Question 5

What does the principle of parsimony state?

Answer 5

The principle of parsimony suggests that models should be simple and interpretable, keeping the number of predictors manageable.

Question 6

What is overfitting in regression models?

Answer 6

Overfitting occurs when too many predictors are included in the model, hindering generality to new data.

Question 7

What is the risk of missing the bigger picture in data analysis?

Answer 7

Focusing solely on individual predictors may overlook the fundamental relationships among them, which can be grouped into components.

Question 8

What are the three main objectives of dimension reduction methods?

Answer 8

• Reduce the number of predictor items
• Ensure that these predictor items are uncorrelated
• Provide a framework for interpretability of the results.

Question 9

What does multicollinearity lead to in regression analysis?

Answer 9

Multicollinearity leads to instability in the solution space, causing unreliable regression coefficients.

Question 10

What happens to regression coefficients when predictors are correlated?

Answer 10

The coefficients can vary widely across different samples, making them unreliable for interpretation.

Question 11

How can variance inflation factors (VIFs) indicate multicollinearity?

Answer 11

A large VIF indicates that a predictor is highly correlated with other predictors, with VIF ≥ 5 indicating moderate and VIF ≥ 10 indicating severe multicollinearity.

Question 12

What is the formula for calculating VIF?

Answer 12

VIF_i = 1 / (1 - R_i^2), where R_i^2 is the R-squared value obtained by regressing predictor i on the other predictors.

Question 13

What does a VIF of 6.85 indicate?

Answer 13

A VIF of 6.85 indicates moderate-to-strong multicollinearity for the predictor variable.

Question 14

What is principal components analysis (PCA)?

Answer 14

PCA seeks to account for the correlation structure of a set of predictor variables using a smaller set of uncorrelated linear combinations, called components.

Question 15

What is the significance of the first principal component?

Answer 15

The first principal component accounts for the greatest variability among the predictors.

Question 16

True or False: PCA considers the target variable during analysis.

Answer 16

False. PCA acts solely on the predictor variables and ignores the target variable.

Question 17

What should be done to predictors before applying PCA?

Answer 17

The predictors should be either standardized or normalized.

Question 18

Fill in the blank: The total variability produced by the complete set of m predictors can often be mostly accounted for by a smaller set of k < m __________.

Answer 18

[components]

Question 19

What is PCA?

Answer 19

Principal Component Analysis (PCA) is a technique for dimension reduction.

Question 20

What does PCA act on?

Answer 20

PCA acts solely on the predictor variables and ignores the target variable.

Question 21

What is the characteristic of the first principal component?

Answer 21

The first principal component accounts for greater variability among the predictors than any other component.

Question 22

How does the second principal component relate to the first?

Answer 22

The second principal component accounts for the second-most variability and is uncorrelated with the first.

Question 23

What is the purpose of varimax rotation in PCA?

Answer 23

Varimax rotation helps in the interpretability of the components.

Question 24

What is the cumulative variance explained by the first two components in the example?

Answer 24

The first two components account for about 52.2% of the variance.

Question 25

What does the eigenvalue criterion suggest for extracting components?

Answer 25

Only components with eigenvalues greater than one should be retained.

Question 26

What is the eigenvalue of 1.0 indicative of?

Answer 26

An eigenvalue of 1.0 indicates that the component explains about 'one predictor's worth' of variability.

Question 27

What is the proportion of variance explained criterion?

Answer 27

The criterion specifies the proportion of total variability that the principal components should account for.

Question 28

How many components should be extracted based on the consensus in the example?

Answer 28

k = 4 components should be extracted.

Question 29

What does the principal component matrix indicate after rotation?

Answer 29

The rotated component matrix provides clearer interpretations of the principal components.

Question 30

What should be done to validate PCA results?

Answer 30

The PCA results should be validated using the test data set.

Question 31

What is indicated by VIFs equal to 1 in the regression of Sales per Visit on principal components?

Answer 31

All VIFs equal 1 indicate the elimination of multicollinearity.

Question 32

What is the purpose of standardizing predictor variables before PCA?

Answer 32

Standardizing ensures that all predictors contribute equally to the analysis.

Question 33

What command is used to perform PCA in Python?

Answer 33

The PCA() command from the sklearn.decomposition module is used.

Question 34

What is the significance of the cutoff in PCA loadings in R?

Answer 34

The cutoff suppresses small PCA weights to enhance interpretability.

Question 35

What is the first step in performing PCA using R?

Answer 35

Import the required data sets and separate the predictor variables.

Question 36

True or False: PCA can increase multicollinearity in the data.

Answer 36

False

Question 37

What does the command 'scale()' do in R?

Answer 37

It standardizes the predictor variables.

Question 38

Fill in the blank: The first principal component is a combination of _______.

Answer 38

Different Items Purchased and Purchase Visits

Question 39

What is the minimum number of predictors needed for eigenvalue criterion to be valid?

Answer 39

At least 20 predictors.

Question 40

What is the purpose of the varimax rotation in PCA?

Answer 40

To simplify the interpretation of the components by maximizing variance among them. ## Footnote Varimax rotation is a method used in factor analysis to achieve a simpler and more interpretable factor structure.

Question 41

What command is used to obtain the correlation of the components in the training data set?

Answer 41

round(cor()) ## Footnote This command calculates the correlation matrix for the scores obtained from PCA.

Question 42

What does the VIF stand for in regression analysis?

Answer 42

Variance Inflation Factor ## Footnote VIF measures how much the variance of an estimated regression coefficient increases when your predictors are correlated.

Question 43

True or False: Multicollinearity adversely affects the ability of the sample regression equation to predict the response variable.

Answer 43

False ## Footnote Multicollinearity does not significantly impact the predictive ability of the regression model.

Question 44

What should be strictly limited when using a multicollinear model?

Answer 44

Estimation and prediction of the target variable ## Footnote Interpretation of the model is inappropriate due to nonsensical individual coefficients.

Question 45

What is the first step in running a regression model after obtaining PCA scores?

Answer 45

Save each component as its own variable ## Footnote This allows for the use of PCA components as predictors in the regression model.

Question 46

Fill in the blank: PCA replaces the original set of m predictors with _______.

Answer 46

principal components ## Footnote Principal components are linear combinations of the original variables that capture the maximum variance.

Question 47

What does the eigenvalue criterion help determine in PCA?

Answer 47

The number of components to retain based on eigenvalues greater than 1 ## Footnote Components with eigenvalues greater than 1 explain more variance than a single original variable.

Question 48

What is the proportion of variance explained criterion in PCA?

Answer 48

It helps determine the number of components to retain by explaining a specified percentage of total variance. ## Footnote Common thresholds are 70%, 80%, or 90% of total variance.

Question 49

What does a high VIF indicate in regression analysis?

Answer 49

That multicollinearity is a problem ## Footnote A VIF value above 5 or 10 is often considered indicative of multicollinearity.

Question 50

What is the relationship between the first and other principal components?

Answer 50

Some principal components may be correlated with the first principal component. ## Footnote Correlation among components can indicate shared variance or underlying relationships.

Question 51

How is multicollinearity defined in regression analysis?

Answer 51

The presence of high correlations among predictor variables ## Footnote Multicollinearity can inflate the variance of coefficient estimates, making them unstable.

Question 52

What does PCA stand for?

Answer 52

Principal Component Analysis ## Footnote PCA is a technique used for dimensionality reduction while preserving as much variance as possible.

Question 53

Which command is used to run a linear regression model in R?

Answer 53

lm() ## Footnote The lm() function is used to fit linear models in R.

Question 54

What is the significance of the output from the vif() command?

Answer 54

It shows the variance inflation factors for the regression model. ## Footnote VIF output helps identify potential multicollinearity issues among predictors.

Question 55

What should be done to predictors before running PCA?

Answer 55

Standardize or normalize them ## Footnote Standardization ensures that each predictor contributes equally to the analysis.

Question 56

What does the correlation matrix show in PCA analysis?

Answer 56

The relationships between predictor variables ## Footnote High correlations among variables indicate redundancy and potential multicollinearity.

Question 57

What is the target variable in the analysis of the red wine dataset?

Answer 57

Wine quality ## Footnote This dataset typically includes various chemical properties of red wine as predictors.

Question 58

How can the number of components to retain be determined?

Answer 58

By combining recommendations from the eigenvalue criterion and the proportion of variance explained criterion. ## Footnote This approach ensures a comprehensive evaluation of component significance.

Question 59

What does the term 'high dimensionality' refer to in data science?

Answer 59

Having a large number of features or predictors in a dataset ## Footnote High dimensionality can lead to overfitting and increased computational cost.

Question 60

What is the main purpose of dimension reduction methods?

Answer 60

To reduce the number of predictors while retaining essential information ## Footnote This helps improve model performance and interpretability.

Question 61

What does a plot of eigenvalues help determine?

Answer 61

How many components to retain based on their explained variance ## Footnote The eigenvalue plot visually represents the variance accounted for by each component.

Question 62

What is the correlation matrix used for in the context of predictors?

Answer 62

To identify highly correlated variables ## Footnote Understanding correlations helps in diagnosing multicollinearity issues.