Lecture 13: Dimension Reduction Flashcards
Principal Components Analysis
Along with EFA and CFA a technique for reducing multiple items to a smaller number of variables.
PCA is a data rotation technique designed to transform original items into uncorrelated components.
The primary goal is dimension reduction, where a small number of components are used to explain most of the variance in the items. This allows us to represent the variance in the items more efficiently.
E.g., if 10 items measure extraversion and one components explains most of the variance, we can keep that one component and discard the remaining nine.
One way to understand PCA is as a method of lossy compression of data. The components are ordered in such a way that the first component has the largest variance, the second component with the second-largest variance, and so on.
Exploratory Factor Analysis (EFA)
EFA is a latent variable method that assumes that latent variables (factors) cause people’s responses to the items. Example: extraversion may cause individuals to respond positively to questions about partying and socialising.
Each observed variable has a “factor loading” for each underlying factor. The factor loading represents how much that variable is influenced by the underlying factor.
High factor loadings indicate a strong relationship between the variable and the factor.
It is called “exploratory” because all variables are allowed to load on (contribute to) all factors, without a predefined structure. In practice, well-constructed questionnaires will exhibit high loadings of items on one factor and low loadings on others.
Confirmatory Factor Analysis (CFA)
CFA tests a theory about the specific association between latent variables and observed indicators. Unlike PCA and EFA which are exploratory, CFA is a confirmatory approach that tests how well a hypothesised measurement model fits the data.
The primary goal of CFA is to evaluate whether the data supports the hypothesised model. By doing so, researchers can determine if their theoretical model fits the observed data well, providing evidence for the validity of the underlying construct and the measurement instrument.
Comparing methods
Purpose:
PCA: Dimensionality reduction
EFA: Exploration of relationships among items and identification of latent constructs
CFA: Taking a predefined theory about which items relate to specific latent constructs
Assumption:
PCA: Does not assume latent variables; dropping components assumes they are irrelevant tor represent error variance
EFA: Assumes all items are caused by a smaller number of latent variables (factors)
CFA: Assumes specific items are caused by specific latent variables
Interpretation:
PCA: Components are mathematical constructs with no further meaning
EFA: Factors represent theoretical latent constructs
CFA: Factors represent known theoretical latent constructs
Strategies to determine the number of components to retain (within PCA)
1) Kaiser’s criterion
- You retain all the components with an Eigenvalue > 1. In other words, retain all items that explain more variance than just a single item’s worth.
This technique is subjective, but often helpful.
2) Horn’s Parallel Analysis (1965)
- Conduct many PCAs on random (fake) data with the same number of cases and variables as real data
- Retain components whose Eigenvalue exceeds the 95th percentile of Eigenvalues of random data
This is the best data-driven strategy for selecting the number of components, but it is not by default downloaded in SPSS.
3) Cattell’s scree plot
- Inflection point: the scree plot displays the eigenvalues of the principal components in descending order.The number of principal components to retain is typically chosen at or before the inflection point. Components before the inflection point capture the substantial amount of variance, and those after contribute relatively little.
Eigenvalue (labda)
Reflects how much variance a component explains. Each item contributes 1 to the total variance. The sum of all Eigenvalues is equal to the total number of items, k.
In other words, 6 items means a total eigenvalue of 6.
The proportion of variance in items explained by each component is equal to the Eigenvalue divided by the number of items, so r-square = labda / k
We can compute Eigenvalues in EFA just as in PCA by taking the column sums of the squared loadings and indicate the amount of variance explained by each factor. Eigenvalues are always smaller than the initial eigenvalues in PCA because some variance is now attributed to error variance.
Bayesian Information Criterion (BIC)
To directly compare models, one can compute the Bayesian Information Criterion (BIC) - a relative model fit index designed for comparing models, which balances model fit and complexity. It is computed from the chi square as follows:
BIC = X2 - df * log(n)
Estimating unknown factor loadings (EFA)
To conduct EFA, we estimate the unknown factor loadings. Two common estimation methods are Principal Axis Factoring (PAF) and Maximum Likelihood (ML). PAF is a default method in SPSS. It provides a solution, even when the model is complex or the data are non-normal. ML is the same estimator used for CFA, but may not perform well when the model is overly complex.
Factor loadings
Represent the correlations between each item and the extracted factors. They indicate the strength and direction of the relationship between the observed item and the (assumed) underlying factor. Factor loadings range from -1 to +1, with values closer to 1 indicating a stronger relationship.
Determinant
Used to detect multicollinearity. It has been argued that the determinant should be greater than 0.00001, which indicates multicollinearity is not too high.
Kaiser-Meyer-Olkin (KMO) statistic
Provides an estimate of the proportion of common variance among items. A higher KMO value indicates that more of the variance among items can be explained by common factors, making the data more suitable for factor analysis. Researchers can interpret the KMO value as follows:
0.00 - 0.49: unacceptable
0.50 - 0.59: miserable
0.60 - 0.69: mediocre
0.70 - 0.79: middling
0.80 - 0.89: meritorious
0.90 - 1.00: marvellous
Rotation
A method used to improve interpretability, which applies a linear transformation to the original factor loadings.
Two main types of rotation are orthogonal and oblique rotation.
Orthogonal rotation produces uncorrelated factors. The most common technique is VARIMAX rotation.
Oblique rotation allows factors to correlate; the most common technique is oblimin rotation. In the social sciences, it is often sensible to allow factors to correlate (e.g., different personality dimensions are probably associated).
