Chapter 2 Flashcards
What does explanatory data analysis do and how?
Takes the available information and analyse it to summarise the whole data set.
It uses descriptive statistical techniques.
How does EDA compare to data mining?
EDA uses descriptive statistical techniques.
Whereas data mining uses descriptive and inferential methods.
What is the purpose of EDA?
To describe the structure and the relationships present in the data, for eventual use in a statistical model
What is univariate exploratory data analysis?
Analysis of individuals.
It is an important step in preliminary data analysis.
What does univariate data analysis usually consist of?
Graphical displays and a series of summary indexes.
What kind of graphical univariate analysis is carried out for qualitative nominal data?
Bar charts and pie charts
What kind of graphical univariate analysis is carried out for ordinal qualitative and discrete quantitative variables?
Frequency diagrams (ie bar charts where order on the horizontal axis is important)
What are the main unidimensional / univariate statistical indexes?
- Measures of location
- Measures of variability
- Measures of heterogeneity
- Measures of centrality
What are the measures of location?
Mean, mode and median
What is the formula for the mean?
[See flashcard]
How is univariate data classified?
In terms of a frequency distribution
What are the measures of variability?
The range, interquartile range and variance.
What is the range?
The difference between the maximum and minimum observations
What is the interquartile range?
The difference between the third and first quartile
What is the formula for the sample variance?
[See flashcard]
What types of data can measures of location and measures of variability be used for?
Only for continuous/quantitative data.
They cannot be used for qualitative data.
How can the dispersion of qualitative data be calculated?
Measures of heterogeneity - the spread of data
What is null heterogeneity?
When all observations have X equal to the same level (ie they all belong to the same category, so there is no spread).
Pi = 1 for a certain I
Pi = 0 for all other k-1 levels
What is maximum heterogeneity?
When all observations are uniformly distributed among the k levels.
Pi = 1/k for all i
How can you asses the dispersion of qualitative data?
- Gini index of heterogeneity
- Entropy
What is the formula for the Gini index of heterogeneity?
[See flashcard]
What does G = 0 represent?
Perfect homogeneity ie everything belongs to the same category
What G = 1 - 1/K or (k-1)/k represent?
Maximum heterogeneity - all categories represented equally
How do we normalise the Gini index of heterogeneity?
By dividing by the maximum value
What values does the normalised Gini index take?
0 to 1
What is the formula for the normalised Gini index?
[See flashcard]
What is the formula for Entropy?
[See flashcard]
What does E = 0 represent?
Perfect homogeneity
What does E = log K represent?
Maximum heterogeneity
What is the normalised index called?
The relative index of heterogeneity
How do you obtain the normalised index E
Rescale by the maximum value (log K)
[See flashcard]
What are the measures of concentration?
The Gini Coefficient R (a summary index of concentration)
What are measures of concentration?
They help understand the concentration of the characteristic among the N quantities
What are the two extreme situations which can occur for measures of concentration?
Minimum concentration - equal income for all (everyone has equal salary) x1 = x2 = …. = xn = x
Maximum concentration - someone gets all the income x1 = x2 = … xn-1 = 0 and xn = N*x_bar
The degree of concentration can lie between these two extremes
What is the equation for the Gini Coefficient R and the relevant steps associated with it?
[See flashcard]
What are the conditions for the variables investigated for the Gini Coefficient R?
There are N non-negative quantitates measuring a transferable characteristic (eg a fixed amount of income among N individuals) placed in an increasing (non-decreasing) number.
What is Fi?
The cumulative proportion of considered units, up to unit i
What is Qi?
The cumulative proportion of characteristic that belongs to the first I units
What do you need to remember about the sums associated with the Gini concentration index R?
They sum up to N-1, don’t include the final value
What does R = 0 represent?
Minimum concentration
What does R = 1 represent?
Maximum concentration
What is a challenge of multivariate data?
The sheer complexity of the information
What are the methodologies for exploring and simplifying complex multivariate data?
- Principal components
- Exploratory factor analysis
What is principal components analysis (PCA)?
A data-reduction technique that transforms a larger number of correlated variables into a much smaller set of uncorrelated variables called principal components.
What is exploratory factor analysis (EFA)?
A collection of methods designed to uncover the latent structure in a given set of variables.
It looks for a smaller set of underlying or latent constructs that can explain the relationships among the observed variables.
eg a dataset of 24 variables has intercorrelations that can be explained by 4 underlying factors.
What are principal components?
Uncorrelated composite variables, used to reduce dimensionality.
They aim to ratio as much information from the original set of variables as possible.
They are linear combinations of the observed variables. The weights used to form the linear composites are chosen to maximise the variance each PC accounts for, while keeping the components uncorrelated.
What are factors?
Factors are assumed to underlie or “cause” the observed variables in exploratory factor analysis, rather than being linear combinations of them.
Errors represent the variance in the observed variables unexplained by the factors.
The factors and errors aren’t directly observable but are inferred from the correlations among the variables.
Curved arrows between factors indicate that they are correlated.
Describe the process of principal component analysis
A statistical technique that linearly transforms an original set of p correlated variables into a new set of k uncorrelated variables called principal components. These are a substantially smaller set of variables that represent most of the information in the original set - they maximise the variance accounted for in the original p variables.
What order are the principal components derived in?
Decreasing order of importance so that the 1st PC accounts for as much as possible of the variation in the original data.
What is the objective of PCA?
To see if the first few components account for most of the variation in the original data. If they do, then it is argued that the effective dimensionality of the problem is less than p (the original number of correlated variables).
What are the goals of PCA?
Reduce the dimensionality of the original data set
A smaller set of uncorrelated variables is much easier to understand and use in further analysis than a larger set of correlated variables.
What does reducing the dimensionality of the problem do?
Simplifies the complexity of the data.
Makes it easier to visualise.
What do PCs reveal about the structure of the data?
Principal components are the underlying structure in the data.
They are the directions where there is the most variance, the directions where the data is most spread out.
How do we use mathematics to find the principal components?
Using eigenvectors and eigenvalues - we can destruct the set of data points into eigenvectors and eigenvalues.
What is the relationship of eigenvectors to each other?
Orthogonal to each other.
The eigenvectors have to be able to span the whole [x-y] area. In order to do this most effectively, the directions need to be orthogonal.
The eigenvectors then provide a much more useful axis to frame the date in.
What do eigenvectors and eigenvalues represent?
Eigenvector - direction
Eigenvalue - number, telling us how much variance there is in the data in that direction. Telling us how spread out the data is on the line.
Which eigenvectors is the principal component?
The eigenvectors with the highest eigenvalues
How many eigenvectors/values exit?
The same number of dimensions that the data set has.
The eigenvectors put the data into a new set of dimensions, so these new dimensions have to be equal to the original amount of dimensions.
Does investigating eigenvectors change the data?
No, we are just looking at it from a different angle.
We are shifting from one set of axes to another. We rearrange the axes to be along the eigenvectors.
These new axes are much more intuitive to the shape of the data.
How do the eigenvectors compare to the original axes?
They are more intuitive to the shape oof the data.
However, the original axis were well defined (we explicitly measure these things), and the new axes are not.
There is often a good reason why the new axes represent the data better, but the maths won’t tell us why. Data scientists have to work out the meanings of the new axes.
What is the main way PCA and eigenvectors help in the analysis of data?
Dimensionality reduction.
What does dimension reduction do?
Reduces the data down into its basic components, stripping away any unnecessary parts.
What matrix do we use to carry out PCA?
The correlation matrix, R
This is the variance - covariance matrix of standardised variables.
We have to standardised the matrix of data X (with n rows and p columns) to give matrix Z so that each column has variance 1 and mean 0.
What is the first principal component of the data matrix Z described by?
It is a vector described by a linear combination of the variables.
In matrix terms: Y1 = Z * a1
Z - the original standardised matrix
A1 - the vector of coefficients (weights)
How are the weights chosen?
Chosen to maximise the variance of the variable Y1.
Y1 is maximised when the weights are chosen to be the eigenvectors corresponding to the largest eigenvalues of the correlation matrix.
How is the 1st PC determined?
By the vector of weights a1 such that the variance of Y1 is maximised, under the constraint a1 ‘ a1 = 1
(a1 to itself = 1)
How is the 2nd PC determined?
Y2 = Za2
where the vector of coefficients is chosen in such a way that the variance of Y2 is maximised, under the constraints a2 ‘ a2 = 1 and a2 ‘ a1 = 0 (they are perpendicular)
It can be shown that a2 is the eigenvector (normalised and orthogonal to a1) corresponding to the second largest eigenvalues of R.
What is the general formula for the vth principal component for v = 1, … , k?
It is the linear combination
Yv = Zav
In which the vector of coefficients av is the eigenvectors of R corresponding to the vth largest eigenvalues.
This eigenvectors is normalised and orthogonal to all the previous eigenvectors.
What is the variance of the vth principal component?
The vth eigenvalues
Var(Yv) = lambda v
What does the covariance between the principal components satisfy?
Cov(Yi, Yj) = 0
Should be 0 since they are perpendicular
How can you describe the variance-covariance matrix of Y?
A diagonal matrix - where the lambdas 1-K appear along the diagonal and all other values are 0.
What is the proportion of variability that is maintained in the transformation from the original p variables to k <= p components?
(1/p) * sum(lamda i)
eg 10 original variables to 3 PCs
Proportion of variability maintained by 3PCs is L1 + L2 + L3 / 10
This equation expresses a cumulative measure of the quota of variability “reproduced” by the first k components, with respect to the overall variability present in the original data matrix.
Therefore it can be a measure of importance of the chosen k PCs, in terms of “quantity of information” maintained by passing p variables to k components.
How do we calculate the relative importance of each PC (with respect to the original variables)?
Consider the general covariance between a PC and the original standardised variables Z.
This helps us interpret what the new PCs mean - if PC1 is highly correlated with age, we know it tells us a lot about age.
What is the linear correlation between a PC (Yj) and the original standardised variable (Zi)?
Corr(Yj, Zi) = sqrt(lambda-j) * a-ij
The linear correlation between PC Yj and the original variable Xi
Corr(Yj, Xi) = Corr(Yj, Zi) = sqrt(lambda-j) * a-ij
What is the loading?
The algebraic sign and value of the coefficient a-ij.
This determines the sign and strength of the correlation between the jth PC and the ith original variable.
What is the formula for the portion of variability of an original variable Xi explained by k principle components?
Portion of variability = sum (lambda-j * a-ij^2)
The portion of variability is the sum of the square of the appropriate correlation terms.
This describes the quota of variability of each explanatory variable that is maintained in passing from the original variables to the principal components.
What does knowing the portion of variability enable?
We can interpret each PC by referring it mainly to the variables with which it is strongly correlated.
If all correlations are high between variables (as displayed in a correlation matrix), what might we suspect?
That a few principal components would explain most of the variation in the original variables.
What should all the eigenvalues add up to?
The total number of original values - the information of the original values has been redistributed.
How do you calculate the percentage of the original variance that a principal component accounts for?
Take the eigenvalues for that PC and divide it by the total number of original variables.
What do the eigenvalues represent?
The variance accounted for by each principal component
How do we compute principal component scores?
Y1 = Z * av
where av is a vector of coefficients, the eigenvector of R corresponding to the Vth largest eigenvalue
What is sometimes referred to as component loading?
The correlations of the original variables with the principal components
How do you obtain the correlations of the original variables with the principal components?
Corr(Xi, Yj) = sqrt(lambda) * aij
ie multiply the elements of the eigenvectors by the square root of the corresponding eigenvalue.
What does a principal component loading of relatively large magnitude (approaching magnitude of 1 indicate)?
Indicate that the corresponding original variable was important in defining that particular principal component the element relates to.
If the original variables are all positively correlated with a particular PC, what will the elements of the corresponding eigenvector be?
They will all be positive
If the correlations of the original variables and a particular PC are all of the same magnitude, then what can be said of the elements in the corresponding eigenvector?
They will be of about the same magnitude.
That is if we were to create a principal component score using the elements of the eigenvector, it would essentially be an equally weighted average of each variable.
What is the size factor?
The largest principal component [in these circumstances]
Why might the second PC be less uniform than the first?
It has to be orthogonal to the first (and all other PCs). In order for this to be fulfilled, the sum of the cross product of the elements of 1 PC eigenvector to another must be zero.
So, since the first PC has all positive values, the second must be a mix of positive and negative.
When we have an overall size factor, the succeeding principal components with alternating positive and negative signs are usually interpreted as contrasts.
Why are the first few PCs in many instances more interpretable?
They explain most of the variation in the set of variables.
Frequently, the smaller components are more difficult to interpret as to what they represent.
Why is it useful to examine the sum of the squares of the loadings of each row of the principal component loading matrix?
The row sum of squares indicates how much variance for that variable is accounted for by the retained PCs.
Eg if we retain 2 PCs, add together the two corresponding values int he row to see how much variation is maintained (cumulative)
How many principal components should you choose?
We want to choose enough to adequately represent the data. There are many different criteria suggested to do this
- Include any component with an eigenvalue (lambda-i) >= 1
- Cattell’s scree criterion
- Retain enough PCs to account for at least X% of the overall variability
- Retain enough PCs to account for at least X% of the variation in each variable
Describe the rationale for including any component that has an eigenvalue (lambda-1) >= 1
The overall variance is equal to p, so the average variance should be equal to at least 1.
What is Cattell’s scree criterion?
Look at the scree plot of the ordered eigenvalues.
The components which make up the steepest part of the curve are included, whilst those on the flatter part are discarded.
Sometimes the elbow itself is included and sometimes it is not.
What is the point of separation of the Scree plot called?
An elbow
What does the number of components retained depend on?
The eventual use of the PCs
If they were going to be used as independent variables in a regression analysis, then we might want to retain components such that all the variables are adequately represented by the PCs.
What are principal component scores calculated for?
Individual rows
Why would you plot the principal component scores for any pair of principal components?
Check for outlying observations, searching for clusters and in general understanding the structure of the data.
What are reasons data can be missing?
- Participants forget to answer one or more questions
- Refuse to answer sensitive questions
- Grow fatigued and fail to complete a long questionnaire
- Study participants miss appointments or drop out
- Recording equipment fails
- Data miscoded
- Data may be lost for reasons you may never be able to ascertain
What do most statistical methods assume?
That you are working with complete data
What is a comprehensive approach for data handling?
- Identify the missing data
- Examine the causes of the missing data
- Delete the cases containing missing data, or replace (?impute) the missing values with reasonable alternative data values
What are the three missing data-mechanisms defined in Rubin’s classification system?
- Missing completely random (MCAR)
- Missing at random (MAR)
- Missing not at random (MNAR)
It depends on how the missing data process is related to the underlying hypothetical complete data.
When are the data described as missing completely at random (MCAR)?
If the missingness of a variable Y is unrelated to either the value of Y or that of other measured variables.
ie the observed data points are a simple random sample of the data had the data been complete. Missing cases are no different than non-missing cases.
These can be thought of as randomly missing. The only really penalty in failing to account for missing data is loss of power.
What are some examples of MCAR?
- There are random missing questions throughout a survey
When are the data described as missing at random (MAR)?
When the missingness of a variable Y is unrelated to the value of Y itself after conditioning on other observed values.
ie missing data depends on known values and thus it is fully described by variables in the dataset. Accounting for the values which “cause” the missing data will produce unbiased results in an analysis.
What are some examples of MAR?
- Education survey where GCSE question was left blank - it didn’t make sense for global people
When are data described as missing not at random (MNAR)?
When the missingness of a variable Y still depends on the value of Y even given the observed variables.
When data is missing in an unmeasured fashion, this is also termed “non-ignorable”. Since the missing data depends on events or items which the researcher has not measured this is a damaging situation.
Can’t infer from the dataset why
Which of the mechanisms is it possible to verify?
- Can verify whether data are MCAR or not
- Impossible to test the MAR mechanism (with exception)
Which mechanism is assumed in many popular techniques?
MAR
Describe accessible vs inaccessible missing data mechanisms?
- An accessible mechanism is one where the cause of missingness can be accounted for
- MCAR and most MAR
- An inaccessible mechanism is one where the missing data mechanism cannot be measured
- Nonignorable mechanisms and MAR mechanisms
Often the missing data mechanism is made up of both accessible and inaccessible factors.
What can you do to evaluate potential mechanisms leading to missing data and the impact of missing data on your ability to answer questions?
Identify the amount, distribution and pattern of missing data
- What % of data is missing?
- Are the missing data concentrated in a few variables or widely distributed?
- Do the missing values appear to be random? - this may be seen when plotted
- Does the data suggest a possible mechanism that’s producing the missing values?
If missing data are concentrated in a few relatively unimportant variables, what can you do?
Delete the variables and continue the analysis normally
If a small amount of data (< 10%) are randomly distributed throughout the dataset (MCAR) what can you do?
Limit the analysis to cases with complete data and still get reliable and valid results
If you have MCAR or MAR data, what can you do?
Apply multiple imputation methods and arrive at valid conclusions
If the data are NMAR what can you do?
Turn to specialised methods or collect new data
What are some examples of NMAR
- Depression study where questions are omitted describing depressed mood by depressed people and older people
- Cancer questionnaire where < 10 was omitted from dataset
What are the three most popular approaches for dealing with missing data?
- A rational approach for recovering data
- A traditional approach involving deleting missing data or simple data imputation
- A modern approach that is based on simulations to perform the missing data imputation
What is the rational approach for recovering data?
Using mathematical or logical relationships among variables to attempt to fill in or recover missing values. It may be exact or approximate
- Eg variables which are part of an equation, can use one to calculate another
- Eg date of birth and age etc.
- Eg using numerical answers to answer binary questions
What are traditional approaches for dealing with incomplete data?
List-wise and pair-wise deletion methods
The advantage of these methods is that they are convenient and are standard options.
What is list wise deletion?
A case is dropped from an analysis because it has a missing value in at last on of the specified variables.
What is pair wise deletion?
Approach which uses cases which contain some missing data.
eg looking at the subset of the dataset that contains eg two columns that we want to find correlation between
What are many multivariate techniques, eg multivariate regression, computationally based on?
A correlation matrix (pair-wise correlations of all variables) that is computed as the first step
What does list-wise deletion assume?
MCAR data
If this assumption does not hold it can produce distorted parameter estimates
It is not recommended unless the portion of missing data is very small.
What are the advantages/disadvantages of pair-wise deletion?
- It allows you to use more of your data
- Each computed statistic may be based on a different subset of cases which can be problematic - things may become irrelevant
What is simple imputation?
You substitute a value for each missing value. Standard statistical procedures for complete data analysis can then be used with the filled-in data set.
- Eg input the variable mean of complete cases
- Eg input the mean conditional on observed values of other variables
Simple imputation does not reflect the uncertainty about the predictions of the unknown missing values, and the resulting estimated variances of the parameter estimates will be biased toward zero.
What is the modern approach for dealing with incomplete data based on?
Simulations
What is multiple imputation (MI)?
An approach to dealing with missing values based on repeated simulations.
Instead of filling in a single value for each missing value, replace each missing value with a set of plausible values that represent the uncertainty about the right value to input.
The multiple imputed data sets are then analysed using standard procedures for complete data and combined,
How many datasets are usually generated in multiple imputation (MI)?
3 - 10
How many distinct phases are there of multiple imputation inference and what are they?
Three
1 - the missing data are filled m times to generate m complete data sets
2 - the m complete data sets are analysed by using standard procedures
3 - the results from the m complete data sets are combined for the inference
What are imputation methods dependent on?
Types of missing data pattern / types of missingness
What is used for monotone missing data patterns?
- Parametric regression method - assumes multivariate normality
- Nonparametric method which uses propensity scores
What is used for an arbitrary missing data pattern?
A Markov chain Monte Carlo (MCMC) method that assumes multivariate normality.
This creates multiple imputations by using simulations from a Bayesian prediction distribution for normal data.
Another way to handle a data set with arbitrary missing data pattern is to use the MCMC approach to impute enough values to make the missing data pattern monotone.
What is a monotone missing pattern?
When a variable Y is missing for the individual I implies that all subsequent variables are missing for the individual i.
You have greater flexibility in your choice of strategies.
You can implement a regression model without involving iterations as in MCMC
