exploratory data analysis (week 1-2)

Question 1

types of data analysis

Answer 1

descriptive, inferential, predictive

Question 2

what is descriptive data

Answer 2

summarize data, highlight any patterns, have central tendency, dispersion, and shape of distribution

Question 3

what is inferential data

Answer 3

collect sample to represent the wider population, estimate parameter, testing hypothesis

Question 4

what is predictive analysis

Answer 4

use past data to make predictions, divide data into training and testing set

Question 5

process of data analysis

Answer 5

a. develop clear analysis
b. identify data required
c. collect data (external and internal)
d. process/format/clean data
e. perform exploratory or preliminary data analysis (basic)
f. fit the model into the data
g. communicate the result
h. monitor ongoing experience
i. comply with professional guidance and legal requirements

Question 6

4 data resources?

Answer 6

a. simple random sampling
b. stratified sampling
c. cross-sectional data
d. longitudinal data

Question 7

what is simple random sampling?

Answer 7

random, so have equal chance to be selected

Question 8

what is stratified sampling?

Answer 8

split group to specific criteria, then pick random

Question 9

what is cross-sectional data?

Answer 9

different variable of interest are recorded across all objects at a single point of time

Question 10

what is longitudinal data?

Answer 10

different variables of interest of particular object are recorded closely

Question 11

type of error in the data collected (3)

Answer 11

a. censored data (only partially known)
b. truncated data (some values are missing)
c. big data (need machine learning)

Question 12

reproducility

Answer 12

all information to produce the same step of research is given and produce the same result, so third party can start from scratch

Question 13

replication

Answer 13

data to generate the same result is provided

Question 14

what does exploratory data analysis do

Answer 14

it analyze the data and identify any basic patterns or relationship. It also find most important variables, detect any data error.

Question 15

exploratory data analysis on univariate var

Answer 15

mean, median, quantile, sd, skewness, dsb

Question 16

exploratory data analysis on multivariate var

Answer 16

scatterplot, association measures, PCA

Question 17

what is the diff between correlation and association/relationship

Answer 17

correlation for uni variable, while association/relationship can be used for multilinear relationship

Question 18

what is bivariate analysis

Answer 18

assess the strength / shape of relationship between 2 RV
(if lurus then perfect corr)

Question 19

covariance

Answer 19

E[XY] - E[X]E[Y]

Question 20

correlation

Answer 20

Covariance / SD[X]/SD[Y]
between -1 and 1

Question 21

type of correlations (2)

Answer 21

pearson and kendall

Question 22

how to answer question for pearson corr?

Answer 22

1. find the corr (p hat)
2. find calc test stats
3. find the t using table

Question 23

pearson correlation used for?

Answer 23

quantitative work, measure of linear dependance for multivariate dist

Question 24

dependance vs corr

Answer 24

if X and Y are independent, corr is 0 but not vice versa

Question 25

kendall tau used for?

Answer 25

measure ordinal relationship (order, but how much not important), discordance vs concordance, less affected by extreme values, can be used as a measure of overall dependance

Question 26

how do we calculate kendall tau?

Answer 26

a. calc number of dis and con
b. plug it in to the formula sample tau
c. calc test stats
d. find the N(0.1) in the table

Question 27

Multivariate analysis

Answer 27

same as bivariate, to assess strength of relationship but it between several random RV

Question 28

PCA (Principal Component Analysis)

Answer 28

due to overlapping information between correlated multivariate RV, we make new uncorrelated var and using first few of them which explain most of data variability.

Question 29

each new var on PCA is

Answer 29

linear combination of original var

Question 30

PCA can be used for

Answer 30

summarises high dimensional data, identifying major patterns, analysing trends, performing regresison

Question 31

how to perform PCA?

Answer 31

a. centre the matrix X
b. compute X^tX
d. diagonalnya diminus lamda
e. diagonal dikali dikurangi perkalian diagonal sebelah
f. cari lamda
g. use the b matrix and minus the diagonal with the lambda (if hv 2 lambda, do step g-i 2 times)
h. kalikan matrix dgn matrix ab
i. eigenvectors dgn cara ..b = ..a terus itu dibuat matrix (b)
(a)
j. matrix di h dipangkat 2 each trs di tambah hasilnya = k trs 1/sqrt(k)
k. hasil j ditaruh depan hasil i
l. hasil k dikali hasil dri a terus setiap valuenya itu adalah z. misal top left row is z(11) trs top right itu z(12)

Question 32

Singular Value Decomposition

Answer 32

decompose X into U (PCs), D (contains sqrt(eigenvalues)), V (contain loadings)

Question 33

How to choose PC?

Answer 33

Find that at least 90% of data variability
Or perform scree test (var vs PCs), choose PCs after which scree plot levels off