EDA

Question 1

Continuous vs Discrete Data

Answer 1

Numerical data

1) Data that can take on any value in an interval
2) Data that can be taken only as an integer value, such as counts

Question 2

Binary vs Ordinal Data

Answer 2

Categorical data - specific set of values representing a set of possible categories

1) special case of categorical data with just 2 categories of values
2) categorical data that has an explicit ordering

Question 3

Advantages of explicit identification of data

Answer 3

1) tells software how statistical procedures, such as making a chart of fitting a model should behave
Sk.learn.preprocessing.OrdinalEncoder
2) storage and indexing can be optimized
3) possible values given a categorical variable can take are enforced in the software

Question 4

Types of nonrectangular data structures

Answer 4

1) time series - successive measurements of the same variable, raw material for statistical forecasting methods
2) spatial data structure - mapping and location
3) graph data structures - physical, social, and abstract relationships

Question 5

Estimate

Answer 5

Getting a typical value for each features: an estimate of where most of the data is located

Question 6

Mean

Answer 6

Average

Sum of all values divided by the number of values

Question 7

Weighted mean

Answer 7

weighted average

Sum of all values times a weight divided by the sum fo the weights

Question 8

Median

Answer 8

Value such that 1/2 of the data lies above and below

Question 9

Percentile

Answer 9

Quantile

Value such that p percent of the data lies below

Question 10

Weighted median

Answer 10

Value such that one half of the sum of the weights lies above and below the sorted data

Question 11

Trimmed mean

Answer 11

truncated mean
Average of all values after dropping a fixed number of extreme values
Eliminates influence of extreme values

Question 12

Robust

Answer 12

Not sensitive to extreme values

Median is a robust estimator

Question 13

Outlier

Answer 13

Data value that is very different from most the data

Question 14

Metrics vs estimates

Answer 14

Statisticians estimate - account for uncertainty - draw a distinction between what we see from the data and the theoretical true or exact state of affairs
Metric - concrete business or organizational objectives at the focus of data science

Question 15

anomaly detection

Answer 15

Points of interest that are the outliers, and the greater mass of data serves primarily to define the “normal” against which anomalies are measured

Question 16

Dispersion/variability

Answer 16

Measures whether the data values are tightly clustered or spread out
Heart of stats: measure, reduce, and distinguishing random from real variability, identify various sources of real variability and making decisions in the presence of it

Question 17

Deviations

Answer 17

Difference between the observed values and the estimate of location (mean)
Errors, residuals

Question 18

Variance

Answer 18

Sum of squared deviations from the mean divided by n-1 where n is the number of data values

an average of the squared deviations

Mean squared error

Question 19

Standard deviation

Answer 19

Square root of the variance

I2-norm, Euclidean norm

Question 20

Mean absolute deviation

Answer 20

Mean of the absolute value of the deviations from the mean

I1norm, manhattan norm

Question 21

Median absolute deviation from the median

Answer 21

The median of the absolute value of the deviations from the mean

Question 22

Range

Answer 22

Difference between the largest and the smallest value in a dataset

Question 23

Order statistics

Answer 23

Metrics based on the data values sorted from smallest to biggest

Question 24

Percentile

Answer 24

Value such that P percent of the values take on this value or less and (100-P) percent take on this value or more

Quantile

Question 25

Interquartile range

Answer 25

Difference between the 75th percentile and the 25th percentile IQR

Question 26

Degrees of freedom

Answer 26

N-1 in denominator instead of n If you use n you will underestimate the true value of the variance and the standard deviation in the population - biased When you do n-1, the variance becomes an unbiased estimate Degrees of freedom = takes into account the number of constraints in computing an estimate One constraint - standard deviation depends on calculating the sample mean

Question 27

Boxplot

Answer 27

Visualize distribution of the data Top and the bottom of the box are 75th and 25th percentiles, respectively Had Median is the horizontal line in the box Whiskers - extend from the top and bottom to indicate the range of the bulk of the dataw

Question 28

Frequency table

Answer 28

Tally of the count of numeric data values that fall into a set of intervals

Question 29

Histogram

Answer 29

Plot of the frequency table with the bins on the x-axis and the count on the y-axis

Question 30

Density plot

Answer 30

Smooth version of the histogram, often based on a kernel density estimate

Question 31

Why make bins?

Answer 31

Both frequency tables and percentiles summarize the data by creating bins In general, quartile and deciles will have the same count in each bin (equal-count bins), but bin size will be different Small bins = result is too granular and the ability to see bigger pictures is lost

Question 32

Statistical moments

Answer 32

1) location 2) variability 3) skew ness 4) kurtosis

Question 33

Skewness

Answer 33

Refers to whether the data is skewed to larger or smaller values

Question 34

Kurtosis

Answer 34

Propensity of the data to have extreme values

Question 35

Density estimates, density plot

Answer 35

Smoothed histogram A density plot corresponds to plotting the history ram as a proportion rather than counts

Question 36

Mode

Answer 36

The most commonly occurring category or value in a data set

Question 37

Expected value

Answer 37

When the categories can be associated with a numeric value, this give an average value based on a category’s probability of occurence 1) multiply each outcome by its probability of occurring 2) sum these values - future expectations and probability weights

Question 38

Bar charts

Answer 38

Frequency or proportion for each category plotted as bars

Question 39

Pie charts

Answer 39

Frequency or proportion for each category plotted as wedges in a pie

Question 40

Correlation coefficient

Answer 40

A metric that measures the extent to which numeric variables are associated with one another (ranges from -1 to +1) Multiply deviations from the mean for variable 1 times those for variable 2 and divide by the product of the standard deviations

Question 41

Correlation matrix

Answer 41

A table where the variables are shown on both rows and columns, and the cell values are the correlations between the variables

Question 42

Scatterplot

Answer 42

A plot in which the x-axis is the values of one variable, and the y-axis the value of another

Question 43

Contingency tables

Answer 43

A tally of counts between 2 or more categorical variables

Question 44

Hexagonal binning

Answer 44

A plot of two numeric variables with the records binned into hexagons

Question 45

Contour plots

Answer 45

A plot showing the density of 2 numeric variables like a topographical map

Question 46

Violin plots

Answer 46

Similar to a boxplot but showing the density estimate Plot a numeric variable against a categorical variable

Question 47

Boxplot

Answer 47

Visually compare the distributions of a numeric variable grouped according to a categorical variable