data science - statistics I

Question 1

EDA

Answer 1

Exploratory Data Analysis - first step of the data science project, familiarizing yourself with the data

Question 2

continuous data

Answer 2

data that can take any value in an interval (float)

Question 3

discrete data

Answer 3

data that can only take integer values, such as counts (int)

Question 4

categorical data

Answer 4

data that can take on only a specific set of values representing a set of possible categories

Question 5

binary data

Answer 5

special subset of categorical data that with just two category values

Question 6

ordinal data

Answer 6

categorical data that has an explicit ordering

Question 7

feature

Answer 7

often a column in a table, attribute/predictor of a row of data

Question 8

record

Answer 8

a row in a table

Question 9

data scientists use features to predict a target, while statisticians..

Answer 9

use predictor variables in a model to predict a response/dependent variable

Question 10

trimmed mean

Answer 10

avg of all values after dropping a fixed number of extreme values

Question 11

robust

Answer 11

not sensitive to extreme values

Question 12

x-bar

Answer 12

sample mean of a population

Question 13

reasons to use a weighted mean

Answer 13

1) observations that are highly variable may be given a lower weight (ex: a sensor that is less accurate)
2) data collected doesn’t represent different groups that we are interested in measuring (ex: give greater weight to underrepresented minorities )

Question 14

why is the median more robust than mean as an estimate of central tendency

Answer 14

It isn’t influenced by outliers / extreme cases that could skew the results

Question 15

what is thought to be a compromise between mean and median

Answer 15

trimmed mean - robust to extreme values in data, but uses more data to calculate the estimate for central tendency than median

Question 16

variance

Answer 16

sum of squared deviations from the mean divided by n - 1 (aka: mean-squared-error)

Question 17

standard deviation

Answer 17

the square root of variance (aka: 12-norm, euclidean norm)

Question 18

range

Answer 18

difference between largest and smallest value in a dataset

Question 19

interquartile range

Answer 19

the difference between the 75th percentile and the 25th percentile

Question 20

mean absolute deviation

Answer 20

mean of the abs value of the deviations from the mean

Question 21

Why use n-1 instead of n when calculating variance?

Answer 21

When using n, you will underestimate the true value of the variance and the std in the population.

Question 22

what measure of variability is most robust to extremes / outliers?

Answer 22

median abso value = median (| x1 - m |, |x2 - m |,… | xN -m |)

Question 23

Graph that shows the min/max, IQR, median

Answer 23

box plot, box and whisker plot

Question 24

tally of the count of data falling into intervals/bins

Answer 24

frequency table

Question 25

plot of the frequency table

Answer 25

histogram

Question 26

smoothed version of a histogram

Answer 26

density plot

Question 27

When the categories can be associated with a numeric value, this gives an average value based on a category’s probability of occurrence.

Answer 27

expected value

Question 28

How is the expected value calculated (2 steps)

Answer 28

A marketer for a new cloud technology, for example, offers two levels of service, one priced at $300/month and another at $50/month. The marketer offers free webinars to generate leads, and the firm figures that 5% of the attendees will sign up for the $300 service, 15% for the $50 service, and 80% will not sign up for anything. This data can be summed up, for financial purposes, in a single “expected value,” which is a form of weighted mean in which the weights are probabilities.

The expected value is calculated as follows:

1. Multiply each outcome by its probability of occurring.
2. Sum these values.

In the cloud service example, the expected value of a webinar attendee is thus $22.50 per month, calculated as follows:
EV = (0.05)(300)+(0.15)(50)+(0.8)(0) == 22.5

Question 29

Exploratory data analysis often begins with what 3 things?

Answer 29

1. univariate analysis
2. examining correlation among predictors (features)
3. examining correlation among features and the target

Question 30

A metric that measures the extent to which numeric variables are associated with one another (ranges from –1 to +1).

Answer 30

correlation coefficient (R)

Question 31

A table where the variables are shown on both rows and columns, and the cell values are the correlations between the variables.

Answer 31

correlation matrix

Question 32

A plot in which the x-axis is the value of one variable, and the y-axis the value of another.

Answer 32

scatterplot

Question 33

How do we compute the correlation coefficient?

Answer 33

we multiply deviations from the mean for variable 1 times those for variable 2, and divide by the product of the standard deviations & (n-1):

(Summation of (xi - xbar)(yi - ybar)) / ((n-1) * SDx * SDy)

Question 34

what is an example where correlation coefficient isn’t a useful metric

Answer 34

when the relationship is not linear

Question 35

A tally of counts between two or more categorical variables.

Answer 35

Contingency tables

Question 36

A plot of two numeric variables with the records binned into hexagons.

Answer 36

Hexagonal binning

Question 37

A plot showing the density of two numeric variables like a topographical map.

Answer 37

Contour plots

Question 38

Similar to a boxplot but showing the density estimate.

Answer 38

Violin plots

Question 39

scatterplots are good for smaller amounts of data, what are good alternatives when having large amounts of data

Answer 39

hexagonal binning, contour plots,heat maps