lecture 2: data engineering

Question 1

types of data

Answer 1

continuous, discrete, ordinal, categorical, missing, censored

Question 2

what is continuous data

Answer 2

measured on a quantitative scale, can be any fractional number

Question 3

what is discrete data

Answer 3

data points have a countable number of values between any 2 points

Question 4

what is ordinal data

hint ordinal = ordered

Answer 4

have a fixed number of possible values (<100), called levels that are ranked/ordered

Question 5

what is categorical data

Answer 5

multiple categories that are not ordered

Question 6

what is missing data and how do we deal with it

Answer 6

a missing data point that we do not know the mechanism of

we should use a non number code to denote such date, eg. NA

Question 7

what is censored data and how do we deal with it

Answer 7

a missing value but we know the mechanism on some level

coded as NA as well, add a column for censored TRUE or FALSE

Question 8

what is a top down vs bottom up approach

Answer 8

top down: starting from a problem/question and then finding to date to solve that problem
bottom up: starting from the data set, study and analyse it to see what problem/question it can solve

Question 9

what is data wrangling

Answer 9

process of transforming/mapping raw data into another format to make it more appropriate for downstream analytics
*downstream analytics: just means the data is used after some processing step, for a specific purpose

Question 10

types of data wrangling

Answer 10

scaling(eg. min-max), clipping(eg. feature clipping), z-score

Question 11

when to use z-score standardisation

Answer 11

when the population of each independent dimension of data is normally distributed

Question 12

what is data cleaning/cleansing

Answer 12

process of detecting and correcting/removing corrupt or inaccurate records from a data set

Question 13

what are the methods of dealing with missing features

Answer 13

1. removing the examples with missing features (only if dataset is big enough)
2. use a learning algorithm
3. using data imputation

Question 14

what is data imputation

Answer 14

replacing the missing value of a feature with an average value of this feature in the dataset
OR
replace the missing value with a value outside the normal range of values

Question 15

what is data integrity

Answer 15

maintenance and assurance of the accuracy and consistency of data over its entire life cycle, eg. credit card numbers