Lecture 2 - Data Understanding I

Question 1

What are the goals of data understanding?

Answer 1

• Gain general insight
• Check the assumptions
• Check the specific domain knowledge
• Suitability of data for goals

Question 2

What are the rows and columns called?

Answer 2

Rows: Instances, records, samples, data objects
Columns: Features, attributes, variables

Question 3

What are the types of attributes?

Answer 3

• Categorical (nominal): Classes or categories {Finnish, Swedish}
• Ordinal: Linear order {BSc, MSc, PhD}
• Numerical: Values are numbers
• Discrete: Categorical attribute or numerical attribute whose domain is a subset of the integer number
• Interval scale: the definition of value 0 is arbitrary, ratios are meaningless (like celsius)
• Ratio scale: 0 has a canonical meaning, ratios make sense (distance)
• Absolute scale (for example number of people)

Question 4

What does the level of granularity mean?

Answer 4

More defined information, for example:
Drinks -> water -> producer -> size 0.33l
More detailed has more information, but might not help discover general associations (like wine and cheese bought together)

Question 5

What can be a probelm for categorical attributes?

Answer 5

Dynamic domains: the possible values, like the products name or amount of sales for new product can change over time

Question 6

What is a categorical variable with two possible values?

Answer 6

Binary attribute

Question 7

Does the order of binary attribute matter?

Answer 7

No, can be 1 = FALSE 0 = TRUE or 1 = TRUE 0 = FALSE

Question 8

How can you encode categorical attributes? Example of {Finland, Sweden, Denmark, Norway}.

Answer 8

One hot encoding.
Can’t be 1,2,3,4 because there is no order.
Transform each into binary variable.
isFIN = 0/1, isSWE = 0/1 etc.
Categorical feature with k possible values can be transformed into:
Fin = 1000, Swe = 0100, Den = 0010, Norway = 0001
Then people like Björn can be 1100, because they are FinSwe

Question 9

How can you encode ordinal attributes?

Answer 9

There is order, so {BSc, MSc, PhD} = {1,2,3}.

Question 10

What is syntactic accuracy?

Answer 10

It is the accuracy that the entry belongs to the domain of the attribute. For example entry fmale for sex violates synctactic accuracy. Text entries “one” for numerical data 1 also violate synctactic accuracy.
Can be checked quite easily

Question 11

What is semantic accuracy?

Answer 11

Semantic accuracy means whether or not the entry is correct even though it correctly belongs to the domain of attribute. For example PhD and age 10 can be correct for studies and age, but it’s not possible to be PhD at the age of 10.
Hard to check.

Question 12

What does completeness of the data mean?

Answer 12

It means that there is no missing entries, for example null entries.

Question 13

What is unbiased and representative data?

Answer 13

The data contains all information about the inherent pattern and rules in the data.

Question 14

Why is it sometimes hard to get unbiased and representative data?

Answer 14

Machine monitoring: nuclear power plant runs a lot normally, you can’t manually make it run terribly.
Natural disasters: How to predict earthquake in a place where there has not been earthquakes?
Mortgage/insurance: Certain types of customers are missing from the data

Question 15

What causes missing values?

Answer 15

• Broken sensors
• Refusal to answer
• Test not done for person
• Combined data with different variables between databases
• Irrelevant attribute (gas/l for tesla)

Question 16

What are the types of missing values?

Answer 16

MCAR (Missing completely at random, aka OAR observed at random), MAR (Missing at random) and nonignorable

Question 17

What does MCAR mean?

Answer 17

Missing completely at random. The probability that a value for X is missing does not depend on the true value of X or on other variables.

For example battery of a sensor runs out and is replaced at random times.

Question 18

What is the probability of MCAR at each instance?

Answer 18

The same, missing values follow same distribution as observed

Question 19

What methods can be used for MCAR values?

Answer 19

Data imputation or ignoring will not bias the analysis, but often MCAR is not realistic assumption.

Question 20

What is MAR?

Answer 20

Missing at random. The probability that value for X is missing depends on Y. For example battery sensor runs out, and it is replaced during specific times. Or higher earning people are less likely to report their income

Question 21

What distribution MAR values follow?

Answer 21

Not the same as the distribution for the observed values, so the analysis can be biased.

Question 22

What methods for MAR values?

Answer 22

We can estimate values for day, if values from night are missing, but the problem is that can we be sure that Y provides all the required information?

Question 23

What is nonignorable missing value?

Answer 23

The probability that value X is missing depends on the true value of X.

For example temperature sensor doesn’t work at below zero, person with over 1M€ income does not report their income.

Question 24

Can nonignorable values be estimated?

Answer 24

No, it is impossible to provide any statements about temperatures below 0, if the sensor never works at below 0.

Question 25

What strategy can be applied, if the domain knowledge doesn’t tell the kind of missing values we can expect?

Answer 25

1. Turn the considered attribute X to binary (all measured to YES and all missing to NO)
2. Build a classifier with binary attribute X as the target and use the other attributes for the prediction of YES/NO
3. Determine the misclassification rate.
   - MCAR: other attributes should not provide any information, so the misclassification should not differ from pure guessing.
   - If the misclassification is a lot better than pure guessing, then other attributes affect X, so it is not MCAR. But MCAR and MAR cannot be distinguished from nonignorable this way.

Question 26

Why is data visualisation imoprtant for data understanding and data quality evaluation?

Answer 26

Visualisations reveal patterns or exceptions, there is “something” in the dataset

Question 27

What causes outliers?

Answer 27

• Data quality problems, like erroneous data from wrong measurements
• Exceptional or unusual situations

Question 28

What should be done to outliers?

Answer 28

Erroneous outliers should be excluded from the analysis. Correct outliers (aka exceptional situations) are sometimes useful to exclude also.

Question 29

Can outliers be useful?

Answer 29

Yes, for example “whales” in mobile games.