lecture 3 - feature engineering

Question 1

types of feature engineering in the time domain

Answer 1

1. numerical
2. categorical
3. mized

we can look at a combination of values in a window and base the prediction on that

Question 2

numerical time domain

Answer 2

• summarize values of a numerical attribute in a certain window
• mean, max, min, std

Question 3

small step sizes

Answer 3

• the smaller the window size, the closer you’ll be to the original data
• small window size = more data

Question 4

categorical time domain

Answer 4

• generate patterns that combine categorical values over time
• pattern types: succession and co-occurrence
• we consider support to identify patterns that occur frequently enough to be considered significant

Question 5

support Θ

Answer 5

[sum of the number of times the pattern occurs inside window throughout N instances] / [N - λ]
–> by itself and in the historical window

• we extend patterns with sufficient support ( > Θ) to more complex patterns of size k
• k = number of rows considered as a pattern within a window
• k grows over time, the bigger the more complex. bigger k also decreases support.

Question 6

frequency domain

Answer 6

considers periodic behavior

Question 7

frequency domain: fourier transformation

Answer 7

• any sequence of measurements can be represented by a combination of sinusoid functions
• find which frequencies are available in the window (frequency decomposition)

Question 8

fourier transformation steps

Answer 8

1. assume a base frequency
2. compute a frequency per second for each value of k
3. find the amplitudes a(k) associated with the k different frequencies - this way, when we multiply the amplitude with the value for the sinusoid functions, we end up with the original signal again.
4. summation formula explains the original signal (x^i_t) as a sum of separate amplitudes
5. get feature values (highest amplitude frequency, frequency weighted signal average, power spectrum entropy)

Question 9

base frequency

Answer 9

• this is the lowest frequency with a complete sinusoid in it
• we can look at k multiples of this base frequency
• k * f_0 represents the k-th multiple of the base frequency
• k is directly related to the number of full sinusoid periods within the given window
• k runs from 0 to λ

Question 10

frequency per second

Answer 10

• (k * Nsec)/ (λ + 1)
• output in Hz

Question 11

how many frequencies do we need to cover the entire range of frequencies from the base frequency up to the highest multiple of it?

Answer 11

λ + 1 different frequencies

Question 12

finding best values for a(k) for a given window of time points

Answer 12

best done with fast fourier transform

Question 13

definition: highest amplitude frequency

Answer 13

feature that identifies the frequency that has the highest amplitude in the signal

Question 14

definition: frequency weighted signal

Answer 14

feature that calculates a weighted average of the frequencies, where the weights are the amplitudes of the frequencies

• frequencies with high amplitudes get more weight

Question 15

definition: power spectrum entropy

Answer 15

• quantifies the amount of signal in the data
• calculates the intensity of each frequency component
• checks whether one or a few discrete frequencies are standing out
• high value = more complex signal

Question 16

unstructured data: preprocessing pipeline for text data

Answer 16

1. tokenization
2. lower case
3. stemming
4. stop word removal

Question 17

tokenization

Answer 17

identify sentences and words within sentences

Question 18

lower case

Answer 18

change the uppercase letters to lowercase

Question 19

stemming

Answer 19

identify the stem of the word to reduce words to their stem and map all different variations of e.g., verbs to a single term

Question 20

stop word removal

Answer 20

remove known stop words as they are not likely to be predictive

Question 21

features/approaches for text data

Answer 21

1. bag of words
2. TD-IDF (inverse document frequency)
3. topic modeling

Question 22

bag of words

Answer 22

• count occurrences of n-grams within text irrespective of order. this is the value of the attribute
• does not account for uniqueness of words: some words occur more frequently than others. low frequency words are often more predictive than words that you see in every text.

Question 23

TF-IDF (inverse document frequency)

Answer 23

• does account for uniqueness of words
• gives more weight to unique words and avoids very frequent words to become too dominant

1. TF = number of occurrences of a certain n-gram (= a number a^j_i)
2. idf_j = log([total number of documents or instances in question] / [number of documents that contain the n-gram])
3. tf_idf = TF * IDF

Question 24

IDF score

Answer 24

• 0 = word occurs in all documents
• high score = word is more rare/unique

Question 25

topic modeling

Answer 25

instead of looking at a lot of words, look at the topics the free text is about

• we assume that the text contains k topics
• for each topic(k), a word has a certain weight
• a topic is defined as a combination of words that have weights assigned to them -> set of weights for each word in the corpus

Question 26

scoring topics

Answer 26

topic_k(i) = sum over m attributes(count of word m in in document i x weight of word m in topic k)

-> for all k topics, a weight is assigned to each and every word in the corpus

Question 27

overlap (problem, consequence, solution, implication)

Answer 27

• if we have a large window size, the next data point will not be very different from the current one
• the consequence is that the learning process slowed down and contains a lot of uninformative data points, as well as data leakage.
• this is avoided by allowing for a certain percentage of overlap (typically 50%)
• this means that we have less remaining instances than if we would have used 90% overlap

Question 28

fourier transformation: λ

Answer 28

• λ = [x_{t-λ},…, x_t]
• λ = 40, means 41 datapoints

Question 29

n-gram

Answer 29

• n consecutive words
• n represents the number of words we consider as a single unit/attribute

Question 30

fourier transform: feature values

Answer 30

• every time a fourier transformation is done, you end up with amplitudes.
• these amplitudes are aggregated into three values

1. highest amplitude frequency
2. frequency weighted signal average
3. power spectrum entropy

Question 31

TF-IDF score

Answer 31

gives more weight to n-grams that are unique

Question 32

definition of ‘topic’

Answer 32

• a topic is defined as a combination of words.
• the way topics are represented is that for each topic, these words each have certain weights assigned to them.

Question 33

how to find topics

Answer 33

• latent dirichlet allocation (LDA)
• LDA assumes texts are generated with a poisson distribution over words, and a dirichlet distribution over topics
• we assume words belong fully to a single topic initially
• we then update the weights to maximize the probability of observing the texts
• i.e., topics are found based on distributions of words and topics