preprocessing

Question 1

What is the main goal of the Savitzky-Golay filter?

Answer 1

To smooth data while preserving features like peaks and edges better than simple moving averages.

Question 2

How does the Savitzky-Golay filter compute smoothed values?

Answer 2

By fitting a low-degree polynomial to a subset of the data points (window) using least squares and evaluating the polynomial at the central point.

Question 3

Do the Savitzky-Golay filter coefficients depend on the data?

Answer 3

No, the coefficients depend only on the window size and the degree of the polynomial.

Question 4

What does mutual information measure?

Answer 4

The amount of information shared between two variables.

Question 5

How is mutual information related to entropy?

Answer 5

I(X;Y)=H(X)+H(Y)−H(X,Y), where H represents entropy.

Question 6

What is the range of mutual information values?

Answer 6

Non-negative, I(X;Y)≥0, and it is
0 if X and Y are independent.

Question 7

How is mutual information used in feature selection?

Answer 7

By ranking features based on their MI with the target variable to select the most informative ones.

Question 8

What does the Chi-Square test measure?

Answer 8

The difference between observed and expected frequencies in categorical data.

Question 9

What is the Chi-Square statistic formula?

Answer 9

𝜒^2= ∑(𝑂𝑖−𝐸𝑖)^2/𝐸𝑖
where 𝑂𝑖 is the observed frequency and 𝐸𝑖 is the expected frequency.

Question 10

What is the purpose of the Chi-Square test in feature selection?

Answer 10

To assess the independence of a feature and the target variable.

Question 11

What are the assumptions of the Chi-Square test?

Answer 11

Data must be categorical, and expected frequencies should generally be greater than 5.

Question 12

What does the correlation coefficient measure?

Answer 12

The strength and direction of the linear relationship between two variables.

Question 13

What is the range of the correlation coefficient?

Answer 13

[−1,1], where−1 indicates a perfect negative correlation,
1 indicates a perfect positive correlation, and 0 indicates no linear relationship.

Question 14

What is the formula for the Pearson correlation coefficient?

Answer 14

r = (SUM(xi - x)(yi - y))/(sqrt(SUM(xi - x)^2 * SUM(yi - y)^2)

Question 15

What is the goal of PCA?

Answer 15

To reduce the dimensionality of data while retaining as much variance as possible.

Question 16

How does PCA work?

Answer 16

By projecting data onto a new set of orthogonal axes (principal components) ordered by the amount of variance they capture.

Question 17

What are eigenvalues and eigenvectors in PCA?

Answer 17

Eigenvalues represent the amount of variance captured by each principal component, and eigenvectors define the directions of the principal components.

Question 18

How do you decide the number of components to keep in PCA?

Answer 18

Using techniques like the explained variance ratio or a scree plot.

Question 19

What is Chi-Merge used for?

Answer 19

To discretize continuous variables by merging adjacent intervals based on a Chi-Square test.

Question 20

How does Chi-Merge determine whether to merge intervals?

Answer 20

By calculating the Chi-Square statistic between adjacent intervals and merging them if the statistic is below a threshold.

Question 21

What are frequent patterns, and why are they important in data mining?

Answer 21

Frequent patterns are sets of items, subsequences, or substructures that appear in a dataset with a frequency higher than a specified threshold (support).
They form the foundation for discovering association rules, correlations, and sequential patterns.

Question 22

What are support and confidence, and how are they used in association rule mining?

Answer 22

Support: The proportion of transactions in which an itemset appears. It measures how frequently the itemset occurs in the dataset.
S(x) = transactions containing x / total transactions

Confidence: The likelihood that items in Y are purchased when
X is purchased. It measures the strength of an association rule X→Y.
Confidence(X→Y) = S(XUY)/S(X)

Question 23

Explain the key steps of the Apriori algorithm.

Answer 23

Candidate Generation: Start with single items (1-itemsets) and generate larger k-itemsets iteratively.
Prune Infrequent Itemsets: Eliminate candidates that have subsets not meeting the minimum support.
Count Support: Count occurrences of each itemset in the dataset.
Repeat: Continue generating and pruning k+1-itemsets until no new frequent itemsets can be found.
Generate Rules: Use frequent itemsets to derive association rules and calculate their confidence.
Key Insight: Apriori uses the downward closure property (if an itemset is frequent, all its subsets are frequent) to reduce the search space.

Weaknesses:
Computationally expensive due to candidate generation and multiple database scans.
Inefficient for datasets with a large number of items or high-dimensional data.
Memory-intensive as it stores many intermediate candidates.

Question 24

What is FP-Growth, and how does it address the inefficiencies of the Apriori algorithm?

Answer 24

The FP-Growth (Frequent Pattern Growth) algorithm avoids candidate generation by using a compact FP-Tree (Frequent Pattern Tree) structure.
Steps:
Build an FP-Tree:
Scan the database to count support and order items by descending frequency.
Insert transactions into the tree, sharing paths when possible.
Mine the FP-Tree:
Extract frequent patterns recursively using conditional pattern bases and conditional FP-Trees.

Benefits:
Reduces the number of database scans (only two passes).
Handles dense datasets more efficiently.
Memory-efficient due to compact tree structure.

Question 25

Describe how an FP-Tree is constructed and mined for frequent patterns.

Answer 25

Constructing the FP-Tree:
Scan the dataset to calculate support for each item.
Remove infrequent items and sort remaining items in descending frequency.
Insert transactions into the tree:
Shared prefixes of transactions are represented as branches.
Each node contains:
An item name.
A count (frequency of the item).
Links to its children and a header table for connections.
Mining the FP-Tree:
Start with the least frequent item (bottom of the tree).
Extract its conditional pattern base (prefix paths leading to that item).
Build a conditional FP-Tree from this base.
Repeat recursively for each conditional FP-Tree.