Preprocessing Flashcards
Welche Dimensionen gibt es, um die Qualität von Daten zu messen?
- Completeness: is the data fully available? What to do if not?
- Consistency: differences in data units or name conventions?
- Timeliness: measurements from different epochs?
Old measure devices? - Believability: is the data source reliable?
- Interpretability: how easily can the data be understood?
Für das Data Cleaning, über welche Typen an Fehlern sollte man Bescheid wissen?
- Incomplete: lacking attribute values, lacking certain attributes of
interest, or only aggregate data available - Noisy: containing noise, errors, or outliers
- Inconsistent: containing discrepancies in codes or names
- Intentionally imprecise
– Jan. 1 as everyone’s birthday
Wie kann man mit fehlenden Daten umgehen?
- Ignorieren: kein großer Effekt bei großen Daten
- manuell die Einträge überarbeiten
- automatisch die Einträge überarbeiten (global constant, mean, most probable value using inference such as Bayesian
formula or decision tree based on other attributess)
Was ist Data Integration?
Data integration combines data from
multiple sources into a coherent store
Mit welcher Methode kann man redundante Attribute erkennen?
chi-square test (nominal)
correlation analysis
Was sind die Vorteile von Data Integration?
- reduce/avoid redundancies and inconsistencies and
- improve mining speed and quality
Beschreib den Chi-Square Test mathematisch
Was bedeutet ein hohes Chi-Quadrat?
→ data distributions are statistically different
Was bedeutet ein niedriges Chi-Quadrat?
distributions are similar
Wie funktioniert ChiMerge?
Man hat Intervalle und checkt rekursiv, ob die Verteilung der Label in den beiden ähnlich ist anhand des Chi-Quadrat tests und mergt diese, falls dies stimmt
Beschreib den Pearson’s product
moment coefficient mathematisch
Pearson’s product moment coefficient
Was bedeutet es, wenn r > 0?
A and B are positively correlated
Pearson’s product moment coefficient
Was bedeutet es, wenn r = 0?
uncorrelated, not necessarily independent
Pearson’s product moment coefficient
Was bedeutet es, wenn r < 0?
negatively correlated
Wie berechent man die Kovarianz?
Was bedeutet eine Kovarianz von größer als null?
A and B tend to be together
larger or together smaller than their expected values
Was bedeutet eine Kovarianz von kleiner als null?
if A is larger than its expected
value, B is likely to be smaller than its expected value.
Wie kann die Kovarianz vereinfacht werden?
Wie berechnet man element ij einer Kovarianzmatrix?
it computes the covariance between feature i and feature j
Welche Strategien für das Binning existieren?
- equal-width
- equal-first (same number of samples)
Welche Smoothing Strategien im Anschluss des Binnings existieren?
Welche zwei Wege zur Dimensionsreduzierung existieren?
- Feature selection: A process that chooses an optimal
subset of features according to an objective function - Feature extraction: refers to the mapping of the original
high-dimensional data onto a lower-dimensional space
Was minimiert deskriptive Dimensionsreuzierung?
den Informationsverlust
Was maximiert prädiktive Dimensionsreduzierung?
die Klassendiskrimination
Wie bestimmt man die separation quality?
Was ist entity identification, or entity resolution?
Entity identification, or entity resolution, is a common challenge in data integration. It involves identifying and linking records that refer to the same entities (such as individuals, companies, or products) across different data sources. This process can be complex due to variations in how entities are represented (different names, addresses, etc.) in various sources.
Which of the following statements on boxplots are correct?
- The whiskers extend to Q1 and Q3.
- It is possible to create a boxplot for ordinal data.
- The box of a boxplot contains half of the data.
- Boxplots show the min, max, Q1, Q3, and mean of a distribution.
- The box of a boxplot contains half of the data.,
- It is possible to create a boxplot for ordinal data.
Which of the following statements are correct?
1. Boxplots contain more information than histograms.
2. The area of the bars in a histogram matters.
3. Buckets of histograms can have varying size.
4. In histograms, the buckets can be reordered.
- The area of the bars in a histogram matters.
- Buckets of histograms can have varying size.
Which of the following statements are correct?
1. Equal width binning results in bins with the same number of elements.
2. Binning is a simple way of data discretization.
3. Equal width binning is not influenced by outliers.
4. Bins can be used for data smoothing.
2 und 4
Wie können Bins für data smoothing sorgen?
Data smoothing is a technique used to reduce the noise within a dataset to highlight the underlying trend, and binning is one method to achieve this. By grouping data points into bins, and then replacing each data point by a central value of the bin (like the mean or median), the minor fluctuations or the noise in the data can be smoothed out. This process simplifies the data, making it easier to observe long-term trends and patterns.
Which of the following statements about outliers is correct?
1. Graphical outlier detection is suitable for high-dimensional data.
2. Outlier detection can be used to prevent credit card fraud.
3. It is always useful to automatically remove all outliers to reduce noise.
- Outlier detection can be used to prevent credit card fraud.
Which of the statements are not correct?
1. It is never possible to automatically detect redundant attributes.
2. Data integration may lead to redundancy in the dataset.
3. Data integration combines data from multiple sources.
4. Entity identification is a problem in data integration.
- It is never possible to automatically detect redundant attributes.
Wie berechnet man die Expected werte in der confusion tabelle bei chi quadrat
