07_unsupervised learning

Question 1

What is the goal for unsupervised problems?

Answer 1

fins a transformation (T)
that builds a compact internal representation
of unlabeled data (x)
to unveil its internal structure

• we want a mathematical function that does something for you, model that builds a representation, way to represent data in an efficient way

Question 2

What is the difference between supervised and unsupervised learning?

Answer 2

in unsupervised learning…

• no labels are required, whereas those are crucial in supervised learning
• no specific task is learned (no classification, regression etc)
• train/val/test splits are not as common, but can still be useful depending on your goals

(split data sets still allow you to evaluate how well your model generalizes to unseen data)

Question 3

Why use unsupervised learning?

Answer 3

unsupervised learning unveils structure in the data, which we want to exploit to make better use of it

• clustering
• dimensionality reduction

Question 4

What is the goal of clustering in unsupervised learning?

Answer 4

reveal agglomerates in the data set that might indicate populations of samples with distinct properties

Question 5

What is the goal of dimensionality reduction in unsupervised learning?

Answer 5

reduce the number of dimensions in the data for better interpretability and/or better performance of supervised learning approaches

Question 6

What is Clustering?

Answer 6

is the task of grouping a set of objects
in such a way that objects in the same group (called a cluster)
are more similar (in some sense)
to each other than to those in other groups (clusters)

• identify latent (not observable) classes inherent to the data
• identify taxonomies in the underlying data
• identify noise/outliers

Question 7

What are different approaches to clustering? (4)

Answer 7

• centroid-based: clusters are represented as mean vectors
• connectivity-based: clusters are defined based on distance connectivity
• distribution-based: clusters are defined by statistical distributions
• density-based: clusters are high density regions

Question 8

What is a centroid-based clustering method?

Answer 8

k-means clustering

clusters are represented by vectors that point to their centers (centroids)

Question 9

What is k-means clustering?

Answer 9

similarly to nn-models and many other clustering methods, k-means clustering is based on a DISTANCE METRIC
eg. euclidian distance metric

k-means does not see labels in a dataset

Question 10

What are the steps to implement k-means algorithm?

Answer 10

0) initialization: pick k random data points as cluster center locations

1) cluster assignment: compute distances between cluster ceonters and data points; assign each data point to closest cluster

2) recompute cluster centers: recompute cluster center locations from all cluster members

3) reiterate steps 1 and 2

4) termination: terminate algorithm if cluster assignments do not change anymore –> it self-organizes itself

Question 11

What happens if cluster colors are mismatched with k-means results?

Answer 11

some data points are impossible to recover

–> k-means clusters resemble ground-truth labels very much, however it ignores mismatch in the clusters: k-means is unsupervised

Question 12

What is the right value of k in k-means discussion?

Answer 12

-qualitative approach: visual inspection and human intuition (works well on low-dimensional and well-behaved data)

-quantitative approach: BIC (see hierarchical clustering)

Question 13

What are limitations of k-means method?

Answer 13

• relying on a distance metric, k-means intrinsically expects radially symmetric (“gaussian”) clusters
• proper data scaling is crucial

Question 14

What is a distribution-based clustering method?

Answer 14

expectation maximization clustering

Question 15

What is a big disadvantage of k-means/agglomerative clustering?

Answer 15

hard cluster assignment

–> each data point is assigned to a single cluster, no information on likelihood

how to compute likelihoods for each data point to belong to individual clusters?
–> EM (expectation maximization) clustering

Question 16

How can we compute the likelihoods for each data point to belong to an individual cluster?

Answer 16

EM clustering
(expectation maximization clustering)

Question 17

What is expectation maximization clustering?

Answer 17

class affiliation probability for each of k clusters can be approximated by a Gaussian N (x; µi, Summe i) ;
EM Clustering is therefore a parametric clustering method

probability for data point x to belong to cluster i?
Intuition: if x-µi small, probability that it belongs to cluster i is large
if x-µi large, probability that it belongs to cluster i is small

we treat the data distribution as a mixture of Gaussians (Gaussian mixture model)

Question 18

What are the steps to implement expectation maximization clustering?

Answer 18

0) initialize parameters: pick random data points for centroids µi, and adopt default values for covariances Summe i

1) “expectation step”: for each data point j calculate probability to belong to cluster i

2) “maximization step”: recalculate model parameters to better fit the previously derived probability distributions (maximize probability)
–> changes the centroids and also the shape of the clusters

3) reiterate steps 1 and 2

4) termination: terminate algorithm if cluster assignments do not change anymore

Question 19

What are the differences on the assumptions on the data from k-means clustering and expectation maximization clustering?

Answer 19

k-means:
distribution of radially symmetric (gaussian) clusters

EM:
distributions of elongated (multi-normal) clusters

Question 20

How do k-means and Expectations maximization clustering relate?

Answer 20

similar algorithm, iterative two-step process for fitting of model
(k-means is actually a special case of EM)

with K-means, covariants are fixed and you only change the centroids

Question 21

What is the difference on cluster assignments between k-means and expectation maximization clustering?

Answer 21

k-means:
Hart Assignment (each data point is assigned to one cluster)

EM:
Soft assignment (each data point has a probability associated to each cluster)

Question 22

What is a connectivity-based clustering method?

Answer 22

hierarchical clustering

Question 23

What is hierarchical clustering?

Answer 23

builds a hierarchy of clusters based on the provided data set. the number of found clusters, k, varies in each iteration and can be considered its hyperparameter.

hierarchical clustering is non-parametric

assumes that data point that are close to each other are more likely to be part of the same cluster

Question 24

What are two different hierarchical- clustering approaches that work very similarly?

Answer 24

• agglomerative clustering (bottom-up)
  –> clusters are merged based on a distance function
• divisive clustering (top-down)
  –> clusters are split based on distance function

Question 25

What are steps to agglomerative clustering?

Answer 25

0) initialization: each data point forms its own cluster

1) merge closest neighbors: find closest neighbors and merge them (euclidean distance metric with “single-linkage” criteria)

2) merge closest neighbors: find closest neighbors and merge them, if neighbor is already merges in the same step, simply omit)

3) merge closest neighbors: find closest neighbors and merge them (same as step 2)

4) terminate: all data points form a single cluster

–> forms a dendrogram, “family tree”

at different iterations we find different numbers of clusters

Question 26

What is the “best-fit” number of clusters k for agglomerative clustering?

Answer 26

• qualitative approach: visual inspection and human intuition (works well for low-dimensional and well-behaved data)
• quantitative approach: Bayes Information Criterion (BIC)

Question 27

What is BIC?

Answer 27

Bayes Information Criterion

BIC = number of model parameters + ln(number of data points) - 2ln(maximum likelihood of model describing data)

Idea: minimize BIX to find the least complex model that fits the data well

Question 28

What is an advantage of agglomerative clustering?

Answer 28

much better able to deal with non-Gaussian clusters

Question 29

What is a disadvantage of agglomerative clustering?

Answer 29

data points belonging to the same cluster might be assigned to different clusters if highly elongated

Question 30

What is a density-based clustering method?

Answer 30

DBSCAN

Density-based spatial clustering of applications with noise

Question 31

What does DBSCAN stand for?

Answer 31

Density-based spatial clustering of applications with noise

Question 32

What is DBSCAN?

Answer 32

define clusters based on local over-densities in the data

supports the notion of noise: data points that are not part of any cluster will be considered noise

uses two hyperparameters:
- the distance within which to define a neighborhood, epsilon (radius)
- the number of data points per neighborhood to consider it a cluster

N = minimum number of data points per neighborhood to form a cluster

Question 33

What are steps to the DBSCAN algorithm?

Answer 33

0) initialization: pick a random data point and place an epsilon-neighborhood around it

1) find number of neighbors: check how many neighbors the data point ahs within the epsilon-neighborhood (if not enough neighbors, data point is considered noise)

2) create a new cluster: and assign it to the current data point. repeat this step with same cluster label for all data points in the neighborhood

3) reiterate steps 1 and 2: data points are either noise or there is a new neighborhood to create

4) terminate? (assumed: when all data points are categorized)

Question 34

How sensitive are the results of DBSCAN to the set of hyperparameters?

Answer 34

high sensitivity to epsilon:

small e: clusters are too small
large e: clusters are too big

–> needs a considerable amount of fine-tuning

Question 35

What are advantages of DBSCAN? (2)

Answer 35

• often provides clearly delineated clusters of different shapes, if the hyperparameters were chosen properly
• ability to identify noise

Question 36

What is a dimensionality reduction method?

Answer 36

principal components analysis

Question 37

What is dimensionality reduction?

Answer 37

helps to reduce the number of dimensions (features) in the data set

closely related to feature selection: provides a means to identify the important features in the data set

Question 38

What is principal component analysis (PCA)?

Answer 38

fits a multinormal ellipsoid to the data set. each axis of the ellipsoid is called a PRINCIPAL COMPONENT of the data set and its length relates to the variance along that axis (the more important a principal component, the longer it is)
–> important = descriptive for the data set

each principal component is a linear combination of feature vectors –> orthonormal basis for the data set

to achieve dimensionality reduction, only a subset of all principal components are utilized to represent data

allows you to transform (rotate) a data set into a more meaningful representation

Question 39

How is PCA done? (principal component analysis)

Answer 39

mathematically, principal components are EIGENVECTORS of the data set’s covariance matrix

–> eigendecomposition or singular value decomposition

required the underlying data to be scaled properly (zero mean, unity variance) –> otherwise mean offset strongly affects the principal components: the first component will most likely point to the mean of the data set