lecture 4 - clustering

Question 1

two types of setup

Answer 1

1. per instance
2. per person

Question 2

setup per instance

Answer 2

• per timepoint, over all QS
• Each instance (time point) is treated as a separate observation.
• For each instance, there is a feature vector.
• These feature vectors are combined into a large matrix
  X where each row corresponds to an instance (measurement at a particular time). (X_N, qs_n)

Question 3

setup per person

Answer 3

• for finding types of people
• data is organized by individual, where each person’s data is grouped together
• each person has multiple sets of features, representing different measurements or time points
• matrix X now corresponds to submatrices, each corresponding to a different person (X_qs_n)

Question 4

individual distance metrics (instance based)

Answer 4

1. euclidean distance
2. manhattan distance
3. minkowski distance
4. gower’s similarity

Question 5

euclidean distance

Answer 5

shortest straight path

Question 6

manhattan distance

Answer 6

block based structure

Question 7

minkowski distance

Answer 7

• generalized form of euclidean and manhattan distance
• q =1: manhattan distance
• q = 2: euclidean distance

Question 8

important to consider for euclidean, manhattan, and minkowski distance

Answer 8

scaling the data, since all of these metrics assume numeric values

Question 9

gower’s similarity

Answer 9

• does not assume numeric values, so we can use this to find distances for different types of features

1. dichotomous attributes
2. categorical attributes
3. numerical attributes

Question 10

value of s(x^k_i, x^k_j) for dichotomous attributes

Answer 10

• 1 when x^k_i and x^k_j are both present
• 0 otherwise
• i.e., similar when both instances indicate presence

Question 11

value of s(x^k_i, x^k_j) for categorical attributes

Answer 11

• 1 when x^k_i = x^k_j
• 0 otherwise
• i.e., similar when instances are of the same category

Question 12

value of s(x^k_i, x^k_j) for numerical attributes

Answer 12

• 1 - ((absolute difference between x^k_i and x^k_j) / (range of the attribute))
• 1 - normalized absolute difference
• automatically scaled!

Question 13

gower’s similarity: final similarity

Answer 13

gower’s similarity of two instances

[sum over all instances s(x^k_i, x^k_j)] / [sum of all times when x^k_i and x^k_j can be compared]

Question 14

person level distance metrics (person-dataset based)

Answer 14

• how do we compare similarity between datasets (qs1, qs2)

1. without explicit ordering
2. with temporal ordering

Question 15

person-dataset similarity: without explicit ordering

Answer 15

1. summarize values per attribute over the entire dataset into a single value with the same distance metrics as before
   –> you lose a lot of information this way
2. estimate parameters of distribution per attribute and compare parameter values with same distance metrics as before
3. compare distributions of values for an attribute with a statistical test (e.g., kolmogorov-smirnov). take 1-p as distance metric.
   –> low p = very different distributions = distance metric will be close to 1

Question 16

person-dataset similarity: datasets with temporal ordering

Answer 16

1. raw-data based
2. feature based: same as non-temporal case. extract features from temporal data set and compare those values.
3. model-based: fit a time series model and use those parameters. again in line with the non-temporal case, except for the type of model being different

Question 17

raw-data based similarity

Answer 17

1. simplest case: assume equal number of points, and compute the euclidean distance of qs1 and qs2 per point of an attribute, then sum over attributes
   –> i.e., calculate the euclidean distance of each time point
2. if the time series are more or less the same, but shifted in time. To handle this, we use the concept of lag and the cross-correlation coefficient to get the cc_distance.
3. for different frequencies at which different people perform their activities, we can use dynamic time warping

Question 18

shifted time series: lag

Answer 18

• Lag τ is the amount of time by which one time series dataset is shifted relative to another.
• The goal is to find the best lag τ that maximizes the similarity between the two time series. This is an optimisation problem

Question 19

shifted time series: cross-correlation coefficient (ccc)

Answer 19

• measures the similarity between two time series attributes after shifting one of them by τ
• for each time point of an attribute, multiply that value of qs1 with the time point + τ value of qs2, then sum these products.

Question 20

shifted time series: cross-correlation distance (cc_distance)

Answer 20

• gives us the distance between two time series as one is shifted by a certain time lag
• we are testing all possible shifts τ from 1 up to the smaller length of the two data sets
• We sum the inverse of the cross correlation coefficient (ccc) between two qs for each attribute: 1/ ccc
• the best time lag is related to the smallest cc_distance

Question 21

dynamic time warping

Answer 21

• for different frequencies at which different persons perform their activities
• finds the best pairs of instances in sequences to find the minimum distance

Question 22

dynamic time warping: pairing conditions

Answer 22

1. monotonicity condition: time order should be preserved.
   –> i.e., we can’t go back to a previous instance. you can only move left, up, or up-diagonal, but not backwards
2. boundary condition: first and last points should be matched. this requires that we start at the bottom left and end at the top right. we cannot move outside of our time series

Question 23

dyamic time warping: cheapest path in the matrix

Answer 23

1. start at (0,0)
2. per pair, compute the cheapest way to get there given the costraints and the distance between each pair

• cost = [distance between the two points] + [cheapest previous path (from left, from below, or diagonally)

Question 24

dynamic time warping: DTW distance

Answer 24

• the value in top-right cell of the matrix is the DTW distance
• this represents the minimum cost to align the two series.
• finding this distance is computationally expensive: solved with the keogh bound

Question 25

DTW: keogh bound

Answer 25

allows you to estimate what the cheapest path will be

this makes DTW less computationally expensive

Question 26

clustering approaches

Answer 26

1. k-means
2. k-medoids
3. hierarchical (divisive & agglomerative)
4. subspace clustering

Question 27

k-means clustering

Answer 27

• The goal of k-means clustering is to partition a set of data points into k clusters, where each data point belongs to the cluster with the nearest mean.

1. initialization: Start by selecting k random points in the data space. These points will serve as the initial centers of the clusters.
2. assign points to clusters: Each data point is assigned to the cluster whose center is nearest. This is typically done using the Euclidean distance.
3. update centers: Recalculate the centers (centroids) of the clusters. The new center of each cluster is the mean of all the data points assigned to that cluster.
4. assign points to clusters: Data points are reassigned based on the updated centers. This process repeats until convergence.

Question 28

k-means: selecting the best value for k

Answer 28

using the silhouette score

Question 29

k-means: silhouette score

Answer 29

• measure used to determine how similar an object is to its own cluster (cohesion) compared to other clusters (separation).
• The silhouette score ranges from -1 to 1, where a value closer to 1 indicates better clustering.
• you want to be close to point in your own cluster, and far away from points in other clusters: b > a = score closer to 1
• based on the silhouette score for each k, we can decide which k is best

Question 30

silhouette score a(x_i) and b(x_i)

Answer 30

• a: average distance from x_i to all other points in the same cluster
• b: average distance from x_i to all points in the nearest neighboring cluster

Question 31

k-medoids clustering

Answer 31

• whereas k-means uses random points as centers, k-medoids takes actual points as centers
• This makes it more robust to noise and outliers.

Question 32

types of hierarchical clustering

Answer 32

1. divisive clustering
2. agglomerative clustering

these can take a more iterative approach than k-means and k-medoids

Question 33

divisive clustering

Answer 33

• start with one big cluster C, make a split each step
• calculate dissimilarity of a point to other points inside cluster C
• create a new cluster C’ and remove the most dissimilar points from C to C’ until there is no point left in C that is less dissimilar to the points in C’
• select the C with the largest diameter for this process
  –> the diameter of a cluster is the maximum distance between points in the cluster

Question 34

divisive clustering: dissimilarity of a point to other points within cluster C

Answer 34

sum(distance(x_i, x_j)) / |C|

• i.e., this is just the average distance in a cluster

Question 35

divisive clustering: largest diameter

Answer 35

for all the points in C

diameter(C) = max distance(x_i, x_j)

• i.e., largest cluster

Question 36

agglomerative clustering

Answer 36

• start with one cluster per instance and merge into larger clusters

1. Initialization: Start with each data point in its own cluster.
2. Merge Clusters: At each step, find the pair of clusters that are closest to each other and merge them (based on critera).
3. Continue this process until the desired number of clusters is achieved or a stopping criterion is met.

Question 37

agglomerative clustering: criteria for which clusters to merge

Answer 37

1. single linkage: distance between two clusters is defined as the minimum distance between two points in separate clusters
2. complete linkage: distance between two clusters is defined as the maximum distance between two points in separate clusters
3. group average: distance between two clusters is defined as the average distance between all pairs of points, one from each cluster. This method balances between single and complete linkage.
4. ward’s criterion: defines the distance between clusters as the increase in the standard deviation when the clusters are merged.

Question 38

subspace clustering

Answer 38

• handles a large number of features (high dimensional data)
• uses the CLIQUE algorithm

Question 39

CLIQUE algorithm

Answer 39

1. create units (u): we split the range of each feature up into ε distinct intervals.
   –> a unit u is defined by means of boundaries per dimension: u = {u1,…,uk}.
2. define selectivity and density of each unit u

Question 40

CLIQUE algorithm: units

Answer 40

• defined by upper and lower boundaries per feature
• u = {u_1, … ,u_p}
• an instance x is part of this unit when its value falls within the boundaries for all features

Question 41

CLIQUE algorithm: selectivity(u)

Answer 41

[number of points in u] / [total number of points]

• defines the proportion of points inside a unit

Question 42

CLIQUE algorithm: dense(u)

Answer 42

• 1: when selectivity(u) is larger than threshold τ
• 0: otherwise
• a density score of 1 tells you that the unit holds a lot of points and therefore how relevant/how much information a unit holds

Question 43

CLIQUE algorithm: subspaces

Answer 43

• we want subspaces (subsets of attributes) so our units do not have to cover all attributes
• we can have units that cover p-k attributes

Question 44

CLIQUE algorithm: common face

Answer 44

• units have a common face when all specifications of units are the same, except for one. for this one, either the upper bound is the same as the lower bound of another unit, or the lower bound is the same as the upper bound of another unit.
• i.e., they are adjacent
• we define a cluster as a maximal set of connected dense units.

Question 45

CLIQUE algorithm: connected

Answer 45

units are connected when;

1. they share a common face (they are each other’s common face)

or

2. when they share a unit that is a common face to both (i.e., connected through the common face)

Question 46

visualizing hierarchical clustering

Answer 46

• dendogram
• can be done for both divisive and agglomerative clustering

Question 47

ward’s criterion algorithm

Answer 47

1. define clusters A, B, and the merged cluster AB
2. take the sum of squared differences between all points in each cluster and the center of that cluster
3. subtract the within cluster variances (AB - A - B)
4. if the difference is small, it indicates that clusters A and B should be merged.

Question 48

problems with k-means, k-medoids, and hierarchical clustering (+ solution)

Answer 48

1. will take a long time to compute
2. calculating distances over a large number of attributes can be problematic and distances might not distinguish cases very clearly
3. the results will not be very insightful due to the high dimensionality.

Hence, we need to define a subset of the attributes (or subspace) to perform clustering (CLIQUE)