Data Mining

Question 1

What is data mining used for?

Answer 1

to make decisions, classifications, diagnoses, and recommendations that affect our lives

Question 2

Is data meaningful?

Answer 2

no – all information is meaningless without someone to analyze, and make sense of it

Question 3

What is data mining?

Answer 3

process of looking for patterns in large data sets

Question 4

What are two data mining tasks?

Answer 4

• classification (decision tree method) – supervised technique
• clustering (k-means method) – unsupervised technique

Question 5

What is classification?

Answer 5

using previously categorized data to determine how to categorize new data

putting things into groups that already exist

Question 6

What is clustering?

Answer 6

partitioning a set of items into subgroups to ensure certain measures of quality (ie. ‘similar’ items are grouped together)

Question 7

Compare classification vs. clustering.

• data
• output groups
• goal

Answer 7

data:

• classification: labelled
• clustering: unlabelled

output groups:

• classification: known
• clustering: unknown

goal:

• classification: used to predict future observations
• clustering: used to understand or describe observations

Question 8

Why might clustering be used?

Answer 8

explore data for hidden patterns or correlations

• once you see something interesting, you can delve further by other means
• quickly see if there are any possible missed relationships

helps organize data

reduces number of data points (ie. can reduce a cluster to a representative data point)

results might be fed into other data mining techniques

Question 9

What is clustering by numbers?

Answer 9

clustering points, typically in a high-dimensional space (2D, multidimensional)

Question 10

What is the goal in clustering data?

Answer 10

points that are ‘near’ each other

Question 11

What are the 3 possible criteria used to measure cluster quality?

Answer 11

• intra-class (intra-cluster) similarity
• inter-class (inter-cluster) dissimilarity
• size similarity

Question 12

What is intra-class (intra-cluster) similarity?

Answer 12

• points within same cluster are close to each other (or at least to their closest neighbours)
• distances are minimized
• items in same cluster are similar

Question 13

What is inter-class (inter-cluster) dissimilarity?

Answer 13

• points in two different clusters are far from each other (or at least to their closest neighbours in other clusters)
• distances are maximized

Question 14

What is size similarity?

Answer 14

clusters have similar size

Question 15

What are some limitations of data mining?

Answer 15

dirty data (obsolete, inaccurate, and missing info) – results are not meaningful

Question 16

What do data scientists call cleaning up dirty data?

Answer 16

data wrangling
data munging
data janitor work

Question 17

What are issues with dirty data?

Answer 17

• cleaning dirty data up is an increasingly important and overlooked job that can help prevent costly mistakes
• data scientists spend from 50-80% of their time doing labor of collecting and preparing unruly digital data, before it can be explored for useful nuggets

Question 18

What can the K-means clustering algorithm be used on?

Answer 18

any number of data points – works for 2D and multidimensional data

Question 19

What is K in the K-means clustering algorithm?

Answer 19

number of clusters you want to end up with – you choose it

depending on data points, it is possible that there is no final answer so you should pick the max number of times you want to run it

Question 20

What are some limitations of K-means clustering?

Answer 20

algorithm may give different cluster solutions depending on how initial centroids are chosen
- studies will do many runs with different interior centroids

not always clear how to choose K (number of clusters)

• if size of data set is small, different values of K can be chosen
• OR, large value of K can be chosen, then clusters can be merged to yield hierarchical cluster structure

Question 21

Can the final clusters assigned depend on initial positions of centroids in k-means clustering?

Answer 21

yes