ch1

Question 1

big data

Answer 1

data sets that are so voluminous and complex that traditional data processing application software are inadequate to deal with them

Question 2

machine learning

Answer 2

a set of methods thst can automatically detect patterns in data, and then use the uncovered patterns to predict future data, or perform other kinds of decision making under uncertainty

Question 3

the long tail

Answer 3

property in data sets of a few instances being highly common/likely while other instances are quite rare

this is apparently shared in data generated across a wide variety of domains (why?)

Question 4

supervised learning

Answer 4

the machine learning task of inferring a function from labeled training data

Formally put, the task is to learn a mapping from inputs x to outputs y, given a labeled set of input-output pairs D = {(Xi, Yi)} for i=1 through N

Here, D is the training set, and N is the number of training examples.

Question 5

features, attributes. covariates

Answer 5

input-output pairs that make up the training set in a supervised learning problem

Question 6

response variable

Answer 6

output as a function of input in a supervised lesrning problem

Question 7

categorical/nominal variable

Answer 7

a variable that can take on one of a limited, and usually fixed, number of possible values, assigning each individual or other unit of observation to a particular group or nominal category on the basis of some qualitative property

Question 8

classification vs regression

Answer 8

When the response variable is categorical (Ie takes on a finite range of values), supervised learning is called classification. When the response variable is resl-valued, the problem is known as regression.

Question 9

ordinal regression

Answer 9

supervised learning when the response variable occurs over a finite range of values but they have some natural ordering (eg grades A-F)

Question 10

descriptive/unsupervised learning, or knowledge discovery

Answer 10

learning where only inputs D - {Xi} over i[1 to N are provided and the goal is to find “interesting patterns” in the data.

A much less well-defined problem, since we aren’t told what kinds of patterns to look for and no obvious error metric exists

Question 11

reinforcement learning

Answer 11

learning a decision based on occasional reward or punishment signals

Question 12

binary classification

Answer 12

classification between two classes

Question 13

multiclass classificstion

Answer 13

classifcation between more than two classes

Question 14

multi-label classifcation

Answer 14

classification where class labels aren’t mutually exclusive

often best viewed as predicting multiple related binsry class labels from the same inout (multiple output model)

Question 15

multiple output model

Answer 15

model of multi-label classification treating problem as involving prediction of multiple related binary class labels

Question 16

function approximation

Answer 16

formalization of classification w/ assumption y = f(x) for some unknown function f and goal to estimate the function f given a labeled training set and predict using y^ = f^(x)

Question 17

maximum a posteriori probability (MAP) estimate

Answer 17

an estimate of an unknown quantity, that equals the mode of the posterior distribution

Question 18

posterior probability

Answer 18

the conditional probability that is assigned after the relevant evidence or background is taken into account.

Question 19

posterior probability distribution

Answer 19

the probability distribution of an unknown quantity, treated as a random variable, conditional on the evidence obtained from an experiment or survey

Question 20

central idea of bayesian statistics

Answer 20

probability is orderly opinion, and that inference from data is nothing other than the revision of such opinion in the light of relevant new information.

Question 21

bag-of-words / vector space model

Answer 21

a simplifying representation used in natural language processing and information retrieval (IR). Also known as vector space model[1]. In this model, a text (such as a sentence or a document) is represented as the bag (multiset) of its words, disregarding grammar and even word order but keeping multiplicity.

Documents and queries are represented as vectors. Each dimension corresponds to a separate term. If a term occurs in the document, its value in the vector is non-zero.

Question 22

tf–idf or TFIDF, short for term frequency–inverse document frequency

Answer 22

often used as a weighting factor in searches of information retrieval, text mining, and user modeling. The tf-idf value increases proportionally to the number of times a word appears in the document, but is often offset by the frequency of the word in the corpus, which helps to adjust for the fact that some words appear more frequently in general.

Question 23

stop words

Answer 23

words which are filtered out before or after processing of natural language data (text)

Question 24

feature extraction

Answer 24

starts from an initial set of measured data and builds derived values (features) intended to be informative and non-redundant, facilitating the subsequent learning and generalization steps, and in some cases leading to better human interpretations

Question 25

dimensionality reduction or dimension reduction

Answer 25

the process of reducing the number of random variables under consideration[1] by obtaining a set of principal variables. It can be divided into feature selection and feature extraction.

Question 26

feature selection, also known as variable selection, attribute selection or variable subset selection

Answer 26

the process of selecting a subset of relevant features (variables, predictors) for use in model construction

Question 27

curse of dimensionality

Answer 27

various phenomena that arise when analyzing and organizing data in high-dimensional spaces (often with hundreds or thousands of dimensions) that do not occur in low-dimensional settings such as the three-dimensional physical space of everyday experience

The common theme of these problems is that when the dimensionality increases, the volume of the space increases so fast that the available data become sparse. This sparsity is problematic for any method that requires statistical significance. In order to obtain a statistically sound and reliable result, the amount of data needed to support the result often grows exponentially with the dimensionality. Also, organizing and searching data often relies on detecting areas where objects form groups with similar properties; in high dimensional data, however, all objects appear to be sparse and dissimilar in many ways, which prevents common data organization strategies from being efficient.

Question 28

overfitting

Answer 28

the production of an analysis that corresponds too closely or exactly to a particular set of data, and may therefore fail to fit additional data or predict future observations reliably”

The essence of overfitting is to have unknowingly extracted some of the residual variation (i.e. the noise) as if that variation represented underlying model structure.

Question 29

Underfitting

Answer 29

occurs when a statistical model cannot adequately capture the underlying structure of the data. An underfitted model is a model where some parameters or terms that would appear in a correctly specified model are missing.[2] Underfitting would occur, for example, when fitting a linear model to non-linear data. Such a model will tend to have poor predictive performance.

Question 30

density estimation

Answer 30

formulation of unsupervised learning as involving the construction of an estimate, based on observed data, of an unobservable underlying probability density function

Question 31

Cluster analysis or clustering

Answer 31

the task of grouping a set of objects in such a way that objects in the same group (called a cluster) are more similar (in some sense or another) to each other than to those in other groups (clusters)

Question 32

latent variables

Answer 32

variables that are not directly observed but are rather inferred (through a mathematical model) from other variables that are observed (directly measured).

Question 33

model based clustering

Answer 33

fitting a probabilistic model to the datw, rather than running some ad hoc algorithm

advantages include that one can compare different kinds of models in an objective way (in trrms of the likelihood that they assign to the data)

Question 34

imputation

Answer 34

the process of replacing missing data with substituted values

Question 35

matrix completion

Answer 35

the task of filling in the missing entries of a partially observed matrix

Question 36

collaborative filtering

Answer 36

In the newer, narrower sense, collaborative filtering is a method of making automatic predictions (filtering) about the interests of a user by collecting preferences or taste information from many users (collaborating).

In the more general sense, collaborative filtering is the process of filtering for information or patterns using techniques involving collaboration among multiple agents, viewpoints, data sources, etc

Question 37

parametric model

Answer 37

A statistical model with a fixed number of parameters

Question 38

non-parametric model

Answer 38

statistical model whose number of parameters increase with the amount of training data. More flexible than parametric models (ie make weaker assumptions), but are computationally intractable for large data sets.

Question 39

K nearest neighbor classifier

Answer 39

Non parametric classifier that looks at the K points of the training set nearest to the test input X, counts how many members of each class are in the set, and returns the empirical fraction as the estimate.

Loses effectiveness in high dimensions or whenever the nearest neighbors are in fact distant from the test input.

Question 40

instance-based or memory-based learning

Answer 40

a family of learning algorithms that, instead of performing explicit generalization, compares new problem instances with instances seen in training, which have been stored in memory.

Question 41

inductive bias

Answer 41

a learning algorithm is the set of assumptions that the learner uses to predict outputs given inputs that it has not encountered

Question 42

inductive bias

Answer 42

of a learning algorithm is the set of assumptions that the learner uses to predict outputs given inputs that it has not encountered

Question 43

linear regression

Answer 43

parametric model asserting that the response is a linear function of the inputs

Question 44

error

Answer 44

the deviation of the observed value from the (unobservable) true value of a quantity of interest

Question 45

residual

Answer 45

the difference between the observed value and the estimated value of the quantity of interest

Question 46

scalar product

Answer 46

dot product of two vectors

Question 47

basis function expansion

Answer 47

using linear regression to model nonlinear relationships by replacing x with some nonlinear function of the inputs ø(x)

Question 48

bernoulli distribution

Answer 48

the probability distribution of a random variable which takes the value 1 with probability 
p
p and the value 0 with probability 
q
=
1
−
p
q=1-p

Question 49

sigmoid function, aka logistic or logit function

Answer 49

S shaped “squashing function” that maps the whole real line to [0,1], which is necessary formthe output to be interpreted as a probsbility

Question 50

logistic regression

Answer 50

a regression model where the dependent variable (DV) is categorical (usually binary)

result of doing a basis (sigmoid) function expansion on a bernoulli distribution,

actuslly classification, not real regression

Question 51

decision rule

Answer 51

a function which maps an observation to an appropriate action

Question 52

linearly separable

Answer 52

two sets are linearly separable if there exists at least one line in the plane with all of the blue points on one side of the line and all the red points on the other side

Question 53

generalization error

Answer 53

a measure of how accurately an algorithm is able to predict outcome values for previously unseen data

Question 54

cross validation

Answer 54

partitioning a sample of data into complementary subsets, performing the analysis on one subset (called the training set), and validating the analysis on the other subset (called the validation set or testing set). To reduce variability, in most methods multiple rounds of cross-validation are performed using different partitions, and the validation results are combined (e.g. averaged) over the rounds to estimate a final predictive model.

often used when there is not enough data available to partition it into separate training and test sets without losing significant modelling or testing capability

Question 55

LOOCV leave-one-out cross validation

or replace ‘one’ w ‘p’

Answer 55

using p observations as the validation set and the remaining observations as the training set. This is repeated on all ways to cut the original sample on a validation set of p observations and a training set.

Question 56

k-fold cross-validation

Answer 56

the original sample is randomly partitioned into k equal sized subsamples. Of the k subsamples, a single subsample is retained as the validation data for testing the model, and the remaining k − 1 subsamples are used as training data. The cross-validation process is then repeated k times (the folds), with each of the k subsamples used exactly once as the validation data. The k results from the folds can then be averaged to produce a single estimation.

The advantage of this method over repeated random sub-sampling (see below) is that all observations are used for both training and validation, and each observation is used for validation exactly once. 10-fold cross-validation is commonly used,[7] but in general k remains an unfixed parameter.

Question 57

no free lunch theorem

Answer 57

a result that states that for certain types of mathematical problems, the computational cost of finding a solution, averaged over all problems in the class, is the same for any solution method. No solution therefore offers a “short cut”.