Lecture #2 - ML Basics

Question 1

Describe the differences between AI/ML/DL in your own words.

Answer 1

AI:

1. Sims to create intelligent machines capable of performing tasks that typically require human intelligence.
2. Focuses on developing systems that can exhibit intelligent behaviour and adapt to different stituations.

Machine Learning:

1. Subset of AI; development of algo. and models that allow computers to learn and make predictions or decisions without being explicitly programmed.

Deep Learning:

1. Specicialised subfield of MLthat utilises artificial neural networks inspired by the structure and functioning of the human brain.
2. Learn hierarchial representations of data by building complex computational models called deep neural networks.

Question 2

Describe the concept of a rule-based system

Answer 2

A machine learning approach that operates based on a set of predefined rules, typically created by human experts or domain knowledge.

Used to make decisions or predictions.

1. Input
2. Hand-designed program
3. Output

Question 3

Describe the concept of a hand-designed program.

Answer 3

A hand-designed program refers to a software program that is explicitly created and written by human programmers, specifying the precise steps and instructions to accomplish a particular task or solve a specific problem. In a hand-designed program, every aspect of the program’s behavior and functionality is carefully crafted and coded by human experts.

Question 4

What’s the advantage of the rule-based system.

Answer 4

1. Reply heavily on pre-defined rules and may struggle with handling complex or amiguous situations where the rules may not cover all possible scenarios.

Question 5

Give an example of a rule-based system.

Answer 5

If the ambient temperature is larger than 31 degrees celsius then

If the relative humidity is larger than 65 degrees celsius then

Thunderstorms are likely.

Question 6

Give the definition of classical ML in the lecture.

Answer 6

A computer program is said to learn from experience E with respect to some class of tasks T and performance measure P, if its performance at tasks in T, as measured by P, improves with experience E.

Question 7

Talk about task T

Answer 7

1. How should the ML system process an input/example?

We have experience from example.

2. An example is a collection of features that have somehow been obtained.
3. Represent the example as a vector x (which is an element of a real number), each entry x_i is a feature.
4. n denotes the input dimensionality.

Question 8

When talking about the task T, explain the classification tasks.

Answer 8

The task of classification involved assigning input data to predefined categories or classes.

e.g. classifying emails as spam or non-spam, or classifying images as containing cat or dogs.

1. Specify to which class (out of k possible), x belongs to.
2. Real number and there is 1 to k solutions.

Question 9

When talking about the task T, explain the clustering tasks.

Answer 9

The goal is to find a suitable set of k classes that seperate input vectors belonging to the same or similar classes.

e..g quantizer codebook optimizaion, grouping words by similarity.

Question 10

When talking about the task T, explain the regression tasks.

Answer 10

Regression tasks involve predicting a continuous numerical value based on input features.

1. Predict a numerical value from x

e.g. estimation of position, channel estimation, house price estimation

Question 11

What is the target function f

Answer 11

The target function f which should be implemented by the ML system is unkown.

The goal of the ML system is to learn an approximation of f hat = f from Experience E

Question 12

Explain what the Experience E is.

Answer 12

ML algorithms experience a training dataset, i.e. a collection of examples

Question 13

Explain the difference between unsupervised and supervised learning.

Answer 13

Supervised learning involves training a machine learning model using labeled data. Labeled data consists of input examples (features) and their corresponding correct output labels. The goal is to learn a mapping or relationship between the input features and the output labels, enabling the model to make accurate predictions or classifications on new, unseen data.

Unsupervised learning, on the other hand, deals with unlabeled data. The objective is to discover patterns, structures, or relationships within the data without the guidance of predefined output labels. The algorithm explores the inherent structure in the data to identify meaningful patterns or groupings.

Question 14

Explain reinforcement learning

Answer 14

In reinforcement learning, the learning system is referred to as an “agent,” which interacts with an “environment.” The agent takes actions in the environment, and the environment responds by providing feedback in the form of rewards or penalties. The agent’s objective is to learn the optimal strategy or policy that maximizes the long-term cumulative reward.

Question 15

Explain the Performance Measure P

Answer 15

The performance P measures quantitatively the performance of the ML algorithm.

1. Classification: accuracy or error rate
2. Regression: how well is the function approximated? MSE

Question 16

What’s important with the performance P when evaluating examples.

Answer 16

That the examples are not used for training.

The true performance measure P is typically only known after the model has been trained and tested on a seperate validation.

It may cause overfitting. The model may end up memorising the training data rather than generalizing well to new, unseen data.

Question 17

What’s the implication when defining P

Answer 17

Defining P to obtain the desired system behavior is often difficult.

Sometimes, we cannot measure the quantity we are interested in. An alternative quality that still achieves the design objectives need to be found.

Question 18

What’s the difference between training set and testing set?

Answer 18

A training set is used to train the machine learning model.

1. training set size > test set size -> sufficient data for model to learn.
2. performance model on training set is not a reliable indicator to ability to generalise unseen data; already seen training set.

A testing set is only used to evaluate the performance of ML system.

1. Serves as an unbiased benchmark to assess how well the model performs on new, unseen data.

Question 19

Give the parametric model of ML and explain the different variables within this model.

Answer 19

The model is written in f^(x, θ)

with input x and parameter θ

Question 20

Explain the learning algorithm

Answer 20

The objective function to measure the performance P:

J(θ, X[test], Y[test])

J is called the loss (or reward) and it should be minimised (or maximised).

Question 21

Explain a particular feature when it comes to classical machine learning?

Answer 21

Generating new features from existing ones is particular to classical machine learning.

Question 22

Give examples of hand-crafted features.

Answer 22

1. Transformation to polar coordinates.
2. Transformation in higher dimensional space.

Question 23

What does feature engineering require?

Answer 23

It requires domain knowledge and expertise and many educated guesses which features could be useful.

Question 24

Give the assumptions used for capacity, overfitting and underfitting.

Answer 24

1. The examples in each dataset are independent from each other.
2. The training and test set are identically distributed.
3. The underlying distribution P_data is the data-generating distribution.

Question 25

Why is the expected training error is equal to the expected test error.

Answer 25

This is due to the fact that both training and test data stem from from the same underlying distribution p_data.

Question 26

Name the process that the expected test error is greater than or equal to the expected value of training error.

Answer 26

Sample a training set of fixed size, adapt model parameters to the training set such that training error is reduced, and then sample the test set.

Question 27

Name the factors determining how well an ML algorithm will perform.

Answer 27

1. Make small training error
2. Make the gap between training and test error small.

Question 28

Name the two factors that correspond to the two central challenges in ML in terms of fitting

Answer 28

1. Underfitting

Underfitting occurs when the model is not able to obtain a sufficiently low error value on the training set.

2. Overfitting

Overfitting occurs when the gap between the training error and the test error is large.

Question 29

Define the capacity of a model.

Answer 29

It determines whether it is more likely to overfit or to underfit.

It’s the ability to fit a wide variety of functions:

1. Think of the max degree of a poly. as its capacity.
2. Models with low cap. may struggle to fit the training set.
3. Models with high cap. can overfit by memorising properties of the training set that do not appear on the test set.

Question 30

What’s the “No Free Lunch” Theorem.

Answer 30

Averaged over all possible data generating distributions p_data, every classification algorithm has the same error rate when classifying previously unobserved examples.

i.e.

The most sophisticated classifier has the same performance as just guessing (in average)

This means that no ML algo. is universal. Need to adapt it to the real-world problem at hand.

Question 31

Describe what Regularisation is.

Answer 31

Regularisation is a modification to a learning algorithm that is intended to reduce its generalisation error but not its training error.

Question 32

What are hypermeters?

Answer 32

The settings that we have at our disposal to control our ML algorithm are called hyperparameters.

Hyperparameters are parameters that are not learned b the ML algo; otherwise it would cause overfitting.

Question 33

When having data, what’s usually the split for the training and validation set.

Answer 33

• 80% training set.
• 20% validation set.

Question 34

What is the KL divergence

Answer 34

The Kullback-Leibler (KL) divergence, also known as relative entropy, is a measure of how one probability distribution differs from another.

Question 35

What is the assumption in the Naive Bayesian Classifier?

Answer 35

That x_i are conditionally independent given y

Question 36

How can we estimate P(y) and P(x_i|y)?

Answer 36

1. Using histograms
2. By fitting a distribution (e.g. Gaussian or Gaussian mixture models) and using this distribution

Question 37

In practice for hte Naive Bayesian Classifier, x_i are usually not independant. How can we make it independent?

Answer 37

They can be made independent by pre-processing with the independnet compenent analysys or approximately independent using the (simpler) principal component analysis