AI Flashcards
1
Q
Define deep learning
A
Specific sub-field of machine learning: a new take on learning representations from data that puts an emphasis on learning successive layers of increasingly meaningful representations.
2
Q
What is binary cross entropy
A
A loss function commonly used in deep learning for binary classification tasks. It measures the difference between the predicted probabilities by the model and the actual binary labels in the data.
3
Q
What are some key points about binary cross-entropy
A
-It’s a differentiable function, allowing optimization algorithms like gradient descent to efficiently adjust the model’s weights during training.
-Lower binary cross entropy indicates better model performance, meaning the predictions are more aligned with the true labels. High deviations from expected properties are punished
-It’s suitable for problems where the outcome can be classified into two categories.
4
Q
What must be present in the compliation step of a deep learning model?
A
A loss function
An optimiser
Metrics to monitor during training and testing
5
Q
What is Categorical cross entropy
A
A loss function that works very similar to binary cross-entropy for multi-classification tasks. It measures the difference between the probability distribution that the model predicts (often called the softmax layer) and the actual probability distribution of the correct class.
6
Q
When and why might Categorical cross entropy be used?>
A
-It is used for multi-classification problems
-It is a differentiable function, allowing optimization algorithms to efficiently adjust the models weights during training
7
Q
What is backpropagation and how does it work
A
-Training technique for deep neural networks
-Works to minimize the loss function by adjusting weights and biases within the network
-Uses a reversed flow of data and calculates the error at the output layer then propagates back through the network
8
Q
Define hyperparameter tuning
A
For a given neural network, there are several multiple parameters that can be optimised including the number of hidden neurons, BATCH_SIZE, number of epochs.
Hyperparameter tuning is the process of finding the optimal combination of those parameters that minimise the loss function.
9
Q
Define learning rate for gradient decent algorithms
A
Learning rate
Gradient descent algorithms multiply the magnitude of the gradient by a scalar known as learning rate (also sometimes called step size) to determine the next point.
10
Q
When should we use SGD(Stochastic Gradient Descent) vs Mini-batch SGD
A
SGD:
-simpler to implement
-Can escape local minima more easily due to the noisy updates
-Slow for large datasets
Mini-Batch SGD:
-faster than SGD for large datasets( fewer updates per epoch)
- requires tuning batch size
- may be less accurate
11
Q
Define Overfitting and some signs it’s occuring
A
Overfitting occurs when the model becomes too focused on memorizing the specific details and noise present in the training data, rather than learning the underlying patterns and relationships that generalize well to unseen data.
Signs:
High training accuracy, low validation accuracy - The model performs well on the training data but struggles on the validation data
12
Q
How can we avoid overfitting
A
Reduce model complexity: This can involve using fewer layers, neurons, or connections in the network. A simpler model has less capacity to overfit.
Data augmentation: Artificially increasing the size and diversity of your training data by techniques like flipping images, adding noise, or cropping.
Regularization: Techniques like L1/L2 regularization penalize large weights, discouraging the model from becoming too complex and overfitting the data.
Early stopping: Stop training the model before it starts to overfit. Monitor the validation accuracy and stop training when it starts to decrease.
13
Q
What are activation functions, and why are they necessary in deep learning?
A
They add non-linearity, crucial for complex learning.
Without them, networks can only learn simple patterns.
Activation functions transform neuron output (e.g. squashing values, using thresholds).
Different activation functions (ReLU, sigmoid, tanh) exist for various tasks.
14
Q
What is meant by non linearity
A
Non-linearity means the relationship between the input and output of a neuron is not a straight line.
15
Q
What do non linear activation functions do?
A
Non-linear activation functions solve the following limitations of linear activation functions:
They allow backpropagation because now the derivative function would be related to the input.
They allow the stacking of multiple layers of neurons as the output would now be a non-linear combination of input passed through multiple layers.
16
Q
What is the vanishing gradient problem
A
Vanishing Gradient Problem: In deep learning, gradients are used to train the network. This problem occurs when these gradients become very small as they travel through the network layers during training.
Impact: Small gradients make it difficult to update weights in earlier layers, hindering the network’s ability to learn complex patterns.
17
Q
What are some causes of the vanishing gradient problem
A
Blame the Activation Function: Certain activation functions (like sigmoid) have outputs that flatten out at extremes (very positive or negative inputs).
Backpropagation Culprit: During backpropagation, gradients are multiplied by the activation function’s derivative. With flattening activation, these derivatives become very small, shrinking the gradients as they travel back through layers.
Small Gradients, Big Problem: Tiny gradients make it hard to adjust weights in earlier layers, hindering learning in those crucial parts of the network.
18
Q
What is the difference between classification and regression in supervised machine learning
A
Classification:
Goal: Predict discrete categories (classes)
Output: Labels (e.g., spam/not spam, cat/dog)
Think of: Sorting things into groups
Regression:
Goal: Predict continuous values
Output: Numbers (e.g., house price, temperature)
Think of: Estimating a value on a spectrum
19
Q
What does a loss function do??
A
The loss function measures how badly the AI system did by comparing it’s predicted output to the ground truth
20
Q
What is the ground truth?
A
Ground truth refers to the correct or true information that a model is trained on and ultimately tries to predict.
21
Q
How does the mean-squared error loss function work
A
Mean squared error (MSE) is a common loss function used in machine learning, particularly in regression tasks. It measures the average of the squared differences between the predicted values by a model and the actual values (ground truth).
-Effective for continuous tasks
-Sensitive to outliers
22
Q
What is the Log Loss loss function(binary cross entropy)
A
Log loss leverages the logarithm function to penalize models for predicting probabilities that are far from the actual labels (0 or 1 in binary classification). The core idea is that the loss should be higher when the predicted probability diverges from the truth and lower when it aligns with the truth. The logarithmic function inherently satisfies this property because:
Logarithms return smaller values for smaller inputs (closer to 0).
Conversely, logarithms return larger negative values for inputs further away from 1.
-Used for binary tasks
23
Q
What is the difference between binary and multi-class cross entropy
A
Binary cross entropy deals with two classes (0 or 1), while multi-class cross entropy handles scenarios with more than two possible categories.
In a multi-class problem, the model outputs a vector of probabilities, where each element represents the probability of the input belonging to a specific class.
The individual loss terms are averages to get an overal multi-class cross entropy
24
Q
How does a vector output work for multi-class classification tasks?
A
the model will output a vector with the probability it believes the object being classified is for each possible class. e.g if there were apples, oranges and pears the model might output [0.9, 0.02, 0.01] assuming the object was an apple.
25
What is a convoluted Neural network used for and why
It's commonly used in image classification problems as it's able to sucessfully capture the spatial and temportal dependancies of an image through the application of relevent filters
26
Do Convoluted neural networks need more or less pre-processing compared to other classification algorithms
It will require much less
27
Note: the role of a Convoluted neural network is to reduce the images to a form that is easier to process
28
What are some key points about Convuluted neural networks
-CNNs learn features automatically
-Processing images in terms of RGB channels allows the network to capture color dependent features needed for image recognition
-progressively builds a hierarchy of features
29
What is a basic overview of how CNNs work
-The image is entered as a 3D cube, the height represents the pixel location and the depth represents the three color channnels (RGB)
-filters will slide across the image looking for patterns for each color
-the filters will create feature maps showing where they found interesting features
-pooling layers will shrink these maps and grab the key points
-This will be repeated and then the network will flatten everything and feed it to a regular neural network
30
What is max pooling vs average pooling in a CNN?
max pooling will take the maximum value in a specific window and will capture the most prominent features within a local area
-is good for object recognistion tasks where key features are crucial but might lose some spacial information
Average pooling will the average value for a specific region of the feature map and will capture a more generalised representation of the features within a local area
-Will provide more spatial information than max pooling but may blur sharper features
31
Note pooling is the step in a CNN where the data is shrunk, works by summerising the data in a region feature map
32
What is the class imbalance problem?
occurs when the dataset has a significant skew in the distribution of classes.
This is problematic as most algorithms will favor the majority class and poorly identify the minority class
33
What is an issue with having limited data
-The model may not be able to capture all the complexities present in the data
-The model may perform well on data it's seen during training but may struggle with unseen data
33
What is overfitting and why is it a problem?
Overfitting is where the model will memorise the training data and learn from random noise or quirks in that data set.
This means that the model will perform well on the training data but will fail on anything new.
34
What are some solutions to a class imbalance?
-collect more data
-Delete data from the majority class
-Create synthetic data
35
Random over/undersampling is a solution to the class imbalance problem, what does it mean?
Random oversampling is where random data points from the minority class are duplicated
Random undersampling is where random datapoints from the majority class are deleted
36
What are some problems with over/under sampling
It can cause loss of information(undersampling)
It can cause overfitting and fixed boundaries(for over sampling)
37
When training a CNN on small/class imbalanced data overfitting and bias are two issues that can arise. What are they and how can they be solved
Overfitting is caused by having too few samples to learn from, rendering the model unable to generalise to new data
Bias is caused by having class imbalance, the model is unable to learn the boundaries between the classes
These can be solved by regularisation and data augmentation
38
What are some techniques for data augmentation
Rotation range - takes a value in degrees which signifies a range in which to rotate pictures
width shift and height shift - takes a value within which will randomly translate pictures vertically or horizontally
Shear range - will randomly apply shear transformations
zoom range- will randomly zoom into pictures
horizontal flip - will randomly flip half the images horizontally
fill mode- Will fill in newly created pixels which can appear after a rotation or width shift
39
What is Synthetic Minorty Oversampling Technique(SMOTE)
It's a data augmentation technique used to address a class imbalance in machine learning tasks
-Should only be performed after the train-test split
-Should only ever be performed on the training data
40
How does SMOTE work (4 steps)?
1. Identifiying minority class Instances
2. Select instances randomly for oversampling
3. Find the nearest neighbours of the selected instance from the minority class
4. Generate Synthetic Samples
-Create synthetic samples by interpolating between the selected instance and it's K nearest neighbour e.g. take the difference between the two
The synthetic samples can then be added to the dataset
41
What are three performance metrics used for a CNN?
Accuraccy - can be misleading for imbalanced data sets
recall - measure of the model corectly identifiying true positives
precision - the ratio between true positives and all the positives
42
What is a pretrained model and what is it commonly used for?
It's a saved network that was previously trained on a large dataset
it's commonly used when attempting deep learning on smaller datasets
43
What is an example of a pretrained model?
ImageNet - has 1.4 million labeled images and 1000 different classes
Contains many different animal classes and will perform well at animal identification
44
Give three examples of pre-trained models and their uses
Faster R-CNN:
Faster R-CNN is a widely used model for object detection.
Mask R-CNN:
An extension of Faster R-CNN, providing pixel-wise segmentation
masks along with bounding boxes.
YOLO (You Only Look Once):
YOLO is an object detection algorithm that divides the input
image into a grid and predicts bounding boxes and class
probabilities directly. YOLOv3 and YOLOv4 are some of the
popular versions.
45
Define feature extraction
The process of identifying and extracting meaningful characteristics from data. Features help a deep learning model understand data by focusing only on the relevant parts, making the data easier for the model to process and analyse. The model can then learn patterns and relationships more effectively
46
Why is feature extraction a useful part of pre-trained models?
As these features are often learned from massive datasets pre trained models will already have learned useful features
47
How can we use extracted features and pre-trained models to improve our models
We can freeze the the weights(parameters) of the pre-trained models initial layers(convolutional base) so they can act as a feature extractor and their knowledge will remain unchanged. The later layers can be trained specifically for the required task(This is pretty much fine-tuning tbh)
48
How can we create a feature extractor?
We can remove the final layers of a pre-trained model, removing the decision making sections and leaving only a feature extractor
49
How can we use fine-tuning in pre-trained models?
Fine-tuning is about using the pre-trained models large amount of generalised knowledge and adapting it to a specific problem. This can be done by freezing the initial layers, keeping the feature extraction knowledge intact. Later layers responsible for classifying categories are left unfrozen and additional layers can be added specifically for the classification task`
50
What are some benefits of fine tunning?
Reduced Training Data Need: Since the pre-trained model already has a strong foundation, you can often achieve good results with a smaller amount of your own data for fine-tuning.
Faster Training Times: By only training a portion of the model, fine-tuning is generally much faster than training a model from scratch.
Improved Performance: By leveraging pre-trained knowledge and adapting it to your specific task, fine-tuning can significantly improve the accuracy of your final model.
51
When reusing a convulutional base in a CNN what is the difference between deep and shallow mode?
In deep mode, also known as heavy fine-tuning, you essentially fine-tune a significant portion of the pre-trained convolutional base along with your newly added layers.
-Improved performance
-Overfitting risk & increased training time
Shallow mode, also known as light fine-tuning, focuses on making smaller adjustments to the pre-trained model.
-Fast training
-Reduced overfitting risk
-limited performance
52
What is dimensionality reduction?
Dimensionality reduction is the technique of representing multi-dimensional data (data with multiple features having a correlation with each other) in a lower dimension.
53
Why do we need dimensionality reduction?
The curse of dimensionality is a phenomenon that happens because the sample density decreases exponentially with the increase of the dimensionality. When we keep adding (or increasing the number) features without increasing the size/number of training samples, the dimensionality of the feature space grows and becomes sparser. Due to this sparsity, it becomes much easier to find a perfect solution for the machine learning model which highly likely leads to overfitting.
e.g.
As the number of dimensions increases, the amount of data needed to effectively train a model grows exponentially. This can lead to issues like overfitting and computational inefficiency. Dimensionality reduction helps alleviate this curse by reducing the number of features the model needs to learn from.
54
What are autoencoders?
Autoencoders are a type of unsupervised ANN. Its primary objective is to learn a representation (encoding) of the input data in a way that it captures the essential features, reducing the dimensionality of the data. This encoding is then used to reconstruct the original input as closely as possible.
55
What are the four parts of an autoencoder
Encoder: is the part of the autoencoder responsible for encoding/compressing the input data into a lower-dimensional representation.
Decoder: it takes the compressed representation generated by the encoder and attempts to reconstruct the original input data from this representation.
Latent Space (code): the compressed representation learned by the encoder is often referred to as the "latent space."
Objective Function: to minimize the difference between the input data and its reconstruction, where could be achieved by using a loss function such as mean squared error (MSE) to measure the difference between the input and the reconstructed output.
56
What are denoising autoencoders?
A specific type of autoencoder architecture designed to tackle noisy data. In addition to dimensionality reduction and feature extraction(like a normal autoencoder) they can remove noise from the data iteself
57
How do denoising autoencoders work?
An autoencoder is trained on blurry or noisy images. The DAE tries to reconstruct a clean, clear version of the image from the corrupted input. By forcing this reconstruction, it learns to identify and remove the noise while capturing the underlying features of the data.
-They are trained on a corrupted version of the data, this can be anything from salt-and-pepper noise to occluding or masking part of the data
58
One application for AutoEncoders is adding color to black and white images
59
What is the goal of natural language processing?
To make machines understand and interpret human language the way it is written or spoken
60
What are the two levels of linguistic analysis that must be done before performing NLP
syntax- what part of the text is grammatically correct
semantics - What is the meaning of the given text
61
One thing that NLP has to deal with is morphology, what does that mean?
The formation of words and how they relate with each other
62
What is natural language understanding and what are the ambiguities present in natural language?
Understanding the meaning of a given text
The following are ambiguities NLP will attempt to resolve
Lexical ambiguity - words have different meanings
syntactic - sentence has multiple parse trees
sematic- sentence has multiple meanings
anaphoric - phrase or word previoulsly mentioned which has a different meaning
63
What are 4 stages of Natural Language Understanding?
Syntax Analysis: identify the structure of the sentence, including parts of speech and sentence structure.
Semantics: Understand the meaning of words and phrases
Named Entity Recognition(NER): recognise named entities in the input. e.g. Exeter is a location
Intent Recognition: Understand the Users Intent
64
What are the stages of the NLP pipeline?
Import text file
Sentence Segmentation- identify sentence boundaries in the given text e.g full stops new lines etc
Tokenisation - Identify different words numbers and punctuation
Stemming - Strip the ending of words e.g. eating is reduced to eat
Part of speech(POS) tagging - assign each word in a sentence it's own tag such as designating word as noun or adverb
Parsing - Divide text into different categories to answer a question e.g. this part of sentence modifies another part
Named Entity Recognition - Identifies entities such as persons, location and times
co-reference - Define the relationship of the given words in a sentence with previous and next sentence
65
How does Sentence Segmentation work?
Will split a large section of text into component sentences, often using punctuation like fullstops, to apply sentence tokenisation with NLTK we can use the NLTK.sent_tokenise function
66
How does Tokenisation work?
Will split the segmented sentences up into "tokens". Tokenised sentences are essentially just arrays of words
67
What are stop words?
Stop words usually refer to the most common words such as “and”, “the”, “a” in a language, but there is no single universal list of stopwords. The list of the stop words can change depending on your application.
Stop words are removed before filtering text data as they are considered to have little meaning
67
What does Lemmatization and Stemming do?
The goal of both stemming and lemmatization is to reduce inflectional forms and sometimes derivation-ally related forms of a word to a common base form.
am, are, is => be
dog, dogs, dog’s, dogs’ => dog
67
Define Lemmatization
Lemmatization usually refers to doing things properly with the use of a vocabulary and morphological analysis of words, normally aiming to remove inflectional endings only and to return the base or dictionary form of a word, which is known as the lemma.
e.g. The word better has "good" as it's lemma
68
Define Stemming
Stemming usually refers to a crude heuristic process that chops off the ends of words in the hope of achieving this goal correctly most of the time, and often includes the removal of derivational affixes
69
How does Part of Speech(POS) tagging work?
Part of speech (POS) tagging is a fundamental technique used to understand the grammatical function of each word in a sentence. It's like dissecting a sentence and labeling each word based on its role.
Each word in a text is assigned a category based on its grammatical function. Common categories include nouns, verbs, adjectives, adverbs, pronouns, prepositions, conjunctions, interjections, and determiners
-By knowing the POS of each word NLP applications can grasp the sentence structure and how words relate to each other
70
How does the bag-of-words model work?
The bag-of-words model is a popular and simple feature extraction technique used when we work with text. It describes the occurrence of each word within a document.
Any information about the order or structure of words is discarded. That’s why it’s called a bag of words. This model is trying to understand whether a known word occurs in a document, but don’t know where is that word in the document.
The intuition is that similar documents have similar contents. Also, from a content, we can learn something about the meaning of the document.
71
Using the bag of words model(sentence Level) how would these sentences be represented
The quick dog jumped over the lazy fox
The dog is quick
Note: im not sure whether stop words should be removed so i won't here but keep in mind that they can be
[2, 1, 1, 1, 1, 1, 1, 0]
[1, 0, 1, 0, 0, 0, 0, 1]
72
How can we measure the similarity of a BOW model?
measured as the cosine of the angle between two vectors
73
What is TF-IDF
TF-IDF, short for term frequency-inverse document frequency is a statistical measure used to evaluate the importance of a word to a document in a collection or corpus.
The goal is to highlight words that are both important within a specific document and distinctive in the entire corpus.
74
How can Term frequency be calculated?
A scoring of the frequency of the word in the current document.
This part measures how often a word appears in a document.
The more frequently a word appears, the higher its TF score for that document.
TF = times term appears in doc/total items in doc
75
How can Inverse Term frequencey be calculated?
Aa scoring of how rare the word is across documents.
This part measures how unique or rare a word is across multiple documents in the corpus.
The rarer the word, the higher its IDF score.
IDF = log(no. documents/ documents with term in it)
76
How can TFIDF be calculated?
TFIDF(term) = TF(term)*IDF(term)
77
What is the basic idea of N-gram word prediction?
Basic idea underlying the statistical approach to word prediction is to use the probabilities of SEQUENCES OF WORDS to choose the most likely next word/correction of spelling error
for all words we predict the next word for which this (conditional) probability is highest. We can get this probability by estimating by relative frequency
78
What is MLE and how does it work?
MLE is a statistical technique for estimating the parameters of a probability distribution. In NLP, it's used to estimate the likelihood of a sequence of words occurring in a natural language.
By analyzing a large corpus of text, MLE estimates the probability of each word appearing in a specific context. This information is then used to predict the most likely next word in a sequence.
79
How can we use the chain rule to calculate the probability of a sequence of words?
Imagine a sentence as a sequence of words, like "The quick brown fox jumps." The chain rule allows you to calculate the probability of this entire sentence by breaking it down into the probability of each word appearing given the words that came before it.
Using the chain rule:
P(S) = P(w1) * P(w2 | w1) * P(w3 | w1, w2) * ... * P(wn | w1, w2, ..., wn-1)
Directly calculating all these individual probabilities can be impractical, especially for long sentences.
In practice, NLP models estimate these probabilities using techniques like n-grams that rely on statistical learning from large text corpora.
80
Define the Markov assumption
only prior local context – the last few words – affects the next word
making the Markov assumption for word prediction means assuming that the probability of a word only depends on the previous N-1 words (N-GRAM model)
81
What is the difference between a unigram, bigram and trigram n-gram model?
Unigram: This is the simplest type of n-gram, where n = 1. It just considers the probability of a single word appearing in isolation. For example, the probability of the word "the" appearing in a text.
Bigram: Here, n = 2. This considers the probability of a two-word sequence appearing together. For instance, the probability of the word "the" being followed by the word "cat" is likely higher than "the" being followed by "pineapple."
Trigram: This type of n-gram uses a sequence of three words (n = 3). So, it would consider the probability of a three-word sequence like "I went to" being followed by the word "the" or "the store."
82
What are some drawbacks of N-gram language models?
The higher the N, the better is the model usually. But this leads to lots of computation overhead that requires large computation power in terms of RAM.
N-grams are a sparse representation of language. It will give zero probability to all the words that are not present in the training corpus.
83
Give an overview of a Recurrent Neural Network
They are designed to handle sequential data where the order matters. They will use internal memory to store information about past inputs and influence how they process future ones
The idea is to have multiple copies of the same network, each passing a message to a successor. In this way the network can remember information.
RNN can be demonstrated as a graph of RNN cells, where each cell performs the same operation on every element in the input sequence.
84
How does an RNN process sequences?
RNN processes sequences by iterating through the sequence elements and maintaining a state containing information relative to what it has seen so far.
85
What is padding sequences?
it's a common pre-processing step when working with sequence data. The purpose of it is to ensure that all sequences have the same length which is required for feeding data into networks that expect fixed-size inputs
86
How does word embedding work?
The embedding layer represents words as dense vectors where each dimension encodes information about how the word relates to others in the given context
87
What is the training objective of word embedding?
The objective of the model during training is typically a language modeling task, where it predicts the probability distribution over the vocabulary for the next word given the context. This task encourages the model to learn representations that capture semantic and syntactic relationships.
88
How can we create an RNN used for Word prediction?
-Token text into words represented by a unique integer index
-Create input output sequences where the input is a sequence of words and the output is the next word in the sequence
Building the model:
-Use an Embedding layer to convert word indices into dense vectors.
-Include one or more recurrent layers to capture sequential dependencies.
-Use a Dense layer with a softmax activation for predicting the next word.
-Choose an appropriate loss function, optimizer, and metrics for your problem.
-Fit the model on the training data.
89
What is the long term dependency problem
Also known as the vanishing gradient problem. This is where RNN struggles to learn dependencies in data points far away from each other in long sequences.
Technical explanation:
For long sequences, the multiplication of gradients across many time steps can lead to extremely small values. If these values become close to zero, they may cause the gradients for early time steps to vanish.
90
What is a Long Short-Term memory network?
Long Short-Term Memory (LSTM) networks are a special kind of Recurrent Neural Network (RNN) specifically designed to address the vanishing gradient problem that hinders traditional RNNs. They excel at handling long-term dependencies in sequential data, making them a powerful tool for tasks like machine translation, speech recognition, and time series analysis.
LSTM networks are a type of RNN that uses special units that include a 'memory cell' that can maintain information in memory for long periods of time.
A set of gates is used to control when information enters the memory, when it's output, and when it's forgotten.
91
What are the three gates an LSTM uses to manipulate input information
Input gate to control the intake of new information.
-Determines what new information to store in the cell state. It considers the current input and the previous state, deciding which elements are relevant and assigning weights between 0 and 1.
Forget gate to determine what part of the cell state to be updated.
-Decides what information to discard from the previous cell state. It analyzes the current input and the previous state, assigning a value between 0 and 1 to each element in the cell state. Values close to 0 indicate forgetting, while 1 means retaining the information.
Output gate to determine what part of the cell state to output.
-Controls what information from the cell state is used to update the hidden state of the LSTM. It examines the current cell state and the previous hidden state, assigning weights between 0 and 1.
92
What is the difference between machine and deep learning?
One major difference is that in machine learning feature extraction must be done manually, the quality of the models perfromance relies on the quality of the chosen features
In deep learning neural networks can automatically learn features from raw data
93
How does the amount of data effect the performance of deep vs classical learning?
While deep learning and classical learning will perform at similar levels with a small amount of data, as the amount of data increases deep learning models will benefit much more
94
Define regression in predictive modelling
Estimating numerical value of an attribute, based on the history of values assigned to this attribute and other affecting measurements
95
Define classification in predictive modeling
Assigning a label to each instance depending on the values
of a set of attributes
96
How does logistic regression work for classification?
It will create a linear decision boundary(e.g a straight line). This will be based on a formula that considers features of the objects being classified, objects below this score will be classed as something different to something above the line
97
How do Decision trees work for classification
Is almost like a binary tree of yes/no decisions with each level being a question about the object being classified. The model will continue asking questions until it reaches a leaf node containing a classification for the object.
98
What are the steps to create a decision tree?
Determine attribute to select as the root;
Partition input examples into subsets according to values of the root attribute;
Construct Decision Tree recursively for each subset;
Connect the roots for the sub-trees to the root of the whole tree via Labeled links
99
What is the first attribute of a decision tree called and how can we determine what it should be
It's called the root and it should be the attribute considered the best by some metric of goodness e.g. information gain for the attribute. Root node can be determined by computing the entropy of the classification data and creating a frequency table for the classes
100
When does the exit decision for the decision tree occur?
Exit condition occurs when all examples belong to one class (other conditions exist to stop growing the tree earlier).
101
How can we calculate the entropy for a decision tree
E = -Σ (pi * log2(pi))
102
How can we calculate the information gain for attributes in a decision tree
IG(Feature) = E - Σ (pi * E(Di))
103
How can we use decision trees for regression
Decision tree regression is a supervised learning technique used for predicting continuous values.
Objective Function:
Common metrics for regression include Mean Squared Error (MSE) or Mean Absolute Error (MAE), or other regression-specific metrics.
Leaf Node Values:
In classification trees, the leaves represent the predicted class. In regression trees, the leaves should represent a continuous value.
Splitting Criteria:
Regression trees aim to minimise the chosen regression metric (e.g., MSE or MAE).
104
What is the random forest model and how does it work?
The random forest model is a powerful ensemble learning technique used for both classification (predicting discrete categories) and regression (predicting continuous values) tasks.
A random forest is built by creating a multitude of decision trees at training time. Each tree is like an individual expert, and the final prediction is made by combining the predictions of all the trees (like a democratic vote).
105
How can we create a random forest model?
Random Subsets: When creating each tree, the algorithm samples a random subset of data points (with replacement) from the original training data. This technique, called bagging (bootstrap aggregating), helps to introduce diversity among the trees and reduce overfitting.
Random Features: At each node of a tree, the algorithm randomly selects a subset of the features (instead of considering all features) to determine the best splitting rule. This randomness further diversifies the trees and prevents them from becoming too similar.
Growing the Trees: Each tree is grown to full depth without pruning (unlike some decision tree approaches). This ensures that each tree captures a unique aspect of the data.
Prediction: For classification tasks, the most frequent class predicted by the individual trees is chosen as the final prediction. For regression tasks, the average of the individual tree predictions is used as the final prediction.
106
What are some advantages and disadvantages of the random forest model?
Advantages:
High Accuracy: By combining multiple decision trees, random forests can achieve higher accuracy than a single decision tree.
Robust to Overfitting: Randomization through bagging and feature selection helps to reduce overfitting, a common problem in decision trees.
Handles Missing Data: The algorithm can handle missing data points effectively.
Interpretability: While not as interpretable as a single decision tree, feature importance scores can provide insights into which features are most influential for the model's predictions.
Disadvantages:
Can be Black Box: The inner workings of the individual trees can be complex, making it challenging to understand exactly why the model makes specific predictions.
Computationally Expensive: Training a random forest can be computationally expensive, especially for large datasets.
107
What is nearest neighbor classification
Nearest Neighbor (NN) classification is a fundamental machine learning technique used to classify new data points based on their similarity to existing labeled data points.
108
What are some key issues with nearest neighbor classification?
no model is built! all the data is retained
training could take up to O(np) per observation
will not do well when number of features is large
109
What is the k-NN algorithm(also known as nearest neighbor regression)?
User inputs k:an integer representing the number of nearest neighbors (instances) to search for
With each unlabeled instance: calculate the distance between it and all the instances in the data set
Find the k nearest neighbors
Count the assigned class labels in k nearest neighbour for each class
The class with the highest count (majority vote) is the output
110
What is nearest neighbor regression
Nearest Neighbor (NN) regression, also called k-Nearest Neighbors (KNN) regression, is a technique for predicting continuous values using the k nearest neighbors of a new data point.
111
How does nearest neighbor regression work?
Calculate Distances:
Compute the distances between the query point and all other points in the training dataset based on the chosen distance metric (e.g., Euclidean distance).
Identify Neighbours:
Select the K nearest neighbours based on the calculated distances.
Aggregate Target Values:
For regression, instead of counting votes, take the average (or another aggregation measure) of the target variable values of the K neighbours. This aggregated value is the prediction for the query point.
112
Define a Multi-layer perceptron and the it's components
A feed-forward multi-layer perceptron is essentially a simple kind of neural network with:
input vector {x1, x2, ..., xn}
output value y
intermediate “hidden” values {h1, ..., hm}
113
How are multi layer perceptrons used for classification?
a multi-layer perceptron is a non-linear function that maps the input vector to the output value
the hidden values are also non-linear functions
non-linear sigmoid threshold function ensures that the output is
a value between [0; 1], i.e., 0 <= y <= 1
weights are applied at each layer
114
Describe the learning process of an MLP
the learning process follows a steepest descent approach
the idea is that we search the landscape of possible output values, and we want to descend from large errors to small (ideally 0) error
backpropagation is the classic example
the weights are adjusted by moving backwards:
first from the hidden > output layer
and then from the input > hidden layer
amount of adjustment is proportional to the value of error function
big errors > big adjustments
small errors . small adjustments
the error rate should decline during the learning process
115
Define unsupervised learning
Unsupervised learning is a type of machine learning where algorithms learn patterns and insights from unlabeled data. Unsupervised learning allows the model to discover hidden structures or groupings within the data on its own.
116
What are some advantages of unsupervised learning
Unsupervised learning excels at finding hidden patterns and relationships within unlabeled data
Can handle a wide variety of data types like text images and sensor data
Good for anomaly detection
117
What are some disadvantages of unsupervised learning?
There is no ground truth and therefore no clear "correct answer", evaluating the result can be subjective
118
Define clustering
Clustering is a type of unsupervised learning
A cluster is a collection of objects which are similar in some way
Clustering is the process of grouping similar objects into groups
Clustering is a type of unsupervised learning
A cluster is a collection of objects which are similar in some way
Clustering is the process of grouping similar objects into groups
119
What is the purpose of clustering algorithms
To see whether the data fall into distinct groups, with members within each group being similar to other members in that group but different from members of other groups.
120
How do clustering algorithms work?
We have to define a similarity or distance metric that computes how close two instances are to each other. Each instance is assigned to a cluster(group) based on this metric. Some instances are considered outliers and do not belong in any cluster
121
What is the formula for euclidean similarity measures?
How can we calculate the ecludian distance for two instances p = (7,10 ) and q (5, 14)
sqrt((q1 - p1)^2 + (q2 - p2)^2 +...+ (qn - pn)^2)
for p=(7,10) q=(5, 14)
= sqrt((5 - 7)^2 + (14 - 10)^2) = 4.47
122
What is the formula for manhattan similarity measure?
How can we calculate the manhattan similarity measure for two instances p = (7,10 ) and q (5, 14)
Σ(n, i = 1) |pi - qi|
for p=(7,10) q=(5, 14)
= |7 - 5 | + |10 - 14|
123
What do each of these terms mean in clustering?
Centroids
size
variations
Centroids: The centre of each cluster(average of each feature of all instances belonging to the cluster )
size: number of instances in the cluster
variations: the variance or standard deviation of the instances belonging to each cluster
124
How do i calculate standard deviation and variance
Variance is the mean of the squares minus the square of the mean
Ex^2/n - (Ex/n)^2
Standard deviation is the sqrt of variance
125
What is the cluster center?
A single data point that manages to describe best a collection of objects(stays in the centre)
For numeric data the centre of mass
-calculated using the mean of all data points
For nominal or ordinary data it could be the most frequent mode
Is a score function
126
What are Cluster variations and how are they calculated?
Cluster variation is a score function that can show how compact/tight clusters are, it's calculated as follows:
Calculate the difference between each data point and the centriod of the structure then square it
Add up the squared distance for every data point for a single cluster
We can then do this for ever cluster and sum them to find the variation for the entire clustering
127
What is the K-means algorithm(5-steps)
1. define number of clusters
2. choose k data objects randomly to serve as initial centriods for k clusters
3. assign each data object to the cluster represented by it's nearest centriod
4. find a new centriod for each cluster by calculating the mean vector of it's members
5. Remove all memberships and repeat from step 3 until cluster membership does not change/ max iterations are reached
128
What are some strengths of the k-means algorithm?
simple and easy to implement
quite efficient
129
What are some weaknesses of the k-means algorithm?
Meed to specify the value of k, but we may not know what the value should be beforehand
You may want to experiment with k value (i.e., number of clusters)
Sensitive to the initialisation
Sensitive to noise
Clusters of different size
130
What is a hierarchical based clustering algorithm
Imagine a family tree. Hierarchical clustering builds a similar structure for your data points, where data points are grouped based on similarities, forming a hierarchy of clusters.
At the bottom of the hierarchy, each data point is in its own individual cluster.
As you move up the hierarchy, clusters are merged based on their similarities, forming larger and more general clusters.
The final result is a dendrogram, which shows the merging process and the relationships between clusters at different levels.
131
What are the two main approaches for a hierarchical based clustering algorithm
Agglomerative Hierarchical Clustering(Bottom-up): start with each data point in it's own cluster and iteratively merge similar clusters until one remains
Divisive Hierarchical Clustering(top-down): start will all data in a single cluster and recursively split the cluster into subclusters based on dissimilarities
132
What are the steps for an Agglomerative Clustering algorithm
Take n data objects as individual clusters and build an n x n dissimilarity matrix storing distances between any pair of objects
While there is more than one cluster:
-find the two data objects with the minimum distance and merge into a bigger cluster
-replace the entries in the matrix for the original clusters or
objects by the cluster tag of the newly formed cluster
-re-calculate relevant distances and update the matrix
This whole process will produce a dendogram and relies of the definition of a distance metric between clusters
133
What are the strengths of Agglomerative clustering algorithms
Deterministic results
Multiple possible versions of clustering
Mo need to specify the value of a k beforehand
Can create clusters of arbitrary shapes (single-link
134
What is the weakness of Agglomerative clustering algorithms
Does not scale up for large data sets
135
What do these metrics about Clustering algorithms mean:
nearest neighbor(single link)
furthest neighbor(complete link)
centriod distance measurement
group average
nearest neighbor(single link) - Distance between two closest points in distinct clusters
furthest neighbor(complete link) - Distance between two furthest away points in 2 clusters
distance between centriods for clusters
Group average- average of all distances between pairs of points in two clusters(every points links to each other)
136
What is the algorithm for self organising maps(this is like one slide so idk what's going on here)
Select the size and type of the map.
Initialise all node/neuron weight vectors randomly.
Choose a random data point from training data and present it to the SOM.
Find the “Best Matching Unit” (BMU) in the map – the most similar node. Similarity is calculated using the Euclidean distance formula.
Determine the nodes within the “neighbourhood” of the BMU.
Adjust weights of nodes in the BMU neighbourhood towards the chosen datapoint. Repeat Steps 2-5 for N iterations / convergence.
137
What can we do to solve the issue of summary statistics being deceptive?
We can visualize our data attributes
138
Why should we use data visualisation?
It is a good way to communicate complex information.
It is critical tool in AI, it provides an effective way to identify summaries, structure, relationships, differences, and abnormalities in the data.
139
What type of chart can we use to show categories as percentage?
Pie charts show categories as a proportion or a percentage of the whole. Use pie charts to show the composition of categorical data with each segment proportional to the quantity it represents.
140
What is a grammer of graphics?
A framework that enables us to concisely describe the components of any graphics
141
Why is the grammer of graphics important?
Learning the grammar will help you not only create graphics that you know about now, but will also help you to think about new graphics that would be even better. Without the grammar, there is no underlying theory and existing graphics packages are just a big collection of special cases.
142
What are the key components in the grammar of graphs
Data: The raw information you want to visualize.
Aesthetic Mappings: How visual properties (aesthetics) like color, size, position, etc. are linked to the data variables
Geometrical Shapes: The basic visual marks used to represent data points (like points, lines, bars, etc.).
Scales: How data values are mapped to visual properties (e.g., color scale for temperature).
Coordinates: The system that defines the position of visual elements (e.g., x and y axes in a scatter plot).
Themes: The overall visual design elements that provide a consistent look and feel to the graphic (e.g., color palettes, fonts, backgrounds).
143
What is principle component analysis?
Principal Component Analysis (PCA) is a dimensionality reduction technique used in machine learning and data analysis. It helps simplify complex datasets by focusing on the most important features that capture the majority of the variation in the data.
144
What is mean centering in PCA
Imagine a dataset with features like height and weight. Without centering, features with larger scales (e.g., height in cm) can dominate the analysis.
Mean centering subtracts the mean value of each feature from all the data points in that feature.
This essentially shifts the data such that each feature has a mean of zero.
By centering the data around the origin (0, 0), PCA focuses on the direction of the spread (variance) rather than absolute values.
145
What is Covariance Matrix in PCA?
This matrix captures how much two features in your data vary together.
After centering, the covariance matrix shows the relationship between features in terms of their deviations from the mean.
A positive covariance indicates features tend to move together (higher values in one feature correspond to higher values in the other).
A negative covariance indicates features move in opposite directions (higher values in one feature correspond to lower values in the other).
A value close to zero suggests little linear relationship between the features.
146
What is Eigenvalues and Eigenvectors in PCA?
Find the eigenvalues and eigenvectors of the covariance matrix. Eigenvalues represent the amount of variance captured in each principal component, and eigenvectors give the direction of these components
147
What are the Principle Components in PCA?
The eigenvectors are the principal components, and they are ranked based on their corresponding eigenvalues. The first principal component captures the maximum variance in the data, followed by the second, third, and so on.
148
What is a search algorithm?
Search algorithms provide search solutions through a sequence of actions that transform the start state to the goal state.
149
What do these properties of search algorithms mean?
Completeness
Optimal
Time Complexity
Space Complexity
Completeness: A search algorithm is complete if it provides a solution for a given input when there exists at least one solution for this input.
Optimality: Optimal solutions are the best solutions given by the search algorithms at the lowest path cost.
Time Complexity: The time needed to accomplish a task or provide a solution.
Space Complexity: Memory or storage space needed when conducting a search operation.
150
What do these terms mean?
Initial state
State space
Actions
Goal State
Goal Test
Path Cost
Initial state: This is the start state in which the search starts.
State space: These are all the possible states that can be attained from the initial state through a series of actions.
Actions: These are the steps, activities, or operations undertaken by AI agents in a particular state.
Goal state: This is the endpoint or the desired state.
Goal test: This is a test conducted to establish whether a particular state is a goal state.
Path cost: This is the cost associated with a given path taken by the agents
151
What are three uninformed/blind search algorithms
Breadth-first search
Depth-first search
Uniform cost search
152
What are two Informed search algorithms
Greedy search
A* tree search
153
How does Breadth first search work?
It starts at the root node and systematically explores all the neighbouring nodes at the current depth level before moving on to nodes at the next depth level.
154
How does Breadth first search do according to these metrics?
Completeness
Optimality
Time complextiy
Space complexity
Completeness:
BFS is complete. If there is a solution in the search space, BFS will eventually find it. This is because BFS explores all nodes at a given depth level before moving on to the next level, ensuring that it thoroughly searches the entire space.
Optimality:
BFS is optimal when all actions have the same cost. Since BFS systematically explores nodes level by level, it guarantees that the first solution found is one with the fewest possible steps or edges.
Time Complexity:
The time complexity of BFS is typically O(V+E), where V is the number of vertices (nodes) and E is the number of edges in the graph. In the worst case, BFS might visit all vertices and edges.
Space Complexity:
The space complexity of BFS is O(V) where V is the number of vertices. This is because BFS stores all vertices at a given depth level in a queue. In the worst case, the queue may contain all vertices at the maximum depth.
155
How does depth first search work?
it starts at the root node, explores along a branch until it reaches a leaf node, and then backtracks to explore other branches.
156
How does Depth first search do according to these metrics?
Completeness
Optimality
Time complextiy
Space complexity
Completeness:
DFS is not necessarily complete. DFS may get stuck in an infinite loop if the graph has cycles.
Optimality:
DFS is not optimal. It may find a solution, but it does not guarantee that it is the shortest path. The first solution encountered may not be the optimal one.
Time complexity and space complexity are the same as breadth first search
157
How does uniform cost search work?
It explores nodes in a way that considers the cost associated with each node, prioritizing nodes with lower costs before those with higher costs.
It is guided by the cost to move from one node to another. The goal is to find a path where the cumulative sum of costs is the least.
cost(node) = cumulative cost of all nodes from root.
158
How does Uniform Cost search do in terms of completeness and optimality?
Completeness:
UCS is complete. This is because UCS explores paths in increasing order of cost, ensuring that it finds the lowest-cost solution.
Optimality:
UCS is optimal. It guarantees that the first solution found is the one with the lowest cost. This is because UCS prioritizes paths with lower costs and explores them first.
159
How does Greedy search work?
picks the next step by focusing on the best-looking option at each moment, mainly based on heuristic values, without caring much about the total path cost.
the closest node to the goal node is expanded, where closeness factor is calculated using a heuristic function h (x).
h (x) is an estimate of the distance between one node and the end or goal node. The lower the value of h (x), the closer the node is to the endpoint
160
How is the completeness and optimality of Greedy search?
Completeness:
Greedy search is not guaranteed to be complete. It may get stuck in an infinite loop or fail to find a solution, especially if it encounters cycles or cannot backtrack.
Optimality:
Greedy search is not necessarily optimal. It makes locally optimal choices at each step, but these choices may not lead to the globally optimal solution. The algorithm tends to prioritize immediate gains without considering the overall cost.
161
How does A* tree search work?
A* search is like a smart explorer. It considers both the cost to reach a point and a heuristic estimate of how far that point is from the goal.
It combines the attributes of the uniform cost algorithm and the greedy algorithm. Here, the heuristic is simply an integration of the greedy search cost (h (x)) and the cost in the uniform cost algorithm (g (x)).
h(x) is the forward cost, which is an estimate of the distance of the current node from the goal node.
g(x) is the backward cost, which is the cumulative cost of a node from the root node.
The cumulative cost is denoted as f (x) = h(x) + g(x).
The strategy is to select the node with the lowest total cost value (f (x)).
162
How does A* tree search work with completeness and optimality
Completeness:
A* Search is complete. If there is a solution in the search space, A* will eventually find it.
Optimality:
A Search:* A* Search is optimal. It guarantees finding the optimal solution by using a heuristic function to guide the search. The heuristic helps prioritise paths likely to lead to the best solution.
163
What are some examples of Breadth first search and Depth first search applications
Shortest Path in Unweighted Graphs: Identifying the shortest path in a network.
Web Crawling: Indexing web pages level by level.
Puzzle Games: Solving puzzles with multiple states.
Maze Solving: Finding a path through a maze.
Game AI: Exploring possible moves in games like chess or tic-tac-toe.
Network Routing: Exploring paths in computer networks.
164
What are some examples of Uniform Cost Search aplications
Dijkstra's Algorithm: Finding the shortest path in a graph with weighted edges.
Resource Allocation: Optimising the use of resources in project management.
Network Routing with Variable Costs: Routing in computer networks where edges have varying costs.
165
What are some examples of A* search and greedy search applications
Robotics: Path planning for robots in an environment with obstacles.
Maps and Navigation Systems: Finding optimal routes on maps.
Video Game Pathfinding: Navigating characters in video games efficiently.
Traveling Salesman Problem: Determining the most efficient route to visit multiple locations.
Job Scheduling: Assigning tasks in a way that optimises a specific criterion.
Network Design: Optimising the layout of communication networks.
166
What is a proposition?
A proposition is a declarative sentence (a sentence that declares a fact) that is either true or false, but not both.
167
What is a propositional function?
A proposition that depends on unknown values like variables
168
What does the equivalence mean in propositional logic
denoted as ≡ or ↔
Equivalence requires two variables and will return “true” if both variables have the same value
169
What is a tautology?
Tautology is very similar to logical equivalence
When all values are “true” that is a tautology
170
Why might we want to translate English into compound statements?
English (and every other human language) is often ambiguous. Translating sentences into compound statements removes the ambiguity.
171
What does soundness mean in propositional logic?
Soundness ensures that if the initial statements and rules are true, then the conclusion will also be true.
172
What does completeness mean in propositional logic?
Completeness ensures that any true statement can be derived within the system.
In our example, if we know the true statement "I have free time" (Q), the system allows us to derive the true statement "It's the weekend" (P).
Completeness guarantees that the system captures all possible true statements within its rules.
173
