tenta

Question 1

what is Data science

Answer 1

Data science is an interdisciplinary field with focus on extracting knowledge and insights from data. Includes: Computation, statistics, understanding of domain. (Domain knowledge is the understanding of a specific industry, discipline or activity.) and scientific methods.

Findings within data science is driven from different domain areas often to drive business decisions.

Question 2

What is Exploration, Inference and Prediction in Data science?

Answer 2

Exploration
Initial phase where focus is gaining a preliminary understanding of the dataset.
Identifying patterns in information
Uses Visualizations.

Inference
Inference is data analysis and statistics to make conclusions based on observed data. **Quantifying where those patterns are reliable. **
Uses randomization

Prediction
Making informed guesses
Uses machine learning.

Question 3

Relations: what is Association?

Answer 3

“Any relation” == association
If phenomenon x has any relation to y there is an association.

Question 4

Relations: what is Causality?

Answer 4

“lead to” == Causality

If phenomenon x leads to y there is casual relationship. While correlation indicates that there is a statistical relationship, it doesn’t necessarily imply causation. Causation implies a direct influence of one variable on another. Data science is very much about looking for cause and effect.

Question 5

What are the 4 V’s of big data

Answer 5

Usually characterized by the 4 V’s:’

Volume
Size, amount of data. Exceeds the processing capacity of conventional databases. The ability to handle and process large volumes of data efficiently is a fundamental aspect of big data analytics.

Velocity
Represents the speed at which data is generated, collected and processed.

Variety
Refers to the diversity of data types and sources. Is the data structured? unstructured? semi? Design of solutions made for handling this diversity.

Veracity
Deals with the quality and reliability of the data.

Question 6

what is data mining?

Answer 6

“Data mining is the process of sorting through large data sets to identify patterns and relationships that can help solve business problems through data analysis. Data mining techniques and tools enable enterprises to predict future trends and make more-informed business decisions.”

Question 7

KDD refers to extraction of implicit previously unknown and potentially useful information from data. What are the 5 steps of KDD?

Answer 7

1. Data selection
   Selecting appropriate data from various sources.
2. Data pre-processing
   Cleaning, removing errors, removing irrelevant data.
3. Transformation
   Transformation, Transforming the data into a format usable by the data mining method. Like normalization. normalization is a technique used in DM to transform the values of a dataset into a common scale. If a dataset has multiple attributes but the attributes have values on different scales this may lead to poor data models while preforming data mining operations.

4.Data mining
The Actual application of the appropriate DM methods.

5.Interpretation and analysis.

Question 8

what does it mean if a task is predictive?

Answer 8

Predictive
Is to predict a value of an attribute (target valuable) based upon the values of other attributes. Methods of predictive tasks usually fall under the category of Supervised learning.

Question 9

what does it mean if a task is Descriptive?

Answer 9

Descriptive**
Detects patterns that summarize (describe) the underlying relationship in the data**. Methods of Descriptive tasks usually fall under the category of Unsupervised learning.

Question 10

what is unsupervised learning?

Answer 10

Unsupervised learning. Explores patterns and relationships with unlabeled data. While primarily descriptive (uncovers hidden structures) its finding can indirectly be used in predictive tasks. No explicit feedback on the correctness of predictions. Objective: The primary goal is to explore the inherent structure of the data. Unsupervised learning seeks to find patterns, groupings, or relationships within the data without relying on predefined output labels. Use cases for this could be for example clustering or dimensionality reduction.

Question 11

what is supervised learning?

Answer 11

Supervised learning. In supervised learning, the algorithm is trained on a labeled dataset, where each input is paired with the corresponding correct output or label. **The algorithm learns the mapping between inputs and outputs. **The algorithm receives feedback during training. Objective: The primary goal is to learn a mapping or relationship between inputs and outputs based on the labeled training data. The learned model can then make predictions on new, unseen data.Use Cases here could be classification and regression (predicting a continuous output)

Question 12

What is CRISP-DM

Answer 12

CRoss-Industry Standard Process for Data Mining: Open standard and can be used freely. Intended as a model for best practice. Modeled as an ongoing, iterative cycle.
The Model outlines the stages involved in a typical data mining project. CRISP-DM provides a structured framework for guiding organizations and data scientists through the data mining process. The process is iterative.

Question 13

what are the 6 steps of CRISP-DM

Answer 13

1. Business Understanding:
   Understand the business objectives and goals of the DM project. Defining the problem, understanding the requirements and scope. Produce plan.
2. Data Understanding
   Understanding of the available data. Exploring the nature of the data, relationships and potential issues. This includes initial data preprocessing. Verify data quality
3. Data Preparation
   The data preparation involves transforming the data to make it suitable to analysis. Merge data from different sources. Deal with missing data.
4. Modeling
   Various DM techniques are applied to build and train models based on the dataset.
5. Evaluation
   Assessing the performance of the models based on the business objectives. This point is closely tied to what we learned in 1. Business Understanding.
6. Deployment
   The successful models are implemented into the operational environment. Plan monitoring and maintenance. Documentation.

Question 14

what is the DIKW pyramid?

Answer 14

“Refers loosely to a class of models for representing purported structural and/or functional relationships between data, information, knowledge, and wisdom. “Typically information is defined in terms of data, knowledge in terms of information, and wisdom in terms of knowledge”.”- wiki

Question 15

what is Ordinal (qualitative) data?

Answer 15

Ordinal attributes, on the other hand, represent categories with a clear order or ranking, but the intervals between the categories may not be uniform or meaningful.

Question 16

what is Nominal (qualitative) data?

Answer 16

Nominal (qualitative)
Could be for example labels, names even when denoted by numerical values. Operations based on arithmetic are not applicable here. Can be binary aswell, TRUE / FALSE. When transforming the data labels can be freely changed ex green = 1, Blue = 2.

Question 17

Datasets. Terminologi:

Dimensionality

Answer 17

The number of attributes, Dimension reduction may occur in some processing.

Question 18

Datasets. Terminologi:

Sparsity

Answer 18

Sparsity:
Asymmetric features -> Refers to the proportion of zero or empty values in a dataset. The more values are missing sparsity increases.

Question 19

Datasets. Terminologi:

Resolution

Answer 19

Resolution:
Level of detail -> Ex higher resolution could here mean more decimal places that enable more precise calculations.

Question 20

Datasets. Terminologi:

Record Data

Answer 20

Record Data:
When each record represents a distinct unit of information.

Question 21

Datasets. Terminologi:

Web scraping

Answer 21

Web scraping: Web scraping allows us to programmatically extract data from public Web pages, provides semi-structured data usually. “Social listening” can potentially give early-warnings to events that are not yet reported in the media by scraping information of the internet that users generate.

Question 22

Datasets. Terminologi:

Data exhaust:

Answer 22

Data exhaust: Data exhaust is the trail of activity, or residual data, left behind by some
other kinds of business or computing process. ex: Transactions, calls, locations.

Question 23

Terminologi
Document term matrix:

Answer 23

A matrix (as above) that represents the frequency of terms / words in a collection of documents.

Question 24

what is
Machine learning:

Answer 24

Machine learning is about methods that can be used to improve the performance of an intelligent agent over time, based on stimuli (data) from the environment.

Question 25

what is the difference between supervised and unsupervised learning?

Answer 25

In supervised learning, the algorithm is trained on a labeled dataset, where each input has a corresponding output. The goal is to learn a mapping from inputs to outputs. In unsupervised learning, the algorithm is given unlabeled data and must find patterns or relationships within the data without explicit guidance on the output.

Question 26

what is Linear Regression (supervised learning):

Answer 26

Fitting a line to describe the relationship between variables. “The goal of linear regression is to find the best-fitting linear relationship that can be used for making predictions.”

Main idea: If classes of points can be separated by a line, you can use a linear model to classify data points.
Is best suited for problems where the goal is to predict a continuous value.

Question 27

what is Support Vector Machines (supervised learning)?

Answer 27

An algorithm used for classification tasks. SVM helps draw a line, as linear regression in a way to separate data objects from different groups. The line drawn is called a decision boundary and it is drawn to have the maximum distance between the decision boundary and the nearest data point from either group.

Question 28

what is the difference between linear regression and SVM?

Answer 28

Difference from Linear regression. Linear regression deals with a continuous output for example predicting prices. while the SVM is of a categorical label. (Is this a spam email or not?) SVM is designed to find the optimal decision boundary to separate different classes.

Question 29

what is Clustering? (Unsupervised learning)

Answer 29

groups of data points usually of similar values.

Question 30

what is K-means clustering:

Answer 30

K-means clustering is an unsupervised learning algorithm that falls under descriptive modeling.

Iteratively work towards finding the optimum cluster centers for a specified number of clusters / groups. Data points belong to a cluster that is defined by the closest centroid.

Question 31

what is DBSCAN?

Answer 31

Groups data points that are close together. Density-based spatial clustering.

Question 32

what is Hierarchical clustering?

Answer 32

Hierarchical clustering is a type of clustering algorithm used in unsupervised machine learning to group similar items into clusters. The term “hierarchical” is used because the algorithm creates a hierarchy of clusters. This clustering technique builds a tree-like structure of clusters, known as a dendrogram, which visually represents the relationships and similarities between different data points.

Question 33

what is Association rule mining (Unsupervised)?

Answer 33

DM technique that identifies interesting relationships, patterns / associations among a set of items in large datasets. For example: An association between which products are frequently purchased together?

Question 34

what is Support within association rule mining?

Answer 34

Support == A measure of how frequently a set of items appear in the dataset.

Question 35

what is Confidence in association rule mining?

Answer 35

Confidence
Confidence is that if there is a rule 𝐵𝑒𝑒𝑓,𝐶ℎ𝑖𝑐𝑘𝑒𝑛→𝐴𝑝𝑝𝑙𝑒
and has a confidence of 33%, we mean that if there is beef and chicken bought together, there is 33% chance that there are also apples in the shopping cart.

Question 36

what is lift in association rule mining?

Answer 36

Lift gives us a metric about how good a rule is. If the lift is >1 then the rule is better than guessing. If the lift is ≤1 the rule is pretty much as good as guessing.

Question 37

Is regression or classification a predictive or descriptive task?

Answer 37

Predictive

Question 38

is Clustering or rules a descriptive or a predictive task?

Answer 38

Descriptive

Question 39

What is the difference between Ordinal and Nominal attributes?

Answer 39

Nominal Attributes:

Nominal attributes are categorical variables that represent different categories or groups with no inherent order or ranking among them.

Nominal data can be represented by labels or names, and mathematical operations like addition or subtraction are not meaningful in this context.

Ordinal Attributes:

Ordinal attributes, on the other hand, represent categories with a clear order or ranking, but the intervals between the categories may not be uniform or meaningful. While there is a meaningful order among the categories, the differences between them may not be consistent.

Question 40

What are the two types of quantitative attributes?

Answer 40

Interval:
The distance between each step is the same size but with no absolute zero. Zero is arbitrary. (0 Celsius is not absence of temperature)

Ratio:
As Interval, But zero is a meaningful property indicating the absence of a property.

Question 41

What is cross-validation?

Answer 41

Cross-Validation is a technique used to assess the preformance of a predictive model. In a typical ML model the dataset is divided into traning and test set. Cross validation divides these further multiple times into subsets using each subset as a test set.

Question 42

What is the advantage of cross-validation?

Answer 42

The advantage of using cross-validation is a more reliable way of estimating a models performance. This provides a accurate assessment of how well a model will do on unseen data. “Overfitting” is when a model does good on seen data and bad on unseen.

Question 43

What is the training set?

Answer 43

The training data is used to train the model, during training the model learns patterns, relationships and attributes. When predictions are iteratively made on the training data, the difference between the prediction and the actual outcome is used to update the models parameters (optimization) Training data is typically 70-80% of the dataset.

Question 44

what is test set?

Answer 44

The test set is used to evaluate the performance of the trained model on new, unseen data. It stimulates how the model is to preform when applied to real-world scenarios.

Question 45

What is Stratification?

Answer 45

The objective is to ensure that the proportion between classes is maintained correctly in different subsets of the data. This is done to prevent biases.

For example in cross validation each subset that is created aims to have a proportional distribution of classes as the whole dataset. The same is true for when splitting the data into training and test sets

Question 46

What is re-weighing?

Answer 46

Re-Weighing adjusts the weights assigned to different instances / classes in a dataset. This is done to address class imbalances. This makes underrepresented classes more influential during learning and helps the model prioritize learning patterns from the minority class.

Question 47

What is SVR and what is it used for?

Answer 47

The idea behind Support Vector Regression is to extend the concept of SVM. To predict continuous outcomes rather than handle classification problems. The objective of SVR is the same as linear regression.

Question 48

What are some of the advantages of using SVR instead of linear regression?

Answer 48

Linear regression assumes a straight-line relationship whereas SVR is more flexible and can capture non-linear relationships.

SVR is also less sensitive to outliers, data points that deviate significantly from the overall pattern.

Question 49

What are some of the advantages of using linear regression over SVR?

Answer 49

Computationally it is less expensive. If real-time prediction is needed in a large dataset Linear regression might be preferable.

Easy to implement, interpret. Works fine if the data is not to noisy, truly linear or has few features.

Question 50

How does identifying a control group and
a treatment group then help us establish causality

Answer 50

Treatment group is a subset of individuals exposed to something that we would like to know effects that group. A new interface, a new drug etc. The goal is to observe the impact of this “treatment”.

The Control Group serves as a comparison / baseline for the treatment group. Subset of individuals that are not exposed to the “treatment”.

This makes it possible to infer causality by studying the impact of the treatment and compare the results with the control group.

Question 51

what is K-nearest neighbour?

Answer 51

An algorithm that can be used for both regression and classification.

Question 52

how does K-nearest neighbour work for classification?

Answer 52

In k-NN classification, the goal is to assign an object to a specific group or category. Imagine the object asking its nearby neighbours for advice on which group it belongs to. The “k” in k-NN represents how many neighbours it asks. The object then joins the group that the majority of its closest neighbours belong to. If it only asks one neighbour (k = 1), it simply joins the same group as that single closest neighbour.

Question 53

What do we mean by distance in a machine learning context? How can distance be measured?
Provide an example.

Answer 53

In a machine learning context, “distance” generally refers to a measure of dissimilarity or similarity between two data points in a feature space. The idea is to quantify how far apart or close together two instances are in the context of the given features.

Question 54

Advantages:
Stemming

Answer 54

1. Reduces variation. This helps group together variations of the same word. This is the main objective of stemming.
2. Simplifies analysis. It simplifies the analysis of text data by focusing on the core meaning of words.

3.Computational Efficiency: Stemming can improve computational efficiency since the reduced dimensionality makes subsequent processing faster.

Question 55

Disadvantages:
Stemming

Answer 55

1. Over-Stemming and Under-Stemming: Stemming algorithms may sometimes over-stem (remove too many letters, leading to loss of meaning) or under-stem (leave too many letters, failing to reduce related words to the same stem).
2. Loss of Interpretability: The stemmed words may not be easily interpretable, making it challenging to understand the original context.
3. Language Dependence: Stemming algorithms are language-dependent, and the effectiveness may vary across different languages.

Question 56

what is Mean and Median Imputation?

Answer 56

Mean and median imputation are methods used to replace missing values in a dataset with the mean or median of the observed values for that variable. These techniques are commonly employed in data preprocessing to handle missing data, ensuring that the dataset remains suitable for analysis.

Question 57

𝑀𝐴𝐸?

Answer 57

Mean Average error. Average difference mellan prediction och outcome

Question 58

MSE?

Answer 58

Mean Square Error
Take the difference between the actual and predicted values for each data point.
Square each of these differences.
Take the average of the squared differences.

Question 59

Calc recall

Answer 59

TP / (TP + FN)

Question 60

Calc Precision

Answer 60

TP / (TP + FP)

Question 61

F1

Answer 61

2 * (Recall * precision)
/
(Recall + precision)

Question 62

Manhattan distance.

Answer 62

Manhattan distance.
(abs(x1 - x2) + (y1 - y2))

Question 63

what are Decision Trees (DTs)

Answer 63

Decision Trees (DTs) are a predictive supervised learning method used for classification and regression. The goal is to create a model that predicts the value of a target variable by learning simple decision rules inferred from the data features.

Question 64

When is MAE and MSE used and why?

Answer 64

Mean Square Error (MSE) and Mean Absolute Error (MAE) are commonly used metrics for evaluating the performance of regression models. These metrics are useful when you’re dealing when the goal is to predict a continuous numerical value, as opposed to a classification task.

MAE is less sensitive to outliers compared to MSE because it doesn’t square the differences.

Question 65

How is KNN calculated

Answer 65

Put the distances from the new data point to the rest.
Choose the k shortest.
if [yes,no,yes,no]
Calc average distance for yes and no.
Group to the shortest alternative.

Question 66

Listwise deletion

Answer 66

Ta bort om saknas