Big Data Projects

Question 1

Identify asteps in a data analysis project.

Answer 1

Conceptualization of the modeling task.
Data Collecor
Data preparation and wrangling.
Data exploration.
Model training.

Question 2

Conceptualization of the modeling task.

Answer 2

define the problem at hand

Question 3

Data preparation and wrangling

Answer 3

cleaning the data set and preparing it for the model

Question 4

Data exploration.

Answer 4

feature selection and engineering and initial data analysis

Question 5

Model training

Answer 5

determining the appropriate ML algorithm to use, evaluating the algorithm using a training data set, and tuning the model.

Question 6

steps of preparing and wrangling data.

Answer 6

critical step involves cleansing and organizing raw data for use in a model

Question 7

Data cleansing

Answer 7

deals with reducing errors in the raw data

Question 8

Data wrangling

Answer 8

involves preprocessing data for model use

Question 9

Preprocessing includes

Answer 9

data transformation and scaling.

Question 10

scaling

Answer 10

Conversion of data features to a common unit of measurement

Question 11

Two common methods of scaling

Answer 11

normalization and standardization.

Question 12

Normalization scales

Answer 12

normalized X=(X−Xmin)/(Xmax−Xmin)

Question 13

Standardization Scales

Answer 13

standardized Xi=Xi−μσ

Question 14

Text processing

Answer 14

cleansing and preprocessing of text-based data.

Question 15

Text cleansing involves the following steps:

Answer 15

Remove HTML tags.
Remove punctuations.
Remove numbers.
Remove white spaces.

Question 16

Cleansed text is then normalized using the following steps:

Answer 16

Lowercasing.
Removal of stop words
Stemming.
Lemmatization.

Question 17

Stemming

Answer 17

converts all variations of a word into a common value

Question 18

Lemmatization

Answer 18

conversion of inflected forms of a word into its lemma

Question 19

tokenization

Answer 19

is the process of splitting a sentence into tokens

Question 20

Data exploration

Answer 20

seeks to evaluate the data set and determine the most appropriate way to configure it for model training

Question 21

Steps in data exploration include the following:

Answer 21

Exploratory data analysis (EDA)
Feature selection.
Feature engineering

Question 22

Exploratory data analysis (EDA)

Answer 22

involves looking at data descriptors

Question 23

Feature selection

Answer 23

is a process to select only the needed attributes of the data for ML model training.

Question 24

Feature engineering

Answer 24

is the process of creating new features by transforming

Question 25

Data Exploration for Structured Data

Answer 25

With EDA, structured data is organized in rows (observations) and columns (features). With feature selection, we try to select only the features that contribute to the out-of-sample predictive power of the model. Feature Engineering (FE) involves optimizing and improving the selected features.

Question 26

Model fitting errors can be caused by:

Answer 26

Size of the training sample. Number of features.

Question 27

The three tasks of model training are as follows:

Answer 27

1- Method selection 2-Performance evaluation 3- Tuning

Question 28

Techniques to Measure Model Performance

Answer 28

1 - Error analysis. 2 - Receiver operating characteristic (ROC). 3 - Root mean square error (RMSE).

Question 29

Error analysis.

Answer 29

Errors in classification problems can be false positives (type I error) or false negatives (type II error).

Question 30

precision (P)

Answer 30

= TP / (TP + FP)

Question 31

recall (R)

Answer 31

= TP / (TP + FN)

Question 32

accuracy

Answer 32

= (TP + TN) / (TP + TN + FP + FN)

Question 33

F1 score =

Answer 33

= (2 × P × R) / (P + R)

Question 34

Receiver operating characteristic (ROC)

Answer 34

Also used for classification problems, the ROC is a curve that plots the tradeoff between FPs and TPs.

Question 35

TPR

Answer 35

= TP / (TP + FN)

Question 36

FPR

Answer 36

= FP / (FP + TN)

Question 37

Root mean square error (RMSE)

Answer 37

RMSE=⎷(predicted−actual)2/n

Question 38

Parameters

Answer 38

are estimated by the model (e.g., slope coefficients in a regression model) using an optimization technique on the training sample.

Question 39

Hyperparameters

Answer 39

are specified by ML engineers, and are not dependent on the training sample.

Question 40

Ceiling analysis

Answer 40

Ceiling analysis is an evaluation and tuning of each of the components in the entire model-building pipeline.