Data Science for Dummies

Question 1

What are the 3 types of data and examples of each?

Answer 1

Structured - stored, processed, and manipulated in a traditional relational database management system (RDBMS), i.e. a MySQL database that uses tabular data

Unstructured - data generated from human activities that doesn’t fit into a structured database format, i.e. emails, Word docs, AV files

Semistructured - data that doesn’t fit into a structured database system but is organizable by tags that are useful for creating a form of order/hierarchy. Examples include XML (a file used to store data in the form of hierarchical elements) and JSON files ( JSON file is a file that stores simple data structures and objects in JavaScript Object Notation (JSON) format, which is a standard data interchange format. It is primarily used for transmitting data between a web application and a server. JSON files are lightweight, text-based, human-readable, and can be edited using a text editor.)

Question 2

What is big data?

Answer 2

Data that exceeds the processing capacity of conventional database systems because it’s too big or lacks the structural requirements of a traditional database architecture

Question 3

What does it mean to query data?

Answer 3

Write commands to extract relevant datasets from data storage systems (usually you do this with SQL, structured query language)

Question 4

What is Hadoop?

Answer 4

A platform for batch-processing and storing large volumes of data. It was designed to boil down big data into smaller datasets that are more manageable for data scientists to analyze. It’s been in decline in popularity since 2015

Question 5

What are the 3 characteristics that define big data?

Answer 5

Volume, velocity, and variety. Because these 3 Vs are ever expanding, newer, more innovative data technologies must be developed to manage big data problems

Question 6

What’s the size of big data volume?

Answer 6

As low as 1 terabyte and has no upper limit. If your org owns at least 1 terabyte of data, the data technically qualifies as Big Data

Question 7

What does it mean that most big data is “low value?”

Answer 7

Big data is composed of huge number of very small transactions in a lot of formats that only have value once they’re aggregated (data engineers) and analyzed (data scientists)

Question 8

Why is data velocity important?

Answer 8

A lot of big data is created using automated processes, and a lot of it is low value. You need systems that are able to ingest a lot of it quickly and generate timely and valuable insights

Question 9

What is data velocity?

Answer 9

Data velocity is data volume per unit in time. Big data enters an average system at velocities between 30 kilobytes/second to 30 gigabytes per second

Question 10

What does latency refer to?

Answer 10

Related to data velocity. A system’s delay in moving data after it’s been instructed to do so (every data system has this). Many data engineered systems are required to have latency <100 milliseconds from data creation to the time the system responds

Question 11

What is throughput?

Answer 11

Related to data velocity. A characteristic describing a system’s capacity for work per unit of time. The capabilities of data handling and processing technologies limit data velocities

Question 12

What are some of the tools (3) that intake data into a system (data ingestion)?

Answer 12

Apache scoop - quickly transfer data back and forth between a relational data system and the Hadoop distributed file system (HDFS)

Apache Kafka
Apache Flume

Question 13

What does data variety refer to?

Answer 13

High variety data comes from a multitude of sources with different underlying structures (structured, unstructured, or semistructured)

Question 14

What is a data lake?

Answer 14

A nonhierarchical data storage system used to hold huge volumes of multistructured raw data within a flat storage architecture. In other words, a collection of records that come in uniform format and are not cross-referenced in any way. HDFS can be used as a data lage storage reposity as can AWS S3 platform (one of the more popular cloud architectures for storing big data)

Question 15

What is a data warehouse?

Answer 15

A data warehouse is a centralized data repository that you can use to store and access only structured data. A more traditional warehouse system is a data mart, a storage system for structured data that you can use to store one particular focus area of data belonging to one line of business in the company

Question 16

What is machine learning?

Answer 16

The practice of applying algorithms to learn from and make automated predictions from data

Question 17

What is a machine learning engineer?

Answer 17

Hybrid between a software engineer and a data scientist (NOT data engineer). A software engineer who is skilled enough in data science to deploy advanced data science models within the applications they build, bringing ML models into production in a live environment like a SaaS product or a webpage

Question 18

What is a data engineer?

Answer 18

Build and maintain data systems for overcoming data processing bottlenecks and data handling problems that arise from handling the high volume, velocity, and variety of big data. They use SWE to design systems for and solve problems with handling and manipulating big data sets. They often have experience working with and designing real time processing frameworks and massively parallel processing platforms as well as RDBMSs.

Question 19

What programs to data engineers code in?

Answer 19

Java, C++, Scala, or Python. THey also know how to deploy Hadoop MapReduce or Spark to handle, process, and refine big datasets into more manageable sizes

Question 20

What is the purpose of data engineering?

Answer 20

Engineer large scale data solutions by building coherent, modular, scalable data processing platforms that data scientists can use to derive insights

Question 21

True or false: Data engineering involves engineering a built system

Answer 21

False. It involves the designing, building, and implementing of software solutions to problems in the data world.

Question 22

What’s the difference between a data engineer, ML engineer, and data scientist?

Answer 22

The data engineer will store, migrate, and process your data, data scientist will make sense of the data, and ML engineer will bring ML models into production

Question 23

What are the big cloud data storage services?

Answer 23

AWS, Google cloud, Microsoft azure

Question 24

Why is cloud data storage more beneficial than on premise data storage?

Answer 24

-cloud service providers take care of work to configure and maintain computing resources which makes it easier to use the data
-more flexibility - can turn off cloud service for what’s no longer needed vs. having servers on premise you’re not using
-more secure

Question 25

What is serverless computing?

Answer 25

Computing done in the cloud vs. on a desktop or on premise. A physical server exists but is supported by the cloud computing company you retain

Question 26

What is FaaS and some examples?

Answer 26

Function as a service. It’s a containerized cloud computing service that makes it easier to execute code in a cloud environment without needing to set up code infrastructure (data science model runs directly in the container). Examples: AWS Lambda, Google cloud functions, azure functions

Question 27

What is kubernetes?

Answer 27

Open source software suite that manages and coordinates deployment, scaling, and management of containerized applications across clusters of worker nodes. It helps software developers build and scale apps quickly but needs data engineering expertise to set up quickly

Question 28

What does it mean for a system to be fault tolerant?

Answer 28

Built to continue successful operations even if one of sub components fails. Has redundancy in computing nodes.

Question 29

What does it mean for a system to be extensible?

Answer 29

It can be extended or shrunk in size without disrupting its operations

Question 30

What is parallel processing?

Answer 30

Data is processed quickly bc the work required to process the data is distributed across multiple nodes in a system. This configuration allows for simultaneous processing is tasks across multiple nodes

Question 31

Name 3 cloud warehouse solutions.

Answer 31

Amazing Redshift - big data warehousing service running on data sitting in the cloud

Snowflake - SaaS solution providing parallel processing for structured and semi structured data stored in the cloud on snowflakes servers

Google BigQuery

Question 32

True of false: an RDBMS can handle big data

Answer 32

False; rdbms only for tabular data that can be queried by SQL

Question 33

What is NoSQL used for?

Answer 33

Better for big data because it can handle structured, unstructured, and semistructured data.

Question 34

What is a key value pair?

Answer 34

Pair of data items—key is the record identifier and the value is the data identified by the respective key

Question 35

What are the 4 categories of nonrelational databases NoSQL offers?

Answer 35

Graph databases
Document databases (aka document store)
Key-value stores
Column family stores

Question 36

What is a real time processing framework?

Answer 36

A framework that processes data in real time as the data streams and flows into the system. The frameworks process data in micro batches and return very quick results.

Question 37

What is in-memory?

Answer 37

In-memory refers to processing data within the computer’s memory without reading and writing its computational results to a disk; this is much faster but cannot process much data per processing interval

Question 38

What is machine learning?

Answer 38

The practice of applying algorithmic models to data repeatedly so that your computer discovers hidden patterns or trends you can use to make predictions (aka algorithmic learning)

Question 39

What are the 3 steps in machine learning

Answer 39

Setup - acquiring data, preprocessing it, selecting the variable for the task at hand (feature selection), and breaking the data into training/test data sets
Learning - involves model experimentation, training, building, and testing. Use the training data to train the model and the test data to test the accuracy of the model’s predictions
Application - model deployment and prediction

Question 40

How should you break data into test and training sets?

Answer 40

Apply random sampling to 2/3 of the original data set in order to train the model and use the remaining 1/3 for evaluating the model’s predictions.

Question 41

What is a random sample?

Answer 41

A random sample contains observations that all have an equal probability of being selected in the original data set

Question 42

In ML, what is an instance?

Answer 42

The same as a row in a data table, an observation in stats, and a data point. AKA a case.

Question 43

In ML, what is a feature?

Answer 43

The same as a column or field in a data table, a variable in stats. In regression models, a feature is also the independent variable

Question 44

In ML, what is a target variable?

Answer 44

The same as a dependent variable in stats

Question 45

In ML, what is feature selection and how is it different from feature engineering?

Answer 45

Feature selection is the process of selecting appropriate variables, whereas feature engineering is where you design input variables from an underlying data set (used when your model needs a better representation of the problem being solved than is available in the raw data set)

Question 46

What are supervised learning algos? What are their use cases?

Answer 46

Supervised learning algos require that input data has labeled features. These algos learn from known features of data to produce an output model that successfully predicts labels for new, incoming, unlabeled data points, i.e. when you have historical values that are good predictors of future events (like survival analysis (aka event history analysis, desigend to predict the time of an event like a mother’s age at first childbirth, fraud detection, etc.).

You use logistic regression in a supervised learning algorithm.

Question 47

What are unsupervised algos?

Answer 47

Unsupervised learning algos accept unlabeled data and attempt to group observations into categories based on underlying similarities in input features. Examples of this include principal component analysis, k-means clustering, and singular value decomposition. Use cases include recommendation engines, facial recognition, and customer segmentation

Question 48

What is reinforcement learning in ML?

Answer 48

A behavior-based learning model. Given “rewards” based on how it behaves and it learns to behave in a way that maximize rewards

Question 49

What are descriptive statistics?

Answer 49

Provide a description that illuminates some characteristic of a numerical data set (i.e. distribution, central tendency, dispersion, etc.)

Question 50

What’s the difference between standard deviation and variance?

Answer 50

Standard deviation is the spread of a group of numbers from the mean. The variance measures the average degree to which each point differs from the mean. While standard deviation is the square root of the variance, variance is the average of all data points within a group.

Question 51

True or false: Descriptive statistics posit the X causes Y.

Answer 51

False; they only highlight relationships between x and y

Question 52

What are inferential statistics?

Answer 52

They carve out a small section of the dataset and attempt to deduce significant information about the larger datasets. Inferential stats, such as regression analysis, DO try to predict by studying causation.

Question 53

What are the 3 types of distributions?

Answer 53

Normal distribution (numeric continuous) - bell curve

Binomial distribution (numeric discrete) - model # of successes that can occur in a certain number of attempts when only two outcomes are possible. Binary variables (only two outcomes possible) have a binomial distribution

Categorical distributions (non-numeric): non-numerical categorical variables or ordinal variables (ordered categorical variable, i.e. airline classes)

Question 54

What is Naive Bayes and what is it used for? What are the 3 types?

Answer 54

It’s an ML method used to predict the likelihood that an event will occur given evidence defined in your data features (conditional probability). It is based on classification and regression and us useful to classify text data (i.e. model predicts whether an email is spam (the event) based on features gathered from content in a repository (evidence).)

NB is good for classifying text data

3 types:
-MultinomialNB, BernoulliNB, GaussianNB

Question 55

What’s the difference between a binomial and multinomial distribution?

Answer 55

A multinomial distribution can produce 2+ outcomes and binomial can produce only 2

Question 56

TRUE OR FALSE: Many ML methods assume features are independent

Answer 56

True; you need to test if they’re independent by evaluating their correlation. Correlations with an r value close to 0 could indicate variables are indepednent.

Question 57

What is an a priori assumption?

Answer 57

Predictions are based on an assumption that past conditions still hold true

Question 58

What is the Pearson correlation and what assumptions is it based on?

Answer 58

A method for measuring the linear relationship/determining if there is a relationship between two continuous variables

In order to use the Pearson correlation, you must have:
1. Data is normally distributed
2. Have continuous, numeric variables
3. Variables are linearly related

Question 59

What is Spearman’s rank correlation?

Answer 59

A popular test for determining correlation between ordinal variables.

It assumes:
1. Your variables are ordinal
2. Your variables are related nonlinearly (tell by looking at the graph)
3. Data are nonnormally distributed

Imagine you have two sets of data, like the heights of students and their test scores. Spearman’s rank correlation helps us figure out if there’s a relationship between these two sets of data, but it doesn’t assume that the relationship is a straight line.

Here’s how it works:

1. Ranking:
   
   • First, we rank the values in each set. Ranking is like giving each value a position from smallest to largest. If two students have the same height or test score, they get the same rank.
2. Differences between Ranks:
   
   • Then, we look at the differences in ranks for each pair of values. For example, if one student is ranked 3rd in height and 5th in test score, the difference is 2 (5 - 3).
3. Squaring the Differences:
   
   • We square these differences. Squaring just means multiplying a number by itself. So, if the difference was 2, we square it to get 4 (2 * 2).
4. Adding Up the Squares:
   
   • We add up all these squared differences.
5. Calculating the Correlation:
   
   • Finally, we use a formula to calculate a number called Spearman’s rank correlation coefficient (let’s call it ρ). This number tells us how related the two sets of data are.
   • If ρ is close to 1, it means there’s a strong relationship: as one set goes up, the other tends to go up too.
   • If ρ is close to -1, it means there’s a strong relationship, but as one set goes up, the other tends to go down.
   • If ρ is close to 0, it means there’s not much of a relationship.

So, Spearman’s rank correlation helps us understand if there’s a connection between two sets of data, without assuming that the connection is a perfect straight line. It’s useful when the data doesn’t follow a normal pattern or when we’re working with ranks or orderings instead of actual numbers.

Question 60

What does it mean to reduce a dataset’s dimensionality?

Answer 60

Reduce a dataset’s feature count without losing the important information the datasets contains by compressing its features’ information into synthetic variables you can subsequently utilize to make predictions or as input into another ML model

Question 61

What is SVD?

Answer 61

Singular value decomposition.
Allows you to reduce the dimensionality of your dataset (the number of features that you track when carrying out an analysis). It allows you to compress dataset and remove redundant information and noise.

SVD is applied to analyze principal components from large, noise, sparse datasets, an ML approach called Principal Component Analysis (PCA).

Question 62

What’s the difference between PCA and SVD?

Answer 62

PCA = principal component analysis
SVD = singular value composition

PCA assumes you’re working with a 1x1 square input matric; if input matrix is not a square, use SVD

Question 63

What does CVE stand for?

Answer 63

Cumulative variance explained. The lower the CVE, the more you should take your model’s results with a grain of salt. (don’t worry about going into the weeds on this just know roughly what it’s related to)

Question 64

What are the different ways you can reduce dimensionality of your dataset?

Answer 64

SVD = singular value decomposition
Factor analysis
PCA = Principal component analysis

Question 65

What are latent variables?

Answer 65

Meaningful inferred variables that underlie a dataset but are not directly observable

Question 66

What is factor analysis?

Answer 66

The process of fitting a model to prepare a dataset for analysis by reducing its dimensionality and information redundancy. It compresses dataset’s information into a reduced set of non-information-redundant latent variables

Question 67

What is PCA?

Answer 67

Principal component analysis is related to SVD. It’s an unsupervised method that finds relationships between features in your data set then transforms and reduces them to a set of non-information-redundant principal components - uncorrelated features that explain the information that’s contained within the dataset. It’s a refined representation of the dataset without redundancy, noise, or outliers.

Question 68

What does MCDM stand for?

Answer 68

Multiple criteria decision making.

Question 69

What is linear regression?

Answer 69

A ML method you can use to describe and quantify the relationship between your target variable, y (aka the predictant) and the dataset features you’ve chosen to use as predictor variables (aka dataset X in ML).

Question 70

What are the limitations of linear regression?

Answer 70

1. Works only with numerical variables (not categorical)
2. If dataset is missing values, you will have problems
3. Outliers
4. Assumes a linear relationship exists
5. Assumes features are independent
6. Prediction errors (residuals) should be normally distributed

Needs at least 20 observations per predictive feature for reliable results

Question 71

What is logistic regression?

Answer 71

An ML method you can use to estimate values for a categorical target variable based on your selected features. Your target variable should be numeric and contain the target variable’s class/category. In addition to predicting the class of observation of your target variable, it indicates the probability for each of its estimates.

Question 72

What are the requirements for logistic regression?

Answer 72

1. Does not need to be a linear relationship between features and target variable
2. Residuals do not have to be normally disttributed
3. Predictive features aren’t required to have a normal distribution

Question 73

What are 4 limitations of logistic regression?

Answer 73

1. Missing values should be treated or removed
2. Your target variables must be binary or ordinal (1 for yes and 0 for no)
3. Predictive features should be independent of each other

Needs at least 50 observations per predictive feature to generate reliable results

Question 74

What is a least squares regression line?

Answer 74

A method that fits a linear regression line to a dataset. You square the vertical distance values that describe the distances between the data points and the best fit line, add up those squared distances, and adjust the placement of the line so the summed squared distance value is minimized.

Question 75

What is time series?

Answer 75

A collection of data on attribute values over time. They are used to predict future instances of the measure based on past observational data.

Question 76

What are the 5 different patterns in time series?

Answer 76

Constant time series
Trended time series
Untrended seasonal time series
Trended seasonal time series
Nonstationary processes- unpredictable behavior not related to seasonality resulting from economic or industry-wide conditions instead. These can’t be forecasted

Question 77

What’s the difference between multivariate and univariate analysis?

Answer 77

Multivariate analysis is the analysis of relationship between multiple variables; univariate analysis is the quantitative analysis of only one variable at a time.

Question 78

What is ARMA?

Answer 78

Autoregressive moving average is a class of forecasting methods you can use to predict future values from current and historical data. It combines autoregression techniques (analyses that assume the previous observations are good predictions of future values) and moving average techniques (are like a smoothing tool. They help you see the forest (long-term trends) without being distracted by the trees (short-term fluctuations), making it easier to interpret and analyze time-varying data.)

Question 80

What is regression analysis used for?

Answer 80

It allows us to quantify the relationship between a particular variable and an outcome we care about while controlling for other factors