Module 3 - Preparing and Cleaning Data for Analysis

Question 1

Selecting relevant data for your analysis includes determining what?

Answer 1

determining the type(s) of data that you need and finding a source for the data.

Question 2

Explain this “When selecting data for a project, it is important to focus on finding data that may provide insights into your original business question.”

Answer 2

For example, if you are seeking to understand demographic characteristics of people who bought Product X in the past year, you should only be using data that is directly related to Product X.

Question 3

What to do if sometimes the data you need to answer your questions isn’t readily available?

Answer 3

It may be necessary to establish new procedures to collect the data required for your analysis. Other times, it may involve combining data from multiple sources into a format that can be analyzed.

Question 4

Explain on what can be done on this situation: “ an entertainment producer gathering data about the viability of a movie project.”

Answer 4

If the movie is an adaptation of a book, they need data on the sales of books by that author, within that genre, and across a variety of population demographics. They might compare the profitability of other movies with similar plots or characters, and their release dates, to determine the best time of the year to release a picture of that genre. Producers may also analyze data on the actors and locations that appear in the most successful recent movies to make casting and production decisions.

Question 5

Enumerate the questions that you should ask yourself when selecting a data source.

Answer 5

Some questions that you should ask yourself when selecting a data source:
a. What data points are necessary to inform your analysis?
b. Do I already have access to this data, or must I find a dataset from another source?
c. Where are reliable and verifiable sources of this data?
d. How often is the relevant data collected and updated?
e. How the data is licensed for use, and is there a cost?
f. Is the data in a format that I can use, or convert to use, with my tools?

Question 6

What are the two types of data that analysts work with?

Answer 6

static data and streaming data.

Question 7

Data that is received and stored prior to performing analysis on the data is considered ?

Answer 7

static data

Question 8

When each event is processed and analyzed as it is received and subsequent results are used or stored, the data is referred to as?

Answer 8

streaming data.

Question 9

is a data type used to represent text or sequences of characters. It can include letters, numbers, symbols, and spaces. Strings are commonly used to store names, addresses, sentences, and any other textual information. In programming, strings are typically enclosed within single (‘’) or double (“”) quotation marks.

Answer 9

String

Question 10

is a whole number without any decimal or fractional parts. It represents a count or quantity that can be positive, negative, or zero. Integers are used to store values like counts of items, ages, and identifiers. They are typically represented without a decimal point.

Answer 10

Integer

Question 11

is a data type used to represent numbers with decimal points. Floats can represent a wide range of values, including both integers and fractions. They are used for calculations involving precision, such as scientific calculations, measurements, and financial computations.

Answer 11

A floating-point number, often referred to as a “float,”

Question 12

This data type is used to represent specific points in time. It can include information about the year, month, day, hour, minute, second, and sometimes even milliseconds. This data type is crucial for recording events, scheduling, and performing temporal calculations. Formats for representing date and time values can vary depending on the programming language and system.

Answer 12

The date and time datatype

Question 13

This data type represents binary values: either true or false, yes or no, on or off. Booleans are used to make logical comparisons and decisions in programming. They are often used in conditional statements and expressions to control the flow of a program. Booleans help determine the validity of conditions or statements.

Answer 13

The boolean data type

Question 14

refers to data that is entered and maintained in defined fields within a file or record.

Answer 14

Structured data

Question 15

is easily entered, classified, queried, and analyzed by a computer.

Answer 15

Structured data

Question 16

Is data found in relational databases and spreadsheets structured?

Answer 16

Yes, for example, when you submit your name, address, and billing information to a website, you are creating structured data. The database may force you to enter it in a certain format for a computer to interpret it easily.

Question 17

What are the characteristics of structured data?

Answer 17

• There is a well-defined and organized structure.
• It can be stored in tables, usually within vertical columns and horizontal rows.
• The content and format of the data is documented.
• It is organized into files, records, and fields.
• It can be searched, sorted, and queried.
• Input controls can reduce the possibility of invalid data.

Question 18

is a lightweight data-interchange format that is easy for humans to read and write.

Answer 18

JSON

Question 19

is a markup language that is similar to HTML.

Answer 19

XML

Question 20

There are many different types of structured data files that can either be created by humans or machine-generated. Give some few examples

Answer 20

• Relational Databases
• Logs
• Spreadsheets
• Sensor readings
• Transactional Records

Question 21

This data is raw data, not organized in a predefined way. It does not possess a fixed schema that identifies the type of data or its format. This type of data lacks a set way of entering or grouping the data, and then analyzing the data.

Answer 21

Unstructured data

Question 22

Give some examples of unstructured data

Answer 22

Examples of unstructured data include the content of photos, audio, video, web pages, blogs, books, journals, white papers, PowerPoint presentations, articles, email, wikis, word processing documents, and text in general. Even the PDF version of this chapter is unstructured. The text is searchable, but it is not organized in a predefined format. Unstructured data can even be a traffic camera feed that is continuously sending images for processing.

Question 23

is an automated process which uses a bot or web crawler to gather and copy specific data from the web to a database or spreadsheet. The data can then be easily analyzed.

Answer 23

Web scraping

Question 24

These are the two versions for moving or processing data through a pipeline

Answer 24

ETL and ELT

Question 25

Explain Extract, Transform and Load (ETL)

Answer 25

Extract, Transform and Load (ETL) is a process for collecting data from this variety of sources, transforming the data, and then loading the data into a database. One company’s data might be found in Word documents, spreadsheets, plain text, PowerPoints, emails and PDF files. Another company’s data may be housed in relational databases. This data can be stored in a variety of different formats, making it difficult to combine and analyze, so the transformation happens before loading.

Question 26

Explain Extract, Load, Transform (ELT) process

Answer 26

In an Extract, Load, Transform (ELT) process, the load and transform steps are reversed. ELT enables raw data to skip the transformation step and go straight to storage in an unstructured form. Transformation then occurs on the stored data as it is used. The ELT process is used primarily for large amounts of unstructured data.

Question 27

Explain the Step 1: Extract

Answer 27

In this step, data is located and gathered from various sources in order to be converted into a single format for analysis. The data may be extracted from a relational database, NoSQL, flat files, XML files, or other formats.

Question 28

Explain Step 2: Transform

Answer 28

Data usually must be transformed before it can be loaded into a data warehouse for analysis. The transform step uses rules to transform the source data to the type of data needed for the target database. This includes converting any measured data to the same dimension (e.g., Imperial to Metric). The transformation step also requires several additional tasks. Some of these tasks are joining data from several sources, aggregating, sorting, determining new values that are calculated from aggregated data, and then applying validation rules.
Data (possibly including some empty or error data) may go through another part of the transform step known as ‘cleaning’ or ‘scrubbing’ data, and validation lets you know whether the data needs cleaning. Some examples of data cleaning are removing blank records and standardizing formats such as date, time, and location. The cleaning part of the transform step further ensures the consistency of the source data.