Understanding the concept of data inspection

Question 1

Data Inspection

Answer 1

Data Inspection is the act of viewing data for verification and debugging purposes, before, during, or after a translation.

Question 2

Data cleansing:

Answer 2

When cleansing a dataset, data-scientists seek to:

Remove null or invalid results
Standardise data within a single relevant, usable format
Unify disparate data sources in a consistent format
Maintain the integrity of the source dataset
As simple as this process might seem, actual data sets are often much larger and typically contain a large variety of disparate values requiring greater scrutiny.

Question 3

Data Extraction:

Answer 3

Data extraction is the process of obtaining data from a database or SaaS platform so
that it can be replicated to a destination — such as a data warehouse designed to support online analytical processing (OLAP).

Question 4

Types of data extraction:

Answer 4

Extraction jobs may be scheduled, or analysts may extract data on demand as dictated by business needs and analysis goals. Data can be extracted in three primary ways:
Update notification
Incremental extraction
Full extraction

Question 5

Data extraction process:

Answer 5

1. Check for changes to the structure of the data, including the addition of new tables and columns. Changed data structures have to be dealt with programmatically.
2. Retrieve the target tables and fields from the records specified by the integration’s replication scheme.
3. Extract the appropriate data, if any.

Question 6

Data loading:

Answer 6

Data loading is the process of copying and loading data or data sets from a source file, folder or application to a database or similar application. It is usually implemented by copying digital data from a source and pasting or loading the data to a data storage or processing utility.

Question 7

Benefits of data loading:

Answer 7

Today, the ETL process — including data loading — is designed for speed, efficiency, and flexibility. But more importantly, it can scale to meet the growing data demands of most enterprises. ETL easily accommodates proliferation of data sources as technologies like IoT and connected devices continue to gain popularity. And it can handle any number of data types and formats, whether structured, semi- structured, or unstructured.

Question 8

Challenges of data loading:

Answer 8

Slow down analysis
Increase the likelihood of errors
Require specialized knowledge
Require costly equipment