Chp.5 Data Generation in Source systems

Question 1

Draw the data engineering lifecycle

Answer 1

Question 2

A file is a ..?

Answer 2

sequence of bytes stored on a disk

Question 3

source systems produce..?

Answer 3

data in several ways

Question 4

Get familiar with your source system and how

Answer 4

it generates data

Question 5

files may store…?

Answer 5

local parameters , events, logs, images, and audio.

Elaborate

Question 6

files are the universal medium of …?

Answer 6

data exchange

Question 7

What are the major file formats you will come across?

Answer 7

excel, csv, txt, json, xml

Question 8

What are standard ways for exchanging data between systems.

Answer 8

API’s

Question 9

typically an application database is an …?

Answer 9

online transaction processing system - OLTP

Question 10

OLTP are referred to as ….?

Answer 10

transactional databases. Why?

Question 11

OLTP db’s work well as …?

Answer 11

application backends when thousands or even millions or users might be interacting. Why ?

Question 12

What does ACID stand for?

Answer 12

atomicity, constancy, isolation, and durability

Question 13

With respect to ACID what does consistency relate to?

Answer 13

Consistency means that any database read will return the last written version of the retrieved item. Why?

Question 14

what is a atomic transaction?

Answer 14

it is a set of several changes that are committed as a unit. Why?

Question 15

in the data engineering : fundamentals data application stands for what?

Answer 15

applications that hybridize transactional and analytics workloads. Why?

Question 16

What does CDC stand for?

Answer 16

Change Data Capture

Question 17

What is CDC ?

Answer 17

it is method for extracting each change event. (insert, update, delete). Why?

Question 18

CDC is often used to … ?

Answer 18

replicate between databases in near real time or create an event stream for downstream processing. Why?

Question 19

What does OLAP stand for?

Answer 19

online analytical processing system. Why ?

Question 20

What is the difference between an OLAP and OLTP?

Answer 20

OLAP is for doing large scale analytics and OLTP is for doing large scale reads and write of individual records. Why?

Question 21

Typically OLAP are …?

Answer 21

inefficient in handing look ups of individual records. Why?

Question 22

A log captures ….?

Answer 22

information about events that occur in systems.

Question 23

A log captures ….?

Answer 23

information about events that occur in systems. Why?

Question 24

Logs are a …?

Answer 24

rich data source, potentially valuable for downstream data analysis.

Question 25

What are three common ways logs are encoded?

Answer 25

1. Binary-encoded logs
2. Semistructured logs
3. Plain-text (unstructured ) logs

Question 26

Relational DB’s often store…?

Answer 26

event log stored directly on the database server that can be processed to create a stream.

Question 27

All logs track …>?

Answer 27

events and metadata

Question 28

What is log resolution?

Answer 28

it referred to the amount of event data stored/captured in the log.

Question 29

At a minimum a log should capture…?

Answer 29

who , what, and when.

Question 30

describe binary-encoded logs:

Answer 30

These logs encode data in a custom compact format for space efficiency and fast I/O. Why?

Question 31

Tables are typically indexed by a ..?

Answer 31

primary key

Question 32

what is a primary key?

Answer 32

a unique field for each row of the table

Question 33

What does RDBMS stand for?

Answer 33

relational database management system

Question 34

what is the most common db for application backends?

Answer 34

relational database management system