Data Engineering

Question 1

Deploy Data Lake (s)

Answer 1

Distributed Data Objects

Use S3 in AWS

Question 2

Data Vault (s)

Answer 2

Schema on Read to Schema on Write
Constitutes - Hub, Link, Satellites
Hubs - Business keys, Links link Business keys as link tables
Satellites store the actual data

Question 3

Spark (s)

Answer 3

Cluster computing framework
Spark core, Spark SQL, Spark streaming, MLlib, GraphX
Spark Core - Scheduling & Dispatching, Cluster Computing, Distributed processing
Spark SQL - abstracts data as Dataframe

Question 4

Mesos

Answer 4

Cluster Controller

Resource sharing at fine grained levels across cluster

Question 5

Akka

Answer 5

Actor based Message driven Runtime

Question 6

Cassandra

Answer 6

Distributed database

Question 7

Kafka

Answer 7

Messaging & Streaming platform
Confluent.io is the provider
Kafka Core, Kafka Streams, Kafka Connect

Question 8

CRISP - DM

HORUS

Answer 8

Data Lake Data Science Standard

A Standard that reduce the number of converters required for source data. Vouches a common data format.

Question 9

Layered Approach (s)

Answer 9

Business, Utility, Operations Management, Audit/Balance/Control, Functional Layer,

Question 10

Functional Layer

Answer 10

Sun Model, Conformed Dimension, Availability, Capacity, Extendability, Interoperability,

Question 11

Data Democratization (s)

Answer 11

Treat all data equally relevant and source into a single platform to cut thru barriers of architecture, framework and models.

Question 12

Argument against data democratization

Answer 12

Soon the lake will become swamp.

But this can be controlled with better governance.

Question 13

In 3 words, what makes Unstructured data complex ?

Answer 13

Searching, Fetching, Consolidating

Question 14

2 Main pieces of Data Ingestion

Answer 14

Data Collection and Data Integration

Question 15

File Formats used in Data Lake

Answer 15

Parquet, ORC
Both are columnar file store, supports predicate pushdown, stripe indexes. When Sorted and Inserted, gives better read performance, but adds overhead during write.

Question 16

Data Ingestion from Relational Source

Answer 16

Almost same as Extraction for DWH
Use Predicate filtering.
Capture Changed records thru a flag.
Perform Incremental extraction with timestamp or id field.
Perform full extraction if source volume is low.

Question 17

Sqoop

Answer 17

SQL to Hadoop Bidirectional
Uses Mapper and Reducer jobs in HDFS
--Split-by Column splits across mappers
--autoreset-to-one-mapper
Can use Character based --split-by

Question 18

Ingestion utilities by database vendors

Answer 18

Compared to Native utility, Ingestion Utility from db vendors capitalize on their database architecture.
Oracle provides CopyToBDA , Greenplum gphdfs,gpfdist
Edge Node , Writeable external table

Question 19

Ingesting Unstructured Data

Flume

Answer 19

Collects and Transfers Streaming data as events.

Question 20

Flume Components

Answer 20

Flume Event - The data to be transferred.
Source - PUTS data events from source system into Channel
Channel - Collects and persists events (temp storage)
Sink - TAKES (pulls) data from channel and pushes to HDFS.

Question 21

Flume Design Considerations.

Answer 21

Choice of Channel is Memory, File, Kafka Topic

Question 22

Kafka

Answer 22

A distributed processing, data Streaming tool
Topics, Producers, Consumers.
Provides APIs to Produce, Consume and process streams.