Data Pipelines on cloud Streaming

Question 1

Log

Answer 1

Append-only data structure, applications ignore details of source

Question 2

Unifed log

Answer 2

collect events from many source systems, enable applications to operate on these event streams as they wish

events automatically deleted after certain time, read once

Question 3

Distributed Unified Log

Answer 3

Log lives across a cluster of machines

Good for scalability and durability (replication)

Question 4

Ordered events

Answer 4

Events in a shard (partition of unified log) have sequential IDs unique to their shard, meaning the ordering is local

Question 5

Single-Event processing

Answer 5

Single event produces zero or more events

Question 6

Multiple-event processing

Answer 6

multiple events collectively produce zero or more events.

Aggregate events, pattern match, reorder

Question 7

Amazon Kinesis Data Streams

Answer 7

real-time streaming service that allows the ingestion and processing of large data streams. Composed of shards and can be scaled by splitting or merging shards

Question 8

Shard

Answer 8

Unit of capacity in Kinesis data streams.

Each shard provides 1MBps of data input

Question 9

Partition Key

Answer 9

User-defined key that determines how records are distributed across shards in Kinesis stream (load balancing support)

Question 10

Re-sharding

Answer 10

Process of splitting or merging shards to scale a data stream either up or down

Question 11

Data Blob

Answer 11

The actual data being streamed through Kinesis

Question 12

Sequence Number

Answer 12

Number assigned to each record by the shard to maintain order within shard

Question 13

Retention period

Answer 13

Maximum amount of time (up to 7 days) that data can be stored in Kinesis before being deleted

Question 14

Kinesis Data Firehose

Answer 14

Service that automatically delivers streaming data to AWS services like S3, Redshidt, ElasticSearch

Question 15

AWS Lambda

Answer 15

Serverless computing (FaaS), used to build modular back-end systems, can process streaming data without need to manage any server infrastructure (scales automatically with size of data stream, paying only for compute time, event-driven)

Question 16

What are some of the operational and administrative activities that Lambda takes care of for you so you can focus on only your code?

Answer 16

Balances memory and CPU
Provisioning Capacity
Monitoring fleet health
Applying security patches

Question 17

FaaS

Answer 17

Write single-purpose stateless functions

(function does just one thing! thats why its modular)

Question 18

Data Pipeline Pattern

Answer 18

Architectural solution to problems in software design

Question 19

Command Pattern

Answer 19

The Command Pattern encapsulates a request as an object, allowing it to be executed later or passed around in the system without knowing when or how it will be executed.

Question 20

Pipes and filters pattern

Answer 20

It decomposes a complex process into a sequence of smaller, manageable steps (filters) connected by pipes, where each filter processes and passes data to the next

Question 21

Messaging Pattern

Answer 21

It decouples different parts of a system by using a queue to send and receive messages, allowing independent processing and handling of tasks.

Question 22

Priority queue pattern

Answer 22

It allows for tasks or messages to be prioritized, ensuring that high-priority items are processed first, while lower-priority tasks wait.