Streaming

Question 1

What is data streaming in big data?

Answer 1

Data streaming refers to the continuous processing of data as it arrives in real-time, involving simpler computations over smaller, continuously incoming data sets as opposed to batch analysis

Question 2

Why is streaming data processing important?

Answer 2

It helps achieve lower latency, enabling timely data analysis
More consistent resource consumption
Real time decision making
Workload balancing across system

Question 3

What are some use cases of data streaming?

Answer 3

Use cases include operational monitoring
real time user interaction in web analytics
online advertising with real time bidding
social media analysis
IoT applications like health monitoring

Question 4

What is the cash register model in data streams?

Answer 4

The cash register model represents data as increments to a variable
Such as counting packages sent to IP addresses
You INCREMENT the state

Question 5

What is the turnstile model in data streaming?

Answer 5

The turnstile model includes updates to variables that can both increment and decrement values
(tracking people entering and exiting a gym)

Question 6

How are data streaming systems classified based on latency tolerance?

Answer 6

Systems are classified as hard (no delay tolerance)
Firm (some tolerance for delay, seconds maybe)
Soft (still valuable even when late, weather monitoring)

Question 7

What are the key characteristics of streaming data?

Answer 7

Generated continuously
One pass processing
Often requires approximation (memory and time constraints)
Handle infinite computation
Low latency

Question 8

What components are involved in data streaming architecture?

Answer 8

Data Collection
Messaging Queue
Analysis
Data Consumer
(in memory storage)
(long term storage)

Question 9

What is the role of message queuing in data streaming?

Answer 9

It facilitates the safe and organized exchange of data between the collection and analysis tiers, decoupling the system components

Also, its worth messaging that it acts as Unified Log in addition to managing data flows between components in the architecture, it also provides an append-only record of events to be accessed by consumer (analytical tier, apache flink)

Question 10

What are common message queuing systems?

Answer 10

Apache Kafka (store data in log as topics) and Apache Flume (move data to long term storage like hdfs)

Question 11

What are the message delivery semantics in streaming?

Answer 11

Exactly once (guarantees no data loss and single reads)
At most once (event read once, but maybe also not read)
At least once (ensures data is read, even if multiple reads)

Question 12

What is a continuous query in data streaming?

Answer 12

A continuous query is executed once and runs continuously over incoming data

Updating results in real-time, often requiring state management for intermediate results

Question 13

What challenges are associated with continuous queries?

Answer 13

Must handle state updates efficiently with memory and time constraints

Question 14

What is windowing in data streaming?

Answer 14

Windowing divides a data stream into finite chunks for processing based on time or number of items

Question 15

What are sliding windows?

Answer 15

Sliding windows overlap and process data at regular intervals (time based)

Question 16

What are data-driven windows?

Answer 16

These windows are defined by data content and are typically used for session analysis, updating only when specific events occur

Question 17

What is event time versus stream time?

Answer 17

Event time is when the data event occurs
Stream time is when data is processed

Question 18

What is a watermark in event time windowing?

Answer 18

A watermark indicates the progress of event time, used to manage how long a system waits for late data before closing a window

Question 19

What are triggers in event time windowing?

Answer 19

Triggers define when intermediate results for a window are processed, based on criteria like watermark progress or element counts

Question 20

What strategies are used for result accumulation in windowing?

Answer 20

Strategies include discarding (independent results)
accumulating (results build on previous ones)
accumulating with retraction (includes both totals and deltas)

Question 21

What is reservoir sampling in data streaming?

Answer 21

It is a method to select a random SAMPLE from an incoming data stream, maintaining a fixed-size reservoir of times with probabilistic replacement as new data arrives

Question 22

What is HyperLogLog used for?

Answer 22

An algorithm used for approximate counting of DISTINCT items in a data stream, providing space efficiency with an accuracy trade off

Question 23

What is the Count-Min Sketch?

Answer 23

A space-efficient algorithm used to approximate FREQUENCY counts of items in data streams, maintaining counters incremented by hash functions and taking the minimum for estimates