Streaming

Question 1

What techniques are required for stream processing?

Answer 1

• A component that acquires events from producers and forwards to consumers
• A component that processes events

Question 2

Describe the Publish/subscribe model

Answer 2

It connects mulitple producers to multiple consumers. Message brokers are systems that sit between producers and consumers and deal with reliable message delivery.

Question 3

What are two types of Broker based messaging?

Answer 3

• Fire and forget: the broker acks the message immediately
• Transaction based: the broker writes the message to permanent storage prior to acking it

Question 4

What does the broker do?

Answer 4

• Buffers the message using disk if appropriate
• Routes the messages to the appropriate queues
• Notifies consumers when messages have arrived

Question 5

What do the consumers do?

Answer 5

• Subscribe to the queue that contains their desired messages
• Acks the message receipt

Question 6

Describe the Competing workers messaging pattern

Answer 6

Multiple consumers read from a single queue competing for incoming messages.

Question 7

Describe the fan out messaging pattern

Answer 7

Each consumer has their own queue, which is replicated

Question 8

Describe the message routing messaging pattern

Answer 8

The producer assigns keys to message metadata and creates topic queues.

Question 9

What are some drawbacks of broker based messaging

Answer 9

• Once the message is received it disappears
• Reprocessing a message is impossible
• Cannot prove a message was disappeared

Question 10

What is log based messaging?

Answer 10

• producers append to a log
• All consumers connect to the log and pull from it
• A new client starts processing from the beginning of a log
• The broker partitions the log to a cluster
• The broker keeps track of current message offset for each consumer per partition

Question 11

What are some programming models for stream processing?

Answer 11

• Event sourcing/command query segregation
• Reactive programming
• DataFlow model

Question 12

Describe Event sourcing/Command Query Segregation

Answer 12

It captures all changes to an application state as a sequence of events:
* Instead of mutating the state we store the event that causes th emutation in an immutable log
* State is generated by processing the events

Question 13

What is Reacrive Programming?

Answer 13

It is a declarative programming paradigm concerned with data streams and propagation of change
* Reactive APIs model event sources as infinite collections on which observers subscribe to and receive events

Question 14

What are the four dimensions of the DataFlow model?

Answer 14

• What: Reslts are being computed
• Where: In event time they are being computer
• When: In processing time they are materialized
• How: Earlier results relate to later refinements

Question 15

What are the two notions of time in stream processing?

Answer 15

• Processing time: The time at which events are observed in the system
• Event time: The time at which events occurred

Question 16

How are skew and lag calculated for a progessing time t?

Answer 16

• Skew = t - s where s is the timestamp of the latest event processed
• Lag = t - s where s is the actual timestamp of the event

Question 17

What are the 4 types of windows?

Answer 17

• Tumbling (repeats non-overlapping window), range = slide
• Jumping has an overlapping interval
• Sliding
  > range > slide
  > Time sliding triggers at regular interval
  > Eviction sliding triggers on a count
• Session windows -> aggregate batches of user activity and end after a session gap time

Question 18

What is a window trigger?

Answer 18

It defines when in processing time the results of a window are processed