Events, Messages, Kafka

Question 1

What is an event?

Answer 1

An event can be defined as a significant change in state. An event is a notification that data has been processed and some objects’ state has changed.

Question 2

What is a message?

Answer 2

A message is a request from one system to another for an action to be taken. The sender includes in the message a full payload of data that needs to be processed, and formats that data appropriately for the receiver to process

Question 3

What are the differences between event and message?

Answer 3

1) Events are lightweight in that they do not include all aggregate data – their payload can either be just the ID of the aggregate that changed, or the ID and just the properties that have changed so the receiver can update their cached copy.
2) An important difference between events and messages is that there is no expectation of further processing for an event.
3) Another difference is that messages can affect 0 or more aggregates while an event reflects a change to exactly one aggregate.

Question 4

Describe how kafka works

Answer 4

Kafka distinguishes producers, consumers, and brokers. In short, producers publish data to Kafka brokers, and consumers read published data from Kafka brokers. Producers and consumers are totally decoupled, and both run outside the Kafka brokers in the perimeter of a Kafka cluster. A Kafka cluster consists of one or more brokers. An application that uses the Kafka Streams API acts as both a producer and a consumer.
The data: Data is stored in topics. The topic is the most important abstraction provided by Kafka: it is a category or feed name to which data is published by producers. Every topic in Kafka is split into one or more partitions. Kafka partitions data for storing, transporting, and replicating it. Kafka Streams partitions data for processing it. In both cases, this partitioning enables elasticity, scalability, high performance, and fault tolerance.

Kafka Streams uses the concepts of partitions and tasks as logical units of its parallelism model based on Kafka topic partitions

Question 5

Kstream

Answer 5

A KStream is an abstraction of a record stream, where each data record represents a self-contained datum in the unbounded data set.
Using the table analogy, data records in a record stream are always interpreted as an “INSERT” – think: adding more entries to an append-only ledger – because no record replaces an existing row with the same key. Examples are a credit card transaction, a page view event, or a server log entry.

Question 6

Ktable

Answer 6

A KTable is an abstraction of a changelog stream, where each data record represents an update. More precisely, the value in a data record is interpreted as an “UPDATE” of the last value for the same record key, if any (if a corresponding key doesn’t exist yet, the update will be considered an INSERT). Using the table analogy, a data record in a changelog stream is interpreted as an UPSERT aka INSERT/UPDATE because any existing row with the same key is overwritten. Also, null values are interpreted in a special way: a record with a null value represents a “DELETE” or tombstone for the record’s key.

Question 7

What is exactly-once processing?

Answer 7

Exactly-once stream processing is simply the ability to execute a read-process-write operation exactly one time. In this case, “getting the right answer” means not missing any input messages or producing any duplicate output.

Question 8

How do we obtain exactly once processing with Kafka?

Answer 8

If a messaging system always delivers all messages without duplication, that is exactly-once processing.

Initially, Kafka only supported at-most-once and at-least-once message delivery.

However, the introduction of Transactions between Kafka brokers and client applications ensures exactly-once delivery in Kafka.