Fundamentals

Question 1

What’s a topic?

Answer 1

streams of collection of related messages or related events.
Can think of it as a sequence of events.

Question 2

What are partitions?

Answer 2

Topics divided into smaller pieces, then you can allocate the different partitions to different brokers in the cluster (key to how Kafka scales)

Question 3

What are the main responsibilities of a Broker?

Answer 3

Broker receive the data from producers and store it temporarily in the page cache, or permanently on disk after the OS flushes the page cache.

How long the data is kept around is determined by the retention time (default 1 week)

Question 4

How does Producer work?

Answer 4

Producers send messages into topics. Different producers can send messages into same topics or different topics.

Question 5

How does Consumers work?

Answer 5

Consumers poll/read periodically data from Kafka.
Many consumers can poll data from Kafka at the same time.
To allow for parallelism, consumers are organised in consumer groups that split up the work.

Question 6

One key feature of Kafka is that Producers and Consumers are decoupled, what does that mean?

Answer 6

• They don’t know about the existence of each other.
• They don’t depend on each other.
• They need to agree on the data format of the records produced and consumed.

Question 7

What does Kafka use ZooKeeper for?

Answer 7

• Cluster management
• Failure detection and recovery (eg. when a broker goes down)
• Store Access Control Lists (ACLs) used for authorization in the Kafka cluster

Question 8

What are Segments?

Answer 8

Physical files.

The broker stores the messages as they come in memory (page cache), then periodically flushes them to a physical file.
Data can potentially be endless so the broker is using a “rolling-file” strategy. It creates/allocates a new file and fills it with messages. When the segments is full or expires, the next one is allocated by the broker.
Kafka rolls over files in segments to make it easier to manage data retention and remove old data.

Question 9

What’s a log?

Answer 9

a data structure that is like a queue of elements. New elements are appended at the end of the log, and once written they are never changed,
Append only, write once data structure.

Question 10

What’s a stream?

Answer 10

a sequence of events. The sequence has a beginning somewhere in the past.

Question 11

Stream is immutable. What does it mean?

Answer 11

You can’t change a stream.
In stream processing one never modifies an existing stream, but always generates a new output stream.

Question 12

What’s a record in Kafka?

Answer 12

A data element in a log or topic. But other words being used are message and event.

Question 13

What are the Consumer Offset?

Answer 13

it keeps track of the latest message read and it’s stored in a special Kafka topic.
It can be changed to reread eg. messages

Question 14

What is a consumer group?

Answer 14

A consumer group consists of 1 to many consumer instances. All consumer instances in a consumer group are identical clones of each other.

Question 15

What’s the purpose of consumer group?

Answer 15

To allow to increase the throughput in downstream consumption of data flowing into a topic.

Question 16

What has to be done for a consumer to be added into a consumer group?

Answer 16

To be transparently added to a consumer group, an instance needs to use the same group.id

Question 17

What’s an offset?

Answer 17

the id in a partition, aka Kafka Partition offset.
Id starts from 0 to infinite.

Question 18

What’s important in partitons?

Answer 18

. Once the data is written to a partition, it cannot be changed (immutability)
· Data is kept only for a limited time (default is one week - configurable)
. Offset only have a meaning for a specific partition.
. E.g. offset 3 in partition 0 doesn’t represent the same data as offset 3 in partition 1
· Offsets are not re-used even if previous messages have been deleted
. Order is guaranteed only within a partition (not across partitions)

Question 19

To produce data to a topic, what must a producer provide to the Kafka client?

Answer 19

need to connect to one broker (any broker) and provide the topic name you want to write to.

Question 20

To get acknowledgment of writes to only the leader, we need to use the config…

Answer 20

acks=1

Question 21

What are the config a consumer need to read data from a topic?

Answer 21

any brokers to connect to, and the topic name

Question 22

what’s the default number of partitions in a topic?

Answer 22

6