Kafka

Question 1

How is partitions maintained?

Answer 1

Each topic is split into partitions. Producer decides what partition to send message to. This can be done in a round-robin fashion simply to balance load or it can be done according to some semantic partition function (say based on some key in the message).

Each partition is an ordered, immutable sequence of messages that is continually appended to—a commit log.

Question 2

What metadata in partitions?

Answer 2

The only metadata retained on a per-consumer basis is the position of the consumer in the log, called the “offset”. This offset is controlled by the consumer: normally a consumer will advance its offset linearly as it reads messages, but in fact the position is controlled by the consumer and it can consume messages in any order it likes. For example a consumer can reset to an older offset to reprocess.

Question 3

How is Fault tolerance achieved?

Answer 3

Each partition has one server which acts as the “leader” and zero or more servers which act as “followers”. The leader handles all read and write requests for the partition while the followers passively replicate the leader. If the leader fails, one of the followers will automatically become the new leader. Each server acts as a leader for some of its partitions and a follower for others so load is well balanced within the cluster.

Question 4

How is consumers designed?

Answer 4

Consumers label themselves with a consumer group name, and each message published to a topic is delivered to one consumer instance within each subscribing consumer group.

If all the consumer instances have the same consumer group, then this works just like a traditional queue balancing load over the consumers.

If all the consumer instances have different consumer groups, then this works like publish-subscribe and all messages are broadcast to all consumers.

Question 5

How durability is implemented?

Answer 5

Kafka relies heavily on the filesystem for storing and caching messages. There is a general perception that “disks are slow” which makes people skeptical that a persistent structure can offer competitive performance.

Sequential disk access can in some cases be faster than random memory access!

Furthermore on top of the JVM:

• The memory overhead of objects is very high, often doubling the size of the data stored (or worse).
• Java garbage collection becomes increasingly fiddly and slow as the in-heap data increases.