Kafka

Question 1

Kafka

What is a broker?

Answer 1

• A broker is a physical or virtual server in the Kafka cluster
• Each broker is responsible for storing data and serving clients

Question 2

Kafka

What is a partition?

Answer 2

• A partition is an ordered, immutable sequence of messages that is continually appended to.
• Think of partitions as a log file
• They are why Kafka can scale as they allow message to be consumed in parallel

Question 3

Kafka

What is a topic?

Answer 3

• A topic is a logical grouping of partitions

Question 4

Kafka

What is the difference between a partition and a topic?

Answer 4

• A topic is a logical group of messages
• A partition is a physical grouping of messages
• A topic can have multiple partitions and each partition can be on a different broker
• Topics are a way to organize data

Question 5

Kafka

What is a producer?

Answer 5

• Producers write data to topics

Question 6

Kafka

What is a consumer?

Answer 6

• Consumers read data from topics

Question 7

Kafka

What are the two main use cases for Kafka?

Answer 7

• Message queue
• Steam

Question 8

Kafka

What is a message/record?

Answer 8

• A message is the data structure stored in partitions
• Messages consist of 4 fields
  
  • headers
  • key
  • value (required)
  • timestamp
• The key is used to which partition a message is sent to, if absent, it will be randomly assigned to a partition

Question 9

Kafka

How does Kafka determine which partition to write a message to?

Answer 9

1. Partition Determination: If a message key is present, the key is hashed and assigned to a partition; if no key is present, a round-robin algorithm will be used or another algorithm defined in the producer configuration.
2. Broker Assignment: After the partition is identified, the Kafka cluster metadata contains a mapping of partitions to specific brokers. The Kafka controller holds this information and the producer can use this to route the message to the proper partition.

Question 10

Kafka

What are some important benefits of partitions acting as append-only log files?

Answer 10

1. Immutability: messages cannot be altered or deleted once written. This simplifies replication
2. Efficiency: By restricting to append only at the end of files, this minimizes seek time
3. Scalability: An append-only log mechanism enables horizontal scaling to handle increasing load. Partitions can be replicated to enhance fault tolerance.

Question 11

Kafka

Describe the leader-follower model for replication.

Answer 11

1. Leader Replica Assignment: For each partition, a leader replica is assigned. The leader replica handles all of the reads and writes for the partition. The cluster controller handles assignment of the leader replica and ensures that leader replicas are evenly distributed across the cluster.
2. Follower Replication: Several follower replicas exist, for each partition, residing on different brokers. Follower replicas do not handle reading or writing, they passively replicate data from the leader replica.
3. Synchronization and Consistency: Followers continually sync with the leader replicas. If a leader fails, a fully synced follower can be promoted to the leader.
4. Controller’s Role in Replication: The controller manages the replication process. When a leader fails, it promotes an in-sync follower to ensure continued availability.

Question 12

Kafka

What is a push model

Answer 12

When a consumer is subscribed to a topic, new messages are sent to the consumer.

Question 13

Kafka

What is a pull model

Answer 13

A consumer can poll the topic, at regular intervals, to get the latest messages.

Question 14

Kafka

When to use Kafka in your interview

Answer 14

• Message queue
  
  • A process that can be done asynchronously: YouTube processing videos
  • To ensure messages are processed in order: Ticketmaster virtual queue
  • Decouple producer and consumer so they scale independently: This means that the producer is producing messages faster than the consumer can consume them
• Streams
  
  • Continuous and immediate processing of incoming data, treat it as a real time flow: And Ad Clicker
  • Messages need to be processed by multiple consumers simultaneously: FB live comments