Integrating Message Queue: Kafka Flashcards
What are the key components of Kafka?
The key components of Kafka include:
Producer: Publishes messages to Kafka topics.
Consumer: Subscribes to Kafka topics to receive messages.
Broker: A Kafka server that stores and manages topics and messages.
Topic: A category or feed name to which messages are published.
Partition: Each topic is divided into partitions, which are distributed across Kafka brokers.
ZooKeeper: Manages and coordinates Kafka brokers and provides metadata storage.
How does Kafka ensure fault tolerance?
Kafka replicates messages across multiple brokers to ensure fault tolerance. Each partition has multiple replicas, and if one broker fails, another replica can take over.
What is the role of ZooKeeper in Kafka?
ZooKeeper manages and coordinates Kafka brokers, maintains metadata, and handles distributed synchronization within the Kafka cluster.
How does Kafka handle message persistence?
Kafka persists messages to disk, allowing messages to be stored for a configurable retention period. This ensures that messages are not lost even if the consumer is unable to process them immediately.
What are some use cases for Kafka as a message queue?
Some common use cases for Kafka as a message queue include real-time data processing, event sourcing, log aggregation, stream processing, and microservices communication.
How does Kafka support both batch processing and stream processing?
Kafka supports both batch processing and stream processing use cases. It can handle high-throughput, low-latency stream processing with Kafka Streams, as well as batch processing with tools like Apache Spark or Apache Flink.
What is the difference between Kafka and traditional message queues?
Kafka is designed as a distributed streaming platform with support for horizontal scalability, fault tolerance, and persistent message storage. Traditional message queues may prioritize in-memory storage and point-to-point messaging models.
How does Kafka ensure fault tolerance?
Kafka replicates messages across multiple brokers, ensuring fault tolerance. Each partition has multiple replicas, and if one broker fails, another replica can take over.
What configuration options are available for Kafka topics?
Kafka topics can be configured with settings such as the number of partitions, replication factor, retention policy, cleanup policy, compression settings, etc.
How does Kafka internally implement partitions in topics?
Internally, each partition is implemented as a segmented log of messages, with messages appended to the end of the log and older segments periodically compacted and archived.
