Section 3 : Consumers

Question 1

How do Kafka Consumers work?

Answer 1

• Kafka consumers is an application that consumes messages (generated by producers) by subscribing to a topic.
• Step 1: Create a Kafka consumer object
• Step 2: Subscribe/assign to the topic (Depending upon whether it’s standalone consumer or part of consumer group)
• Step 3: Start receiving messages
• Step 4: Validate message
• Step 5 : Send for further processing

Question 2

What is Kafka Consumer?

Answer 2

• Consumer can be either standalone consumer or can be part of a consumer group but not both at the same time
• Standalone consumer:
  
  • Simple to implement
  • No need for rebalancing
  • Needs to find partitions by itself:
    
    • Uning consumer.partitionsFor() periodically
  • Consumer assigns partitions to itself
    
    • using assign() method as it’s the only consumer which needs to read all records from the all the partitions

Question 3

What are Kafka Consumer Group?

Answer 3

• Consumer group is a group of consumers all of which read from the same topic.
• You add consumer to help deal with the load
• Kafka can scale to a large number of consumer and consumer groups without reducing performance.
• If Number of consumer in the group > number of partitions of the topic, some of the consumer may be iddle [Though having 1 or 2 idle consumers in the consumer group is recommended]
• Create a new consumer group for each application that needs all the messages from one or more topics consumers to an existing consumer
• Consumer group is to scale the reading and processing of messages from topics.
• To maintain membership to consumer group and to maintain ownership of the partitions assigned, consumers send heartbeats to the Group coordinator.
  
  • Heartbeats are sent:
    
    • When the consumer polls
    • When consumer commits records it has consumed
    • Release 0.10.1 onwards, heartbeats can be sent in between polls as well
    • If heartbeat are not received for soemtime, the repartition is triggered

Question 4

What is Partition Rebalance?

Answer 4

• Moving partition ownership from one consumer to another is called Partition Rebalancing
• When Partition Rebalancing comes into picture?
  
  • ​When a new consumer is added to the group
  • When a consumer shuts down
  • When a consumer crashes
  • When a consumer leaves the consumer group
  • When the topics are modified
  • When a new partition is added
• Why Partition Rebalancing?
  
  • To enable consumer group for high availability and scalability

Question 5

What is Partition Rebalance Functioning?

Answer 5

• The consumer sends “JoinGroup request” to the group Coordinator
• The first consumer in the group becomes the group leader
• Leader consumer gets the following:
  
  • List of consumers in the groups
  • List of partition assignments
• Group leader decides partition/ set of partition to the other consumers in the group.

Question 6

What is Partition Rebalance Issues?

Answer 6

• Unavailability
  
  • During rebalancing process entire consumer group is unavailable
• Loosing state
  
  • Consumer looses it’s current state
• Impact on performance
  
  • Looses cache and hence slows down the preformance for a while
• Partition rebalance is not desired and should be avoided