OpenSearch Consumer

Question 1

What does ‘At most once’ delivery semantics mean?

Answer 1

Offsets are committed as soon as the message batch is received. If the processing goes wrong, the message will be lost (it won’t be read again).

Question 2

What does ‘At least once’ delivery semantics mean?

Answer 2

Offsets are committed after the message is processed. If the processing goes wrong, the message will be read again, which can result in duplicate processing of messages. Make sure to use idempotent!

Question 3

What does ‘Exactly once’ delivery semantics mean?

Answer 3

Can be achieved for Kafka workflows using the Transactional API (easy with Kafka Streams API). For Kafka Sink workflows, use an idempotent consumer.

Question 4

What are the two common patterns for committing offsets in a consumer application?

Answer 4

1. enable.auto.commit = true & synchronous processing of batches (easy)
2. enable.auto.commit = false & manual commit of offsets (medium)

Question 5

What happens with auto-commit in the first pattern?

Answer 5

Offsets will be committed automatically at regular intervals (auto.commit.interval.ms=5000 by default) every time you call .poll(). If you don’t use synchronous processing, you will be in ‘at-most-once’ behavior.

Question 6

What does the second pattern allow you to do?

Answer 6

You can control when you commit offsets and the condition for committing them.

Example: accumulating records into a buffer and then flushing the buffer to a database + committing offsets asynchronously.