Transactional messaging Flashcards
Transactional messaging
Transactional outbox (application publish events)
Transaction log tailing
Polling publisher
Transactional outbox: context
A service typically need to atomically update the database and publish messages/events. For example, perhaps it uses the Saga pattern. In order to be reliable, each step of a saga must atomically update the database and publish messages/events. Alternatively, it might use the Domain event pattern, perhaps to implement CQRS. In either case, it is not viable to use a distributed transaction that spans the database and the message broker to atomically update the database and publish messages/events.
Transactional outbox: problem
How to reliably/atomically update the database and publish messages/events.
Transactional outbox: forces
2PC is not an option
Transactional outbox: solution
The application that uses a relational database inserts messages/events into an outbox table (e.g. MESSAGE) as part of the local transaction. An application that uses a NoSQL database appends the messages/events to attribute of the record (e.g. document or item) being updated. A separate Message Relay process publishes the events inserted into database to a message broker.
Transactional outbox: result benefits
High level domain events
No 2PC
Transactional outbox: result drawback
Potentially error prone since the developer might forget to publish the message/event after updating the database.
Transactional outbox: issues
The Message Relay might publish a message more than once. It might, for example, crash after publishing a message but before recording the fact that it has done so. When it restarts, it will then publish the message again. As a result, a message consumer must be idempotent, perhaps by tracking the IDs of the messages that it has already processed. Fortunately, since Message Consumers usually need to be idempotent (because a message broker can deliver messages more than once) this is typically not a problem.
Transactional outbox: related
The Saga and Domain event patterns create the need for this pattern.
The Event sourcing is an alternative solution
There are two patterns for implementing the message relay:
* The Transaction log tailing pattern
* The Polling publisher pattern
Transactional log tailing: context
You have applied the Application events pattern. In order to be reliable, each step of a saga must atomically update the database and publish messages/events. It is not viable to use a distributed transaction that spans the database and the message broker.
Transactional log tailing: problem
How to publish messages/events into the outbox in the database to the message broker?
Transactional log tailing: solution
Tail the database transaction log and publish each message/event inserted into the outbox to the message broker.
The mechanism for trailing the transaction log depends on the database:
MySQL binlog
Postgres WAL
AWS DynamoDB table streams
Transactional log tailing: example
Eventuate Local implements transaction log tailing.
Transactional log tailing: result benefits
No 2PC
Guaranteed to be accurate
Transactional log tailing: result drawback
Relatively obscure although becoming increasing common
Requires database specific solutions
Tricky to avoid duplicate publishing
