Chapter 5, Event-Driven Architecture Patterns Flashcards
What are the different event delivery guarantees provided by Message brokers?
- At-most-once delivery
The event is delivered to the consumer only once or not at all. If the consumer is not online during a delivery attempt or if network failures occur, the consumer will not get the event. Most important, the message broker will not try to send the same event again. - At-least-once delivery
The event is guaranteed to be delivered to the consumer. However, the consumer may consume the same event multiple times because if the message broker does not get an acknowledgment from the consumer for the event delivery, it will assume that the consumer did not receive the event and will resend it. In this case, the consumer should be intelligent enough to handle duplicate events.
Unfortunately, we cannot achieve an exactly once delivery guarantee, which ensures that the event is delivered to the consumer once and only once, because of the uncertain nature of the network and systems.
265
What does exactly once processing ensures?
exactly once processing ensures that the event is processed once and only once.
266
Describe a way of achieving exactly once processing
We can also achieve exactly once processing when events are idempotent: the outcome of receiving the same event multiple times is no different from receiving the event only once.
266
What are the two main Message Broker Categories?
- Standard (store-backed) message brokers
These are the standard message brokers that store events in a data store to enable serving to intended consumers. Most important, they purge events from their store upon delivery to consumers. Apache ActiveMQ and RabbitMQ are examples of these brokers. - Log-based message brokers
These brokers store events in commit logs. The events persist even beyond their being consumed. Therefore, these brokers allow consumers to replay events from a previous point in time. Apache Kafka and NATS are examples of this type.
Regardless of the category, different message brokers, and at times even the same message broker, can support various delivery guarantees.
267
Describe the Producer-Consumer Pattern
The Producer-Consumer pattern enables producer applications and consumer applications to communicate asynchronously by using event queues. The queue manages which consumer processes which event, and which procedures need to be followed when the consumers fail during the processing of the event.
269
How does the Producer-Consumer Pattern work?
This pattern requires an intermediate message queue that is managed by a message broker (Figure 5-1). One or more producers can send events to the queue. The message broker typically persists the queued events in a durable store, thereby guaranteeing that the events will eventually be delivered to consumers. The message broker then delivers one event at a time, mostly following FIFO order, to the consumers on request. This helps consumers process events as they have capacity and to not become overloaded.
When consumers are done processing the event, they can also send an acknowledgment to the message broker, so that the message broker can purge the event from its store.
269 Figure 5-1. Event delivery from producers to consumers
How is Producer-Consumer Pattern used in practice?
This pattern can be used for various scenarios, such as asynchronously delivering events from one application to another, making sure only one application processes a piece of data, ensuring event delivery, sharing workload among applications, handling sudden bursts of events, and decoupling applications.
270
What are some related patterns to the Producer-Consumer Pattern?
- Publisher-Subscriber pattern
Can send the same event to multiple consumers for processing. - Fire and Forget pattern
Used when events need to be delivered to a single consumer with an at-most-once delivery guarantee without the help of a message broker. - Store and Forward pattern
Allows asynchronous at-least-once event delivery without a message broker.
274
Describe the Publisher-Subscriber Pattern
The Publisher-Subscriber pattern enables applications to communicate asynchronously by using topics. The topic delivers every event to every subscriber.
275
How does the Publisher-Subscriber Pattern work?
This pattern uses topics to propagate the events from publishers to subscribers. The topic is a message broker concept. Multiple publishers can submit events to a topic hosted in the message broker (Figure 5-3). The topic then publishes all those events to all its subscribers, and makes sure that every subscriber receives all incoming events.
The standard behavior of the pattern is best effort: the events are delivered at most once. When a subscriber misses an event due to unavailability or network failure, they will never receive the event. But we overcome this problem with a durable subscription, which guarantees that all messages are delivered to all the consumers at least once, accounting for subscribers who are temporarily unavailable.
275 Figure 5-3. Event delivery from publishers to multiple subscribers
How is the Publisher-Subscriber Pattern used in practice?
This pattern is used for scenarios such as broadcasting notifications in parallel to multiple recipients with both the best effort and at-least-once delivery guarantees. Let’s see how we can achieve these.
276
What are some related patterns to the Publisher-Subscriber Pattern?
The Publisher-Subscriber pattern is related to the preceding Producer-Consumer pattern, which provides the capability to send an event to only one consumer for processing.
279
Describe the Fire and Forget Pattern
The Fire and Forget pattern enables clients (producers) to send events to respective consumers (services) with an at-most-once delivery guarantee without the use of a message broker.
279
How does the Fire and Forget Pattern work?
Let’s imagine that a weather sensor periodically sends current temperature and humidity readings to a weather-prediction service hosted in the cloud. Because of technical limitations, instead of using a message broker, as depicted in Figure 5-5, it is designed to invoke the API of the service by using protocols such as HTTP. The client is interested only in whether the server received the events, and is not interested in the final outcome. When a client publishes an event to the service, it expects only an acknowledgment of the event by a relevant HTTP status code such as 202 Accepted.
279 Figure 5-5. Fire and Forget event delivery from client to service
How is the Fire and Forget Pattern used in practice?
This pattern is useful when we need best-effort delivery of noncritical data, or when the receiving service does not possess the capability to subscribe or pull events from a client.
280
Describe the Store and Forward Pattern
The Store and Forward pattern enables clients to send events to services with an at-least-once delivery guarantee. As with Fire and Forget, this pattern does not use message brokers but uses APIs to directly send events.
282
How does the Store and Forward Pattern work?
This pattern requires a complex client design to achieve the at-least-once event delivery guarantee. The client in this pattern first persists the events to a durable store, such as a database or queue in a message broker, before attempting to send them to the service (Figure 5-6). Upon successful event delivery, the client purges the events from the store. If delivery is unsuccessful, it retries to send the event. During this, as the client receives more events to send, it persists them to the store. Once the connection to the service is established, it will deliver all pending events, receive acknowledgment of the event consumption, and purge the events from its store.
282 Figure 5-6. Store and Forward event delivery from client to service
How is the Store and Forward Pattern used in practice?
This pattern is useful when delivering critical data with a message broker, or when the receiving service cannot subscribe or pull events from the client.
283
What are some related patterns to the Store and Forward Pattern?
- Fire and Forget pattern
Publishes events to service endpoints with an at-most-once delivery guarantee. - Producer-Consumer pattern
Allows events to be delivered to a single consumer with higher delivery guarantees when the consumer can subscribe to an event queue.
285
Describe the Polling Pattern
The Polling pattern enables clients such as web browsers to initiate a long-running job, periodically checking completion.
285
How does the Polling Pattern work?
The frontend client or browser sends a request to initiate the process, such as insurance claim processing (Figure 5-7). Because the processing takes time, the backend service immediately sends an acknowledgment stating that it has accepted the request and initiated the asynchronous job processing. Along with the acknowledgment, it sends a job ID and, potentially, an estimated time of job completion. Based on this information, the client periodically queries the backend to check if the claim processing has completed. Upon completion, the backend returns the results as part of the response to the query, or provides a redirection to an endpoint containing the results.
285 Figure 5-7. Frontend client/browser repeatedly calling backend service to retrieve asynchronous job results
How is the Polling Pattern used in practice?
This pattern is useful when we need to retrieve the result of an asynchronous job without using a subscription or callback.
286
What are some related patterns to the Polling Pattern?
- Producer-Consumer pattern
This is an alternative used when the participating applications publish and subscribe to a queue. - Request Callback pattern
This is also an alternative that’s used when the clients and services are capable of using WebSockets or webhooks.
288
Describe the Request Callback Pattern
The Request Callback pattern enables applications to communicate asynchronously. The application provides the callback information with the request so responses can be delivered to the given callback.
289
How does the Request Callback Pattern work?
The Request Callback pattern enables applications to communicate asynchronously. The application provides the callback information with the request so responses can be delivered to the given callback.
289
Describe how WebSockets are used in the Request Callback Pattern
To use WebSockets, both client and service should have the capability to communicate via the WebSocket protocol (Figure 5-8). The client initiates the connection to the service and establishes a long-running connection. Both the client and service persist the connection and communicate by sending events. This approach is used for clients requesting information via an event and waiting on the service to respond, or for exchanging multiple events. WebSocket is an HTTP-based technology, but HTTP2 and gRPC also provide similar callback-based communication.
289 Figure 5-8. Client and service communicating via the WebSocket protocol
Describe how Webhooks are used in the Request Callback Pattern
In this approach, the client application issues a request and has the response delivered to a callback endpoint (Figure 5-9). The client sends the request with a callback URL. If the callback URL is consistent, we configure that on the service side, so we do not have to redundantly send the URL with the request. The response, when generated, is delivered to the callback URL.
290 Figure 5-9. Client and server communicating via webhook
How is the Request Callback Pattern used in practice?
We use this pattern to deliver responses asynchronously or when we need to receive continuous updates. This pattern mimics registering our telephone number as a callback so that the insurance agent can call to inform us of the status of our insurance claim. Following are some use cases in practice.
291
What are some related patterns to the Request Callback Pattern?
- Store and Forward pattern
Complements this pattern by providing guaranteed callback event delivery (covered previously in this chapter). - Polling pattern
Provides an alternative when applications cannot establish callbacks (covered previously in this chapter). - Asynchronous Request-Reply pattern
An alternative approach to communicate asynchronously by using a message broker.
292
When to use the Producer-Consumer pattern
There is a particular event to be consumed and processed by only one of the available consumers.
We cannot confirm the availability of the consumers and producers.
We see burst event production over a short period.
We need to ensure fairness in the processing of events.
293
How does the Event Sourcing Pattern work?
Every time an event updates the application state, the event is also recorded in a persistence store in the order of operation. Figure 5-10 shows a banking use case, in which transactions are performed on an account. Events occur, such as Bob depositing $110, withdrawing $20, and then again depositing $50.
If we consider only the current state of the application, we know only that Bob now has a balance of $140 in his account, but we do not know the events that led up to this. Now, as each event is stored in a persistence store (or event log), in the sequence of its occurrence, we can gain an understanding of how his account balance changed over time.
Events are usually stored on a commit log, such as Apache Kafka, which allows us to read the events back from any point in time in a sequential manner by passing event sequence IDs. These events are read again by the application or other systems so that they can process the events that have occurred or re-create an application state with either a subset or all events.
296 Figure 5-10. Current account microservice generating an event log while updating application state
How does the Mediator pattern work?
The key element of this pattern is the mediator microservice runtime built as part of the cloud native application. It operates by interacting with all the microservices that integrate, via event queues, topics, and APIs. It connects on various protocols and transforms events for applications accordingly. These mediator microservices are usually stateless, and might need to perform only filtering, sorting, and event transformations. But when required, they can coordinate sequential and parallel tasks.
Mediators sometimes need to combine events from multiple systems, and so need to be stateful.
303 Figure 5-11. Mediator microservice orchestrating a new insurance request
How does the Pipe and Filter Pattern work?
This pattern uses event queues and topics to connect microservices. With this approach, we can build a very large graph of microservices via topics and queues to fulfill our business requirements.
307 Figure 5-12. A credit card application processing flow using the Pipe and Filter pattern
How does the Priority Queue Pattern work?
This pattern combines multiple queues, as in the Producer and Consumer pattern, to enable prioritized event processing. We achieve this by building a polling client that uses multiple event queues to process events based on priority.
311 Figure 5-13. Priority-based customer-request handling
