Chapter 5, Event-Driven Architecture Patterns Flashcards
What are the different event delivery guarantees provided by Message brokers?
- At-most-once delivery
The event is delivered to the consumer only once or not at all. If the consumer is not online during a delivery attempt or if network failures occur, the consumer will not get the event. Most important, the message broker will not try to send the same event again. - At-least-once delivery
The event is guaranteed to be delivered to the consumer. However, the consumer may consume the same event multiple times because if the message broker does not get an acknowledgment from the consumer for the event delivery, it will assume that the consumer did not receive the event and will resend it. In this case, the consumer should be intelligent enough to handle duplicate events.
Unfortunately, we cannot achieve an exactly once delivery guarantee, which ensures that the event is delivered to the consumer once and only once, because of the uncertain nature of the network and systems.
265
What does exactly once processing ensures?
exactly once processing ensures that the event is processed once and only once.
266
Describe a way of achieving exactly once processing
We can also achieve exactly once processing when events are idempotent: the outcome of receiving the same event multiple times is no different from receiving the event only once.
266
What are the two main Message Broker Categories?
- Standard (store-backed) message brokers
These are the standard message brokers that store events in a data store to enable serving to intended consumers. Most important, they purge events from their store upon delivery to consumers. Apache ActiveMQ and RabbitMQ are examples of these brokers. - Log-based message brokers
These brokers store events in commit logs. The events persist even beyond their being consumed. Therefore, these brokers allow consumers to replay events from a previous point in time. Apache Kafka and NATS are examples of this type.
Regardless of the category, different message brokers, and at times even the same message broker, can support various delivery guarantees.
267
Describe the Producer-Consumer Pattern
The Producer-Consumer pattern enables producer applications and consumer applications to communicate asynchronously by using event queues. The queue manages which consumer processes which event, and which procedures need to be followed when the consumers fail during the processing of the event.
269
How does the Producer-Consumer Pattern work?
This pattern requires an intermediate message queue that is managed by a message broker (Figure 5-1). One or more producers can send events to the queue. The message broker typically persists the queued events in a durable store, thereby guaranteeing that the events will eventually be delivered to consumers. The message broker then delivers one event at a time, mostly following FIFO order, to the consumers on request. This helps consumers process events as they have capacity and to not become overloaded.
When consumers are done processing the event, they can also send an acknowledgment to the message broker, so that the message broker can purge the event from its store.
269 Figure 5-1. Event delivery from producers to consumers
How is Producer-Consumer Pattern used in practice?
This pattern can be used for various scenarios, such as asynchronously delivering events from one application to another, making sure only one application processes a piece of data, ensuring event delivery, sharing workload among applications, handling sudden bursts of events, and decoupling applications.
270
What are some related patterns to the Producer-Consumer Pattern?
- Publisher-Subscriber pattern
Can send the same event to multiple consumers for processing. - Fire and Forget pattern
Used when events need to be delivered to a single consumer with an at-most-once delivery guarantee without the help of a message broker. - Store and Forward pattern
Allows asynchronous at-least-once event delivery without a message broker.
274
Describe the Publisher-Subscriber Pattern
The Publisher-Subscriber pattern enables applications to communicate asynchronously by using topics. The topic delivers every event to every subscriber.
275
How does the Publisher-Subscriber Pattern work?
This pattern uses topics to propagate the events from publishers to subscribers. The topic is a message broker concept. Multiple publishers can submit events to a topic hosted in the message broker (Figure 5-3). The topic then publishes all those events to all its subscribers, and makes sure that every subscriber receives all incoming events.
The standard behavior of the pattern is best effort: the events are delivered at most once. When a subscriber misses an event due to unavailability or network failure, they will never receive the event. But we overcome this problem with a durable subscription, which guarantees that all messages are delivered to all the consumers at least once, accounting for subscribers who are temporarily unavailable.
275 Figure 5-3. Event delivery from publishers to multiple subscribers
How is the Publisher-Subscriber Pattern used in practice?
This pattern is used for scenarios such as broadcasting notifications in parallel to multiple recipients with both the best effort and at-least-once delivery guarantees. Let’s see how we can achieve these.
276
What are some related patterns to the Publisher-Subscriber Pattern?
The Publisher-Subscriber pattern is related to the preceding Producer-Consumer pattern, which provides the capability to send an event to only one consumer for processing.
279
Describe the Fire and Forget Pattern
The Fire and Forget pattern enables clients (producers) to send events to respective consumers (services) with an at-most-once delivery guarantee without the use of a message broker.
279
How does the Fire and Forget Pattern work?
Let’s imagine that a weather sensor periodically sends current temperature and humidity readings to a weather-prediction service hosted in the cloud. Because of technical limitations, instead of using a message broker, as depicted in Figure 5-5, it is designed to invoke the API of the service by using protocols such as HTTP. The client is interested only in whether the server received the events, and is not interested in the final outcome. When a client publishes an event to the service, it expects only an acknowledgment of the event by a relevant HTTP status code such as 202 Accepted.
279 Figure 5-5. Fire and Forget event delivery from client to service
How is the Fire and Forget Pattern used in practice?
This pattern is useful when we need best-effort delivery of noncritical data, or when the receiving service does not possess the capability to subscribe or pull events from a client.
280
Describe the Store and Forward Pattern
The Store and Forward pattern enables clients to send events to services with an at-least-once delivery guarantee. As with Fire and Forget, this pattern does not use message brokers but uses APIs to directly send events.
282
How does the Store and Forward Pattern work?
This pattern requires a complex client design to achieve the at-least-once event delivery guarantee. The client in this pattern first persists the events to a durable store, such as a database or queue in a message broker, before attempting to send them to the service (Figure 5-6). Upon successful event delivery, the client purges the events from the store. If delivery is unsuccessful, it retries to send the event. During this, as the client receives more events to send, it persists them to the store. Once the connection to the service is established, it will deliver all pending events, receive acknowledgment of the event consumption, and purge the events from its store.
282 Figure 5-6. Store and Forward event delivery from client to service
How is the Store and Forward Pattern used in practice?
This pattern is useful when delivering critical data with a message broker, or when the receiving service cannot subscribe or pull events from the client.
283
What are some related patterns to the Store and Forward Pattern?
- Fire and Forget pattern
Publishes events to service endpoints with an at-most-once delivery guarantee. - Producer-Consumer pattern
Allows events to be delivered to a single consumer with higher delivery guarantees when the consumer can subscribe to an event queue.
285
Describe the Polling Pattern
The Polling pattern enables clients such as web browsers to initiate a long-running job, periodically checking completion.
285
How does the Polling Pattern work?
The frontend client or browser sends a request to initiate the process, such as insurance claim processing (Figure 5-7). Because the processing takes time, the backend service immediately sends an acknowledgment stating that it has accepted the request and initiated the asynchronous job processing. Along with the acknowledgment, it sends a job ID and, potentially, an estimated time of job completion. Based on this information, the client periodically queries the backend to check if the claim processing has completed. Upon completion, the backend returns the results as part of the response to the query, or provides a redirection to an endpoint containing the results.
285 Figure 5-7. Frontend client/browser repeatedly calling backend service to retrieve asynchronous job results
How is the Polling Pattern used in practice?
This pattern is useful when we need to retrieve the result of an asynchronous job without using a subscription or callback.
286
What are some related patterns to the Polling Pattern?
- Producer-Consumer pattern
This is an alternative used when the participating applications publish and subscribe to a queue. - Request Callback pattern
This is also an alternative that’s used when the clients and services are capable of using WebSockets or webhooks.
288
Describe the Request Callback Pattern
The Request Callback pattern enables applications to communicate asynchronously. The application provides the callback information with the request so responses can be delivered to the given callback.
289
How does the Request Callback Pattern work?
The Request Callback pattern enables applications to communicate asynchronously. The application provides the callback information with the request so responses can be delivered to the given callback.
289
Describe how WebSockets are used in the Request Callback Pattern
To use WebSockets, both client and service should have the capability to communicate via the WebSocket protocol (Figure 5-8). The client initiates the connection to the service and establishes a long-running connection. Both the client and service persist the connection and communicate by sending events. This approach is used for clients requesting information via an event and waiting on the service to respond, or for exchanging multiple events. WebSocket is an HTTP-based technology, but HTTP2 and gRPC also provide similar callback-based communication.
289 Figure 5-8. Client and service communicating via the WebSocket protocol
Describe how Webhooks are used in the Request Callback Pattern
In this approach, the client application issues a request and has the response delivered to a callback endpoint (Figure 5-9). The client sends the request with a callback URL. If the callback URL is consistent, we configure that on the service side, so we do not have to redundantly send the URL with the request. The response, when generated, is delivered to the callback URL.
290 Figure 5-9. Client and server communicating via webhook
How is the Request Callback Pattern used in practice?
We use this pattern to deliver responses asynchronously or when we need to receive continuous updates. This pattern mimics registering our telephone number as a callback so that the insurance agent can call to inform us of the status of our insurance claim. Following are some use cases in practice.
291
What are some related patterns to the Request Callback Pattern?
- Store and Forward pattern
Complements this pattern by providing guaranteed callback event delivery (covered previously in this chapter). - Polling pattern
Provides an alternative when applications cannot establish callbacks (covered previously in this chapter). - Asynchronous Request-Reply pattern
An alternative approach to communicate asynchronously by using a message broker.
292
When to use the Producer-Consumer pattern
There is a particular event to be consumed and processed by only one of the available consumers.
We cannot confirm the availability of the consumers and producers.
We see burst event production over a short period.
We need to ensure fairness in the processing of events.
293
When not to use the Producer-Consumer pattern
There is continuous high traffic, and the incoming event rate is much higher than the consumed event rate.
Message brokers cannot be used in the solution, and producers or consumers cannot connect to a message broker.
293
What are the benefits of using the Producer-Consumer pattern
Delivers an event to one consumer without duplicating the events.
Can tolerate availability problems in consumers and producers.
Handles spikes/bursts in traffic.
293
When to use the Publisher-Subscriber pattern
In a notification delivery system.
An at-most-once delivery guarantee is tolerated by subscribers. (It’s possible to miss the events if the subscriber is not available at the time of event notification.)
We need selective delivery of events to the subscribers.
293
When not to use the Publisher-Subscriber pattern
You cannot tolerate any missed events by the subscriber.
Message brokers cannot be used in the solution, and producers or consumers cannot connect to a message broker.
293
What are the benefits of using the Publisher-Subscriber pattern
Helps build an independent and decoupled system that can publish and subscribe.
Scales well with multiple subscribers interested in the same topic.
293
When to use the Fire and Forget pattern
At-most-once delivery guarantee is tolerated.
Dropping events is acceptable, such as when delivering non-business-critical events.
The consumer cannot pull any updates from the message broker.
Only a set of consumers is available to be notified.
293
When not to use the Fire and Forget pattern
Any issues in delivering or processing business-critical events cannot be ignored.
Message brokers can be included in the system, and producers and consumers can connect to a message broker.
293
What are the benefits of using the Fire and Forget pattern
No need to have a message broker to transfer messages from producer to consumer.
Simple to implement and no need to have additional deployment and maintenance complexities.
293
When to use the Store and Forward pattern
At-least-once delivery is required when both publisher and consumer are online and reachable at any time.
Message brokers cannot be used in the solution.
The consumer cannot pull any updates from the message broker.
293
When not to use the Store and Forward pattern
Message brokers can be included in the system, and producers and consumers can connect to a message broker.
293
What are the benefits of using the Store and Forward pattern
No need to have a message broker to transfer messages from producer to consumer.
293
When to use the Polling pattern
Clients do not have the capability to subscribe to a message broker or to expose an endpoint to receive updates from the backend system.
The service does not have the capability to call other endpoints upon completion of a job.
You have long-running jobs.
293
When not to use the Polling pattern
You have short jobs where success/failure can be reported immediately.
The applications can support callbacks such as webhooks, or WebSocket for communication.
293
What are the benefits of using the Polling pattern
Executes a long-running job and gets the response without having an additional infrastructure.
293
When to use the Request Callback pattern
Handling the request can take more time than the typical connection time-out of a standard request.
Clients are expecting updates from the services on one or more jobs.
The applications have the capability to communicate by using WebSocket, or clients have the capability to expose a callback URL and services can call that URL to send updates.
293
When not to use the Request Callback pattern
Applications do not have the capability to communicate via WebSocket, or the clients cannot provide a callback.
293
What are the benefits of using the Request Callback pattern
Executes a long-running job and gets the response without increasing the traffic to service to continuously check for updates.
More scalable approach, as updates are sent when the job is completed.
293
Describe the Event Sourcing Pattern
The Event Sourcing pattern enables us to store all changes to the application state as a sequence of events. This pattern not only is used to re-create application state at various points in time, and with different domain models, but also serves as an audit history to illustrate how we ended up in the current application state.
295
How does the Event Sourcing Pattern work?
Every time an event updates the application state, the event is also recorded in a persistence store in the order of operation. Figure 5-10 shows a banking use case, in which transactions are performed on an account. Events occur, such as Bob depositing $110, withdrawing $20, and then again depositing $50.
If we consider only the current state of the application, we know only that Bob now has a balance of $140 in his account, but we do not know the events that led up to this. Now, as each event is stored in a persistence store (or event log), in the sequence of its occurrence, we can gain an understanding of how his account balance changed over time.
Events are usually stored on a commit log, such as Apache Kafka, which allows us to read the events back from any point in time in a sequential manner by passing event sequence IDs. These events are read again by the application or other systems so that they can process the events that have occurred or re-create an application state with either a subset or all events.
296 Figure 5-10. Current account microservice generating an event log while updating application state
How is the Event Sourcing Pattern used in practice?
The Event Sourcing pattern unlocks the ability to system time-travel, build different domain models based on the same sequence of events, re-create a failed application state, run temporal queries, and replay events.
297
What are some related patterns to the Event Sourcing Pattern?
- Periodic Snapshot State pattern
Used to generate data store snapshots so that application state can be rebuilt much faster. This pattern is discussed in Chapter 6. - CQRS pattern
Used to store commands so that multiple applications can be built to serve queries. This pattern is discussed in Chapter 4. - Materialized View pattern
Used to store data so that it can generate materialized views based on the events. This pattern is discussed in Chapter 4.
301
When to use the Event Sourcing Pattern?
Multiple applications use the same data and need different domain models.
Application state should be rebuilt.
Temporal queries need to be executed in historical data.
The system needs to time-travel and change past event occurrences.
We need to keep track of audit information.
302
When not to use the Event Sourcing Pattern?
The data model is simple, and the consuming application can query for the intended data.
The event schema changes in a continuous manner.
We need all consuming applications to have data in a consistent state at all times.
302
What are the benefits of using the Event Sourcing Pattern?
Allows consumers to build application state optimized for their domain models and access patterns.
Replicates the data into multiple applications, thereby increasing availability.
Supports system recovery with event replay.
302
Describe the Mediator Pattern
The Mediator pattern provides centralized management of event orchestration. The mediator will not only understand and route events but also orchestrate events in sequential and parallel order across applications, while also handling failures. This pattern helps keep the coordination logic in a central location, allowing us to more simply change the behavior of the system.
303
How does the Mediator pattern work?
The key element of this pattern is the mediator microservice runtime built as part of the cloud native application. It operates by interacting with all the microservices that integrate, via event queues, topics, and APIs. It connects on various protocols and transforms events for applications accordingly. These mediator microservices are usually stateless, and might need to perform only filtering, sorting, and event transformations. But when required, they can coordinate sequential and parallel tasks.
Mediators sometimes need to combine events from multiple systems, and so need to be stateful.
303 Figure 5-11. Mediator microservice orchestrating a new insurance request
How is the Mediator pattern used in principle?
We can use this pattern to sort and distribute events, split an event into multiple subevents, process those events in various tasks in both parallel and sequential order, and finally combine the results to generate an output.
305
What are some related patterns to the Mediator pattern?
- Pipe and Filter pattern
Provides a decentralized approach to orchestrate events across applications. - Event-delivery patterns
The Mediator pattern uses event-delivery patterns to communicate among applications.
307
Describe the Pipe and Filter Pattern
The Pipe and Filter pattern orchestrates events in a decentralized manner. It uses multiple event queues and topics to streamline the event flow across multiple microservices in a cloud native application.
307
How does the Pipe and Filter Pattern work?
This pattern uses event queues and topics to connect microservices. With this approach, we can build a very large graph of microservices via topics and queues to fulfill our business requirements.
307 Figure 5-12. A credit card application processing flow using the Pipe and Filter pattern
How is the Pipe and Filter Pattern used in practice?
Use this pattern when you need to build large-scale asynchronous systems that are managed by multiple teams.
308
What are some related pattern to the Pipe and Filter Pattern?
- Mediator pattern
Provides a more centralized approach for orchestrating event flow in event-driven architecture. This pattern was covered previously in this chapter. - Event-delivery patterns
The Pipe and Filter pattern uses event-delivery patterns, also covered in this chapter, to communicate among microservices. - Saga pattern
Uses this pattern to implement data processing pipelines that can support compensation transactions. Chapter 3 describes this pattern.
310
Describe the Priority Queue Pattern
The Priority Queue pattern handles events based on their priority so that high-priority events are handled first, while low-priority events are processed as capacity allows.
311
How does the Priority Queue Pattern work?
This pattern combines multiple queues, as in the Producer and Consumer pattern, to enable prioritized event processing. We achieve this by building a polling client that uses multiple event queues to process events based on priority.
311 Figure 5-13. Priority-based customer-request handling
How is the Priority Queue Pattern used in practice?
This pattern is needed when we want to preferentially handle some events over others or have insufficient capacity to process all events.
312
What are some related patterns to the Priority Queue Pattern?
- Publisher-Subscriber pattern
With subscription filters, this can provide an alternative design to implement the Priority Queue pattern. - Producer-Consumer pattern
Can be used when prioritization is not necessary.
314
When to use the Mediator pattern?
To simply sort the events among multiple subprocesses.
To split an event into multiple events based on the use case.
Tasks need to be performed in a sequential or parallel order.
The system undergoes rapid changes, and using the Pipe and Filter pattern requires more overhead during each change.
314
When not to use the Mediator pattern?
Central control for orchestration is not necessary.
314
What are the benefits of using the Mediator pattern?
Provides the central system for orchestration, so change management is relatively easy.
314
When to use the Pipe and Filter pattern?
To provide segregation of duties among multiple teams.
To add/remove business logic into multiple stages of processing without impacting other teams.
314
When not to use the Pipe and Filter pattern?
The overall flow of events is changed frequently.
Central control for orchestration is necessary.
314
What are the benefits of using the Pipe and Filter pattern?
Allows seamless addition and removal of applications to the pipeline.
Increases decoupling and reduces impact among multiple teams.
314
When to use the Priority Queue pattern?
To treat one type of an event with urgency compared to another.
Resources are constrained, and we can process only a subset of events.
314
When not to use the Priority Queue pattern?
There is no strong need for priority-based processing.
314
What are the benefits of using the Priority Queue pattern?
Better utilization of resources based on priority of jobs.
314
