Chapter 5, Event-Driven Architecture Patterns gpt

Question 1

What are the key characteristics and benefits of event-driven architecture?

Answer 1

Event-driven architecture is a software paradigm that promotes the generation, detection, consumption, and reaction based on events. It enables the building of distributed and scalable cloud-native applications. Unlike service composition patterns, which are mostly synchronous, event-driven architectures are asynchronous. This decoupled approach simplifies scaling and is fundamental for designing large-scale distributed cloud-native applications.
Page 262

Event-driven architecture is a software architecture paradigm that promotes generation, detection, consumption, and reaction based on events. An event-driven architecture allows us to build distributed and scalable cloud native applications. In contrast to the service composition patterns, which are mostly synchronous in nature, event-driven architectures are asynchronous. They provide a clean and decoupled way of designing cloud native applications, allowing simpler scaling, and are one of the fundamental architectures for building large-scale distributed cloud native applications.

Question 2

How does event-driven architecture differ from traditional service composition patterns in terms of communication style?

Answer 2

Event-driven architecture differs from traditional service composition patterns by being asynchronous, whereas service composition patterns are mostly synchronous. This asynchrony in event-driven architecture allows for a cleaner and more decoupled design, facilitating simpler scaling and more efficient handling of distributed systems.
Page 262

Event-driven architecture is a software architecture paradigm that promotes generation, detection, consumption, and reaction based on events. An event-driven architecture allows us to build distributed and scalable cloud native applications. In contrast to the service composition patterns, which are mostly synchronous in nature, event-driven architectures are asynchronous. They provide a clean and decoupled way of designing cloud native applications, allowing simpler scaling, and are one of the fundamental architectures for building large-scale distributed cloud native applications.

Question 3

What role do events play in sharing information within an event-driven architecture?

Answer 3

Events are used for sharing information between applications in an event-driven architecture. Typically, the application generating the event notification does not expect a direct response, allowing the consuming application to decide how to process the information. This decoupling of the sender and receiver enhances scalability and flexibility.
Page 262

Events are used for sharing information. In most cases, the application generating the event notification does not expect any response, and it lets the consuming application decide what to do with that information. Even if the applications generating the event notification expect a response, they expect it only indirectly.

Question 4

How do applications generating event notifications handle the need for responses in an event-driven architecture?

Answer 4

Even when applications generating event notifications expect a response, they anticipate it indirectly rather than through direct synchronous communication. This allows for greater flexibility and decoupling between the producer and consumer of the event.
Page 262

Events are used for sharing information. In most cases, the application generating the event notification does not expect any response, and it lets the consuming application decide what to do with that information. Even if the applications generating the event notification expect a response, they expect it only indirectly.

Question 5

How can significant occurrences or changes in a system state be categorized in an event-driven architecture?

Answer 5

Significant occurrences or changes in a system state can be categorized as events. For example, depositing $50 into an account is considered an event. These events can trigger notifications and actions in other systems.
Page 262

Things to note: Events can be categorized as any significant occurrence or change in a system state. Let’s take an example of depositing $50 into Bob’s account. Now, Bob’s account balance has increased by $50; this incident is considered an event.

Question 6

How can the occurrence of an event be communicated to other systems in an event-driven architecture?

Answer 6

The occurrence of an event, such as a deposit, can be sent to other systems, like sending a notification to a user’s cell phone. This is typically done through asynchronous messages that contain the event occurrence information.
Page 262

The occurrence of this event can be sent to other systems, such as to Bob’s cell phone, as a notification. The event notification is typically an asynchronous message produced and transmitted with the event occurrence information.

Question 7

What is the relationship between events and event notifications in an event-driven architecture?

Answer 7

In event-driven architecture, the term event is often used interchangeably to denote the message that notifies the event. This is because the architecture relies on an asynchronous messaging infrastructure to identify and communicate event occurrences.
Page 262

Though events just occur and do not travel, the term event is also used interchangeably to denote the message that notifies the event. This is mainly because event-driven architectures are built on top of an asynchronous messaging infrastructure that uses messages to identify and communicate event occurrences.

Question 8

Why are intermediate systems like message brokers often needed in event-driven architectures?

Answer 8

Intermediate systems such as message brokers are often needed in event-driven architectures to consume, store, and deliver events to their consumers. They ensure that no events are lost across system failures and help manage the decoupled nature of event communication.
Page 265

in event-driven architecture, we cannot always directly send messages to consumers and get an acknowledgment that they have consumed them successfully. In most cases, we need intermediate systems such as message brokers to consume, store, and deliver events to their consumers while ensuring that no events are lost across system failures.

Question 9

What role do defined schemas play in event-driven architectures, especially for cloud-native applications?

Answer 9

Defined schemas for events are crucial in event-driven architectures to ensure seamless interoperability between event producers and consumers. For cloud-native applications, using the CloudEvents specification to define the structure of the event payload is recommended.
Page 268

Like APIs and data, events should have defined schemas. This helps event producers and consumers interoperate seamlessly. When defining events for cloud native applications, we recommend using the CloudEvents specification to define the structure of the event payload.

Question 10

How does the Producer-Consumer Pattern function in asynchronous event delivery?

Answer 10

In asynchronous event delivery, the Producer-Consumer Pattern involves passing events to another application without blocking the producer. This ensures that events are processed by one of the available consumers, allowing for efficient event handling without direct interaction between the producer and consumer.
Page 270

Provide asynchronous event delivery: In asynchronous event delivery, the most common use of the Producer-Consumer Pattern, we pass events to another application without blocking.

Question 11

How does the Producer-Consumer Pattern ensure that each event is processed by only one consumer?

Answer 11

The Producer-Consumer Pattern uses event queues to ensure that each event is processed by only one of the available consumers. This prevents multiple consumers from processing the same event, maintaining efficient and accurate event handling.
Page 270

Process each event by a single application: We can use the Producer-Consumer pattern when we need events to be consumed and processed by only one of the available consumers. Using event queues ensures that events are not delivered to multiple consumers.

Question 12

How do message brokers guarantee event delivery in the Producer-Consumer Pattern?

Answer 12

Message brokers guarantee event delivery by using acknowledgments. When a broker receives an event, it durably persists it and sends an acknowledgment to the producer. Similarly, it expects an acknowledgment from the consumer upon event delivery. If no acknowledgment is received, the broker attempts to redeliver the event.
Page 271

The message broker uses acknowledgments to guarantee that the events are delivered. When the broker receives an event, it persists that durably and then sends an acknowledgment to the producer, stating that it has successfully consumed the event. The producer is updated to wait for the acknowledgment from the message broker when it publishes an event. Similarly, when the message broker delivers the message to the consumer, it expects an acknowledgment, and therefore the consumer is updated to send an acknowledgment when it receives an event. If the message broker does not receive the acknowledgment, it will try to redeliver the event, and so the consumer can get duplicate events.

Question 13

How can the Producer-Consumer Pattern handle sudden bursts of events?

Answer 13

The Producer-Consumer Pattern can handle sudden bursts of events by buffering them in a queue. This allows for the storage and gradual processing of events without overwhelming the consumers, ensuring that the system remains stable and efficient during high-volume periods.
Page 272

Handle sudden bursts of events
We can also use the Producer-Consumer Pattern to buffer event bursts over a short period. We can queue and process events without extensively scaling consumers. In the example in Figure 5-2, a data processing organization periodically pulls logs from servers for processing, resulting in a burst of logs every time we fetch a new log file. If we publish the burst of logs directly to the log processors, they will become overloaded and fail. But by buffering with a queue, we store and process the logs at the capacity of the log processors. Further, as they are using a pull-based approach, the consumers will not be overloaded as excess events are buffered at the event queue.

Question 14

How does the Producer-Consumer Pattern share workloads among multiple workers?

Answer 14

The Producer-Consumer Pattern shares workloads among multiple workers by allowing multiple log processors to consume events from the message broker in parallel and at their own pace. This ensures that events are processed quickly and that no processor remains idle while there are events in the queue. The pattern also uses FIFO semantics to prioritize the oldest events, mimicking real-world queue behavior.
Page 272

the Producer-Consumer Pattern can be used to share the workload among multiple workers. Taking the previous example of processing log events, we have multiple log processors consuming events from the message broker in parallel and at their own speed. As the log processors are competing to pull events, we can ensure that events are processed as soon as possible. This also ensures that events are processed as long as at least one log processor is available, and no log processor will idle as long as events remain in the queue. the Producer-Consumer Pattern also allows us to prioritize the oldest events by using FIFO semantics. This mimics queues in the real world: the first person in the queue will be served first. The Producer-Consumer pattern allows us to model the same event-processing semantics on cloud native applications.

Question 15

What are some considerations when using message brokers in the Producer-Consumer Pattern?

Answer 15

When using message brokers in the Producer-Consumer Pattern, it is important to ensure at-least-once delivery guarantees, separate queues for different use cases, and handle bursts effectively. If the input rate consistently exceeds the consumption rate, it is necessary to scale the number of consumers to increase processing capacity.
Page 273

Considerations: Most message brokers provide support for an at-least-once delivery guarantee. If the consumer fails to acknowledge the event delivery or times out, the broker will send that event to another consumer, making sure that the event is processed by at least one consumer.
A single message broker can host multiple event queues. We recommend that each use case and operation has its own queue; for example, events updating customer information might have their own queue, while events updating payments will have their own queue. Mixing different events in a single queue requires us to improve consumers to differentiate between events and handle them appropriately; this complicates the design, and slows performance as the number of events they receive increases.
When using queues, we have to be careful to handle bursts. While queues buffer events, there should be enough capacity to process all produced events over a certain time period. If we are constantly getting bursts of events (as in the log processing example in Figure 5-2), and if the input rate to the queue is consistently higher than the consumption rate, the Producer-Consumer Pattern fails. To mitigate this risk, we should scale the number of consumers to increase consumption capacity.
The Producer-Consumer Pattern helps us build decoupled systems and allows us to independently add and remove producers and consumers, scaling overall event processing.

Question 16

What is the main use of the Publisher-Subscriber Pattern in event-driven architecture?

Answer 16

The Publisher-Subscriber Pattern is primarily used for broadcasting information to multiple subscribers. For example, it can send notifications of tweets with a specific hashtag to all users who have subscribed to that hashtag, allowing efficient dissemination of information to interested parties.
Page 276

Broadcast events: the Publisher-Subscriber Pattern is ideal for broadcasting information. For example, let’s assume we are building a system like Twitter, and we can use the Publisher-Subscriber Pattern to send a notification of all tweets published about sports with #sports (the topic) to all users who have subscribed to that hashtag. The main advantage of the Publisher-Subscriber Pattern is that it enables us to notify all interested subscribers, instead of only a single user or application.

Question 17

What delivery guarantee is typically associated with events produced by publishers in the Publisher-Subscriber Pattern?

Answer 17

Events produced by publishers in the Publisher-Subscriber Pattern typically have the at-most-once delivery guarantee. This means that subscribers may not receive events published during an outage, which is acceptable for scenarios where missing some updates is not critical, such as periodic weather updates.
Page 276

Deliver events with best effort: Typically, events produced by publishers have the at-most-once delivery guarantee. For instance, subscribers do not receive events published during an outage. Though you risk losing events, this approach is still useful for scenarios such as status updates—like publishing current weather periodically to the weather topic so subscribed people can plan what to wear when they go out. Even when some updates are missed, the subscribers can appropriately take corrective decisions based on future events.

Question 18

When is the event delivery with best effort not suitable in the Publisher-Subscriber Pattern?

Answer 18

Event delivery with best effort is not suitable when missed events impact the function of the system. For example, tracking a topic to find out the closing date to apply for a state exam requires that subscribers receive all relevant events, which can be ensured with durable subscriptions to prevent missing critical information.
Page 277

Make sure all events are delivered to all subscribers: Event delivery with best effort is not suitable when missed events impact the function of the system. For example, someone might be tracking a topic to find out the closing date to apply for a state exam. A subscriber who misses that event will miss the deadline and miss the chance to sit for the exam. You can use durable subscriptions to ensure that subscribers receive missed events when they are online again.

Question 19

How can the Publisher-Subscriber Pattern deliver events selectively to subscribers?

Answer 19

The Publisher-Subscriber Pattern can deliver events selectively using hierarchical topics and filtering logic. Subscribers can choose to receive events from specific subcategories, such as sports news, by subscribing to only the relevant subcategory or by using filtering conditions.
Page 277

Selectively deliver events to subscribers: the Publisher-Subscriber Pattern is useful when only specific events require delivery, in which case the most common approach is to leverage hierarchical topics. The topic name has a hierarchy such as news, news/sports, and news/politics. If the subscriber subscribes to the news topic, they will consume all events from news and all its subcategories, including news/sports and news/politics. But if they are interested in only sports news, they can subscribe to only news/sports.
An alternative approach for achieving selective delivery is to use filtering logic. A filtering condition such as news==sports is passed to the message broker upon subscribing to the topic, so that the message broker publishes only the sport news events to the subscriber.

Question 20

How can the Publisher-Subscriber Pattern be used to share workloads among multiple workers?

Answer 20

The Publisher-Subscriber Pattern can replicate and distribute events across multiple workers using unique client IDs. This ensures that all events are delivered to subscribers and distributed among instances of services to share the processing workload, balancing the load and enhancing efficiency.
Page 277

Share workloads: the Publisher-Subscriber Pattern can also be used to replicate and distribute events across multiple workers. In the example in Figure 5-4, we process periodically published weather data in real time. We need to publish every event to human subscribers (Alice and Bob) so they can keep track of the weather, and, in parallel, distribute events across distributed Weather Processing microservices to share and process the messages.
To achieve this, we use the client ID of the subscribers. Each subscriber has a unique client ID such as alice, bob, or P1. Since the topic keeps track of event delivery based on client IDs, it ensures that all events are delivered to both human subscribers with the client IDs alice and bob. We set the same client ID for both instances of the Weather Processing microservices to P1. By doing this, the topic publishes each event to only one of the microservice instances. This enables us to share the workload across those microservices.

Question 21

What are the considerations for ensuring event delivery during subscriber downtime in the Publisher-Subscriber Pattern?

Answer 21

To ensure event delivery during subscriber downtime, durable subscriptions or commit log-based message brokers such as Kafka or NATS should be used. These methods store and resend missed events to subscribers when they come back online, ensuring no loss of critical information.
Page 278

Considerations: When subscribers cannot miss events published during downtime, we need a durable subscription. The message broker takes the responsibility of storing and delivering the events to subscribers as they come back online. Each durable subscription can be viewed as a dedicated event queue for each subscriber.
Alternatively, we can also use a commit log–based message broker such as Kafka or NATS to receive missed events published during subscriber downtime. These message brokers store all events to commit logs. Because they do not remove the events even after a successful or failed delivery attempt, upon request they can resend/replay previous events to subscribers. To retrieve missed events, subscribers must persist the last-processed event sequence ID, and during restart, they request the message broker to replay all the events since the last-processed sequence ID. This allows the system to achieve a higher delivery guarantee.
the Publisher-Subscriber Pattern enables us to build decoupled systems and independently add or remove publishers and subscribers. Use the Producer-Consumer pattern when you need to share events among multiple consumers; but, as shown in Figure 5-4, you can use topics with subscriptions based on client ID to broadcast events to some subscribers while distributing the events across a subset of subscribers sharing the same client ID.

Question 22

When is the Fire and Forget Pattern particularly useful in event-driven architectures?

Answer 22

The Fire and Forget Pattern is useful when the client sends events to a third-party service that cannot subscribe and pull events from a message broker. This pattern is beneficial for services behind an API that consume events via HTTP, or for clients that cannot initiate a connection.
Page 280

Deliver events to systems that do not support subscription: The Fire and Forget Pattern can also be used when the client is sending events to a third-party service that does not possess the capability to subscribe and pull events from a message broker. Services owned by partner organizations usually are deployed behind an API. They have the capability to consume events only via protocols such as HTTP. We also use The Fire and Forget Pattern when the client is hosted in an internal network and the service cannot initiate a connection to the client.

Question 23

What are the considerations for using the Fire and Forget Pattern for events with depreciating value?

Answer 23

The Fire and Forget Pattern is ideal for events with depreciating value over time, such as current weather updates. It allows for quick discarding of outdated events and prioritizes processing more recent ones. If load balancing across multiple services is needed, it is recommended to use an intermediary like a network load balancer instead of adding complexity to the client.
Page 281

Considerations: The Fire and Forget Pattern is useful when processing events that have depreciating value over time, such as the current weather. These events become outdated when their processing is delayed. Instead, processing more-recent events is significantly more valuable. The Fire and Forget pattern enables us to quickly discard events and pick up the next one for processing. We should use the Producer-Consumer pattern if we do not want to discard events when they cannot be delivered.
Say we need the client to load-balance events across multiple services—for example, a single weather client sending events in a round-robin manner to five weather services. Here, we either have the weather client use an intermediary such as a network load balancer to route the events to available services, or improve the client to keep track of all the weather service endpoints and perform the load-balancing logic within itself. We do not recommend the latter approach, as it introduces additional complexity to the weather client.

Question 24

How does the Store and Forward Pattern ensure event delivery during service unavailability?

Answer 24

The Store and Forward Pattern ensures event delivery during service unavailability by storing all incoming events and delivering them to the services as they become available. This pattern is particularly useful when the availability of partner or third-party services cannot be controlled.
Page 284

Ensure event delivery during service unavailability: In the preceding example, our organization does not have control over the availability of the service hosted by partners and third parties. In this situation, we use the Store and Forward Pattern to store all incoming events and then deliver those events to the services as they become available.

Question 25

What are the best practices for using the Store and Forward Pattern when message brokers cannot be used?

Answer 25

When message brokers cannot be used, the Store and Forward Pattern should be implemented with a separate durable store for each client application when possible, or a common event queue for scalability. Additionally, leader election can be used to prevent duplicate event deliveries when multiple clients are involved.
Page 284

Considerations: We recommend the Store and Forward Pattern when message brokers cannot be used and when we need the events to be delivered with an at-least-once delivery guarantee. This is because using message brokers and adopting the Producer-Consumer pattern greatly simplifies the architecture and reduces the operational cost.
When using the Store and Forward Pattern, use a separate durable store (database or queue) for each client application, when possible, to greatly simplify the design. For scalability reasons, let’s say we use five clients to send the order events to third-party services. But instead of using a dedicated store for each client, we recommend using a common event queue for all five clients to store the events. This allows us to distribute the events among the clients when they try to send events to the services. This also helps overcome client failures, as now other clients can fetch and publish events that are supposed to be sent by the failed client.
If you decide to use databases as the durable store, and still want to use multiple clients for event delivery, you must solve the problem of deciding which client publishes which event. We need to prevent multiple clients loading the same order event from the database and delivering it to the service. This causes duplicate events and risks overloading the service. To overcome this, we elect a single client to deliver a particular subset of events (for example, based on the hash of the event order number). Here, client selection is determined via leader election by using services such as ZooKeeper. Additionally, the same client can also deliver multiple subsets of events (for example, event order number hashes 2, 5, and 7).

Question 26

How can frontend applications retrieve results from long-running backend processes?

Answer 26

When integrating frontend and backend applications, the Polling Pattern is used to repeatedly call the backend for results. This pattern is useful for long-running asynchronous tasks that exceed connection time-outs, allowing the frontend to check the backend for completion without maintaining a persistent connection.
Page 286

Retrieve results from a long-running process that cannot notify of job completion: When we integrate frontend and backend applications, usually only the frontend can initiate the connection to the backend, and backends cannot easily call frontends. When the backends are processing an asynchronous task, they can often take more time than the connection time-out, which means clients cannot stay connected to receive the results. In this case, we use the Polling Pattern to repeatedly call the backend for results.

Question 27

When is the Request Callback Pattern used in event-driven architectures?

Answer 27

The Request Callback Pattern is used when clients cannot subscribe to a message broker or expose an endpoint to receive updates from the backend. This pattern is ideal for scenarios where the client needs to be updated about the status or results of a long-running process.
Page 287

Deliver events to a client that cannot initiate subscription or callbacks: The Request Callback Pattern is used when clients cannot subscribe to a message broker or expose an endpoint to receive updates from the backend.

Question 28

What considerations should be taken into account when implementing the Request Callback Pattern?

Answer 28

When implementing the Request Callback Pattern, ensure the backend maintains the life cycle of the asynchronous job since frontend clients may fail and restart. To reduce resource waste from continuous polling, use long polling if the connection time-out is high and the network is stable. Prefer webhooks or WebSockets for more efficient communication if supported.
Page 287

Considerations: When implementing the Request Callback Pattern, we should ensure that the life cycle of the asynchronous job is maintained in the backend, because the frontend clients may fail and get restarted. For example, when the browser window is refreshed, the user should continue to get the correct status update of the insurance claim. Therefore, when the browser initiates the connection after refresh, the backend application should be able to correlate the new request to the previously initiated job, using the available information on the request, and return the appropriate response to the client.
You should keep in mind that polling backends in a continuous manner is a waste of resources for both the client and service, and adds delay to the response, as the backend service cannot inform the client until the next poll. The amount of continuous polling can be reduced by using the long polling technique: the service does not immediately send a response but holds the connection until the response is available, or until the connection times out. This reduces the number of polls and allows services to immediately respond when they have necessary data. We recommend using the long poll if the connection time-out between the client and service is reasonably high, the network is usually stable, and the service has the capacity to hold the request until connection time-out; otherwise, fall back to periodic polling.
We do not recommend using the Request Callback Pattern when the application supports callbacks such as webhooks or WebSockets for communication, because those options are efficient and much less resource intensive. Callback-based event delivery is discussed in detail next.

Question 29

How does GraphQL use WebSocket for real-time event updates?

Answer 29

GraphQL uses WebSocket with its subscription feature, allowing clients to connect to a service and listen to real-time events according to the GraphQL query submitted when subscribing. Clients continuously receive updates when the data in the service changes, and they can unsubscribe by sending a message to the server or due to errors or time-out.
Page 290

GraphQL uses WebSocket with its subscription feature, allowing clients to connect to a service and listen to real-time events according to the GraphQL query submitted when subscribing. Clients will then continuously get updates when the data in the service changes. A client can unsubscribe by sending a message to the server, or the server can unsubscribe due to errors or time-out.

Question 30

Why is the Request Callback Pattern ideal for asynchronous responses in certain scenarios?

Answer 30

The Request Callback Pattern is ideal for asynchronous responses when the service cannot respond within the connection time-out, such as in insurance claim processing. The backend service acknowledges the request immediately and delivers the results when the processing is completed, allowing the backend to send results instantly when necessary data becomes available.
Page 291

Deliver response asynchronously: The Request Callback Pattern is ideal when the service cannot respond within the connection time-out, such as in our insurance claim processing use case. The backend service acknowledges the request immediately, and delivers the results when the claim processing is completed. This also allows the backend to send the results instantly when the necessary data becomes available.

Question 31

When should WebSocket be chosen over webhooks for real-time updates?

Answer 31

WebSocket should be chosen over webhooks when the client and server need to asynchronously communicate by sending more than one message, such as continuously updating stock prices. Webhooks are better for single responses when the response time is uncertain or exceeds a few minutes.
Page 291

Deliver updates continuously
Though we can use the Request Callback Pattern to update the insurance claim status in the browser, the Request Callback Pattern is much more useful when we need real-time updates, such as monitoring stock prices. The browser establishes a WebSocket connection with the backend service to receive the latest updates and dynamically render them in the web page. The Request Callback Pattern can be used only when the client can receive the response from the server leveraging the WebSocket protocol, or by exposing an endpoint to be used as a callback.
When implementing the Request Callback Pattern, the callback does not need to be an HTTP endpoint; it can be an email address, an event queue, or an event topic. We can also model the Request Callback Pattern so that when the service processing is done, instead of calling the client, the service calls another service to process the results based on the callback information provided by the client. When the result is big, the service can also store results in a durable store such as in Amazon S3, and then pass that URL to the callback so that the client can load the processed data.
The webhooks typically provide only an at-most-once delivery guarantee, as the service has to drop the response events when the callback is not available or if a network failure occurs. We can improve this to an at-least-once delivery guarantee by incorporating the Store and Forward pattern when delivering events to callbacks or by using a message broker when the participating applications have the capability to communicate via a message broker.
We recommend choosing WebSocket over webhooks when the client and server need to asynchronously communicate by sending more than one message. This is because WebSocket keeps the connection live throughout the communication, and reduces the cost of sending each new message. Subscribing to a stock symbol and receiving continuous stock price updates is a good example for this. At the same time, we recommend using webhooks over WebSocket when the client is expecting only a single response, and when the response time cannot be determined or if the response can take more than a few minutes (for example, when expecting the outcome of an insurance claim).

Question 32

How does event sourcing help in re-creating application state after a system failure?

Answer 32

Event sourcing allows for re-creating an application state by replaying all the stored events in order. This process involves disabling external notifications during state rebuilding to prevent duplicate notifications, ensuring the accurate reconstruction of the application state.
Page 297

Re-create application state: Let’s assume the application state got corrupted during system failure. Since we have stored all the events corresponding to the state, we can simply replay all the events in order to re-create a specific application state. We also have to disable the system, prohibiting any notifications to external systems during state rebuilding. For example, while we are re-creating a state for Bob’s bank account, we should not send deposit and withdrawal notifications.

Question 33

How can event sourcing be used to build different domain models without tightly coupling services?

Answer 33

Event sourcing can be used to build different domain models asynchronously, allowing systems like rewards services to evolve independently from core applications. By using events stored in commit logs, specific data views can be created to support various access patterns and reduce load on the core system.
Page 297

Build different domain models: Let’s assume Bob is enrolled in a rewards program that gives points on purchases he makes at affiliated stores. The bank uses a different domain model to calculate and keep track of the points Bob has earned. Instead of tightly coupling the rewards service with the core transaction application, with the Event Sourcing pattern the bank can asynchronously build Bob’s points from the transactions stored in the commit logs. This not only allows us to build specific data views to support different access patterns to reduce load, but also helps consuming systems—such as a rewards service—to evolve and change at their own speed without interfering with the core banking system.

Question 34

What advantages does event sourcing provide for running temporal queries?

Answer 34

Event sourcing allows for running temporal queries on stored events, enabling the tracking of changes over time. For instance, a bank can determine if an account should be charged a monthly fee based on historical balance events, demonstrating a lossless architecture that supports time-travel and historical data analysis.
Page 298

Run temporal queries: The Event Sourcing Pattern allows us to run temporal queries on the stored events. For example, let’s say the bank charges a $5 monthly fee if the account balance falls below $100. Bob had $140 in his account at the end of the month. But just after the initial $20 withdrawal, his account had only $90. Hence, he will be liable for the $5 monthly fee. Without having the events in the commit log, achieving such use cases will become difficult. This visibility into the event log demonstrates a fully lossless architecture, unlocking the ability to time-travel and build various views based on historical data.

Question 35

How does event sourcing help correct mistakes in recorded transactions?

Answer 35

Event sourcing helps correct mistakes by allowing the replay of events to adjust the application state. For example, if a bank incorrectly records a transaction, the events can be replayed to revert the erroneous entry and correct the final state, ensuring accuracy and consistency.
Page 298

Replay events: The Event Sourcing Pattern also allows us to correct mistakes. Let’s say the bank has recorded that Bob withdrew $50 from his account on April 15, even though he successfully canceled the withdrawal. Because of this withdrawal entry, the bank also charged the $5 monthly fee on April 30. Assume that the bank found out about this mistake on May 3. By using the Event Sourcing pattern, the bank can replay all the transactions of Bob’s account starting April 15, correctly reverting the withdrawal and refunding Bob the erroneous $5 monthly fee charged on April 30.

Question 36

What are the key considerations when deciding between using the application state or the event log as the single source of truth in event sourcing?

Answer 36

When deciding between using the application state or the event log as the single source of truth, consider the durability and recovery time. If using a database for the application state, it can be the source of truth, with the event log used for auditing. If the state is kept in memory, the event log should be the source of truth due to its durability and ability to regenerate state by replaying events.
Page 298

Considerations: With the Event Sourcing Pattern, we need to decide whether the application state or the event log is going to be our single source of truth. If we are using a database to keep track of the application state, the database can be the source of truth, as it is durable. We can then use the event log only for auditing purposes and to generate other domain models. But at the same time, if we are keeping the state in memory (such as in a data structure, in-memory database, or cache), then we have to use the event log as the single source of truth, as we can always regenerate the state by replaying the events from the logs.
When using an event log as the source of truth, the recovery of system failure can take a long time; we need to re-create the application state by replaying all the events in the event log. To improve recovery time, we can periodically take application state snapshots, as we’ll discuss in Chapter 6, and during recovery, we can load the latest snapshot and replay only the events logged after that.
Performing event playback and re-creating an application’s state can be tricky, especially when the application is interacting with external services. If we need to stop the application from calling external services (such as notifying Bob again about his $50 deposit), we have to either make the service intelligent enough to know that it is performing a replay and so shouldn’t send any external calls, or gate the external services with APIs and drop the service calls at the APIs. If we are replaying events on multiple services and they need to communicate with one another, we recommend adding some sort of reference point, such as date and time, to their request when calling other services. For example, rather than requesting the current account balance, we can request the account balance on 03/23 at 11:15 a.m. In this case, the responding application will be able to always respond with the correct balance at the given time, increasing the consistency of the application during event playback.
When defining the events for a commit log, they should be modeled as change events. In the banking use case, we should use events that reflect some kind of change, such as deposit of $50 and withdrawal of $20, and not use events such as set bank balance to $150, or set bank balance to $130. If we reverse or remove the events during event playback, we will be able to get the corrected final balance.
When designing the service, we can either store the application state as simple objects and let the application operate on them, or we can model the state within the domain model itself. We recommend building the state within the domain model, as this gives us flexibility, especially when the processing logic is complex. But we cannot use this approach if we need to reverse the events. In this case, we would need to store the state after each update, and then revert to the previous state when a reversal is necessary. This can complicate the application architecture. If event reversal is needed, we recommend storing events separate from the application logic.
When building different domain models from the same event source, we should keep in mind that those models are usually built asynchronously and so can only be eventually consistent. This is because there can be network and application processing delays to write the events to the logs, and then other services to read and populate their application state. Hence, we should not use the Event Sourcing Pattern for use cases that do not tolerate eventual consistency. We can use log-based event queries with Apache Kafka and NATS as the event logs, for example, when building the Event Sourcing Pattern as they provide the capability to store events in order, and allow us to replay past events when necessary.
While the Event Sourcing Pattern allows us to time-travel, it also enforces restrictions in the event schema. For example, we can add new attributes to the event, but we cannot remove or update existing attributes. When the system replays previous events, they will not be compatible with the running application state. If we try to handle multiple versions of the event schema in the application code, the system can soon become very complex and difficult to maintain.

Question 37

How does the Mediator Pattern help in sorting and distributing events among multiple subprocesses?

Answer 37

The Mediator Pattern is used to sort events among multiple subprocesses, helping integrate decoupled producer and consumer applications. For instance, orders from a single queue can be sorted based on region, and mediators can format events and perform required protocol transformations.
Page 305

Sort and distribute events: The Mediator pattern is used to sort events among multiple subprocesses. For example, as we are getting orders from a single queue, the mediator sorts the orders based on region. This also helps integrate decoupled producer and consumer applications, especially when they are external. In this case, mediators format the events and perform required protocol transformations.

Question 38

How can the Mediator Pattern be used to split events into multiple subevents?

Answer 38

The Mediator Pattern can split one event into multiple subevents to be sent to different systems. For example, receiving a new insurance application event can trigger subevents for address verification, credit verification, and other subtasks.
Page 306

Split events into multiple subevents: The Mediator pattern is also used to split one event into multiple events. For example, when we receive a new insurance application event, we need to split it into various subevents and send them to multiple systems to perform subtasks such as address and credit verification.

Question 39

How does the Mediator Pattern ensure the execution order of tasks?

Answer 39

The Mediator Pattern ensures the execution order of tasks by performing some tasks sequentially while allowing others to be executed in parallel. This enables the combination of multiple parent task results before executing a dependent task, ensuring proper task coordination.
Page 306

Ensure task execution order: The mediator pattern is used to perform some tasks sequentially while others are executed in parallel. This enables us to combine the results of multiple parent tasks before executing a dependent task.

Question 40

What are the considerations when deciding between using the Mediator Pattern and the Pipe and Filter Pattern?

Answer 40

The Mediator Pattern should be used when central control for orchestration is required, especially if the system undergoes rapid changes. It allows for modifying the mediator to change the integration logic and operation flow. The Pipe and Filter Pattern, however, is better suited for building large-scale asynchronous systems with independent teams and should be used when central control is not necessary.
Page 306

Considerations: Use the Mediator pattern instead of Pipe and Filter when the system is undergoing rapid changes. The Mediator pattern enables us to change the integration logic and operation flow by modifying the mediator. In the Pipe and Filter pattern, we need to update multiple applications and queues to perform the change. As the mediator contains all the coordination logic, over time it can become complex and difficult to maintain. Split coordination logic among distinct separate mediator microservices so they are more manageable.
Do not use the Mediator pattern when central control for orchestration is not required. When the Mediator pattern is overused, it will provide all the orchestration responsibility to a single team, which can constrain the autonomy of other teams, going against the principles of cloud native application development.

Question 41

How does the Pipe and Filter Pattern contribute to building large-scale decoupled systems?

Answer 41

The Pipe and Filter Pattern contributes to building large-scale decoupled systems by connecting microservices via asynchronous protocols like topics and queues. This decoupling allows for the continuous addition of new microservices to the processing flow without impacting existing ones.
Page 308

Build large-scale decoupled systems: As the microservices are connected to each other via asynchronous protocols such as topics and queues, the Pipe and Filter pattern decouples services. We can continuously add new microservices to the topics and queues and extend the processing flow without impacting existing microservices.

Question 42

What are the benefits of using the Pipe and Filter Pattern for adding and removing functionality in a microservices architecture?

Answer 42

The Pipe and Filter Pattern enables seamless addition and removal of microservices to the pipeline, allowing teams to add new business logic with minimal impact on other microservices. This pattern supports independent development and deployment of microservices, enhancing flexibility and scalability.
Page 309

Seamlessly add and remove functionality: The Pipe and Filter pattern enables seamless addition and removal of microservices to the pipeline, and allows teams to add new business logic with minimal effect on other microservices.

Question 43

How does the Pipe and Filter Pattern support the segregation of duties among teams in a microservices architecture?

Answer 43

The Pipe and Filter Pattern supports the segregation of duties among teams by distributing events via multiple topics and queues. Each team can independently consume events, process them, and output corresponding events, allowing for clear boundaries and responsibilities within the development process.
Page 309

Provide segregation of duties: As the events are distributed via multiple topics and queues, the Pipe and Filter pattern is ideal for providing a segregation of duties among teams. This allows each team to consume events, process them, and output corresponding events independently.

Question 44

What considerations should be taken when choosing between the Pipe and Filter Pattern and the Mediator Pattern?

Answer 44

The Pipe and Filter Pattern is recommended for building large-scale asynchronous systems with independent teams, delegating orchestration responsibilities to each team. The Mediator Pattern should be used when central control is needed across multiple services. Well-defined event schemas and a schema registry are essential for the Pipe and Filter Pattern to ensure autonomous discovery and consumption of events.
Page 309

Considerations: Use the Pipe and Filter pattern instead of the Mediator pattern when building large-scale asynchronous systems with independent teams. This way, we can delegate the orchestration responsibilities to each team, as opposed to centralizing control of the event flow with the Mediator pattern. We recommend using the Mediator pattern when central control should be established across multiple services.
Because the Pipe and Filter pattern enables multiple teams to collaborate, it is vital to have well-defined event schemas. Also use a schema registry to store schemas to enable autonomous discovery and consumption of events.
At times we are interested in only specific events, such as credit card request events related to a specific region, so we can treat them differently according to the laws of that region. The microservices use filters in their subscription to consume only the events that they are interested in, or if they consume events from a topic, they can perform the filtering within the microservices.
Do not use the Pipe and Filter pattern if the flow of events changes frequently—for example, when you are rapidly experimenting or innovating on the event processing logic. More pipes and filters may need to be modified to accommodate the changes, which is more costly than performing changes via the Mediator pattern.

Question 45

How does the Priority Queue Pattern optimize resource utilization in event-driven systems?

Answer 45

The Priority Queue Pattern optimizes resource utilization by ensuring that only the highest-value events are processed when there are constraints on available processing nodes. This prioritization helps manage limited resources effectively and ensures critical events are handled promptly.
Page 312

Optimize resource utilization: Because of financial or other reasons, if we have constraints on the available processing nodes, we use the Priority Queue Pattern to ensure that we process only the highest-value events.

Question 46

What are the considerations when implementing the Priority Queue Pattern for handling events?

Answer 46

When implementing the Priority Queue Pattern, consider the need for an intermediary application to perform prioritization if the client cannot poll. Implement a cleaning task to discard old events based on queue depth or promote events to a higher-priority queue if they remain unprocessed for a long time. Avoid using this pattern unless priority-based processing is essential, as it adds complexity.
Page 313

Considerations: When applying the Priority Queue Pattern, at times the client application will not have the capability to perform a polling operation. This may be because it is an external system outside our control. In this case, implement the polling client as an intermediary application that performs the prioritization and pushes events to other systems.
It is important to implement a cleaning task to discard old events based on queue depth; especially when the input event rate is higher than the processing rate, the low-priority queue can have stale events for a long time. Alternatively, you can design the application to promote events to a higher-priority queue when it has capacity and the events have stayed for a considerable time in the lower-priority queue. We recommend the prior approach when we do not have the mandate to process all events, and recommend the latter when we need to process all incoming events, and if we do not want the lower-priority events to starve when there is a steady flow of high-priority events.
You can also consider implementing the Priority Queue Pattern by using a single topic and subscription filters
We recommend using this alternative only if there is enough capacity to process all incoming events. In addition, do not use the Priority Queue Pattern unless priority-based processing is necessary for your use case, as it would introduce unnecessary complexity to the architecture.

Question 47

What are the best practices for testing event-driven systems?

Answer 47

Best practices for testing event-driven systems include using dedicated topics and queues for tests, running the message broker instance specifically for the test, implementing mock clients and message brokers as containers, and running tests in a namespaced environment. These practices help isolate failures, accelerate troubleshooting, and reduce interference between tests and other systems.
Page 323

We recommend using a dedicated topic and queues for tests. When possible, start the message broker instance just for the test. Even when a shared messaging infrastructure is used, we recommend creating dedicated topics and queues. When running mock clients and message brokers, we recommend implementing them as containers. When possible, run the test in a namespaced environment. All these features will help isolate the cause of failures and accelerate troubleshooting. This also allows us to clean the test environment after the tests. This reduces interference between tests and other systems and increases the deterministic behavior of the test.