Chapter 6, Stream-Processing Patterns

Question 1

What is a stream?

Answer 1

A stream can be defined as a continuous sequence of events ordered by time. The stream consists of a name and version that uniquely identify it, such as StockStream 1.0. All events in a stream have a common message format and structure. For example, StockStream has a JSON format and contains symbol, price, and volume in its structure. Having a consistent format and structure allows events in the stream to be processed in an automated manner, using stream-processing systems. The stream version provides a way to safely modify the structure and evolve the stream over time.

330

Question 2

What Is Stream Processing?

Answer 2

Stream processing is performing operations on events in motion. It can be as simple as a stateless service consuming events and transforming its event format, or as complex as storing and processing stateful data in memory with low latency and reliability.
In contrast to simple event processing, stream processing supports use cases in which events need to be handled in the order they are generated. Stream-processing patterns can also remember and use previous events when making a decision. For example, detecting if a stock price is continuously increasing over the last five minutes requires remembering previous events and processing them in order, both in real time.

330

Question 3

What are Streaming Data Processing Patterns and what do they focus on?

Answer 3

Streaming data processing patterns focus on how we can generate useful output by processing real-time events through transformation, filtering, aggregation, and detecting meaningful sequences of events. These capabilities enable cloud native applications to process events on the fly with low latency.
A key performance consideration is avoiding heavy use of persistent data stores. In a cloud native application, the round-trip time of accessing the data store, and the potential for contention, can add significant processing latency to solutions. For some use cases, it is required, but as a general rule of thumb, it should be avoided.

331

Question 4

Describe the Transformation Pattern

Answer 4

The Transformation pattern helps transform events from an event source and publish them to another system with a different format, structure, or protocol.

332

Question 5

How does the Transformation Pattern work?

Answer 5

This pattern maps the data of one event to another.
These transformations are often achieved purely with the information contained in the incoming event. But at times these transformations need other patterns, such as the Windowed Aggregation pattern.

332 Figure 6-1. XML-to-JSON transformation

For example, say we are to publish weather events to a third-party system that expects the events in JSON format with a particular structure (Figure 6-1). The relevant data from the incoming event can be extracted and mapped to the new event format. We can achieve this by using JSON and XML libraries, or by using a graphical interface or SQL-based data-mapping approaches provided by stream-processing technologies.

Question 6

What are some related patterns to the Transformation Pattern?

Answer 6

The Transformation pattern can be combined with other stream data processing patterns, as data transformations can be required for incorporating results of those patterns, such as enriching events with aggregated data.

335

Question 7

Describe the Filters and Thresholds Pattern

Answer 7

Sometimes we need to filter events based on given conditions, or allow only events with values that fit within a given threshold range. The Filters and Thresholds pattern is useful for extracting only the relevant events we need.

336

Question 8

How does the Filters and Thresholds Pattern work?

Answer 8

Users provide conditions that match against the incoming events. These conditions can include exact string matches, substring matches, regular expressions, or threshold ranges when it comes to numeric values with comparison operations such as <, <=, >, >=, and ==. Often more than a single condition is required, so those conditions are consolidated by using the AND, OR, and NOT logical operations and parentheses to generate more-complex filter conditions.
This pattern extracts and processes the relevant data from the input event stream by using data-mapping techniques

336 Figure 6-3. Filtering car events based on brand and year

If we are processing a real-time stream of car sales and are interested in only 2010 or newer Toyota vehicles, we can define a filtering condition as shown in Figure 6-3 to emit only events that satisfy the condition.

Question 9

What are some related patterns to the Filters and Thresholds Pattern?

Answer 9

The Filtering and Thresholds pattern can be applied with all the other stream data processing patterns, as we often need to filter events for those patterns (for example, to aggregate only a particular type of event).

338

Question 10

Describe the Windowed Aggregation Pattern

Answer 10

The Windowed Aggregation pattern enables us to analyze a collection of events based on a condition. Here, aggregation analysis can include operations like summation, minimum, maximum, average, standard deviation, and count, and the window defines the collection of events used for aggregation.
These windows can be based on the time or event count, such as the last five minutes or the last 100 events. These windows may also have behaviors such as sliding or batching, defining when events are added and removed from the window.
This pattern enables us to aggregate data on the fly and make time-critical business decisions within milliseconds.

338

Question 11

What are some of the most common windows?

Answer 11

• Length sliding
• Length batch
• Time sliding
• Time batch

339

Question 12

How is the Windowed Aggregation Pattern used in practice?

Answer 12

The Windowed Aggregation pattern is stateful, meaning it stores data related to the events in memory. Therefore, when designing noncritical use cases such as monitoring that tolerates data loss, we can implement this pattern on any cloud native application. But when the use case requires reliable event processing, we need to combine this pattern with the reliability patterns

343

Question 13

What are some related patterns to the Windowed Aggregation Pattern?

Answer 13

• Transformation pattern
  Appropriately maps the aggregation to the output.
• Reliability patterns
  Help make the window and aggregation state survive system failures.
• Sequential Convoy pattern
  Allows aggregations to be performed in parallel based on shard keys. This not only helps scale aggregation processing, but also allows us to aggregate different types of events in isolation and produce aggregations per event type.
• Service Orchestration pattern
  Splits the events by different shard keys for processing. This pattern is described in Chapter 3.
• Stream Join pattern
  Aggregates results from different shards.

347

Question 14

Describe the Stream Join Pattern

Answer 14

The Stream Join pattern resembles the join of SQL tables and enables us to join events from multiple streams with different schemas.

348

Question 15

How does the Stream Join Pattern work?

Answer 15

This pattern works by defining a condition to identify the joining events. This condition will pick attributes from each joining event stream and define the condition under which they should be joined. This can be a simple equal condition, like joining events from all event streams having the same ID, or it can be more complex. The join should also define a buffer that determines how long events should wait for corresponding events to arrive from other event streams. This buffer period can be common across all streams or can vary among streams. Most stream-processing systems define this buffer period via windows.
Finally, as in the Windowed Aggregation pattern, it is important for this pattern to use the Transformation pattern to map the joining events and their attributes to the output.

348 Figure 6-5. Stream Join based on events that have arrived during the last minute

Question 16

How is the Stream Join Pattern used in practice?

Answer 16

The Stream Join pattern is stateful, as it buffers events for the join. Like the Windowed Aggregation pattern, this one can be implemented in any cloud native application as long as the use case is not business critical and can tolerate event loss. But when event loss is not acceptable, this pattern should be applied along with reliability patterns, so the application can withstand system failures and restarts without event loss.

349

Question 17

What are some patterns related to the Stream Join Pattern?

Answer 17

• Transformation pattern
  Appropriately maps joining event attributes to build the output.
• Reliability patterns
  Helps the joint state survive system failures.
• Sequential Convoy pattern
  Scales joins by performing them in parallel by allowing relevant joining events to fall into the same shard.

352

Question 18

Describe the Temporal Event Ordering Pattern

Answer 18

The Temporal Event Ordering pattern is unique for stream processing. It tries to detect various interesting complex event occurrences by identifying patterns based on event arrival order. The pattern can also detect occurrence and nonoccurrence of incidents based on events emitted by various systems.

352

Question 19

How does the Temporal Event Ordering Pattern work?

Answer 19

This pattern works on the concept of nondeterministic finite-state machines: the application state changes based on the input event and the current application state. The possible state transitions can be represented as a state graph that traverses from one state to another until it reaches either a success or fail state. Upon reaching the success state, the user is notified, as it means the expected events have occurred in order.
This pattern can also be used to identify sequences of events that are immediately followed by one another or scattered randomly among other events. We can also use this to detect the nonoccurrence of events by combining state transitions with time-outs.
Use cases such as stock monitoring most often require the event sequence to be detected repeatedly. To achieve this, a new state machine instance should be initiated upon each event arrival that triggers the initial state of the state machine.

352 Figure 6-7. Using the Temporal Event Ordering pattern to detect a continuous stock price increase followed by a single drop

Question 20

How is the Temporal Event Ordering Pattern used in practice?

Answer 20

Like the Windowed Aggregation and Stream Join patterns, this pattern should also be combined with reliability patterns to preserve data loss during system failures and restarts. Furthermore, as event arrival order is critical for the success of this pattern, we recommend using patterns like Buffered Event Ordering to guarantee ordering of events before processing them.

354

Question 21

What are some patterns related to the Temporal Event Ordering Pattern?

Answer 21

• Transformation pattern
  Appropriately maps the matched events in the sequence to generate a meaningful output.
• Reliability patterns
  Helps state machines survive system failures.
• Sequential Convoy pattern
  Scales sequence matching by performing it in parallel by allowing relevant events to fall into the same shard.
• Buffered Event Ordering pattern
  Orders events based on event-generation time to facilitate correct behavior of this pattern.

356

Question 22

Describe the Machine Learner Pattern

Answer 22

We can use machine learning models in real time to generate predictions and automate decision making. Machine learning models can be prebuilt to produce predictions without updating themselves based on new input events. Online machine learning models can produce predictions while continuously learning, based on new incoming events, whether or not they’re pre-generated, making our cloud native application much more intelligent.

357

Question 23

How does the Machine Learner Pattern work?

Answer 23

We can generate predictions in cloud native applications in two ways: by executing prebuilt machine learning models and by using online machine learning models.

357

Question 24

How is the Machine Learner Pattern used in practice?

Answer 24

Machine learning has now become an integral part of many applications, and cloud native applications should also be well equipped to incorporate them. One common way of integrating machine learning models is to deploy them as individual microservices and make service calls. Alternatively, machine learning models can be embedded into the applications, which can continuously produce predictions based on incoming events. Some scenarios using this pattern are described next.

359

Question 25

What are some patterns related to the Machine Learner Pattern?

Answer 25

• Transformation pattern
  Appropriately maps the predictions of the machine learning model to generate a meaningful output.
• Reliability patterns
  Store and restore online machine learning algorithm state.

362

Question 26

When to use the Transformation pattern?

Answer 26

To transform the event format, structure, or protocol.
To add or remove partial data to or from the event.
Third-party systems do not support the current event.

362

Question 27

When not to use the Transformation pattern?

Answer 27

The consuming system has the ability to understand the event.

362

Question 28

What are the benefits of using the Transformation pattern?

Answer 28

Allows incompatible systems to communicate with one another.
Reduces event size by containing only relevant information.

362

Question 29

When to use the Filters and Thresholds pattern?

Answer 29

Only a subset of events is relevant for processing.

362

Question 30

When not to use the Filters and Thresholds pattern?

Answer 30

All events are needed for decision making.

362

Question 31

What are the benefits of using the Filters and Thresholds pattern?

Answer 31

Reduces the load on the system by selecting only events that can produce the most value to the use case.

362

Question 32

When to use the Windowed Aggregation pattern?

Answer 32

To aggregate events over time or length.
To perform operations such as summation, minimum, maximum, average, standard deviation, and count on the events.

362

Question 33

When not to use the Windowed Aggregation pattern?

Answer 33

For operations that cannot be performed with fixed memory such as detecting the median of the events.
High accuracy is needed without the use of reliability patterns.

362

Question 34

What are the benefits of using the Windowed Aggregation pattern?

Answer 34

Reduces the load on the system by aggregating events.
Provides data summary to better understand the behavior as a whole.

362

Question 35

When to use the Stream Join pattern?

Answer 35

To join events from two or more event streams.
To collect events that were previously split to parallelize processing.

362

Question 36

When not to use the Stream Join pattern?

Answer 36

Joining events do not arrive in relatively close proximity.
High accuracy is needed without the use of reliability patterns.

362

Question 37

What are the benefits of using the Stream Join pattern?

Answer 37

Allows events to be correlated.
Enables synchronous processing of events.

362

Question 38

When to use the Temporal Event Ordering pattern?

Answer 38

To detect the sequence of event occurrences.
To detect the nonoccurrence of events.

362

Question 39

When not to use the Temporal Event Ordering pattern?

Answer 39

Event sequencing cannot be defined as a finite-state machine.
High accuracy is needed without the use of reliability patterns.
Incoming events arrive out-of-order.

362

Question 40

What are the benefits of using the Temporal Event Ordering pattern?

Answer 40

Allows detecting complex conditions based on event arrival order.

362

Question 41

When to use the Machine Learner pattern?

Answer 41

To perform predictions in real time.
To perform classification, clustering, or regression analysis on the events.

362

Question 42

When not to use the Machine Learner pattern?

Answer 42

We cannot use a model to accurately predict the values.
Historical data is not available for building machine learning models.

362

Question 43

What are the benefits of using the Machine Learner pattern?

Answer 43

Automates decision making.
Provides reasonable estimates.

362

Question 44

Describe Scaling and Performance Optimization Patterns

Answer 44

Cloud native applications that perform stream processing have unique scalability and performance requirements. For instance, these applications require event ordering to be maintained while processing events. Furthermore, as most of these applications have in-memory state, they also need a strategy to scale so they can process more events without compromising their accuracy.

364

Question 45

Describe Sequential Convoy Pattern

Answer 45

The Sequential Convoy pattern scales cloud native stream-processing applications by separating events into various categories and processing them in parallel. It also works to persist event ordering so events can be combined at a later time, while preserving the original order of the events.

364

Question 46

How does the Sequential Convoy Pattern work?

Answer 46

As the name suggests, this pattern sees events as items moving along a conveyor belt. It groups the events into categories based on their characteristics and processes them in parallel.

364

Question 47

How is the Sequential Convoy Pattern used in practice?

Answer 47

This pattern is used for scaling event processing so we can process more events with cloud native applications that have limited memory capacity, and for partitioning events so that each substream is processed differently. Let’s look at how this pattern can be used in various scenarios.

366

Question 48

What are some related patterns to the Sequential Convoy Pattern?

Answer 48

• Producer-Consumer and Publisher-Subscriber patterns
  Can be used as the base for building the Sequential Convoy pattern. These patterns are covered in Chapter 5.
• Buffered Event Ordering pattern
  Provides an alternative way to order events while joining events from multiple event streams together. This pattern is covered next.
• Periodic Snapshot State Persistence pattern
  Stores substream application states and supports scalability. This pattern is covered later in this chapter.

370

Question 49

Describe the Buffered Event Ordering Pattern

Answer 49

Network delays and connection retries can cause events to get out of order. The Buffered Event Ordering pattern allows us to reorder events before processing them downstream. We can order events based on time or on the order they are generated.

371

Question 50

How does the Buffered Event Ordering Pattern work?

Answer 50

For events to be ordered, they must have an incremental value by which to order them. This value can be a sequence number or a timestamp, for example. Sequence numbers will continuously increase, and we can guarantee that each event in a stream will have a unique number. But with a timestamp, we cannot guarantee that all events will have unique values, because multiple events can be generated in the same millisecond.

371 Figure 6-17. Ordering events based on sequence number

Figure 6-17 illustrates the use of sequence numbers. If we have most recently received an event with sequence number 7, and now we receive an event with sequence number 8, we can immediately send it for processing because we know that 8 follows 7. But if after 8 we get 10, we know that we are missing an event and so cannot send 10 for processing. Instead, we need to use a time-out (of 30 seconds, for instance) to wait for the missing event. Then, if the missing event 9 arrives in time, we send it as well as event number 10. But if it does not arrive in time, we have to send the event with sequence number 10 before 9.

Question 51

How is the Buffered Event Ordering Pattern used in practice?

Answer 51

This pattern can be deployed in front of any use case that needs ordered events, as long as the events have attributes that can be used for ordering.

372

Question 52

What are some related patterns to the Buffered Event Ordering Pattern?

Answer 52

• Temporal Event Ordering and Windowed Aggregation patterns
  These patterns can benefit from the Buffered Event Ordering pattern, as they require events to be ordered to produce more-accurate results.
• Reliability patterns
  For storing and retrieving events that are waiting in the buffer for ordering, during system failure and restart.

374

Question 53

Describe the Course Correction Pattern

Answer 53

The Course Correction pattern attempts to report its analysis of events as soon as possible, and then later correct its analysis and report again, as soon as it retrieves missing (or late) events. This produces early analysis with low latency rather than sending an accurate analysis with higher latency.

375

Question 54

How does the Course Correction Pattern work?

Answer 54

This pattern should be combined with patterns like Windowed Aggregation or Temporal Event Ordering. Rather than waiting for all events to arrive, we send aggregation and event sequence detection as soon as we have a result. The results of the aggregation and sequence detection are an early estimate and may not be accurate. Later, when we receive missing events, we send updated results.

375

Question 55

How is the Course Correction Pattern used in practice?

Answer 55

This pattern should be used only when we need events in order, have a requirement for low latency, and can cope with inaccurate early estimates. Let’s consider some example scenarios to understand this in more detail.

375

Question 56

What are some related patterns to the Course Correction Pattern?

Answer 56

• Reliability patterns
  For storing application state holding previous events and previously emitted outputs.
• Buffered Event Ordering pattern
  Can be used instead of this pattern when the use case does not support course correction.
• Temporal Event Ordering and Windowed Aggregation patterns
  These patterns can benefit from the Course Correction pattern as they can use course correction to correct their early estimates.

378

Question 57

Describe the Watermark Pattern

Answer 57

The Watermark pattern is useful for periodically aligning stream processing across multiple microservices within a cloud native application that are connected in a mesh-like structure via event streams. This alignment will help determine whether all microservices have processed all arrived events before a given event, which is commonly referred to as the watermark event. We can use this pattern to sync multiple microservices without using system times.

379

Question 58

How does the Watermark Pattern work?

Answer 58

For watermarks to work, a watermark generator should generate a watermark event periodically and send it through all the external inputs of the cloud native application. This event should be considered special, and microservices should pass it through to their dependent systems. We also need to be sure that each intermediate microservice that consumes this event can resend it in the same position among the sequence of events it has received and processed, and not before or after other events.
When the input systems are time synchronized, we can make those systems independently generate the watermark events at given intervals, such as once every minute or every five minutes

379 Figure 6-19. Generating watermark events and synchronizing events based on them

When the microservice receives a watermark event in a stream (such as the watermark event with sequence number 6 in this example), it should not continue processing any more events from that stream, and process only events from other streams (such as Event B of the second stream), that have not yet received the corresponding watermark event. When we receive all corresponding watermark events on all streams, we can pass that watermark event to all its dependents and continue processing other events from all the input streams until we receive the next watermark event in a stream. This process is repeated, and this approach ensures that event processing is synchronized at each watermark event.
When the preceding options are not possible, we can also make the input sources poll a global counter to fetch the next watermark event sequence number and emit it periodically along with the events. In this case, we should make sure that watermark events arriving in multiple streams are processed in a sequential manner. If we find a sequence number out of sync, we should halt the execution of events from that stream until we receive a watermark event with a lower sequence number on another stream.

Question 59

How is the Watermark Pattern used in practice?

Answer 59

This pattern can be used when multiple source systems are not synchronized on time, or when network latency or processing time can affect event arrival time. In this case, events in one stream can arrive earlier, while events from other streams can arrive later, and this can cause issues when analyzing events across multiple streams. This pattern is ideal for synchronizing the event processing periodically to reduce errors.

381

Question 60

What are some patterns related to the Watermark Pattern?

Answer 60

• Buffered Event Ordering and Course Correction patterns
  Can be used instead of this pattern when event arrival times are not affected by network latency or other processing delays by the systems.
• Temporal Event Ordering and Windowed Aggregation patterns
  These patterns can benefit from the Watermark pattern as they require events to be ordered to produce correct results.
• Periodic Snapshot State Persistence pattern
  The Watermark pattern can be a prerequisite for Periodic Snapshot State Persistence to perform state snapshots in a synchronized manner among multiple streams.

384

Question 61

When to use the Sequential Convoy pattern?

Answer 61

To scale stream-processing applications.
To partition streams so each stream can be used for various use cases.
To allow processing events in parallel and regroup them based on the original order.

384

Question 62

When not to use the Sequential Convoy pattern?

Answer 62

Streaming applications have enough capacity to process the events.

384

Question 63

What are the benefits of using the Sequential Convoy pattern?

Answer 63

Supports scalability of stream processing
Preserves event ordering when events are processed in parallel.

384

Question 64

When to use the Buffered Event Ordering pattern?

Answer 64

To order events based on timestamp or sequence number.
To order events that are already out of order and published via a single event stream.

384

Question 65

When not to use the Buffered Event Ordering pattern?

Answer 65

To group events from multiple ordered event streams.
We need true ordering of events that are generated from distributed sources.
Reliability patterns cannot be applied to the application.

384

Question 66

What are the benefits of using the Buffered Event Ordering pattern?

Answer 66

Can be applied in front of any application that needs events in order.

384

Question 67

When to use the Course Correction pattern?

Answer 67

To correct previously produced results.
To produce early aggregation estimates.
To guess the event-sequence order and correct the decision later.

384

Question 68

When not to use the Course Correction pattern?

Answer 68

The dependent downstream applications cannot handle continuous event updates.

384

Question 69

What are the benefits of using the Course Correction pattern?

Answer 69

Allows us to produce early estimates and correct them as we have more data.

384

Question 70

When to use the Watermark pattern?

Answer 70

To perform aggregation operations on event streams that are out of sync.
Try to order events that are generated by distributed systems.

384

Question 71

When not to use the Watermark pattern?

Answer 71

We cannot inject watermark events closer to the event sources.
Intermediate systems cannot bypass watermark events.
Network bandwidth is a concern.

384

Question 72

What are the benefits of using the Watermark pattern?

Answer 72

Periodically synchronizes events across multiple streams.
Helps overcome network and processing latency added by intermediate systems.

384

Question 73

Describe the Replay Pattern

Answer 73

By using the Replay pattern, the state of a microservice can be restored by replaying the events it has processed in the past, especially when its state depends only on recent events.

396

Question 74

How does the Replay Pattern work?

Answer 74

This pattern works by resending events when the system is down. The number of old events it needs to send depends on the use case. For example, if the microservice is aggregating events over the past three minutes, then during failures, resending the events arrived during the last three minutes is sufficient.

386 Figure 6-22. Deferring acknowledgment until after outputs are generated

When the stateful microservice can store its state periodically, we will be able to identify the last successfully processed event from the latest snapshot, and we should be able to replay all events arrived after that.
To re-create the state of a system, the source of the data should contain the events even after they are retrieved by the microservice. We cannot use standard message brokers with their automatic event acknowledgment feature, because the events will be deleted from the message broker as we read them, unless we use queues or durable subscriptions and differ the acknowledgment of consumed events. As shown in Figure 6-22, we can delay sending acknowledgments to the queues until the microservice is done processing and cleaning out its state, or until it persists its state in durable storage.
We can use this pattern with microservices that consume events from log-based message brokers, such as Kafka and NATS, because these brokers will not delete the events when they are delivered to the microservice, and the microservices can request events to be played back from the last sequence number they have successfully processed. We can also use this pattern when events are read from persistent data stores like RDBMS databases, NoSQL stores, or filesystems.

Question 75

How is the Replay Pattern used in practice?

Answer 75

This pattern can be used to restore an application’s state by replaying the lost events due to system failure or restart. Let’s look at how this pattern could be used in a few scenarios.

388

Question 76

What are some related patterns to the Replay Pattern?

Answer 76

• Publisher-Subscriber pattern
  Can be used to establish durable subscriptions with event sources so they can be replayed during failure. This pattern is covered in Chapter 5.
• Periodic Snapshot State Persistence pattern
  This can be used in conjunction with the Replay pattern to restore application state and reduce the time to bring the application back alive. This pattern is covered next.

389

Question 77

Describe the Periodic Snapshot State Persistence Pattern

Answer 77

Persisting the application state upon processing each incoming event is not feasible, as this introduces extremely high latency to cloud native applications due to the round-trip time of accessing state. The Periodic Snapshot State Persistence pattern allows us to persist the application state in a periodic manner so that we can restore the state reliably after system restarts or failures.

390

Question 78

How does the Periodic Snapshot State Persistence Pattern work?

Answer 78

This pattern periodically makes a copy of its current state and persists that to a durable store between processing events. For this to work, we should ensure that the microservices can read and write state to a persistent storage
To ensure that events are not lost during failures and to guarantee at-least-once event delivery, we must use message brokers to retrieve events. When using a log-based message broker like Kafka, we should store the event sequence number with the snapshot (Figure 6-23). With this approach, upon a restart, we reload the last stored snapshot and request the message broker to deliver events from the stored event sequence number.

390

When using standard message brokers like ActiveMQ, we should acknowledge the processed messages only when storing the snapshot. This way, we can ensure that when the microservice is restarted the message broker sends all unacknowledged events.
When using standard message brokers like ActiveMQ, we should acknowledge the processed messages only when storing the snapshot. This way, we can ensure that when the microservice is restarted the message broker sends all unacknowledged events.

Question 79

How is the Periodic Snapshot State Persistence Pattern used in practice?

Answer 79

This pattern can be used to store state when microservices process data in memory, and when their state cannot be persisted after every event.

394

Let’s say we want our application to detect if a stock price has continuously risen over the last 10 minutes. We need to keep track of only the last time we saw a dip in the stock price, and continuously check if that time is now older than 10 minutes. If so, we will alert the user.
If we are retrieving the events from a Kafka topic, we also need to remember the last-processed event’s sequence number along with the current stock price, and the last time we saw the stock price dip. These three represent the state of the microservice. To recover the microservice from failure, we need to persist all three values to a database or similar storage, in a periodic manner. During the recovery process, the microservice can restore the last stored snapshot and replay the Kafka events from the last stored event sequence number to ensure that the system state is preserved across system failures.

Question 80

What are some things to consider when using the Periodic Snapshot State Persistence Pattern?

Answer 80

We should use this pattern only when we are processing critical data that cannot be lost on system failure, because the pattern introduces significant operational overhead that is not worthwhile if data loss is acceptable.

In some situations, the state itself is quite large and requires significant time to store and retrieve. To mitigate this, use incremental snapshots, store only the delta between the current state and the last snapshot, and then replay incremental snapshots to re-create system state.

We do not recommend making the snapshot interval overly short, as this introduces overhead without significant benefit. But don’t set the snapshot overly long either, as this leads to not only writing and reading bigger snapshots (which takes longer), but also replaying more events on application restoration (which can increase the time for applications to become live again).

395

Example 1) For example, if our processing window is small (say, one minute), system failure impacts for only as long as the system is down. To restore state, we can use the Replay pattern to reprocess the events during the lost period. But if our application state contains data from the previous day, we need to replay events from the previous day to re-create the state, which may not be feasible because of the processing time for the quantity of events. In such cases, we advise using the Periodic Snapshot State Persistence pattern.

Example 2) For example, when using a time window of five minutes with one-minute time shifts for aggregation, store snapshots every minute with only the changes that happened during the last minute. When there is a failure, we load the last five snapshots to re-create the state of the five-minute window.

Question 81

What are some related patterns to the Periodic Snapshot State Persistence Pattern?

Answer 81

• Temporal Event Ordering and Windowed Aggregation patterns
  These patterns can benefit from the Periodic Snapshot State Persistence pattern, as they require state to be stored to achieve reliability.
• Replay pattern
  This is used to re-create states from the missing events based on the last snapshot loaded during application recovery.
• Watermark pattern
  This can be used to synchronize state snapshots across multiple microservices.

396

Question 82

Describe the Two-Node Failover Pattern

Answer 82

Low-latency microservices do not have the luxury of taking a couple of minutes after failure to restart and restore their states. For these microservices, it is operationally superior to run a redundant microservice to allow failover. We can run such a microservice by using the Two-Node Failover pattern.

397

Question 83

How does the Two-Node Failover Pattern work?

Answer 83

This pattern focuses on running a parallel backup microservice. When microservices are deployed, they perform a leader election; we can use systems such as ZooKeeper or native cloud services to designate one microservice as primary and the other as secondary.

397 Figure 6-26. Running microservices as primary and secondary to enable failover

Question 84

How is the Two-Node Failover Pattern used in practice?

Answer 84

This pattern can be used when degradation of latency and system downtime cannot be tolerated. Say we retrieve stock order events from NATS, process the number of stock bids and asks in real time, and publish them to stockbrokers so they can instantly identify changes in market trends. Using the Two-Node Failover pattern, we cannot only switch to the secondary node instantly upon detecting that the primary has failed, but also ensure that no events are dropped. This is because the secondary publishes only data that the primary has not.

398

Question 85

What are some things to consider when using the Two-Node Failover Pattern?

Answer 85

Use this pattern only when low latency is the main requirement. Otherwise, use other patterns such as Periodic Snapshot State Persistence. This pattern is complex to implement, and the architectural complexity is not worthwhile if we can tolerate downtime during system failure.

This pattern requires both microservices to have robust connectivity, as the primary needs to publish its output to the secondary. Furthermore, a risk of network partitioning between the primary and secondary exists. In this case, we require a third system to function as the leadership elector; otherwise, both microservices could become primary in parallel and send outputs downstream.

We should also be mindful that both microservices can fail simultaneously; then the system would become unavailable and we’d lose their state. As a mitigation, we can adopt the Periodic Snapshot State Persistence pattern or Replay pattern to allow for state restoration in such a scenario.

399

Question 86

What are some related patterns to the Two-Node Failover Pattern?

Answer 86

• Periodic Snapshot State Persistence and Replay patterns
  These patterns, covered in this chapter, can be used with the Two-Node Failover pattern to restore the state of the microservice when they restart it as a secondary after failure.
• Sequential Convoy pattern
  Used if we need to scale the stream-processing application. This pattern is covered in this chapter.
• Publisher-Subscriber pattern
  To allow both primary and secondary microservices to consume the same events. Chapter 5 details this pattern.

400

Question 87

When to use the Replay pattern?

Answer 87

The system state contains only recent events.
To restore state only when there is access to the previously processed events.
To process data from persistence stores, filesystems, and log-based message brokers.

400

Question 88

When not to use the Replay pattern?

Answer 88

We cannot guarantee that previously processed data cannot be accessed again.
Dependent systems cannot process duplicate events.
Systems cannot take time to re-create their state.
The system state needs to contain events that span over a long period.

400

Question 89

What are some benefits of using the Replay pattern?

Answer 89

Allows re-creating state without storing large snapshots.

400

Question 90

When to use the Periodic Snapshot State Persistence pattern?

Answer 90

The system state needs to contain events that span over a long period.
To restore state only when there is access to the previously processed events.
To process data from persistence stores, filesystems, and log-based message brokers.

400

Question 91

When not to use the Periodic Snapshot State Persistence pattern?

Answer 91

The system state contains only recent events.
We cannot guarantee that previously processed events can be accessed again.
Systems cannot take time to re-create their state.

400

Question 92

What are some benefits of using the Periodic Snapshot State Persistence pattern?

Answer 92

Allows re-creating state faster.
Supports larger and long-running system states.
Supports dependent applications that cannot process duplicate events.

400

Question 93

When to use the Two-Node Failover pattern?

Answer 93

We cannot take time to restore the application after failure.
The system state needs to contain events that span over a long period.
To restore state only when there is access to the previously processed events.

400

Question 94

When not to use the Two-Node Failover pattern?

Answer 94

We cannot guarantee that previously processed events can be accessed again.
Systems can take time to re-create their state.

400

Question 95

What are some benefits of using the Two-Node Failover pattern?

Answer 95

Supports low-latency and highly available stream processing.
Supports dependent applications that cannot process duplicate events.

400

Question 96

When to use Esper?

Answer 96

To embed into cloud native applications.
To support transformations, filtering and thresholds, windowed aggregations, joins, and temporal event ordering.

406

Question 97

When not to use Esper?

Answer 97

To run as a standalone application.
To run machine learning models.
Built-in reliability is required.

406

Question 98

Describe Esper

Answer 98

Esper is a complex event-processing library released under the GPL v2 license. It can be used to implement stream-processing logic in Java and .NET-based microservice applications. It supports stream-processing constructs including transformations, filtering, thresholds, windowed aggregations, joins, and temporal event ordering.

Esper can be used to reduce the complexity of an application, as we can offload most of the processing logic to it. We can model events as Java or .NET objects and pass them to Esper for processing, and then subscribe to it to receive outputs. It also supports a query language to configure the stream-processing logic. We recommend using Esper for implementing stream-processing logic within our microservices or serverless functions.

402

Question 99

When to use Siddhi?

Answer 99

To embed into cloud native applications.
To run as a standalone cloud native application.
To support transformations, filtering and thresholds, windowed aggregations, joins, temporal event ordering, and machine learnin

406

Question 100

When not to use Siddhi?

Answer 100

High scalability is needed.

406

Question 101

Describe Siddhi

Answer 101

Siddhi is a Java-based stream-processing library and a microservice released under Apache License v2. As a library, Siddhi (like Esper) can be embedded into microservices to process stream-processing logic. It allows stream-processing logic to be defined via Siddhi Query Language and supports stream-processing constructs including transformations, filtering, thresholds, windowed aggregations, joins, temporal event ordering, and machine learning. We recommend using Siddhi for implementing stream-processing logic within microservices or serverless functions.

We also recommend using Siddhi if you want to run stream processing as a standalone microservice supporting all the reliability patterns, including Periodic Snapshot State Persistence and Two-Node Failover. Users can use Siddhi Query Language to configure the sources from which Siddhi should consume events, its processing logic, and where it should publish its output and deploy that to Kubernetes.

402

Question 102

When to use ksqlDB?

Answer 102

Kafka is used in the infrastructure.
To support transformations, filtering and thresholds, windowed aggregations, and joins.
To build materialized views from the input events.

406

Question 103

When not to use ksqlDB?

Answer 103

Kafka is not used in the infrastructure.
Temporal event ordering and machine learning is needed.

406

Question 104

Describe ksqlDB

Answer 104

ksqlDB is a stream-processing and a database system that is part of Kafka. It works only in environments where Kafka is used as the broker for distributing events. We can define rules in ksqlDB to retrieve events from a Kafka stream, and then process and publish them. It supports stream-processing constructs such as transformations, filtering, thresholds, windowed aggregations, and joins. It also provides a feature to build materialized views from the input events, which can be queried on demand by cloud native applications. Its ability to pull data on demand is useful, as it can be modeled as a relational database. We recommend using ksqlDB when Kafka is used as the message broker for the cloud native application, and when you need to query event logs via materialized views.

403

Question 105

When to use Apache Spark?

Answer 105

Support for both stream and batch processing is needed.
To support transformations, filtering and thresholds, windowed aggregations, joins, and machine learning.

406

Question 106

When not to use Apache Spark?

Answer 106

A lightweight system is needed for stream processing.
Temporal event ordering is needed.
To embed into cloud native applications.

406

Question 107

Describe Apache Spark

Answer 107

Spark is a big-data and stream-processing platform released under Apache License v2. It can run on Apache Mesos, Hadoop YARN, and Kubernetes. Though it is strong in batch processing, it can also support stream-processing constructs such as transformations, filtering, thresholds, windowed aggregations, joins, and machine learning.

It uses both queries and a structured programming approach, allowing users to program using Java, Scala, or Python to support both stream and batch processing. It achieves reliable processing by periodically checkpointing data into durable storage. It is an ideal choice when use cases are mainly oriented toward batch processing while having some streaming requirements.

403

Question 108

When to use Apache Flink?

Answer 108

To support transformations, filtering and thresholds, windowed aggregations, joins, and temporal event ordering, along with graph processing.
For high scalability and availability requirements.

406

Question 109

When not to use Apache Flink?

Answer 109

A lightweight system is needed for stream processing.
To embed into cloud native applications.

406

Question 110

Describe Apache Flink

Answer 110

Flink is a fully fledged stream-processing platform released under Apache License v2. It can run on platforms like Kubernetes, Knative, and AWS Lambda. It supports stream-processing constructs such as transformations, filtering, thresholds, windowed aggregations, joins, and temporal event ordering, along with graph processing. It also supports exactly once semantics, and supports reliable data processing using watermarks, and snapshots by storing them in storage such as S3, GCS, and HDFS.

Flink supports simple query language for defining stream-processing logic, the Table API for declarative data processing, and data stream and stream-processing APIs in Java for more-granular level configurations. We recommend using this for large-scale stream-processing use cases that have high scalability and availability requirements.

404

Question 111

When to use Amazon Kinesis?

Answer 111

To support Flink in AWS.
To support transformations, filtering and thresholds, windowed aggregations, joins and temporal event ordering, along with graph processing.

406

Question 112

When not to use Amazon Kinesis?

Answer 112

Other cloud providers are selected.
To embed into cloud native applications.

406

Question 113

Describe Amazon Kinesis

Answer 113

Kinesis is a fully managed scalable stream-processing offering from AWS. It supports SQL-based and Flink-based data processing in the cloud and allows users to build their own cloud native applications and run them in Amazon Lambda or EC2. With its SQL mode, it can support transformations, filtering, thresholds, windowed aggregations, and joins.

With Flink, it supports all standard stream-processing constructs. In addition to streaming events, it can also stream video content. We recommend using Kinesis if AWS is the hosting environment for your cloud native application.

404

Question 114

When to use Azure Stream Analytics?

Answer 114

To support transformations, filtering and thresholds, windowed aggregations, joins, temporal event ordering, and machine learning.
To support stream-processing queries to run in the cloud and on the edge node.

406

Question 115

When not to use Azure Stream Analytics?

Answer 115

Other cloud providers are selected.
To embed into cloud native applications.

406

Question 116

Describe Azure Stream Analytics

Answer 116

Azure Stream Analytics is a fully managed scalable streaming analytics platform offered by Microsoft. It supports defining stream-processing logic by using SQL queries and a graphical user interface. It supports stream-processing constructs such as transformations, filtering, thresholds, windowed aggregations, joins, temporal event ordering, and machine learning.

It also supports hybrid architectures for running stream-processing queries in the cloud and on the edge node. We recommend using Azure Stream Analytics if Azure is your hosting environment.

405

Question 117

When to use Google Dataflow?

Answer 117

To support transformations, filtering and thresholds, windowed aggregations, joins, temporal event ordering, and machine learning.
To support portable stream-processing logic that can also run on on-premises stream-processing systems.

406

Question 118

When not to use Google Dataflow?

Answer 118

Other cloud providers are selected.
To embed into cloud native applications.

406

Question 119

Describe Google Dataflow

Answer 119

Google Dataflow is a fully managed scalable stream-processing platform offered by Google. It supports defining stream-processing logic using Apache Beam SDK, SQL queries, and via GUI. It supports stream-processing constructs such as transformations, filtering, thresholds, windowed aggregations, joins, temporal event ordering, and machine learning.

With its Apache Beam SDK, it allows developers to deploy stream-processing logic into on-premises stream-processing systems such as Apache Flink. We recommend using Dataflow if Google Cloud is your hosting environment.

405