Observability patterns Flashcards

1
Q

Observability patterns

A
Log aggregation
Application metrics
Audit logging
Distributed tracing
Exception tracking
Health check API
Log deployments and changesnew
How well did you know this?
1
Not at all
2
3
4
5
Perfectly
2
Q

Log aggregation: contexts

A

You have applied the Microservice architecture pattern. The application consists of multiple services and service instances that are running on multiple machines. Requests often span multiple service instances.

Each service instance generates writes information about what it is doing to a log file in a standardized format. The log file contains errors, warnings, information and debug messages.

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
3
Q

Log aggregation: problem

A

How to understand the behavior of an application and troubleshoot problems?

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
4
Q

Log aggregation: forces

A

Any solution should have minimal runtime overhead

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
5
Q

Log aggregation: solution

A

Use a centralized logging service that aggregates logs from each service instance. The users can search and analyze the logs. They can configure alerts that are triggered when certain messages appear in the logs.

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
6
Q

Log aggregation: examples

A

AWS Cloud Watch

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
7
Q

Log aggregation: result issue

A

handling a large volume of logs requires substantial infrastructure

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
8
Q

Log aggregation: related

A

Distributed tracing - include the external request id in each log message

Exception tracking - as well as logging exceptions, report them to an exception tracking service

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
9
Q

Application metrics: contexts

A

You have applied the Microservice architecture pattern.

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
10
Q

Application metrics: problem

A

How to understand the behavior of an application and troubleshoot problems?

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
11
Q

Application metrics: forces

A

Any solution should have minimal runtime overhead

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
12
Q

Application metrics: solution

A

Instrument a service to gather statistics about individual operations. Aggregate metrics in centralized metrics service, which provides reporting and alerting. There are two models for aggregating metrics:

  • push - the service pushes metrics to the metrics service
  • pull - the metrics services pulls metrics from the service
How well did you know this?
1
Not at all
2
3
4
5
Perfectly
13
Q

Application metrics: example

A
Instrumentation libraries:
- Coda Hale/Yammer Java Metrics Library
- Prometheus client libraries
Metrics aggregation services:
- Prometheus
- AWS Cloud Watch
How well did you know this?
1
Not at all
2
3
4
5
Perfectly
14
Q

Application metrics: result benefits

A

It provides deep insight into application behavior

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
15
Q

Application metrics: result drawbacks

A

Metrics code is intertwined with business logic making it more complicated

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
16
Q

Application metrics: result issues

A

Aggregating metrics can require significant infrastructure

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
17
Q

Audit logging: context

A

You have applied the Microservice architecture pattern.

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
18
Q

Audit logging: problem

A

How to understand the behavior of users and the application and troubleshoot problems?

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
19
Q

Audit logging: forces

A

It is useful to know what actions a user has recently performed: customer support, compliance, security, etc.

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
20
Q

Audit logging: solution

A

Record user activity in a database.

21
Q

Audit logging: result benefits

A

Provides a record of user actions

22
Q

Audit logging: result drawbacks

A

The auditing code is intertwined with the business logic, which makes the business logic more complicated

23
Q

Audit logging: related

A

Event Sourcing is a reliable way to implement auditing

24
Q

Distributed tracing: context

A

You have applied the Microservice architecture pattern. Requests often span multiple services. Each service handles a request by performing one or more operations, e.g. database queries, publishes messages, etc.

25
Q

Distributed tracing: problem

A

How to understand the behavior of an application and troubleshoot problems?

26
Q

Distributed tracing: forces

A
  • External monitoring only tells you the overall response time and number of invocations - no insight into the individual operations
  • Any solution should have minimal runtime overhead
  • Log entries for a request are scattered across numerous logs
27
Q

Distributed tracing: solution

A

Instrument services with code that

  • Assigns each external request a unique external request id
  • Passes the external request id to all services that are involved in handling the request
  • Includes the external request id in all log messages
  • Records information (e.g. start time, end time) about the requests and operations performed when handling a external request in a centralized service

This instrumentation might be part of the functionality provided by a Microservice Chassis framework.

28
Q

Distributed tracing: result benefits

A
  • It provides useful insight into the behavior of the system including the sources of latency
  • It enables developers to see how an individual request is handled by searching across aggregated logs for its external request id
29
Q

Distributed tracing: result issues

A

Aggregating and storing traces can require significant infrastructure

30
Q

Distributed tracing: related

A

Log aggregation - the external request id is included in each log message

31
Q

Exception tracking: context

A

You have applied the Microservice architecture pattern. The application consists of multiple services and service instances that are running on multiple machines. Errors sometimes occur when handling requests. When an error occurs, a service instance throws an exception, which contains an error message and a stack trace.

32
Q

Exception tracking: problem

A

How to understand the behavior of an application and troubleshoot problems?

33
Q

Exception tracking: forces

A
  • Exceptions must be de-duplicated, recorded, investigated by developers and the underlying issue resolved
  • Any solution should have minimal runtime overhead
34
Q

Exception tracking: solution

A

Report all exceptions to a centralized exception tracking service that aggregates and tracks exceptions and notifies developers.

35
Q

Exception tracking: result benefits

A

It is easier to view exceptions and track their resolution

36
Q

Exception tracking: result drawbacks

A

The exception tracking service is additional infrastructure

37
Q

Exception tracking: related

A

Log aggregation - exceptions should be logged as well as reported to a tracking service

38
Q

Health Check API: context

A

You have applied the Microservice architecture pattern. Sometimes a service instance can be incapable of handling requests yet still be running. For example, it might have ran out of database connections. When this occurs, the monitoring system should generate a alert. Also, the load balancer or service registry should not route requests to the failed service instance.

39
Q

Health Check API: problem

A

How to detect that a running service instance is unable to handle requests?

40
Q

Health Check API: forces

A
  • An alert should be generated when a service instance fails

* Requests should be routed to working service instances

41
Q

Health Check API: solution

A

A service has an health check API endpoint (e.g. HTTP /health) that returns the health of the service. The API endpoint handler performs various checks, such as:

  • the status of the connections to the infrastructure services used by the service instance
  • the status of the host, e.g. disk space
  • application specific logic

A health check client - a monitoring service, service registry or load balancer - periodically invokes the endpoint to check the health of the service instance.

42
Q

Health Check API: result benefits

A

The health check endpoint enables the health of a service instance to be periodically tested

43
Q

Health Check API: result drawbacks

A

The health check might not sufficiently comprehensive or the service instance might fail between health checks and so requests might still be routed to a failed service instance

44
Q

Health Check API: related

A

Service registry - the service registry invokes the health check endpoint

45
Q

Log deployments and changes: context

A

You have applied the Microservice architecture pattern.

46
Q

Log deployments and changes: problem

A

How to understand the behavior of an application and troubleshoot problems?

47
Q

Log deployments and changes: forces

A

It useful to see when deployments and other changes occur since issues usually occur immediately after a change

48
Q

Log deployments and changes: solution

A

Log every deployment and every change to the (production) environment.

49
Q

Log deployments and changes: result benefits

A

Enables deployments and changes to be easily correlated with issues leading to faster resolution.