Observability

Question 1

4 Primary Aspects of Observability

Answer 1

Logs, Metrics, Alarms, Events

Question 2

Logs

Answer 2

a list of application information, system performance, or user activities that provide a verbose, running commentary of activity

Question 3

What interface do we use to query and visualize the log data stored in Elasticsearch?

Answer 3

Kibana

Question 4

What are logs most useful for?

Answer 4

Logs are very useful for diagnosing errors because they contain detailed data about activity as well as stack traces.

Question 5

Are logs or metrics more costly? Why?

Answer 5

Logs contain detailed information which makes them much more costly than metrics, so we should use logs sparingly. For example, we should use a metric to record that a request succeeded. We should use a log when a request fails, because then the stack trace and other context will be useful.

Question 6

Metrics

Answer 6

a set of numbers, measured over intervals of time, that give information about a particular process or activity

Question 7

3 Metric Types

Answer 7

Counters, Timers, Gauges

Question 8

Counters

Answer 8

statsd events which can count. The two most useful outputs are count:sum (sum of all increments) and count:rate (sum of all increments, normalized to increments per second)

How many times did an event occur?

eg. How many times was the database accessed?

Question 9

Timers

Answer 9

pre-computed quartiles/percentiles based on events coming into statsd. They may not represent time, but are used whenever you need to understand the distribution of a piece of data (such as finding the median, 95th percentile, etc). Timer metric names are suffixed with timer:[computed value] (for example, timer:p99).

eg. What was the average response time when accessing the database?
eg. How many miles was the average ride?

Question 10

Gauges

Answer 10

views over an internal value of an application. For most purposes, gauge:mean will be the most useful, representing the arithmetic mean of all received gauge values.

e.g. How many database connections are in our connection pool?

Question 11

Metric names all start with ______________

Answer 11

the environment they are in, specifically production or staging

Question 12

Alarms / Alerts

Answer 12

notifications sent to the on-call engineer when the data emitted by one or more metrics passes a certain threshold. The terms “alert” and “alarm” are used interchangeably

Question 13

Every alert has a condition that determines when it will fire, and that condition is based on ________

Answer 13

one or more metrics.

Question 14

What tool do we use to view alerts?

Answer 14

Lighthouse

Question 15

What tool do we use to manage incidents?

Answer 15

PagerDuty

Question 16

Events

Answer 16

structured, nestable key-value pairs that contain information about what it took for a service (e.g. a mobile app, backend app, web client, etc.) to perform a unit of work. Events capture rich details regarding system state changes or any significant occurrences that need to be tracked and propagated.

In that way, events are similar to structured logs (a log with some key-value structure like JSON), but operate at a higher abstraction.

Whereas a structured log would have information about an individual action that happens in a service — for example, that a service launch succeeded or failed — events contain useful metadata information, such as how long it took to launch the service, or how many times the service failed before it gave up.

Question 17

Hive Query

Answer 17

the process for accessing older logs

Question 18

AWS S3

Answer 18

used as our backup logging datastore. Logs are kept on S3 for 120 days

Question 19

Presto

Answer 19

Query engine used to search through the AWS S3 logs

Question 20

Spellbook

Answer 20

Used to issue search queries

Start a new query in Spellbook. Choose Presto as the Database and raw_events as the Schema. You can then search logs in the event_logging_event Hive table based on date.

Question 21

M3

Answer 21

Our in-house Timeseries database based on Open-source M3 to store our real-time metrics and alerts. We use statsd to aggregate events and relay them to M3 via the wavefrontproxy service.

Metrics can be queried on Lighthouse using the query builder. Queries are written in the open-source PromQL query language- see the query language reference for details.

The charts in Grafana dashboards (such as the Lyft dashboard) are built with M3 queries. To add a new chart to a Grafana dashboard, you’ll need to write its M3 query first. M3 querying can be very useful in tracking everything from communication between services to ride cancellations.

Question 22

Dashboards (Grafana)

Answer 22

The primary visual tool for service health monitoring

Question 23

4 Types of Dashboards in Grafana

Answer 23

Default service dashboards, Managed dashboards, Deploy dashboards, Code Health dashboards

Question 24

Default service dashboards

Answer 24

standardized dashboards generated for all services that have a pillar defined under ops/config/pillar/per_service. These dashboards are intended to provide a good starting point for monitoring standard Python or Go services. Service owners can configure many aspects of their dashboards by updating settings in the pillar.

Question 25

Managed dashboards

Answer 25

one-off, highly customizable dashboards and are kept in ops/config/states/managed_dashboards. They are used when a dashboard is required that does not fit into the usual paradigm of monitoring the health of a Python or Go service.

Question 26

Deploy dashboards

Answer 26

show canary-only metrics, allowing you to compare the metrics of a deployed canary instance with the rest of the cluster. To generate a deploy dashboard for your service, add deploy_dashboard: True to the service pillar (example) and a dashboard named {service name}-deploy will be created in Grafana. Deploy dashboards are deployed during the ‘extra’ deploy step.

Question 27

Code Health dashboards

Answer 27

show overall code health and statistics related to the deploy pipeline. Code Health dashboards are deployed during the ‘extra’ deploy step.