Week 9 - Site Reliability Engineering

Question 1

What does SRE stand for?

Answer 1

Site Reliability Engineering

Question 2

What does Site Reliability Engineering do?

Answer 2

Work that has historically been done by an operations team, but using engineers with software expertise, and banking on the fact that these engineers are inherently both predisposed to, and have the ability to, design and implement automation with software to replace human labour

Question 3

Name 5 principles of Site Reliability Engineering.

Answer 3

Embrace risk

Use service level objectives

Use of error budgets

Eliminate toil

Monitoring

Question 4

What does maximising stability cause?

Answer 4

A reduce in feature deployment speed

An increase in feature deployment cost

Limits feature numbers

Question 5

What does SLO stand for?

Answer 5

Service Level Objectives

Question 6

What does SLI stand for?

Answer 6

Service Level Indicators

Question 7

What are SLOs the target values for?

Answer 7

Service Level Indicators (SLIs)

Question 8

Give 3 examples of SLIs.

Answer 8

Latency, throughput, uptime

Question 9

What does publishing SLOs as SLAs do?

Answer 9

Sets expectations about how a service will perform, reducing unfounded complaints about its performance

Question 10

What does SLA stand for?

Answer 10

Service Level Agreements

Question 11

How is the error budget calculated?

Answer 11

The expected uptime is agreed (1)

The actual uptime is measured (2)

The error budget is the difference between (1) and (2)

Question 12

Why is the error budget useful?

Answer 12

It says how much risk is allowed within a quarter. Once the agreed-upon number is exceeded, the team shifts its focus from the development of updates to improving reliability

Question 13

What type of work counts as toil?

Answer 13

Manual (issuing commands, running scripts)
Repetitive (done over and over again)
Automatable (could be designed away)
Tactical (interrupt-driven and tactical)
Devoid of enduring value (nothing gets better)
Scalable only linearly (with traffic volume)

Question 14

What is a typical goal for toil?

Answer 14

To keep toil below 50% of each Site Reliability Engineer’s time - at least 50% should be spent on engineering project work that will either reduce future toil or add service features

Question 15

Monitoring involves collecting, processing, aggregating and displaying real-time quantitative data about a system. What sort of data could this be?

Answer 15

Query counts and types

Error counts and types

Processing times

Server uptimes

Question 16

What are the four golden signals of monitoring?

Answer 16

Latency (the time for a service to respond)

Traffic (the number of requests/second)

Errors (the number of responses that are errors)

Saturation (the fraction of system capacity in use)

Question 17

Name 5 practices of SRE.

Answer 17

Blameless postmortems
Managing incidents
Emergency response
Being on-call
Practical alerting

Question 18

What does a pull model of data collection do?

Answer 18

Provides one way to monitor large systems - a monitoring device regularly requests service data, often through a “/health” endpoint. Hard to scale as the monitoring service must know about every individual service

Question 19

What does a push model of data collection do?

Answer 19

Provides another way to monitor large systems - a service regularly sends data to a monitoring service (perhaps to a flat file (logs) or to a time-series database (metrics). Easier to scale as every individual service only needs to know about the monitoring service

Question 20

How long should it take an SRE to cope with a user-facing incident?

Answer 20

5 minutes

Question 21

How long should it take an SRE to cope with systems incidents other than user-facing ones?

Answer 21

30 minutes

Question 22

What does a proper response take?

Answer 22

A playbook, regular rehearsal and training

Question 23

What does a proactive approach to emergency response involve?

Answer 23

Deliberately breaking systems to watch how they fail, and make changes to improve reliability and prevent failures from recurring

Question 24

What does incident management do?

Answer 24

Limits the disruption by an incident and restores normal business operations as quickly as possible

Question 25

What 2 things can an incident be?

Answer 25

Unmanaged - characterised by teams fixated on technical problems, with poor communication and much freelancing

Managed - characterised by teams with separate responsibilities, clear communication and a plan

Question 26

What can a postmortem be triggered by?

Answer 26

Downtime beyond a certain threshold

Data loss of any king

Intervention by an on-call engineer

Resolution time above some threshold

Manual incident discovery implying a monitoring failure

Question 27

What is a blameless postmortem?

Answer 27

Focus is on identifying the causes of the incident, without indicting any individual - there is no counter-productive finger pointing or shaming of individuals

Question 28

Name 5 common tenets of DevOps and SRE.

Answer 28

No more silos

Accidents are normal

Change should be gradual

Tooling and culture are interrelated

Measurement is crucial

Question 29

What do silos lead to?

Answer 29

A lack of collaboration, and to incentives for purely local optimisation