Intro

Question 1

Approaches for coping with load?

Answer 1

1. Scaling up or vertical scaling
2. Scaling out or horizontal scaling
3. Using Elastic systems in case of unpredictable load.

Question 2

Scaling up or vertical scaling?

Answer 2

Moving to a more powerful machine

Question 3

Scaling out or horizontal scaling?

Answer 3

Distributing the load across multiple smaller machines

Question 4

What are Elastic systems? And when are they useful?

Answer 4

Automatically add computing resources when detected load increase.
Quite useful if load is unpredictable.

Question 5

What are the three design principles for software systems in terms of maintainability?

Answer 5

1. Operability
2. Simplicity
3. Evolvability

Question 6

What is Operability?

Answer 6

Make it easy for operation teams to keep the system running.

Data systems can do the following to make routine tasks easy e.g.

1. Providing visibility into the runtime behavior and internals of the system, with good monitoring.
2. Providing good support for automation and integration with standard tools.
3. Providing good documentation and easy-to-understand operational model (“If I do X, Y will happen”).
4. Self-healing where appropriate, but also giving administrators manual control over the system state when needed.

Question 7

What is Simplicity?

Answer 7

Easy for new engineers to understand the system by removing as much complexity as possible.

Question 8

What is Evolvability?

Answer 8

Make it easy for engineers to make changes to the system in the future.

Question 9

What are functional and nonfunctional requirements?

Answer 9

1. Functional requirements: what the application should do
2. Nonfunctional requirements: general properties like security, reliability, compliance, scalability, compatibility and maintainability.

Question 10

What is the difference between Latency and response time?

Answer 10

The response time is what the client sees. Always measured on client side.
Latency is the duration that a request is waiting to be handled.

Question 11

When we measure response time, what is a better metric than average response time? And why?

Answer 11

Percentiles are a metric than average response time as percentiles tells how many users actually experienced that delay

1. Median (50th percentile or p50). Half of user requests are served in less than the median response time, and the other half take longer than the median
2. Percentiles 95th, 99th and 99.9th (p95, p99 and p999) are good to figure out how bad your outliners are.

Question 12

What are the common percentiles measures used for response time?

Answer 12

1. Median (50th percentile or p50). Half of user requests are served in less than the median response time.
2. Percentiles 95th, 99th and 99.9th (p95, p99 and p999) are good to figure out how bad your outliners are.

Question 13

Give an example of using 99.99 percentile as a measure of response time? And why is not common practice?

Answer 13

Amazon uses 99.9th percentile for response time requirements for internal services because the customers with the slowest requests often have the most data.

Question 14

What accounts for large part of response times at high percentiles? And why are high percentiles not common practice?

Answer 14

Queueing delays often account for large part of the response times at high percentiles.
Optimizations are expensive at high percentiles.

Question 15

What are SLOs and SLAs?

Answer 15

Service level objectives (SLOs) and service level agreements (SLAs) are contracts that define the expected performance and availability of a service. These metrics set expectations for clients of the service and allow customers to demand a refund if the SLA is not met.

An SLA may state the median response time to be less than 200ms and a 99th percentile under 1s.

Question 16

What is reliability?

Answer 16

Reliability: The system should work correctly (performing the correct function at the desired level of performance) even in the face of adversity.

Question 17

What is Scalability?

Answer 17

Scalability: As the system grows(in data , traffic volume, or complexity), there should be reasonable ways of dealing with that growth.

Question 18

What is maintainability?

Answer 18

Maintainability: People should be able to work on the system productively in the future.

Question 19

How can human errors be reduced?

Answer 19

1. Designing systems in a way that minimize opportunities for error through well-designed abstractions, APIs, and admin interfaces.
2. Decoupling the places where people make the most mistakes from the places where they can cause failures. E.g. by providing a fully-featured non-production sandbox environment where people can explore and experiment safely, using real data, without affecting real users.
3. Testing thoroughly at all levels: from unit tests to integration tests to manual tests to automated tests.
4. Allow quick and easy recovery from human errors, to minimize the impact of failure. E.g. By making it easy to roll back configuration changes, roll out new code gradually ( so bugs do not affect all users).
5. Set up detailed and clear monitoring, such as performance metrics and error rates.