Lecture 5: 29th October 2019

Question 1

What is a workflow?

Answer 1

An orchestrated and repeatable pattern of activity, enabled by the systematic organization of resources into processes that transform materials, provide services, or process information. It can be depicted as a sequence of operations, the work of a person or group, the work of an organization of staff, or one or more simple or complex mechanisms.

From a more abstract or higher-level perspective, workflow may be considered a view or representation of real work. The flow being described may refer to a document, service, or product that is being transferred from one step to another.

Workflows may be viewed as one fundamental building block to be combined with other parts of an organization’s structure such as information technology, teams, projects, and hierarchies.

The coordinated execution of multiple tasks or activities.

Question 2

What is a workload?

Answer 2

The amount of work that a computer or computer system has been given to do at a given time.

Question 3

How can workloads be measured?

Answer 3

• log everything all the time? : generally expensive and infeasible
• log with sampling? : samples may miss events of interest - outliers are important in networking
• replay with logging turned on? : completely overlooks “Heisenbugs”

Question 4

Where could we measure workloads?

Answer 4

Nowhere really at the scale of datacentres of large companies. At 10GB/s and with 84 byte packets, you have ~ 70 ns to process each packet. CPU operations are on the level of 10s of ns, and having to include packet I&O, context switching, packet classification, etc make it infeasible

Question 5

What is the scale of traffic inside datacentres?

Answer 5

Big. Google had a 50x growth in traffic between 2008 and 2014. In 2015, Facebook web servers had 100s to 1000s of simultaneous connections, but their traffic within datacentres is several times larger than that which goes out into the Internet.

Question 6

Is there any relationship between the rate of packet drops and utilisation in datacentres?

Answer 6

No (because of incast)

Question 7

What is a top-of-rack?

Answer 7

A switch on top a rack of servers which connects them to each other and to other servers

Question 8

What is a server rack?

Answer 8

A framework cage that contains a number of specialised servers which slide into bays like shelves. The servers are commodity hardware.

Question 9

What are containers?

Answer 9

The normal atomic unit at which servers are bought for datacentres. They are groups of server racks, usually to the size of a shipping container - and sometimes in a shipping container.

Question 10

When do containers get replaced?

Answer 10

When ~ 10 of their machines fail

Question 11

What are containers aka?

Answer 11

blocks

Question 12

What is locality?

Answer 12

The degree to which network traffic does not travel far topologically, i.e. staying within the same server rack vs container or datacentre vs the wider Internet.

Question 13

What locality properties do servers in datacentres hold?

Answer 13

most traffic from a block within its block cluster, and a fifth of that within the same rack. outside its cluster, 12% within its own DC and 18% outside of it.

Question 14

What do the proprieties of datacentre traffic depend on?

Answer 14

the function of application; scale; network topology; protocols used

Question 15

What are the implications of datacentre networking?

Answer 15

large internal traffic; tight deadlines for (network) I/O; congestion and TCP incast must be prevented; networks are complex and shared by diff apps; centralised control at per-flow level hard

Question 16

What are the pros and cons of using few larger data centres versus using more smaller data centres?

Answer 16

fewer and larger is less management complexity and cost overall but more per site; higher latency with fewer large centres; app complexity greater with few large DCs; may need a hierarchical cache structure for progressively authoritative DCs; more multiplexing with fewer large DCs as all connecting to 1 place (?)

Question 17

What is the biggest design choice within datacentres? Why is it important?

Answer 17

How to connect racks together; need to allow rack and machine-wise addressing and routing and maximise performance.

Question 18

What is bisection bandwidth?

Answer 18

The minimum amount of capacity required to be cut in links to bisect (partition into halves) a network. Make the minimum number of cuts needed to separate the two partitions and then sum the bandwidth of the links cut. it represents the bandwidth available between the two partitions, and, thereby, the true bandwidth available in the entire system.

Question 19

What is the big switch approach?

Answer 19

The idea of connecting racks (ToRSes) with “bigger” switches.

Question 20

What are some problems with the big switch approach?

Answer 20

Presents single points of failure (mitigate this by duplication -> cable management) as well as scaling issues.

Question 21

What is a tree network DS?

Answer 21

When you have a hierarchy of increasingly high capacity switches connect different racks in a DS.

Question 22

What are the cons of a tree network?

Answer 22

It is very expensive and doesn’t scale. Higher capacity switches present even greater points of vulnerability to attack and congestion. There is also a limit on the maximum capacity of a network switch: can only go so far up/large.

Question 23

How can you surpass cost and design difficulties?

Answer 23

Using commodity hardware

Question 24

What are the pros of using commodity hardware?

Answer 24

same machines and switches hardware (and protocols and topologies) making maintenance easier; ensure vertical and horizontal scalability; greater capacity; lower latency

Question 25

What are fat-tree networks?

Answer 25

A network topology in which one type of switch is used to implement a k-ary tree with a chosen k.

Question 26

How are fat-trees arranged?

Answer 26

There will be k switches at the core (top of tree), then k pods of (k/2) aggregation switches and (k/2) edge switches. Aggregation switches connect edge switches to core switches. Edge switches connect aggregation switches to (k/2) nodes. Aggregation switches connect ot all the edge switches in their pod and to (k/2) core switches.

Question 27

What are k-ary trees?

Answer 27

k-ary tree means no node can be connected to > k children nodes.

Question 28

What are the pros and cons of fat-tree networks?

Answer 28

low cost; low complexity; high throughput; high redundancy; any pair of hosts get the full bisection bandwidth; can cut and maintain capacity in partitions if you cut up the network (remains in partitions if split)

Question 29

How are intra and inter datacentre routing different?

Answer 29

Inter DC routing different from intra: may be different in where connections are made in the network topology, as well as protocols used (IP and TCP vs OSPF and QUIC).

Question 30

Why may we want to make paths between nodes shorter?

Answer 30

lower latency and also higher capacity per flow: a packet that travels on a short path consumes a small amount of network capacity”

Question 31

How are throughput per flow, no of flows, total capacity, and mean path length in a datacentre naively related?

Answer 31

throughput per flow ≤ total capacity / [number of flows * mean path length]

Question 32

How can you increase the throughput per flow? What will an increased throughput per flow mean for a datacentre network?

Answer 32

lower the mean path length somehow in design of network topologies. This will increase the total network capacity of the datacentre network.

Question 33

Why is TCP unsuitable for use in datacentres?

Answer 33

uses indirect, loss-based congestion measures and control implemented at sender only; slow start and halving on timeout vs linear increase not optimal for bandwidth probing; incasts leading to timeouts and congestion

Question 34

What are some requirements for transport protocols in datacentres?

Answer 34

high burst tolerance; high throughput, low latency

Question 35

What is incast in TCP?

Answer 35

When a rise in the number of servers making requests in a datacentre increase to a certain level, throughput collapses.

Question 36

Why is TCP incast particularly a problem in datacentres?

Answer 36

meets preconditions: have large number of small requests; high-bandwidth, low-latency network; small number of switch buffers.

Question 37

What is the process by which incast occurs in TCP?

Answer 37

When a high number of (MapReduce) queries are made to servers, some time out, causing retransmissions, congesting the network, so fewer of the next queries get through, repeating until very high congestion leading to very low throughput as the number of requests rise.

Question 38

How would TCP incast be changed if you removed the minimum bound on RTO times from TCP?

Answer 38

improves throughput as number of servers increase, no incast-like collapse. but still starts to drop from ~ 40 servers. This is due to TCP’s clock: time unit in TCP, effectively replaces min RTO. Its default is 10ms, much more than RTTs within datacentres.

Question 39

How would TCP incast be changed if you removed the minimum bound on RTO times from TCP and change its clock time to microsecond magnitude?

Answer 39

Improves throughput throughout as num servers increase, No incast-like collapse or small dropoff seen when only removing min RTO

Question 40

What are mice and elephant flows? What are they sensetive to?

Answer 40

In computer networking, an elephant flow is an extremely large (in total bytes) continuous flow set up by a TCP (or other protocol) flow measured over a network link. Elephant flows, though not numerous, can occupy a disproportionate share of the total bandwidth over a period of time.

In computer networking, a mouse flow is a short (in total bytes) flow set up by a TCP (or other protocol) flow measured over a network link.

Elephant flows are sensitive to throughput. Mice flows are sensitive to delay.

Question 41

How are the total sizes of flows and makeup of total date from below and above 1MB distributed?

Answer 41

65% fo flows are < 1MB but 95% of transmitted bytes are from flows > 1MB: most flows are mice but vast majority of data sent in elephants.

Question 42

What is ECN?

Answer 42

Explicit Congestion Notification = an extension to IP and TCP that allows end-to-end notification of network congestion without dropping packets.

Question 43

How does ECN work?

Answer 43

The ECN bit in TCP headers set to 1 when congestion detected by the receiver, leading to a halving of bandwidth each window where an ECN bit is set to 1.

Question 44

What are some conditions needed for TCP incast to occur?

Answer 44

have large number of small requests; high-bandwidth, low-latency network; small number of switch buffers.

Question 45

What is the single root cause of TCP incast?

Answer 45

An imbalance between low link latency (µs) and RTO (ms)

Question 46

What is the TCP clock?

Answer 46

TCP’s representation of time, effectively its minimum unit of time to increment.

Question 47

How are RTTs distributed in a Bing cluster?

Answer 47

90% of packets had an RTT < 1ms, remaining 10% to max of 15ms.

Question 48

What is DCTCP?

Answer 48

Datacentre TCP = TCP modified for datacentres

Question 49

How does DCTCP work?

Answer 49

DCTCP extends ECN processing to estimate the fraction of bytes that encounter congestion rather than simply detecting that some congestion has occurred. In TCP, halve if any number of ECN bits in acks; in DCTCP try to be proportional: reduce by 5% per ECN bit received. Smaller changes but faster response

Question 50

How do the performance levels of normal TCP and DCTP compare in general and with respect to queues?

Answer 50

In DCTCP, queues are of much smaller and stable lengths; TCP has more variance and larger queue sizes; both get full line rate

Question 51

What are some types of workloads?

Answer 51

queries; lookups.