Performance

Question 1

What are several attributes of performance?

Answer 1

Speed
Throughput
Responsiveness
Memory Consumption

Question 2

To measure performance, we need to design ______ specifically targeting performance objectives

Answer 2

tests

Question 3

When measuring performance, it really comes down to _______

Answer 3

benchmarking

Question 4

What are microbenchmarks?

Answer 4

Focusing on small things, like the cost of an operation in isolation. It can help identify core performance details and explain causes

Question 5

What are macrobenchmarks?

Answer 5

Real world system performance

Question 6

What are some possible issues in the below code?
startTime = getCurrentTimeInSeconds();
doWorkloadOfInterest();
endTime = getCurrentTimeInSeconds();
reportResult(endTime – startTime);

Answer 6

Measurement granularity
Warm up effects
Nondeterminism
Size of workload
System interference
Background programs?

Question 7

When bench marking, why do workloads need to be chosen carefully?

Answer 7

Workloads need to be representative. Some inputs can be pathological to stress test.

Question 8

Measuring something ____ means nothing

Answer 8

once

Question 9

Using ______ instead of time can be better to predict performance on different machines

Answer 9

cycles

Question 10

Why is warm up time necessary?

Answer 10

Code might not be compiled until it’s executed frequently (JIT for Java)
Need to make sure code is paged in`

Question 11

How can we alleviate warm up time issues in tests?

Answer 11

Run the workload many times before benchmarking it (Bad strategy for JVMs)

Question 12

When benchmarking, you should benchmark many times and average the times to combat ______________ behavior. However, this might throw info away and we might want to know each value

Answer 12

Nondeterministic

Question 13

How do we interpret/report a benchmark?

Answer 13

Compare that benchmark with your expectation, different solutions, or the performance over time

Question 14

What is a benchmark suite?

Answer 14

A dataset with multiple benchmarks

Question 15

What are two methods we can use to consider interpreting a benchmark suite?

Answer 15

Hypothesis testing (ANOVA)
Summary statistic (condensing a suite to a single number)

Question 16

What are the 2 different types of means? What are they each used for? Give their formulas

Answer 16

Arithmetic Mean - Good for reporting averages of numbers that mean the same thing: (Summation of X)/N
Harmonic Mean - Good for reporting rates: N/(Summation of (1/x))
Geometric Mean - Good for reporting results that mean different things (timing results across many different tests in the same suite: Nth root(PI(X))

Question 17

If a size of a workload changes, a _______ harmonic mean is required

Answer 17

weighted

Question 18

Why would we want to use a geometric mean?

Answer 18

Review lecture video for slide 75

Question 19

What low level architectural effects can determine the performance? How well does sampling based profiling work for these? What other tools can measure these effects?

Answer 19

Cache misses
Mis-speculations
TLB misses
Not well, due to prediction issues
Tools like perf

Question 20

In what ways can we optimize algorithms in the real world?

Answer 20

Hybrid Algorithms - where constants matter
Caching and precomputing
Better performance modeling and algorithms
Asymptotic complexity is less useful in practice - We want to focus on memory, I/O, consistency and speculation, and shapes of workloads
Cache oblivious algorithms and data structures