Program Analysis Tools

Question 1

Why are bugs costly to fix?

Answer 1

The longer it’s hidden, the more code may rely on it

Finding the root cause is difficult after the fact

Question 2

Why do tests not catch all bugs?

Answer 2

Testing is best effort at best

Question 3

What are some proactive ways to prevent bugs from appearing?

Answer 3

Search for known classes of bugs
Guard against certain classes of bugs
Prove that certain bugs are not present
Identify bad styles that may lead to bugs

Question 4

What is program analysis?

Answer 4

Tools and techniques that allows computers to automatically reason about the program’s behaviour

Question 5

Why would we want to push program analysis onto computers?

Answer 5

Computers excel at repetitive, subtle behaviour

Question 6

Why are errors like Apple’s goto fail issue so hard to catch for humans even though simple program analysis tools would be easily able to find it.

Answer 6

People are bad at identifying subtle details. Computers will analyze everything line by line with glossing over anything

Question 7

What are the 2 main categories of program analysis tools?

Answer 7

Dynamic and static analysis

Question 8

What do dynamic analysis tools do? What is it best used for? What are some examples?

Answer 8

Run the program under test and reason about that single execution
Best used for explaining bugs that are already happening
Debuggers, valgrind

Question 9

What do static analysis tools do? What is it best used for? What are some examples?

Answer 9

Examine source code/binary and reason about all possible executions
Best used for identifying bugs that haven’t struck yet but might in the future

Question 10

What are the limitations of dynamic approaches? How about static approaches?

Answer 10

Dynamic: As it’s driven by a single input, it will miss bugs caused by other inputs (false negatives)
Static: Because of undecidibility, it cannot be totally sure that there is a problem when it detects one (halting problem) (false positives)

Question 11

Most large companies are finding __________ to be unacceptable, which makes static approaches less widely used

Answer 11

false positives

Question 12

What does Valgrind use to analyse code?

Answer 12

Dynamic Binary Instrumentation. It’s like JVM for machine code

Question 13

What are some built in tools for Valgrind?

Answer 13

Memcheck, cachegrind, helgrind

Question 14

What does Valgrind not work for Java or Python by default?

Answer 14

Valgrind modifies a compiled C binary to check for errors, not other binaries

Question 15

What do Clang sanitizers use to analyse code?

Answer 15

Compile time instrumentation and rewrites the program once to perform analyses every time it executes

Question 16

Clang sanitizers are used a lot at ______

Answer 16

google

Question 17

Valgrind cannot detect invalid accesses of _____ ______ because of ________ ______ _______. What can detect these errors?

Answer 17

local variables, dynamic runtime instrumentation

Address Sanitizers

Question 18

When using a sampling tool, sampling ______ matters

Answer 18

frequency

Question 19

What is the general rule for sampling rate?

Answer 19

Twice the frequency of the operations you’re measuring

Question 20

What does the clang static analyzer ‘scan-build’ do?

Answer 20

Uses abstract interpretation to simulate many different paths though the program at once

Question 21

False ______ are no extra burden, but false ______ can waste developer time

Answer 21

negatives, positives

Question 22

What is one way to deal with false positives?

Answer 22

Save time in the future by blacklisting and suppressing previous types of false positives

Question 23

What is program verification? What is a tool that does this?

Answer 23

Proving the absence of certain types of bugs

CBMC

Question 24

What are some issues with program verification tools?

Answer 24

Difficult to use
More complex -> more overhead
Still approximate (will miss bugs in the end)