CH13 Security Testing

Question 1

Four Bounds of Security Testing

Answer 1

Dynamic Testing
Static Testing
Automatic Testing
Manual Testing

Question 2

Manual Dynamic Approach

Answer 2

Penetration Testing

Question 3

Automatic Dynamic Approaches

Answer 3

DAST, IAST, Vulnerability Scanner

Question 4

Automatic Static Approach

Answer 4

SAST

Question 5

Manual Static Approach

Answer 5

Manual Code Review

Question 6

Security Testing light definition

Answer 6

a systematic process for revealing flaws in information systems

Question 7

Dynamic Testing: The Core Idea

Answer 7

The running application is being tested

Strange inputs are made

The outputs are returned and the program behaviour is observed and reported

Dynamic Testing operates similarly to black-box testing, in that you’re comparing outputs against inputs and not necessarily assessing the code itself.
- means that hard-coded aspects of the program may get easily overlooked

Question 8

Static Security Testing Core Idea

Answer 8

Parse all source code and config files
Analyze (an abstract representation of) the parsed files
Report any problems found

Static Testing is similar in execution to program compilation

Question 9

Vulnerability existence against report spectrum

Answer 9

Weakness exists and reported = True Positive
Weakness exists and not reported = False Negative (DANGEROUS)
Weakness doesn’t exist and reported = False Positive (ANNOYING TIME WASTE)
Weakness doesn’t exist and not reported = True Negative

Question 10

For a fully automated security testing tool, we must make compromises regarding False Negatives and False Positives:

Answer 10

In case of doubt, report a potential vulnerability:
- we might “annoy” users with many findings that are not real issues
- we risk “the boy that cried wolf” phenomena

In case of doubt, we stay silent:
- we might miss severe issues

Question 11

Reasons and Recommendations for False Negatives

Answer 11

Fundamental: Under-approximation of the tool
- missing language features (might intercept data flow analysis)
- missing support for complete syntax (parsing errors)
Therefore Report to tool vendor

Configuration: lacking knowledge of insecure frameworks
- insecure sinks (output) and sources (input)
Therefore Improve configuration

Unknown security threats
- XML verb tampering
Therefore Develop new analysis for tool (might require support from tool vendor)

Security expert: “I want a tool with 0 false negatives! False negatives increase the overall security risk”

Question 12

Reasons and Recommendations for False Positives

Answer 12

Fundamental: over-approximation of the tool, e.g.m
- pointer analysis
- call stack
- control-flow analysis
Therefore report tool to vendor

Configuration: lacking knowledge of security framework, e.g.,
- sanitization functions
- secure APIs
therefore improve configuration

Mitigated by attack surface: strictly speaking a true finding, e.g.,
- No external communication due to firewall
- SQL injections in a database admin tool
Therefore should be fixed. In practice often mitigated during audit or local analysis configuration

Developer: “I want a tool with 0 false positives!” False positives create unnecessary effort

Question 13

Prioritization of Findings

Answer 13

A pragmatic solution for too many findings

Classification with clear instructions for each vulnerability has proven to be the most easy to understand.

Can clearly see:
- What needs to be audited
- What needs to be fixed
- as security issue
- quality issue
- Different rules for
- old code
- new code

Question 14

Mainly two patterns that cause security vulnerabilities

Answer 14

Local issues
- insecure functions
- secrets stored in the source code
Data-flow related issues
- XSS
- Secrets stored in the source code

Question 15

generic defects visible in the code

Answer 15

static analysis sweet spot. built-in rules make it wasy for tools to find these without programmer guidance

e.g. buffer overflows

Question 16

generic defects visible only in the design

Answer 16

Most likely to be found through architectural analysis.

e.g. the program executes code downloaded as an email attachement

Question 17

context specific defects visible in the code

Answer 17

Possible to find with static analysis, but customisation may be required
e.g. mishandling of credit card information

Question 18

context specific defects visible only in design

Answer 18

Requires both understanding of general security principles along with domain-specific expertise.

e.g. cryptographic keys kept in use for an unsafe duration.

Question 19

Static Application Security Testing SAST, Pragmatic static analysis is based on

Answer 19

successful developments from research community
- type checking
- property checking (model-checkng, SMT solving, etc.)
- Abstract interpretation
- …

techniques from the software engineering community
- style checking
- program comprehension
- security reviews
- …

Question 20

Type checkers are useful, but

Answer 20

may suffer from false positives/negatives
identifying which computations are harmful is undecidable

Question 21

why will the java compiler flag this as an error?

short s = 0;
int i = s;
short r = 1;

Answer 21

Java is a statically typed, compiled language. types are set at compilation. even prior to running the program, an error will flag as an integer is being assigned seemingly compatible data from an incompatible type short.

Question 22

why will the java compiler flag this as an error?

Answer 22

Object [] objs = new String[1];
objs[0] = newObject();

Question 23

Style Checkers

Answer 23

Enforce more superficial rules than type checkers
- like x == 0 vs 0 == x
Style checkers are often extensible
- PMD
- JSHint
Simple, but very successful in practice

Question 24

Program Understanding

Answer 24

Tools can help with
- understanding large code ases
- reverse engineering abstractions
- finding declarations and uses
- Analysing dependencies
- …
Useful for manual code/architectural reviews

Question 25

Fuzzing (DAST) core idea

Answer 25

Create large random strings
Pipe input into Unix utilities
Check if they crash

e.g.
Bash
$ fuzz 100000 -o outfile | deqn

started in 1988 by Barton Miller at University of Wisconsin

Question 26

Industrial Case Study: Fuzzing Chrome
Chrome’s Fuzzing Infrastructure

Answer 26

Automatically grab the most current Chrome LKGR (Last Known Good Revision)

Fuzz and test Chrome, starting with multi-million test cases

Thousands of Chrome instances on hundreds of virtual machines

Question 27

AddressSanitizer

Answer 27

Compiler which performs instrumentation

Run-time library that replaces malloc(), free(), etc

custom malloc() allocates more bytes than requested and “poisons” the redzones around the region returned to the caller

Heap buffer overrun/underrun (out-of-bounds access)

Use after free

Stack buffer overrun/underrun

Question 28

AddressSanitizer: Result

Answer 28

0 months of testing the tool with Chromium (May 2011)

300 previously unknown bugs in the Chromium code and in third-party libraries

Question 29

SyzyASAN

Answer 29

AddressSanitizer works only on Linux and Mac

SysyASAN uses a different instrumentation (using MS Visual Studio)

Question 30

ThreatSanitizer

Answer 30

Runtime data race detector based on binary translation

Supports also compile-time instrumentation
- Greater speed and accuracy

Data races in C++ and Go code

Synchronization issues

Question 31

libFuzzer

Answer 31

Engine for in-process, coverage-guided, whitebox fuzzing (works on top of fuzzers like addressSanitizer)

Question 32

Cluster Fuzzing: ClusterFuzz

Answer 32

A fuzzing tool making use of the following memory debugging tools with libFuzzer-based fuzzers:
- AddressSanitizer (ASan): 500 GCE VMs
- MemorySanitizer (MSan): 100 GCE VMs
- UndefinedBehaviorSanitizer (UBSan): 100 GCE VMs

Question 33

Fuzzing Challenges

Answer 33

Detecting input channel
Input generation
Deciding if the response is a bug or noot
How to get a test setup that is safe to use?
When did we test enough? (coverage)

Question 34

Random Fuzzing

Answer 34

All randomly generated input data

Very Simple
Inefficient
- random input is often rejected, as a specific format is required
- probability of causing a crash is very low
Unlikely generating random HTML documents that trigger edge cases

Question 35

Mutation-based Fuzzing

Answer 35

mutate existing data samples to create new test data

• little or no knowledge of the structure of the inputs is assumed
• anomalies are added to existing valid inputs
• anomalies may be completely random or follow some heuristics
• requires little to no set up time
• dependent on the inputs being modified
• may fail for protocols with checksums, those which depend on challenge response, etc.
• examples include Taof, Peach Fuzzer, ProxyFuzz

Question 36

Mutation-Based Fuzzing pros and cons

Answer 36

+easy to set up and implement
+requires little to no knowledge of the input format/protocol

-effectiveness limited by selection of initial data set
-has problems with file formats/protocols that require valid checksums

Question 37

Generation Based Fuzzing

Answer 37

define new tests based on models of the input format

Generate random inputs with the input specification in mind (RFC, documentation, etc.)

Add anomalies to each possible spot

Knowledge of the input format allows to prune inputs that are rejected by the application

Input can be specified by a grammar-based fuzzing tool

examples include Peach Fuzzer and SPIKE

Question 38

Generation-Based Fuzzing Pros and Cons

Answer 38

+ completeness (you can measure how much of the specification has been covered)
+ can handle complex inputs (e.g., that require matching checksums)

• building a generator can be a complex problem
• specification needs to be available
• Greybox-Fuzzing (Concolic Testing)
  
  • uses symbolic execution to trigger unused paths
  • invented by Microsoft and used for fuzzing file input routines
• Autodafe
  -Fuzzing by weighting attacks with markers
  
  • Open Source