SE4452 Testing FINAL Flashcards
1
Q
Genniwa’s favourite word
A
What is certainly?
2
Q
What is the purpose of testing?
A
To verify the thing we produce accurately represents its specifications
3
Q
What are the two main testing categories?
A
- statistical testing
2. defect testing
4
Q
What is defect testing?
A
Designed to discover system defects
5
Q
What is statistical testing?
A
Designed to reflect the frequency of user inputs
6
Q
What are the stages of testing?
A
unit
integration
system
user/acceptance
7
Q
What is user/acceptance testing?
A
Testing by users to validate the system meets the business and user needs
8
Q
What is integration testing?
A
Testing to expose problems arising from the combination of components
9
Q
What is unit testing?
A
testing of individual components
10
Q
What is system testing?
A
Testing the complete system prior to delivery
11
Q
What quality requirements does a system test consider?
A
System Operational quality requirements
12
Q
What quality requirements does a deployment test consider?
A
User’s environment quality requirements
13
Q
What is the output of the system test stage?
A
Operational system
14
Q
What is the input of the system test stage?
A
Functioning system
15
Q
What is the output of the Integration test stage?
A
Integrated subsystems
16
Q
What is the output of the deployment test stage?
A
Usable system
17
Q
What is the output of the Acceptance test stage?
A
Accepted system
18
Q
What type of testing is known as Black Box?
A
Functional
19
Q
What type of testing is known as White Box?
A
Structual
20
Q
What is regression testing?
A
Ensures changes in a program do not affect functionality/behavior that it should not affect
21
Q
Regression testing is used in which stage(s) of testing?
A
Unit
integration
system
22
Q
functional testing is used in which stage(s) of testing?
A
unit
integration
system
acceptance
23
Q
structural testing is used in which stage(s) of testing?
A
unit
integration
24
Q
What is the objective of unit testing?
A
verify whether the implementation at the unit-level conforms to the Quality requirements
25
What is the objective of integration testing?
to take unit tested modules and build a system structure based on the prescribed design
verify whether the implementation at the system architecture level conforms to the Quality requirements
26
When is integration testing complete?
- entire modules are fully integrated
- all test cases executed
- all severe and moderated defects found have been fixed
27
what is the objective of system testing?
verify whether the implementation at the system-level conforms to the Quality requirements.
28
What environment is used for system testing?
In the real (production) environment if available, or an emulated environemnt
29
What environment is used for user/acceptance testing?
The production/live environment
30
T/F: Unit testing requires both functional and structural testing
True
31
Whats the difference between testing and debugging?
Testing is concerned with confirming the presence of defects
Debugging is concerned with locating and repairing defects
32
What are the 5 testing activities
IDBEC
```
Identify test conditions
Design test cases
Build test cases
Execute
Compare test with expected output
```
33
What are the 4 debugging activities?
LDRR
Locate defect
Design defect repair
Repair defect
Re-test
34
What is a test condition?
A description of circumstances that could be examined
35
What are the categories of test conditions?
Functionality
Structural/system
User-acceptance
36
What is a test case?
The details of a test condition
Consists of test data and expected output
37
What is a test set?
A collection of test cases
38
What are the two steps to build a test case?
Implement preconditions (set up environment)
Prepare test scripts
39
What kind of data/instructions do test case scripts contain?
```
comparison info
what screen data to capture
when/were to read input
control info
testing concurrent activities
```
40
What are the two steps of test case comparison?
1. observe behavior
| 2. analyze behavior to check if it is correct
41
What is the name of the system that performs the checking of the correctness of observed behavior of a test case?
An oracle
42
What are the different ways for program bahvior to be specified?
- plain language
- state diagram
- formal math specification
43
How do you measure the goodness of test cases?
Does it:
- cover certain quality requirements
- cover certain parts of functionality
- cover certain parts of internal logic
44
How is functional coverage achieved?
Test all possible inputs (or at least partition of inputs)
45
How is structural coverage achieve?
Cover the underlying code (all statements, branches, loops)
46
What are the different test stopping criteria?
- meet deadline, budget (management)
- achieve desired coverage
- achieve desire failure intensity
47
What is equivalence class partitioning?
A test data selection technique, to test if unit correctly implements all functionalities
48
What is boundary value analysis?
Test selection technique that targets failures at the boundaries of equivalence classes
49
T/F: boundary value and equivalence class test cases cannot overlap
False
certainly they certainly can
certainly
50
What is a CFG?
Control Flow Graph
A directed labeled graph in which nodes are statements (or groups of statements) and edges represent flow of control
51
What are the different levels of code coverage for white box testing?
statement
edge (branch)
condition
path
52
What is required for edge coverage?
Each edge is executed at least once (covers true/false conditions)
53
What is required for condition coverage?
Edge coverage
+
all combinations of predicate outcomes
54
What is required for path coverage?
All paths (from initial to final node) are covered at least once
55
What is required for statement coverage?
All statements executed
56
In path coverage, for a loop based path, at least how many test cases are required? What are they?
3 cases that execute the loop:
zero times (skips the loop)
once
more than once
57
In unit testing, what is a stub?
Replaces subordinate modules
Uses modules' exact interface, may do minimal data manipulation, provides verification of entry, returns control
58
In unit testing, what is a driver?
Main program that accepts test case data, passes data to unit being tested, returns results
59
What is the test selection problem?
To select a subset of tests that will reveal all failures in a program
60
Why is the test selection problem difficult?
Input domain is large
Input domain has complex structure
61
What is exhaustive testing?
Testing a program against every element in its input domain
62
What does it mean that Equivalence classes are disjoint?
They do not overlap
63
What are the objectives of test generation techniques?
1. completeness
| 2. efficiency (avoid redundancy)
64
What is weak equivalence class testing?
Using one variable from each equivalence class in a test case
Min # of classes is equal to number of classes in the partition with the most subsets
65
What is strong equivalence class testing?
Using the cartesian product of the partition subsets
66
What are the two main subsets of inputs?
expected (valid) inputs
| unexpected (invalid) inputs
67
T/F: strong equivalence classes use a single fault assumption
False - strong equivalence classes are a multiple fault assumption
68
What is a robust selection strategy?
Classes based on valid and valid inputs
69
What is a normal selection strategy?
classes based on valid values of inputs
70
How is effectiveness in equivalence class testing measured?
ratio of :
number of failures exposed
_______________
total failures that exist
71
How is efficiency in equivalence class testing achieved?
By only using one element from each equivalence class
72
What equivalance classes would be required for a string variable?
One class of valid strings
one class of invalid strings
73
What equivalance classes would be required for an array variable?
one class of legal arrays
one class with the empty array
one class with arrays that are too large
74
What equivalance classes would be required for a compound data type?
Uni-dimensional (weak) partitioning: partition input domain one variable at a time
Multidimensional (strong) partitioning: input domain is the set product of input variables
75
What are the steps for equivalence partitioning?
1. Identify input domain
2. Equivalence classing
3. Combine classes
4. Identify infeasible classes
76
What are the steps for boundary value analysis?
1. partition input domain
2. identify boundaries for each partition
3. select test data
77
For each boundary, how many test cases are required in BVA?
3
The boundary, before, and after
78
When does using only BVA work well?
When the program is a function of independent variables that represent bounded physical quanitities
79
What are the limitations of BVA?
No consideration of the function of program or semantic meaning of variables
80
What does a decision table make it easier to observe?
That all possible conditions are accounted for
81
When is logic based testing considered as structural testing?
When applied to structure
82
When is logic based testing considered as functional testing?
When applied to a specification
83
What are the four quadrants of a decision table?
Conditions
Condition entries
Actions
Action entries
84
What is a limited entry decision table?
Only boolean value conditions
85
What is an extended entry decision table?
Several values as conditions
86
What indicates the action to be taken when no other rules apply in a decision table?
The default rule
87
What are the two requirements of a decision table?
Rules must be complete (every combination of conditions + default rule present)
Rules must be consistent (every combination of conditions results in only 1 set of actions)
88
What are the steps for decision table development?
1. Determine actions, conditions, max # rules
2. Encode possible rules, actions for each rule
3. Simplify rules (reduce # columns)
89
How are test cases built from a decision table?
Each rule is a test case
conditions are equivalence classes of inputs
actions are outputs
90
What are the aspects of unit testing?
- unit test framework
- languages (built in, external modules)
- test protocols (TAP)
- test location (server/client side)
91
What framework is Mocha based on?
Javascript test framework
92
T/F: Mocha tests run serially
True
93
T/F: Mocha testing is asynchronous
True
94
Types of testing for system testing:
Functionality, Performance, Security, Load, Stress, Scalability, Robustness testing (SSSLRP F)
95
What client side platform does Mocha use?
browser
96
Difference between testing and debugging
Testing: Confirming the presence of defects
Debugging: Locating and fixing defects
97
In Mocha, what is TestRunner?
The driver, which runs TestCases and reports test results
98
What is a test fixture in Mocha?
A test environment used by multiple tests
99
What is TAP?
Test anything protocol?
100
What does TAP accomplish?
Allows communication between unit tests and a test harness
101
What are the two basic functions required for writing a test in Mocha?
1. describe (), which is the test suite/group
| 2. it(), which is the individual test
102
What are the four hooks in Mocha?
before
beforeEach
after
afterEach
103
What are the requirements for structural testing?
1. cover all statements
2. cover all branches
3. cover all loops
104
What are the 4 Stopping Criteria for Testing?
1. Meet Deadline
2. Exhaust Budget
3. Achieved Desired Coverage
4. Achieved Desired Failure Intensity
105
What are the 4 testing stages?
1. Unit Testing
2. Integration Testing
3. System Testing
4. User acceptance testing
106
In Mocha, what are dynamic tests?
When test cases are creates in an array, and the it function is inside a loop iterating through array
107
What is boundary value analysis?
targets failures in applications at the boundaries of equivalence classes
108
What are the testing stages of Structural Testing
Unit, Integration Testing
109
What are the 4 coverage objectives with structure testing? (SEC P)
1. Statement testing
2. Edge (decision) testing
3. Condition testing
4. Path testing
110
When is integration testing complete?
1. Entire modules are fully integrated
2. all test cases are executed
3. Severe and moderate defects have been fixed
111
Why is test selection difficult? (two reasons)
1. Large size of test
| 2. complex structure of system
112
Which functional testing technique has the highest test identification effort?
Decision table
113
Which functional testing technique has the lowest test identification effort?
BVA
114
What is the main assumption for equivalence classes?
The program exhibits the same behaviour under testing compared to production. This ensure only one test case for each equivalence class is sufficient
115
Which functional testing technique has the lowest test execution effort?
DT
116
what are the entire inputs of an application divided into
1. All expected inputs
| 2. All unexpected inputs
117
In general, is functional testing effective?
No, because it is highly dependent on clarity of quality requirements
118
If the test variables are logical quantities, which functional testing technique should be used?
EC or DT
119
If the test variables are physical quantities or independent domains, which functional testing technique should be used?
EC
120
If the test variables are dependent, which functional testing technique should be used?
DT
121
If the single fault assumption is plausible, which functional testing technique should be used?
EC weak robust
122
What is the "strong" equivalence class testing strategy?
(multiple fault assumption) one from each class in cartesian product
123
What is data flow testing?
selecting paths through control flow to explore sequences of events related to the status of variables
124
What do structural testing techniques provide?
- rigorous test definitions
- mathematical analysis
- precise measurements
125
Two types environments for system testing:
1. Emulated Environment
| 2. Real Production Environment
126
Difference between testing and debugging
Testing: Confirming the presence of defects
Debugging: Locating and fixing defects
127
What are the components of a test case pair
1. tast data to input
| 2. expected output
128
What is a test set?
Collection of one or more test cases
129
How do you build a test case?
1. Set up the environment
| 2. prepare test scripts
130
What are scripts?
Contain data and instructions for testing
131
What is an oracle?
System that performs checking of correctness of observed behaviour
132
What does an oracle require?
Determination of input-output behaviour
133
What are the requirements for structural testing?
1. cover all statements
2. cover all branches
3. cover all loops
134
What is coverage, as a metric of effectiveness?
The percentage of parts of an implementation exercised by a test suite
135
What does a code coverage model do?
Calls out the parts of an implementation that must be exercised to satisfy an implementation-based test model
136
Structurally, a path is _____
A sequence of statements in a program unit
137
Semantically, a path is _____
An execution instance of a program unit
138
What is the fault assumption for path testing techniques?
A fault may occur to make the program take an unintended path
139
What is a DD Path?
A decision-to-decision path, which is a collection of chains obtained from a CFG?
140
In a DD-path, every interior node has how many indegrees and outdegrees?
1 indegree, 1 outdegree
141
What are the 5 cases for DD-path nodes?
1. single node with indeg=0 (initial)
2. single node with outdeg=0 (final)
3. single node with branch
4. single node, no branch
5. Maximal chain of nodes
142
T/F: A DD-path is an expansion graph
F: It is a condensation graph
143
What is the motivation of using DD-paths?
Enable precise descriptions of test coverage
144
T/F: DD-path testing supports metrics for coverage criteria to quantify testing efficiency
F: metrics quantify testing effectiveness
145
What is the DD-path coverage metric C0
Statement coverage
146
What is the DD-path coverage metric C1P
Every predicate outcome coverage
147
What is the DD-path coverage metric CMCC
Multiple Condition Coverage
148
What is the DD-path coverage metric C1
predicate/edge coverage
| branch coverage
149
What is the DD-path coverage metric Cd
Dependent DD-path pair coverage
150
What is the DD-path coverage metric C2
loop coverage
151
What is predicate coverage?
Tests evaluate each predicate to true and false (does NOT consider every boolean combination)
Edge coverage
152
What is dependent DD-path pair coverage?
Includes edge coverage + coverage of dependent path pairs
153
When are 2 DD-paths dependent?
Iff there is a define/reference relationship between the paths
154
What does C1P test?
Tests all possible outcomes of boolean expressions
155
What is Multiple Condition Coverage?
Tests each possible combination of inputs for each decision
156
For code with n conditions, how many test cases are needed for MCC?
2^n
157
For code with n conditions, how many test cases are needed for MC/DC?
n+1
158
What is MC/DC?
A modification of MCC, which requires each condition to independently affect the outcome of a decision
159
In loop coverage, what are the possible outcomes of the decision of a loop decision?
1. traverse the loop
| 2. Exit (skip) the loop
160
How does the BVA extension to loop coverage work?
```
The loop index is given a:
minimum
minimum +
nominal
maximum -
maximum
```
161
In loop coverage, once a loop is tested, what can you do?
Loop structure can be collasped into a single node
162
In loop coverage, what is done if loops are knotted?
Apply data flow analysis testing techniques
163
In loop coverage, what is done if loops are nested?
Start with inner most loop, proceed outwards
164
What are the objectives of coverage analysis?
- determine gaps in test cases
- identify tests to fill gaps
- establish quantitative measure for efficiency and effectiveness
165
What are the 4 Stopping Criteria for Testing?
1. Meet Deadline
2. Exhaust Budget
3. Achieved Desired Coverage
4. Achieved Desired Failure Intensity
166
What is the major issue of statement coverage?
It is insensitive to some control structures
167
T/F: Statement coverage considers do-while loops the same rank as branching statements
False: same rank as NON-branching statements
168
T/F: Statement coverage does not report whether loops reached termination condition
T
169
It is acceptable to release code based on statement coverage alone?
NO
170
What does decision coverage report?
Whether boolean expressions are evaluated to both true and false. The entire Boolean expression is considered one true/false predicate
171
In addition to boolean expressions, decision coverage includes coverage of what?
- switch cases
- exception handlers
- interrupt handlers
172
What is the issue with decision coverage?
Produces gaps in test code written in languages that support short-circuit logic operators
173
What are short-circuit logic operations?
When the entirety of a statement is not necessarily executed
e.g. (f1() || f2())
If f1() is true, f2() is never invoked
174
What does function coverage report?
Whether each function is invoked
175
What does call coverage report?
Whether each function call is executed
176
What does condition coverage report?
Reports the T/F outcome of each boolean expression (measures each sub-expression independently)
177
What hypothesis is call coverage testing?
Hypothesis: faults commonly occur in interfaces between modules
178
What is the extension of condition coverage?
Condition/decision coverage
A hybrid measure, the union of condition and decision coverage
179
Which coverage technique has higher sensitivity: decision or condition?
Condition
180
Call coverage is also know as ____
call pair coverage
181
What does multiple condition coverage report?
Whether every possible combination of Boolean sub-expressions occurs
182
For languages without short circuit operators, MCC is effectively ___
path coverage
183
For languages with short circuit operators, MCC is effectively ___
Condition coverage
| but MCC has more test cases
184
What is the disadvantage of MCC?
Tedious to determine minimum set of test cases required
185
T/F: MCC includes decision coverage
F
186
T/F: Condition coverage includes decision coverage
F
187
T/F: decision coverage includes statement coverage
T
188
T/F: MCDC was designed for languages containing short-circuit operators
False
189
T/F: For C, C++ and Java, MCC requires exactly the same test cases as condition/decision coverage
True
190
What does path coverage report?
Whether each possible path is executed
191
For boundary-interior path testing, what are the possible loop tests?
1. 0 iterations
| 2. >0 iterations
192
What are the major issues of path coverage?
1. number of paths is exponential to number of branches
| 2. many paths are impossible to execute
193
What are the major issues of path coverage?
1. number of paths is exponential to number of branches
| 2. many paths are impossible to execute
194
T/F: Path coverage includes decision coverage?
T
195
What does n-lenght sub-path coverage report?
Whether test has exercised each path of length n branches
196
T/F: Path coverage includes decision coverage?
T
197
What is the objective of equivalence class partitioning?
to identify the test data for the application and to test if it correctly implements all functionalities
198
Describe the Structural (White box) testing technique
Explicit knowledge of the internal workings of the system (example examining the code) is used to select test data
199
What is a program graph
Given a program, it's graph is a directed and labelled graph where
nodes: are individual or groups of statements and
edges: represent flow of control
200
What are logically feasible paths?
All practically possible paths through a graph, based on program logic
201
T/F: Subdomains of equivalence classes are disjoint (MECE)
True
202
What is the main assumption for equivalence classes?
The program exhibits the same behaviour under testing compared to production. This ensure only one test case for each equivalence class is sufficient
203
In basis path testing, subset B of V must ____
Capture the essence of V
204
In basis path testing, every element of the vector space V can be represented by ___
a linear combination of elements of B
205
What are the steps in the McCabe Algorithm?
1. select a baseline path, with as many decisions as possible
2. retrace the paths visited and flip conditions one at a time
3. repeat until all flips considered
206
Theoretically, what does McCabe's cyclomatic complexity metric identify?
The upper bound on number of test cases to achieve edge coverage
207
In practice, what does McCabe's cyclomatic complexity metric identify?
lower bound of number of test cases
| because of loops
208
What are the inputs to the path based testing process?
source code
| path selection criterion
209
What is the process of path bases testing?
1. generate CFG
2. select paths
3. generate test input data
4. feasibility test of path
5. evaluate output
210
What are topological feasible paths?
All theoretically possible paths through a graph
211
What are logically feasible paths?
All practically possible paths through a graph, based on program logic
212
Does path testing focus on the logic of the code or the graph?
The graph
213
What is the robust strategy?
Targeting faults in the application using regions that contain expected inputs and unexpected inputs
214
What is the "normal" equivalence class testing strategy?
classes of valid inputs
215
What is the "robust" equivalence class testing strategy?
classes of valid and invalid inputs
216
What is the "weak" equivalence class testing strategy?
(single fault assumption) one from each class
217
What is the "strong" equivalence class testing strategy?
(multiple fault assumption) one from each class in cartesian product
218
Three main types of functional tests
1. Equivalence Class Testing (ECT)
2. Boundary Value Testing (BVT)
3. Decision Table Testing (DTT)
219
What are the three metrics to assess functional tests?
1. Testing effort
2. Testing efficiency
3. Testing effectiveness
220
What is the tradeoff in effort testing and how do you measure it?
Identification effort vs. execution effort
Graph 1: identification effort vs. sophistication
Graph 2: execution effort vs. sophistication
(graph 1 and 2 are inversely proportional)
221
T/F: Efficiency is objectively measured for functional testing
False, however structural testing can provide metrics for the quantification of efficiency
222
T/F: In general, Functional testing is effective
False, however structural testing provides metrics for the quantification of effectiveness
223
What type of functional testing should be applied if the single fault assumption is plausible?
Equivalence Class testing
224
What type of functional testing should be applied if the variables refer to physical quantities
Equivalence class testing
225
What type of functional testing should be applied if variables refer to logical quantities?
Equivalence class testing or Decision Table Testing
226
What type of functional testing should be applied if variables are dependent?
Decision Table Testing
227
structural testing techniques provide 3 things:
1. Rigorous Test Definitions
2. Mathematical Analysis
3. Precise measurements
228
Can you completely test a system?
No, nearly impossible within practical time boundary. Instead, heuristics are applied.
229
What is code coverage a metric of?
Effectiveness
230
What is the metric formula for code coverage?
% of code parts executed by a test suite
231
What is a path structurally?
A path is a sequence of statements in a program
232
What is a path semantically?
A path is an execution instance of the program unit
233
What is the motivation behind using DD-paths?
They enable a precise description of test coverage
234
What are test coverage metrics measuring?
The extend to which a set of test cases covers a program
235
What is data-flow testing?
A family of test strategies based on selecting paths to explore sequences of events related to status of variables
236
Data-flow testing uses what kind of graph to explore anomalies?
control flow graph
237
What are three types of variable anomalies?
1. defined then defined again
2. undefined but referenced
3. defined but not referenced
238
What are the the different variable statuses?
(d) Defined, created, initialized
(u) used
- (c) in calculation
- (p) in predicate
(k) killed, undefined, released
239
In a CFG, what is a defining node?
A statement where a variable is defined
DEF(v,n)
240
In a CFG, what is a usagenode?
A statement where a variable is used (predicate=p-use, calculation=c-use)
USE(v,n)
241
What does MCC stand for?
Multiple Condition Coverage
242
What is the objective of coverage analysis?
determine gaps in test cases
243
What does statement coverage report on?
The percentage of executable statements of code excised by set of test cases
244
major issue with statement coverage:
does not report on some control structures
245
What does decision coverage report on?
reports if boolean expressions in control structures are evaluated to true and false values
246
major issue with decision coverage:
produces gaps in tested code if language short circuits logical operators
247
what does function coverage report on?
Reports on whether each function in the program has been invoked
248
what does call coverage report on?
Reports on whether each function call in the program has been executed
249
What does condition coverage report on?
reports on the true-false outcome of each boolean sub expression (Like Decision coverage but better sensitivity to control flow)
250
What does multiple condition coverage report on?
reports whether every possible combination of boolean sub expressions occurs
251
What does modified multiple condition coverage (MCC) report on? (so fucked?)
reports on every possible combination of boolean sub expressions that occur, but requires enough test cases to verify every condition that can effect the result of its encompassing decision
252
what does path coverage report on?
reports on whether each possible path in each function has been followed
253
What are the two major issues of of path coverage?
1. # of paths exponential to number of branches
| 2. Many paths are impossible to exercise due to relationship of data
254
T/F: Statement coverage includes decision coverage
False, DECISION coverage includes STATEMENT coverage
255
T/F: Condition/Decision coverage includes decision coverage and condition coverage
True
256
T/F: path coverage includes decision coverage
True
257
What is the objective of basis paths?
to generate effective & efficient test cases
258
Steps of McCabes's Algorithm
1. select a baseline path corresponding to normal program execution
2. Retrace paths visited
3. Flip the conditions one at a time
4. Process repeats until all conditions have been considered
259
What is McCabe's Cyclomatic Complexity Metric?
Used to identify upper bound on number of test cases to ensure edge coverage. Due to loops, actually usually the lower bound
260
What is the process of path based testing?
1. Generation of CFG
2. Selection of paths
3. Generation of test input data
4. Feasibility test of a path
5. Evaluation of test case output
261
Focuses on properties that are directly derived from the requirements and specifications and not based on source code
Functional testing
262
Focuses on the graph, not on the logic of the code
Path Testing
263
Gives a lower bound of necessary test cases
Basis path testing
264
What are the 3 types of anomalies?
1. Defined then defined again
2. Undefined but referenced
3. Defined but not referenced
265
Data flow testing categorizing: (d)
Defined, created, initialized
266
Data flow testing categorizing: (u)
Used
267
Data flow testing categorizing: (c)
calculation
268
Data flow testing categorizing: (p)
predicate
269
Data flow testing categorizing: (k)
killed, undefined, released
270
What is Integration testing?
Systematic technique for constructing new system architecture; conducting tests to uncover errors associated with interface & interaction among units
271
What are the 3 integration testing areas?
1. Internal (Invocation, Parameters, Invocation return)
2. External (Event, I/O)
3. Interaction
272
What are the integration testing approaches?
1. Based on functional decomposition
2. Based on call graph
3. Based on paths
273
What is a def-use association?
```
a triple (x,d,u)
x is a variable
d is a definition node of x
u is a statement or predicate usage node of x
```
AND there is a subpath from d to u, with no other definition of x between d and u
274
What is a du-path?
A definition-use subpath with respect to v from node m to n, where DEF(v,m) and USE(v,n)
275
What is a dc-path?
A definition-clear subpath. Same as a du-path, but there are no other defining nodes for v in the path
276
What are the types of du-path coverage criteria?
All-defs criteria
| All-uses criteria
277
What is required to satisfy All-Defs criterion?
Each definition of variable has a clear path to A use of variable
278
What is required to satisfy All-Uses criterion?
For every variable, there are dc-paths from every defining node of v to EVERY use of v AND to the successor node of each USE(v,n)
279
What are the two types of All-uses criteria?
All-C-Uses
| All-P-Uses
280
What is AduP?
All du paths criteria
Requires that every du path from every definition to use be exercised under test
281
When can a test criterion C1 be said to subsume another test criterion C2?
If every test suite that satisfies C1 also satisfies C2
282
T/F: if test criteria C2 subsumes C1, they are equivalent
False - only if they both subsume each other
283
T/F: In control flow, path coverage subsumes multiple condition coverage?
True
284
T/F: In control flow, basis path coverage subsumes multiple condition coverage?
False
285
T/F: In control flow, multiple condition coverage subsumes branch/condition coverage?
True
286
In control flow, which coverage criteria is subsumed by all others?
Statement
287
T/F: In Data-flow, all-uses subsumes all-p-uses
T
288
T/F: In Data-flow, all-c-uses subsumes all-p-uses
F
289
Integration testing enables conducting tests to uncover errors associated with ____
interfaces and interaction among units
290
What is the ibjective of integration testing?
- take unit tested modules and build a system structure, replacing stubs
- very whether system architecture conforms to quality requirements
291
What are the three areas of integration testing?
- internal (between units)
- external (events, I/O)
- interaction (among modules)
292
What are the three high-level approaches to integration testing?
1. Based on functional decomposition
2. Based on call graph
3. Based on paths
293
What are two approaches to functional decomposition?
Non-incremental
| incremental
294
What is the technique used for non-incremental integration testing?
Big bang
295
What are the three incremental integration testing strategies?
top down
bottom up
sandwich
296
Explain top down integration testing
- test top layer first (main)
- gradually add units reference/required by tested units
- requires test stubs to simulate input/output of missing subsystem
297
What are the pros of top down integration testing?
- test cases defined in terms of functional requirements
| - verifies major control points early in test process
298
What are the cons of top down integration testing?
- stubs will be discarded (waste)
- writing stubs difficult
- no significant data flow can occur until later in testing process
299
Explain bottom-up integration testing
test lowest level first
gradually include units that reference/require previously tested units
requires test drivers
300
What are the pros of bottom-up integration testing?
verifies low level data processing early
no need for stubs
301
What are the cons of bottom-up integration testing?
driver modules needed (waste)
drivers do no contain complete algorithm --> testing may be incomplete
302
What is a test driver?
A "fake" routine that requires a subsystem and passes a test case to it
303
Explain sandwich integration testing
hybrid of top down and bottom up
proceeds using functional groups of units, based on control/data processing
integration within group alternates between low and high
304
What are the pros of sandwich integration testing?
Advantages of top down and bottom up
Minimizes need for drivers and stubs
305
What are the cons of sandwich integration testing?
Requires disciplined approach so that integration doesnt tend towards big bang scenario
306
What type of dependency is call graph based integration based on?
interaction dependency
307
What does call graph based integration use instead of the functional decomposition tree?
the call graph
308
What are the two approaches to call graph based integration testing?
1. pair wise
| 2. neighborhood
309
What is the objective of pair-wise integration testing?
restrict testing essions to just a pair of units in the call graph
310
In pair-wise integration testing, how many test sessions are required?
one for each edge in the call graph
311
What is the Neighborhood of a node in a graph
the set of nodes that are one call edge away from the given node
312
What is the advantage of neighborhood integration testing?
reduces number of test sessions
313
What type of dependency is path based integration based on?
flow dependency
314
path based integration expands beyond structural and behavioral testing to include ___
interaction among units beyond testing interfaces among seperately tested units
315
Interface-based testing is ___
structural
316
interaction-based testing is ___
behavioral
317
What is a source node?
a statement that begins or resumes execution
318
what is a sink node?
a statement that terminates execution (ends or transfers control to another unit)
319
What is MEP?
Module execution path
| - a sequence from a source to sink node, with no sink node in between
320
In the context of path based integration testing, what is a message?
A mechanism where one unit transfers control to another
321
What is an MM-Path?
A module message path - a sequence of MEPs and messages, that crosses unit boundaries
322
What is an MM-Path graph?
a directed graph where nodes are MEPs and edges are messages
323
MM-Path graphs support _____
composition of units and composition based integration testing
324
What is CI?
continous integration
regular, comprehensive and automatic building and testing
325
continuous integration includes automation of ___
the build
the tests
the deployment
326
What are the three steps of the CI build cycle?
1. commit changes to source code
2. build server executes master build script (or delegates)
3. Notify team of build results thrrough feedback machanism
327
What are the principles of CI?
1. environment based on stability
2. maintain a code repository
3. commit frequently
4. build every commit
5. add testing to build
6. make the build self-testing
328
T/F: Interaction based testing is structural
False: Interaction-based testing is behavioural
329
T/F: Interface-based testing is structural
True Interface -> Structure
330
For continuous integration, what are the requirments for adding new modules?
- new code must be reviewed
- new code mut be unit tested
- release note must be available
331
What are the four DevOps processes?
CI - continuous improvement
CT - testing
CD - delivery & deployment
CM - monitoring
332
DevOps has been describes as a ____-centric philosophy
tool
333
What are the four DevOps processes?
CI - continuous improvement
CT - testing
CD - delivery & deployment
CM - monitoring
334
What is required to achieve All-P-Uses/Some-C-Uses criterion?
For every variable v, T contains all def-clear paths from every def of v to every p-use of v
AND if a def of v has no p-uses, there is a def-clear path to at least one c-use
335
What is required to achieve All-C-Uses/Some-P-Uses criterion?
For every variable v, T contains all def-clear paths from every def of v to every c-use of v
AND if a def of v has no c-uses, there is a def-clear path to at least one p-use
336
What does regression testing ensure?
ensures changes in a program do not affect functionality, features and behaviour that should not be affected
337
what are the 3 regression selection techniques
1. Test All
2. Random
3. Selecting Modification Traversing Tests
338
What is the selecting modification traversing tests test technique?
Select a subset of original tests so that successful execution of modified code implied all functionality carried over from previous version of code is covered
339
What is a safe regression test selection technique?
Does not discard any tests that will traverse a modified or impacted statement
340
what is an execution trace?
execution trace of program p for test t is the sequence of nodes in a CFG that are traversed when p is executed against t
341
What is a safe regression test selection technique?
Does not discard any tests that will traverse a modified or impacted statement
342
what is an execution trace?
execution trace of program p for test t is the sequence of nodes in a CFG that are traversed when p is executed against t
343
How frequently is regression testing one?
many times throughout product life, possibly once after every modification
344
Is completion time higher for regression teting or exhaustive testing?
exhaustive
345
What is test revalidation?
identifying obsolete tests to exclude from regression testing
346
What is the drawback of test-all selection?
not feasible with tight deadlines
347
What is the drawback of random selection test selection?
may not select all tests that exercise modified part of code
348
What is a test vector?
a test vector for node n is the set of tests that traverse n the CFG
349
What is a syntax tree used for?
Used in test selection
Built for each node of CFG(P) and CFG(P')
350
How does SelectTests work?
traverse CFG(P) and CFG(P') using a recursive descent procedure
If n and n' are syntactically different, all tests in test(n) are added to Tr
351
What are the main steps of regression test selection?
1. find execution trace for each test in T
2. Extract test vector Test(n) for each node n
3. Build syntax tree for each node in CFG(P) and CFG(P')
4. Use SelectTests to determine Tr
352
What is a dynamic slice?
The set of statements in P that lie in trace(t) and affect the value of v at location L
DS(t,v,L)
353
```
What is the dynamic slice of P with respect to v, t, and L, if L is not
in trace(t)
```
the dynamic slice is empty
354
What is a DDG?
Dynamic dependence graph
Shows data dependence between statements
Needed to obtain dynamic slice
355
what are the steps to produce a DDG?
1. Add a node for each declaration (no edges)
2. Add the first node in trace(t)
3. For each successive statement in trace(t):
- add a new node n
- add control/data dependence edges from n to existing nodes
356
What is the procedure for dynamic slice algorithm?
1. obtain trace(t)
2. Build DDG
3. Identify node n (from L) that contains last assignment to v
4. Find the set DS(t,v,n) of all nodes reachable from n, including n itself)
If any modified statements are in DS(t), add t to Tr
357
What is a test vector?
Set of tests that traverse node n in a CFG
358
test minimization is based on the coverage of ____
testable entities
359
What are examples of testable entities?
statements
branches
def-use paths
functions
360
What are the steps in the naive algorithm?
1. compute all subsets of Tr, starting with smallest one
| 2. Stop when a subset that covers all entities covered by Tr is found
361
What are the steps in the naive algorithm?
1. compute all subsets of Tr, starting with smallest one
| 2. Stop when a subset that covers all entities covered by Tr is found
362
What are the steps to greedy algorithm?
1. find t in Tr that covers max # of entities
2. Add t to Tr min
3. Remove t from Tr, remove covered entities from consideration
4. Repeat until all entities covered
363
What are the main steps to test minimization?
1. identify testable entity
2. For every t, determine which entities are covered
4. Find minimal subset Tr min that covers every entity
364
What are the goals of test prioritization?
- don't discard tests that could reveal an error
| - increase likelihood of revealing faults earlier in testing
365
What are the two test prioritization criteria?
cost(execution time)
| risk (of NOT running test)
366
What is the procedure for test prioritization?
1. identify testable entity
2. For each test in T, compute number of entities covered
3. Arrange tests in order of respective coverage
367
What is the objective of system testing?
verify whether implementation at system level conforms to QR (quality requirements)
368
In what environment is system testing usualy performed in?
a constructed "reasonably stable" emulated-production enviornment
369
What are the different types of system testing?
```
Functionality
performance
security
load
stress
scalability
robustness
```
370
performance testing is designed to verify qr related to...
performance of system as defined in quality framework
371
What factors affect perceived performance?
loading screens
| multi-threaded UIs
372
What are the two approaches to performance testing?
benchmarking
| profiling
373
Benchmarking measures what?
absolute performance of application
374
Benchmarking is based on what type of measurement?
coarse-grained
375
What are common benchmarking metrics?
execution runtime
memory usage
response time
376
What are three optimization techniques?
- reduce I/O routines
- lazy evaluation (dont read/compute something until you need it)
- hashing, caching (combine database queries)
377
What is preferable for optimization: contiguous or non-contiguous memory access?
contiguous
378
How can datat structures be optimized?
Use hash-based structures
- for sorted order ---> TreeMap
- for insertion order --> LinkedHashSet
379
Name some optimization techniques for avoiding computation
- stop when you know the answer
- hoist loop-invariant code outside the loop
- use lookup tables
380
What is a dynamic programming technique for optimzation?
caching previous results
381
Why is optimization deceptive?
Compiler may already be optimizing code
382
What type of measurements does profiling use?
relative system statistics (fine-grained)
383
What are the two types of profiles?
- point profile (events simple, independent)
| - context profile (simple ordered events)
384
Method invocations would be what type of profile?
context profile
385
What are the different types of profiling?
1. insertion (place profiling code into program)
| 2. sampling (monitor CPU at regular intervals, save snapshot)
386
What are the pros and cons of insertion profiling?
pros: accurate, can be used on multiple platforms
cons: requires recompiling, may affect performance
387
What are the pros and cons of samplingprofiling?
pro: no modification of app required
cons:
- less accurate
- small methods may be missed
- hard to monitor memory usage
388
Name two profilin gtools
Android Profiler
| Java hprof
389
What are the drawbacks of profiling?
- CPU profiling slows down code a lot
- CPU samples dont measure everything
- output files are very large
390
What are the six basic security concepts?
1. confidentiality
2. integrity
3. authentication
4. authorization
5. availability
6. non-repudiation
391
What is non-repudiation?
A measure intended to prevent the later denial that an action happened, or a communication that took place etc.
392
What are the three requirements of software security that must be verified?
1. security architecture adheres to security policy
2. security mechanisms are trustworthy
3. applications that constitute the system are trustworthy (not malware)
393
What is a security architecture?
specification used as a guide to enforce security constraints
394
Security mechanisms can be positioned in ___ and ___
subsystems
communication links between the subsystems
395
Security architecture is verification using ____ process
threat/severity modeling
396
What are the steps in the threat modeling process?
1. identify assets w/ security quality requirements
2. create architecture overview
3. decompose application
4. identify threats
5. document threats
6. rank threats
397
What are some different ways to rank threats?
- damage potential
- reproducibility (probability of success)
- exploitability
- affected users
398
What are some ways to diagnose & mitigate attacks?
- develop software sensors
- create self-diagnostic models
- automatically disable features when attacked
- evolve software by learning from attacks
399
Name some security testing mechanisms
```
vulnerability scanning
security scanning
penetration testing
ethical hacking
risk assessment
security auditing
posture assessment and security testing
password cracking
```
400
What is ethical hacking?
a number of penetration tests over the wide network on the system under
test.
401
What is posture assessment?
It combines Security Scanning, Ethical Hacking and Risk Assessments to show an overall Security Posture of the organization.
402
How is a risk assessment carried out?
interviews, discussions, analysis
403
What are the two ways to verify that the applications in a system are trustworthy?
verify that they...
- have been developed using secure coding practices
OR
-are not malware
404
What type of code is the overwhelming target of vulnerability attacks? Why
C/C++ code
- buffer overflow
- format string
- integer
405
What is a software vulnerability?
An instance of a fault in qr, development, or configuration of software that can violate security policy
406
Name some types of software vulnerability?
```
buffer overflows
heap overflows
format string vulnerabilities
cross-site scripting XSS
cross site request forgery XSRF
Cross site script inclusion XSSI
SQL injection
path traversal
DoS
configuration vulnerabilities
AJAX vulnerabilities
```
407
What is a buffer overflow?
Reading/writing past the end of a buffer causes overflow, so any data allocated near buffer can be read and modified
408
What are the two steps in a buffer overflow attack?
1. arrange for code to be available in buffer
| 2. overflow buffer so program jumps to attack code
409
What are two ways to arrange for code to be available in the buffer, in a buffer overflow attack?
1. inject the code as input string
| 2. use code already in the program
410
What are some defenses for buffer overflow attack?
- write good code
- non-executable buffers testing
- array bounds checking
- code pointer integrity testing
- fault injection tools
- static analysis tools
411
What is XSS?
cross site scripting
attacker injects code into contents of website
412
What are the two types of XSS?
-reflected attack: attack is in the request itself (URL)
stored attack: attack is stored in the application, victim triggers attack by browsing to page on server
413
What is SQL injection?
attacker injects script into SQL query, can read or overwrite database
414
How can SQL injections be prevented?
Don't build queries with string concatenation, use API calls instead
415
What is XSRF?
cross site request forgery
allows attacker to transmit unauthorized commands from a user the website trusts
416
What is XSSI?
Cross site script inclusion
attacker enforces server to perform actions user did not intend to perform
417
What is a path traversal attack?
attacker can read files from folders they should not have access to
418
What is a DoS attack?
denial of service
allows attacker to make a server unable to service ordinary requests
- overload server with requests
- manipulate security holes (crash server, overload memory, etc)
419
What is a configuration vulnerability?
attacker controls system by knowing default system configurations such as usernames, passwords, setting
420
Why are third party softwares especially vulnerable to configuration vulnerabilities?
Attacker can access copy of same application and see default configuration
421
What is malware?
malicious software
program that works against system's user or owner
422
How does antivirus work?
identifies malware by looking for unique signatures in code of each program
maintains a database of virus signatures
423
What are the two ways that malware varies?
nature
| operation
424
What are some examples of malware?
```
viruses
worms
trojan horses
backdoors
mobile code
adware
sticky software
```
425
What is a virus?
self-replicating malware that must be executed by user to propagate
426
What are Worms?
self-replicating (like virus) that uses internet to spread, does not require human intervention.
Can spread in data packets in software or via email
427
What are trojan horses?
seemingly innocent program containing hidden malware
easy to create on windows, as file type determined by extension not header
428
What are backdoors?
create covert access channel that attacker can use to connect, control or spy on user
429
What is mobile code malware?
targets mobile users, pretends to create better browsing experience (java applets, ActiveX)
user may see warning, but most will click okay
430
What is adware?
forces ads on users, usually bundled with free software
gathers user data, slows performance
431
What is sticky software?
prevents users from uninstalling
does not support uninstall feature OR uses key registry to launch malware on boot
432
What is load testing?
verify QR related to systems ability to remain stable for long period under full load
433
What is stress testing?
verify QR related to how gracefully system handles extreme load conditions, and level of data loss under extreme load
434
What is scalability testing?
verify qr related to ability t oscale up to its engineering limits
435
What are major causes of scalability limitation?
data storage
computing capacity
network bandwidth
436
What are the four types of robustness tests?
availability
degraded node
on-line insertion and removal
power cycling
437
What are availability tests?
verify qr related to ability to recover from failures without impacting operation of system
438
What are degraded node tests?
verify qr related to ability to remain operational after portion of system becomes non-operational
(AKA failure containment)
439
What are on-line insertion and removal tests?
verify qr related to handling and recover due to on-line insertion and removal of modules
440
What are power cycling tests?
verify qr related to system recovery after power is restored from glitch in deployment environment
