4452 - Testing Final Flashcards
What does statistical testing reflect and what does it measure?
reflects: frequency of user inputs
measures: system reliability
What does defect testing discover?
system defects
What are the 4 testing stages?
- unit testing
- integration testing
- system testing
- user/acceptance testing
What is unit testing?
the testing of individual components
What is integration testing?
the testing of interactions between components
What is system testing?
testing the complete system prior to delivery
What is acceptance testing?
testing done by end users to ensure that the system delivers on business and user needs
What is black box testing?
Its where something is inputted to the system and then the output is compared to the expected output
What is white box testing?
Its where the input is selected with explicit knowledge of the internal workings of the system
What types of testing are included in the white box test suite?
- edge (decision) testing
- path testing
- statement testing
- condition testing
What is regression testing? What does it ensure?
- Regression testing is used to make sure that a defect fix, added functionality, really any changes to the code does not change functionalities or behaviors that should not be affected
- ensures that modified code has not broken the code it modified
What are the two broad techniques used during unit testing?
functional testing and structural testing
What are the two types of structural testing methods? What is the difference between them?
Control flow testing, data flow testing
- one is based on control flow graphs the other is based on data flow graphs
What does integration testing attempt to construct?
the system architecture
Integration testing wants to uncover defects associated with what?
interfaces
When is integration testing complete?
- when all modules are fully integrated together
- when all the test cases have been executed
- when all the severe and moderate defects have been found and fixed
What does system testing check the system against?
the quality requirements
A system must be constructed in a ________ for system testing to be performed
an emulated environment that is as close to the real production environment as possible
Where is acceptance testing performed and who performs it?
in the production environment by the end users
what is the key difference between testing and debugging?
testing confirms the presence of defects, debugging actually locates and fixes them
What are the activities in testing?
IDBEC:
- identify
- design
- build
- execute
- compare
What is “built” during the build activity in testing?
test cases
What are the activities in debugging?
LDRRe
- locate
- design
- repair
- re-test
What is a test condition?
it is the goal of a test case, it is something you want to verify
What does a test case consist of?
test data to be inputted and an expected output
What is a test set?
a collection of test cases
What do test scripts contain?
data and instructions for testing
What is the system that performs the checking of the observed behavior referred to as?
oracle
program behavior can be specified in which three ways?
- plain natural language
- state diagrams
- formal mathematical specifications
What are automated oracles?
a pre determined input output relationship that is checked for all outputs
What is considered completeness for functionality testing and structural testing?
functional; the behavior of all functional partitions are tested
structural; all statement, all branches, all loops
What are the three reasons for test stopping?
- budget exceeded or deadline needs to be met
- desired coverage reached
- desired level of failure intensity reached
What does boundary value testing do?
targets failures in application at the boundary of the equivalence classes
what does equivalence class partitioning do?
selects partitions from WITHIN equivalence classes
boundary value testing and equivalence partitioning tests may overlap (T/F)
T
What are nodes in a CFG?
statements or groups of statements
what are edges in CFG?
the flow of control; there is an edge between i and j if j can be executed immediately after i completes
what is statement coverage?
run tests until all statements have been executed
what is edge coverage?
run tests until all edges in control flow graph have been executed at least once
what is condition coverage?
like edge coverage but with additional attention being paid to conditionals - ensuring that all combinations have been tested
what is path coverage?
ensuring that all paths (start node to end node) have been executed multiple times
for loop based paths what test cases should you write?
three cases:
- skip the loop
- execute it once
- execute it multiple times
what does the driver component do in unit tests?
it takes test case data and passes it into the component being tested and returns the results
what do stubs do? what MUST they use?
replace the modules that are subordinate to the subsystem being tested. They MUST use the modules exact interface
why is test selection difficult?
because of two aspects of the input domain for the partition; its large size and complexity
how does equivalence portioning solve the input domain size issue?
by creating a relatively small number of disjoint sub-domains that allow the system to be tested with N test cases for N sub domains or partitions
what are the two objectives for test generation?
completeness and efficiency
what is the difference between weak and strong equivalence class testing?
weak; uses one variable from each equivalence class in a test case strong; uses the Cartesian product of the partition subset
what is the minimum number of test cases that can be in weak equivalence class testing?
the number of classes in the partition with the largest number of subsets
in strong equivalence classes the interactions between the represented values and any ______ is tested?
subsets
What is the foundation for partitioning equivalence classes?
expected behavior
what is the difference between normal and robust testing strategies
normal: equivalence classes of valid values of inputs
robust: equivalence classes of valid and invalid values of inputs
what is efficiency for equivalence classes?
identify test cases using only one element from each equivalence classes
what ratio is effectiveness evaluated on for equivalence class testing
failures exposed by equivalence class testing vs total failures in the system
what is the typically effectiveness of equivalence partitioning?
less than 1
what are ways to improve the effectiveness of equivalence class testing?
- unambiguous and complete specifications of the requirements
- carefully selected equivalence partitioning
what is uni dimensional partitioning for compound data types?
when you consider only 1 variable at a time
what is multi-dimensional partitioning for compound data types?
the input domain is the set product of the the variable inputs
what are the 4 steps of equivalence classing?
- identify the input domain
- equivalence classing: partition the set of values of each variable into disjoint subsets
- combine the equivalence classing
- identify the infeasible equivalence classes
what is the main source for identifying inputs and outputs for equivalence testing?
the software requirements
input variables don’t include environmental variables (T/F)
F; they do
how can equivalence classes be defined?
using multidimensional partitioning
what inputs are used in infeasible equivalence classes?
inputs that cannot be generated during the test
what are the three steps to boundary value testing?
- partition the input domain
- identify the boundary
- select test data
when does BVA work well?
when program to be tested is a function of several independent variables that are physically bounded quantities and when the selected test data is chosen with no consideration for the function of the program or the semantic meaning of the variable
what do decision variables make easier?
to make sure all possible conditions have been accounted for
what is logic based modeling?
associating conditions with actions to be performed
what are default rules in limited entry decision tables?
indicate actions to be taken when none of the rules apply
what must you ensure before using a decision table?
that the rules are complete and consistent
how do you create a decision table? (what are the steps)
- determine the actions, conditions and values, and how many rules you want to specify
- encode all the possible rules and the actions for each rule
- reduce where you can
what is TAP?
test anything protocol :)
what does it mean that drivers and stubs are considered “overhead”?
they must be written but are not considered part of the testing software product
what is the mocha testing tool based off of?
the javasript test framework
what runs mocha on server and client side
node-js and browser
mocha tests run asynchronously (T/F)
True
mocha test run in parallel (T/F)
False; they run serially
what is a test runner?
the driver; takes test cases and returns results
what is a test fixture?
a test environment used by multiple test cases
what does TAP allows in terms of communications?
it allows for communication between unit tests and the test harness
what is the test harness?
the automated test framework
what was tap initially designed for and when?
for the perl interpreter in 1987
what are the two inputs of the describe function in mocha?
the name of the test group and the all back function
what are the two inputs of the it function
the name of the test case and the call back function that contains the actual test
what is the difference between the it and describe function
describe –> test group (higher level)
it –> test case
what are the 4 hooks in mocha?
- beforeEach
- afterEach
- after
- before
what runs first beforeEach() or before()?
before
what does SDK testing use for white box testing?
user code test coverage reports
what kind of testing does api testing use?
black box testing
what does api testing use of the server side?
REST
what function are stubs written in for EQT?
beforeEach and afterEach
what does the spy function solve?
it prevents the program from not calling subsequent arguments after an invalid return
what are the three types of functional testing?
- equivalence class
- boundary value
- decision table
what three basic metrics can be used to evaluate functional testing methods?
- effort
- efficiency
- effectiveness
what functional testing method requires the highest identification effort?
decision tables
what functional testing method requires the highest execution effort?
boundary value testing
what is the most efficient functional testing technique?
decision tables
How do we measure efficiency in functional testing?
you can’t really do it
what is single fault assumption?
that faults rarely occur as the result of two or more simultaneous faults
if single fault assumption is assumed what functional testing method should be considered?
equivalence class testing with robustness
if variables are independent and refer to physical domains what functional testing method should be considered?
equivalence class testing
if variables are logical quantities what functional testing methods can be considered?
equivalence class testing or decision table
if variables are dependent what functional testing methods can be considered?
decision table
structural testing provides test definitions relating to what?
- control flow
- data flow
- program language semantics
- coverage criteria
what is used to select test data in structural testing?
the internal workings of the system
data flow testing uses a test strategy based on what?
selecting paths through the programs control flow graph
if a program has compound conditionals what must you ensure?
that all the combinations have been tested
what is the fault assumption in path testing?q
that the program may make take a different path then the one expected
what are the characteristics of a dd path graph
- that all nodes are DD paths from the CFG
2. edges represent control flow between DD paths
what graph enables the precise description of test coverage?
DDpath graphs
DD path testing provides metrics and coverage criteria for what?
effectiveness
what does predicate coverage aim to reduce all program predicates to?
a true or false (single predicate)
for n conditions how many test cases are there typically for MC/DV coverage testing? For regular multiple decision coverage how many should there be?
MC/DC: n + 1
MDC: 2^n
what is the issue with C1 (simple coverage criteria) for DD path testing?
it only covers all edges of the DD path but does not take into consideration dependencies between paths; for example if a variable is declared in one path and then referenced in another, they will be tested independently from each other which isn’t what happens in practice
what is Cd and what does it solve?
data flow dependency modeling coverage criteria for DD path testing; ensures that all edges as well as all dependent DD path pairs are tested
what is C2 and what are the two basic outcomes?
loop testing
- traverse loop
- exit loop
how can C2 be extended to consider the loop index?
by doing a modified boundary value analysis on the index
statement coverage ensures that all loops reach their termination condition (T/F_)
F; only ensures that all statement within the loop were executed
it is not acceptable to release code based exclusively on statement coverage (T/F)
T
what is the flaw with decision coverage?
it can produce gaps in programming languages that support short circuit logic operators (will skip the evaluation of logic if some other condition is accepted)
what is function coverage?
it ensures that all functions in the program have been called
what is call coverage?
ensures that all function calls in the program have been executed
what is mutliple decision coverage?
ensures that all possible combinations of Boolean sub expressions occurs
for languages without short circuit logic operators what is multiple decision coverage equal to ?
path coverage
what is MC/DC?
multiple decision coverage but only combinations whose result can effect the decision are evaluated
what are the problems with path testing?
- the number of paths has an exponential relationship to the number of branches
- some paths cannot be exercised due to the relationship of the data
path coverage includes decision coverage (T/F)
T
statement coverage includes decision coverage (T/F)
F; the other way around
what is a baseline path?
corresponds to a normal execution of the program, from start node to end node, that encompasses as many decisions as possible
what is the difference between topological feasible paths and logically feasible paths?
topological is strictly based on the graph, logical incorporates program logic and identifies constraints that eliminate certain paths that are present in the topological model
what selection criteria exercises every statement or instruction?
C0; statement coverage
what selection criteria exercises each branch and case statement at least once in each direction?
C1; branch coverage
what selection criteria exercises each condition in a decision with all possible outcomes
C1p; predicate to outcome
what selection criteria exercises every compound predicate outcome?
C2; multiple condition coverage (MCC)
what selection criteria exercises every path from entry to exit?
C(infinite), all possible execution paths
what is the core difference between functional and structural (path) testing?
functional testing is directly derived from the software quality requirements and specifications and is not based on the source code, whereas structural testing is directly based on the the source code and the resulting graph
path testing is based on the logic of the code (T/F)
F; it is based on the graph
data flow diagrams explore the events related to what?
the status of variables
what are the 3 types of data flow anomalies?
- variable is defined then defined again
- variable is referenced but no defined
- variable is defined but not referenced
what is the difference between a defining node and a usage node in CFG?
a defining node is DEFINED at the statement
a usage node is USED at the statement
what is the triple that defines def-use?
x - variable
d - node containing definition of x
u - statement or predicate node that uses x
what is a du-path
for variable x, its a path whose first node is a definition for x and last node is a usage node for x
what is a dc-path
a du-path where there are no other defining nodes for x
what is AduP
a testing coverage where every du-path from every definition of every variable to every use of that definition is exercised under some test
what does integration testing attempt to construct?
the system architecture
integration testing attempts to uncover errors associated with what?
interfaces and interactions among units
what are the three testing areas of integration testing
internal, external, interactions
what are the three approaches to base integration testing off of and what is the associated dependency?
- function decomposition (functional dependency)
- call graph (interaction dependency)
- paths (flow dependency)
what are the three types of incremental integration functional decomposition?
top down (module --> stub) bottom up (module --> driver) sandwich testing (mix of above two)
what are the pros and cons of top down integration testing?
pros
- test cases can be created based on the functionality of the system
- makes sure that the majority of control points are tested early on
cons
- stubs are a waste
- stubs are tricky
- stubs make it so no significant data flow can occur until later level testing
what is a test driver?
a fake routine that requires a valid subsystem and passes a test case to it
what are the pros and cons of bottom up integration testing?
pros
- low level data processes are tested early on
- no stubs
cons
- you need to build test drivers which constitutes overhead
- testing may be incomplete
how does sandwich testing work?
functional groups are tested alternatively between high level and low level modules, once complete its on to the next functional group
what is the advantage to sandwich testing?
it has all the positives of top down and bottom down while minimizing the need for stubs or drivers
what are the two methods of call based integration testing?
pair wise and neighbourhood
what is pair wise integration testing?
uses test sessions that are restricted to a particular pair of units, corresponding to one edge in the call graph equally to one integration test session
what is neighborhood integration testing?
where a test session makes up all nodes that are one edge away from the current node, meaning that all predecessor and successor nodes are in the session
neighborhood testing has more test sessions than pair wise testing (T/F)
F; pair wise has way more
interface based testing is behavioral (T/F)
F; its structural
interaction based testing is behavioral (T/F)
T
what is a source node?
a node where the program execution begins or resumes (in the case of control transfer from one unit to another)
what is a sink node?
a node where the program terminates can be both the end of all execution or when control is transfered to another unit
what is a module execution path (MEP)?
a sequence of statements that begins with a source node and ends in a sink node with no intervening sink nodes
what is the program mechanism that transfer flow of control from one unit to another?
a message
what is an MM-path?
a chain of MEP & messages
what are the nodes and edges in an MM-path graph
nodes - MEP
edges - messages
what three things does continuous integration try and automate?
the build, tests, and deployment
what are the three key steps to the CI build cycle?
- developer commits code to the source code repository
- build server executes the master build script which does the following
- checks source code
- builds an executable version
- runs other jobs like testing - team is notified of build results
what is the criteria for adding new models to CI build?
- must be unit tested
- must be inspected/reviewed
- must have some release note
what are the 4 types of devops processes?
CI - uses tools to orchestrate software changes and tests
CT - uses tools to orchestrate test environment and management
CD - uses tools to manage deliverable
CM - uses tools to run analytics
when does regression testing occur?
many times throughout the life of a product
what does regression testing ensure?
that changes do not affect functionalities that should not be affected
what tests should not be included in regression tests?
any tests that have become obsolete as a result to the changes
what is test re validation
identifying obsolete tests in regression testing
what are the 3 types of regression testing?
- test all
- random
- selected modification traversing tests
what tests are included in the test all approach?
all valid tests from the previous versions and any new tests added to test new functionality
what is the problem with the test all approach?
it may not be feasible depending on the size of the program
what tests are included in the random approach for regression testing?
random tests from valid previous and new
what is selected modified traversing testing for regression testing?
selecting a subset of the original test set such that the execution of the modified code against this subset implies that all functionality was carried over from the previous version, and then also testing any new test cases developed to test new functionality
what is a “safe” regression technique?
where no test cases that will traverse modified code are disgarded
the sequence of node that a test traverses in a CFG is called what?
its execution trace
what are some examples of testable entities?
statements, decisions, branches, def-us path
what does the naive algorithm do?
computes all subsets of the test set starting by the smallest subset and stops when one that covers all the entities covered by the test set is found
what does the greedy algorithm do?
finds the local optimum choice at each stage and hopes to find the global optimum
when do you use test prioritization instead of test minimization?
when you can’t discard any test cases due to quality requirements
what is the goal of test prioritization
to identify faults as early as possible in the test process
which tests get maximum priority?
tests with the most coverage
what type of environment is system testing typically performed on?
a constructed by reasonably stable emulated production environment
what is perceived performance
user perspective of system performance
what are the two approaches to performance testing?
bench marking and profiling
comparing performance against some pre determine criteria is called what?
benchmarking
comparing performance against measures relative to the system statistics is called what?
profiling
what type of performance testing measures the absolute performance of the application?
benchmarking
what type of performance testing measures the run-time performance of the application?
profiling
what are the typical criteria used in benchmarking?
execution time (cpu usage), response time and memory usage
what are some ways to optimize to improve performance?
reduce I/O routines, lazy evaluation saves (computing/loading), hashing/cashing save (reloading resource)
what is the difference between contiguous and non contiguous memory testing
contiguous; assigns the consecutive blocks of memory to a process requesting for memory
non contiguous; assigns the separate memory blocks at different locations in memory space in a non consecutive manner for a process requesting memory
how do you optimize memory access?
changing non contiguous memory access to contiguous
how do you improve data structures?
by using hash based functions and dealing with their limitations in an intelligent way
how do you optimize evaluation saves?
- don’t do computation when you already know the answer
- hoist computation heavy code outside of a loop
- use dynamic programming
what is the classic question that define profiling?
where is the most time being spent
what are the two types of profiling (with definitions)
point; events are simple and independent
context; event consist of simple ordered events
what is the difference between insertion and sampling profiling?
- insertion means writing profiling code
- sampling means monitoring cpu at regular intervals
what is hprof?
a tool built into JDK for profiling
why should you design your profiler code to be short?
because profiling slows down code a ton
where should security mechanisms be positioned?
in subsystems and in the communication links between sub systems
what are the 6 steps to the treat modeling?
- identify assets with security quality requirements
- create an architecture overview
- decompose the application in terms of processes
- identify threats
- document threat
- rank threat severity
what are security testing mechanisms (VESPP)
vulnerability scanning, ethical hacking, security scanning, penetration testing, password crackingq
malware exploits arise from what coding problems?
- buffer overflow
- format string vulnerabilities
- integer vulnerabilities
how is a buffer overflow attack done?
injecting attack code into the buffer and overflow the buffer so that it jumps to the attack code
how do you defend against an overflow buffer attack
- non overflow buffers testing
- array bound checking
- code pointer integrity
what does non overflow buffer testing do?
makes attack code injected into input buffer non-executable
what does array bound checking do?
ensure that all read and writes to arrays are within array bounds
what does code pointer integrity testing detect?
if a code pointer has been corrupted before it was de-referenced
what is XSS?
cross site scripting which allows you to inject code directly into contents of a website
what is SQL injection?
when you fuck with queries
what is important do not do when building SQL queries?
string concatenation
what is XSSI?
an attack that forces the server to perform actions that the client never requested
what is path traversal?
a vulnerability that allows attackers to read from folder that they don’t have access to
what is DDOS attack?
an attack where a server is unable to fulfill ordinary requests
what are configuration vulnerabilities?
vulnerabilities that can allow an attacker to take control of a system given that they have knowledge of the default system configuration
what is malware?
any software program that works against the interest of the user
how to antiviruses identify malware?
by looking for specific signatures in the code
what are viruses?
self replicating programs
what are worms?
malware that uses the internet to spread
what is the difference between a worm and a virus (not internet propogation)
a virus requires user intervention to initiate replication, worms do not
what is a trojan horse?
a seemingly innocent program that contains malware
what is a backdoor?
malware that creates a covert access channel
what is sticky software?
malware that prevents the user from uninstalling it manually
what does load testing try and expose?
faults that would not come up under normal conditions (only occur after a long time under full load)
what type of testing is used to evaulate software under full load?
stress testing
what do availability tests verify?
the systems ability to recover from hardware and software failures without having adverse impacts on other functionality
what do degreaded node test verify?
the ability of the system to remain operational after a portion of the system become non-operational
what is a p-use?
predicate use, statement is a predicate statement
what is a c-use?
computational use, statement is a computation
