4452 - Testing Midterm Flashcards
Quality is the ability of a product or service to fulfill its:
Quality Requirements
What are the 3 dimensions of quality?
Entity, Viewpoint, Attribute
An attribute depends on:
the entity and viewpoint
Process quality is the quality factors used to:
develop, operate and maintain software artifacts
What is the purpose of ISO 9126?
to eliminate misunderstandings relating to quality between customer and supplier
What is the key difference between customer and supplier in ISO 9126
customer understands and communicates requirements the supplier does as well but also needs to assess with confidence whether the product can be produced with the right level of quality
Who evaluates the software without knowing any internal aspects of it?
the user
What is the difference between an engineer and manager view of quality?
engineers care about the final product as well as all intermediate artifacts, managers care more about overall quality
What are the quality criteria for functionality?
suitability, accuracy, interoperability, security, compliance
The presence and appropriateness for a set of functions for a specified tasks is
suitability
the provision of right or agreed upon results is
accuracy
the ability to interact with specified systems is
interoperability
the ability to prevent unauthorized access is
security
adhering to application related standards or regulations is
compliance
What are the quality criteria for reliability?
maturity, crash frequency, fault tolerance, recoverability
The frequency of failure by faults is?
maturity
what is crash frequency?
number of system crashes per unit of time
what is fault tolerance?
ability to maintain a specified level of performance in case of software faults or infringements
what is recoverability
ability to re-establish performance and recover lost data
What are the sub-factors to the Reliability quality factor?
maturity, crack frequency, fault tolerance, recoverability
What is understandability?
effort required by user to grasp the logical concept and its applicability
what is learnability?
ability for a user to learn the software (input, output, operational control)
what is operability?
ability for user to perform operations
What are the sub factors of usability?
understandability, learnability, operability
what is time behavior?
response, processing time and throughput rates
what is resource behavior?
amount of resources used and the duration they are used
what are the sub factors of efficiency?
time and resource behaviors
what is analyzability?
ability to diagnose failures and identify things to by modified
what is changeability?
effort needed to actually modify the software
what is stability?
encapsulates the risk of unexpected events occurring when software is being modified
what is testability?
the ability and effort relating to verifying the modified software
what are the sub factors of maintainability?
analyzability, changeability, stability, testability
what is adaptability?
ability to be adapted to other environments
what is installability?
ability for the software to be installed on systems
what is conformance?
a softwares ability to conform to standards surrounding portability
what is replaceability?
effort needed to replace the software in its environment
what are the sub factors for portability?
adaptability, installability, conformance, replaceability
What are the three product states in the McCall model?
operation, revision, transition
What quality factors are grouped under product operation in McCalls model?
reliability, correctness, usability, integrity, efficiency
What quality factors are grouped under product revision in McCalls model?
Maintainability, flexibility, testability,
What quality factors are grouped under product transition in McCalls model?
portability, interoperability, reusability
What is completeness?
degree to which full implementation of requirements has been achieved
what is consistency?
uniform design and implementation
what is traceability?
ability to link software components to its requirements
what is accuracy?
precision of computations of outputs
what is communication commonality?
degree to which standard protocols and interfaces are used
what is access control?
ability to control and protect the software and data
what is conciseness?
compactnss of the source code
what is access to audit?
ease in which software can be checked for compliance
what is data commonality?
standard representation of data
what is error tolerance?
degree to which operations can continue under adverse conditiions
what is execution efficiency?
the run time efficiency of the software
TF: McCall uses a hierarchical framework?
T
TF: ISO 9126 uses a hierarchical framework?
T
Which quality model reflects the user view?
ISO 9126
Which quality model reflects the product view?
McCall
What are the high level characteristics in Boehms’s quality model?
portability, as-is utility, maintainability
what are the three hierarchical levels to Boehms quality model?
high levle, intermediate and primitive characteristics
What are the two difference requirements in Furps+ model?
functional and non functional
What does FURPS stand for?
functionality, usability, reliability, performance, supportability
what does the + in FURPS+ represent?
design, implementation, interface and physical
What are the 4 quality factors in SEI model?
performance, dependability, security , safety
a tool that makes a quality factor is called what?
a quality metric
what is static software measurement derived from?
examination of software artifacts
what is dynamic software measurement derived from?
examination of the execution of software
what is the difference between direct and indirect software measurement?
direct is when you look at things like time, effort, size which are factual frontline measures, indirect is when you measure based on things derived from direct measures such as failure intensity (failures/time)
what is a commonly used metrics when measuring reliability through maturity?
software maturity index
what are the 4 important crash frequency metrics?
expected total failures, mean time to failure, failure intensity, crash free (probability of a failure free operation)
what metric is the expected number of failure experienced in a time period?
failure intensity
the probability of a failure free operation is what metric?
reliability/crash free
what must be considered when measuring fault tolerance?
overall system architecture
what is the difference between serial and parallel system architectures in terms of fault tolerance?
serial - fails if one component fails
parallel - fails if all components fail
TF: in a serial system architecture the reliability of one component is higher than the reliability of the overall system
T
TF: in a parallel system architecture the reliability of one component is higher than the reliability of the overall system
F
What are the recoverability metrics?
system recovery time, service degradation rate, time to switch
How can accuracy be measured?
problem reports per phase, priority, category
reported problems in a period of time
rate of error disovery
reported problems can be classified into what 3 categories?
open, closed, unevaluated
How is security measured?
with the security level metric
What are the dimensions used to measure analyzability? Provide some example metrics
complexity and readability
- cyclomatic number for the flow of control
- size and complexity of logical and physical statements
- comment rate on code
what dimension is used to measure changeability? what are some example metrics?
modularity
- depth of architecture
- fan in fan out
- number of branches, nested levels, shared variables
what dimension is used to measure stability? what are some example metrics?
modularity
- fan in fan out
- number of parameters referenced/changed, global variables, called relationships
- depth of architectue
what dimensions are used to measure testability? what are some example metrics?
modularity, complexity
- cyclomatic number
- number of call-paths, non-cyclic paths, nested levels of architecture
What metric is used to test portability?
Degree of portability
When is portability cost effective
if DP > 0
TF: Cpdoc, Cptest > Crdoc, Crtest (generally)
False
What does it mean if Cmod > Crdesign + Crcode?
a portable design has not been developed
What does it mean if Cmod «_space;Crdesign + Crcode?
an effective portable design exists
Quality factors tend to be what since they are related to description related to applications, components, or users
domain specific
What are the two main approaches to fault tolerance and what is the difference between them?
single approach
- integrating mechanisms into a single unique piece of software
multi approach
- develop different software versions using different techniques
What are some typical exceptions handled through tolerating by acceptance?
deadlocking, corrupting data, failure to provide intended functionality
what does redundancy enable the software to do?
detect and recover from failures
What are the 3 major parts the fault tolerance model?
fault detection, fault recovery, fault correction
What are the sub-aspects of fault correction?
diagnosis and repair
How are faults detected?
using embedded checks
Fault detection can be carried out in which 2 modes?
concurrently (during service delivery) and preemptively (while service is suspended)
What are the two techniques to fault recovery?
backward recovery and forward recovery
What is the backward recovery technique?
consists of discarding the current state in favor of an earlier state
What is the forward recovery technique?
making use of the current corrupted state to construct a recovery state
TF: Forward recovery is typically more difficult than backward recovery?
T
What are the 3 mechanisms used to record and store system states in favor of an earlier state?
undoing transactions, checkpoint/rollback, degraded service
Why is forward recovery generally very difficult to get right?
It is system specific and depends on accurate predictions of the location and cause of errors
What is the core concept of multi-version fault tolerance?
redundancy
What is the though process behind multi version fault tolerance?
components that are built differently should fail differently and therefore at least one of the versions should deliver an appropriate output
What are the two types of redundancy?
serial and parallel
How does multi version fault tolerance work?
design multiple components with the same functionality, implement two distinct versions of the same software executing on the same inputs, if there is a discrepancy in the outputs it triggers an error detection
What are the two key architectural features that relevant to security?
immunity and resilience
What is immunity?
ability to prevent an attempted attack
How does software architecture encourage immunity?
minimizing exploitable security weakness, making sure all security features are included in design
What is resilience?
a systems ability to recover
How does software architecture encourage resilience?
segmentation of functionality to contain an attack, ability to quickly restore functionality
Efficiency describes what 2 system constraints
time and capacity
What aspects are included in efficiency
throughput, response time, and load
Describe throughput
how many processed can be processed per minute
Describe response time
how long does it take to process a request
Describe load
how many users can be supported before response time and load suffer?
What mechanisms can be used to improve response time?
scheduling and caching
what are the different scheduling types
round robin, explicit, fifo, earliest deadline
What is load balancing?
a way to improve the distribution of workloads across multiple workstations
To achieve the quality actor of maintainability a software must be?
easily changed
What two ways can a software unit be affected by a software change? What does each one mean?
directly affected - when the responsibilities of the software unit must change
indirectly affected - responsibilities don’t change, but implementation must be revisited
What are some tactics for minimizing the impact on changeability (and therefore maintainability), of directly affected software units?
clustering anticipate changes
- identify design decisions that likely to be directly affected
- encapsulate them in software units
- ensure that these software units are highly cohesive
- try to confine them in a few units
What is the core strategy for reducing the impact on changeability and therefore maintainability, of indirectly affected software units?
reduce dependencies
What are ways to reduce dependencies between software units?
low coupling, and designing to the interface (units interact only through their interfaces)
How can the MVC framework be applied to achieve high usability?
understandability –> UX information architecture
learnability –> View
operability –> controller
what is a program fault?
an incorrect step, process or data definition in a program
What are the two fundamental ideas (criteria) for the defect based quality model and how are they measured?
software should be defect free (measured by fault count) and suitable for use (measured with failure reports)
a fault __________ an error, and an error _________ failure
activates, propigates
where may a fault originate?
design, specification, implementation (programmer mistake)
what are the two states of a fault?
active –> produces and error
dormant –> exists but is not producing an error
what is an error?
part of an artifact state that propagates failure
what is failure?
an inability to deliver on quality requirements, occurs when artifact is executing
what are the 6 fault categories?
creation phase, system boundaries, domain, phenomenological causes, intent, persistence
what are the two creation phase fault sub-cats?
operational, developmental
what are the two system boundary fault sub-cats?
internal, external
what are the domain fault sub-cats?
hardware, software
what are the phenomenological fault sub-cats?
human made, natural
what are the intent fault sub-cats
malicious (deliberate), accidental (unintentional)
what are the persistence fault sub-cats?
permanent, transient
what is always required in order to generate a good product?
a good process
what is the principal quality determinant for manufactured goods?
the process
in design based activities what must be considered?
the process as well as additional factors, like designer skill
What happens to the cost of fixing an error as the lifecycle of a product progresses?
it gets more expensive
what is the SQM framework?
software quality management and it is the quantitative planning and guiding of the software development
what are the 5 steps to SQM?
- engineer quality factors
- develop operational profiles
- Software quality assurance planning
- Software quality assurance control
- Apply failure data to guide decisions
what is failure defined as under SQM?
the negative drift on the software quality requirements (things that should not be done to them)
How are failure severity classes determined under SQM?
failures with the same degree of per-failure impact on a user are given the same class
what are the criteria used to assign the per-failure impact of a failure?
cost, environment, human-life, system capability
what factors are considered in the system capability criteria?
loss of data, recovery time, total downtime
What does FIO represent?
failure intensity
in terms of failure severity classes, what should be resolved before proceeding to the operational profile?
any disagreements between stakeholders regarding them
what is an operational profile?
the set of operations and their probability of occurrence
what does the operational profile reflect?
how the software will be used in practice
what are the uses of an operational profile?
proportionally distribute test cases, accurate measurement of reliability since it models real use, for a competitive release strategy since you can implement most used operations in early releases
what are the tiers of the operational profile triangle from the top? (CUSFOT)
customer, user, system, function, operation, test selection
what is the customer profile?
the total number of customer groups and the associated probability that they will use the software (number of customer in group / total customers)
what is the difference between a user and a customer?
a customer acquires the software a user “uses” the software
what is a user profile?
the total number of user groups and their associated probabilities
what is a user group?
a set of users that use the software in the same way
what is a customer group?
a set of customers that acquire the software in the same way
Users of a system, a system own internal controller, and external systems are examples of what?
actors in a use case
what is the system mode?
a collection of function grouped in a way that makes it easy to analyze their execution behavior
what is the best source for identifying system functions?
functional and business requirements
tasks in a workflow are modeled as different function if what?
managing their development has different priorities or resource allocation, or if they differ in frequency of use or criticality
what are environmental variables?
they describe conditions that do not relate directly to features (paths taken, data accessed)
what is an operation (within the context of operational profiles)
a system task as observed by the end user
TF: Operational architecture captures how sub systems and modules combine
False: system architecture captures this, operational architecture only captures how a user will invoke operations to complete functions
TF: usually there are more operations than functions
True
what is a key input variable?
a variable that is a common input for two or more operations whose value differentiates them
what is the difference between implicit and explicit representation of key input variables
implicit splits up key input variables into sub profiles with associated probabilities, explicit represents each possible combination of key input variables desperately and assigns a probability to each one
what are occurrence rates based on?
field data, system logs, manually collected data, or experience
how are occurrence rates calculated
take the individual occurrence rate and divide by total occurrence rate
what are the two ways you can estimate a test case and ultimately which one should you chose?
- using team capacity and cost
- using historical data or industry benchmarks
you should go with the lower of the two
where should the majority of test cases be allocated to?
developed components
if acquired components represent a substantial portion of the software and their reliability is questionable what is the general rule for assigning test cases?
20-30% of test cases to these acquired components
what should be utilized for the basis of test case allocation?
the operational profile
what is the methodology for allocating test cases to operations?
assign one test case to each infrequent new operation, pre-assign test cases to rarely occurring critical operations, distribute remaining test cases among remaining new operations based on occurrence probability
what is used to determine whether or not a new operation is infrequent or not?
an occurrence probability threshold
what makes a new operation critical?
its high failure severity
why do we initially allocate test cases only to “rarely occurring” critical operations?
because common occurring ones will get enough test cases due to their occurrence probability in the next step
what is the V&V perspective?
validation and verification
what are the 4 dimension of the V&V perspective (brian marricks quadrants)
business facing vs technology facing
supporting programming vs critiquing product
what are the 4 types of testing that make up the 4 quandrants in V&V perspective (brian marrick)
feature testing, user acceptance testing, functional testing, quality factor testing
what are software review and inspections
quality control processes for written material
what is the difference between a review and inspection
a review is a process or meeting to examine software artifacts where stakeholder can give their input
an inspection is a formal engineering process for detecting defects in software products
what does an inspection confirm?
that software presents overall specifications
what are the outputs of an inspection?
detailed defect list
defect summary list
estimated rework effort and completion date
TF: inspections and testing are opposing verification techniques
False: they are complimentary
TF: inspections cannot check non-functional quality requirements
True
who introduced testing and for what reason?
some dude names Fagan because he thought that execution testing was not robust enough
what are the inspection roles under fagans model? (MART)
moderator, author, reader, tester
what should be used to drive the inspection?
a checklist of common defects
