Unit 11 - Product quality: verification, metrics and testing

Question 1

What does TDD (Test Driven Development) demand?

Answer 1

Before any code is written, tests should be written for that code to meet. Just enough code should then be written to pass those tests – then further tests should be written and so on. On each iteration all of the tests should be run.

Question 2

What advantage does running all test on each iteration have?

Answer 2

This approach has the major advantage that as development proceeds, and after any change, a complete suite of tests can be run at the press of a button

Question 3

How does TDD help the design process?

Answer 3

In practical terms, TDD eases the process of making changes to code at late stages in development, simply because the accumulated tests provide assurance that changes have not broken the code. In object-oriented contexts this whole approach is generally relatively convenient to manage due to the existence of unit testing frameworks such as JUnit.

Question 4

How do Pre- and postconditions in DbC differ from unit tests in TDD?

Answer 4

Pre- and postconditions, like the unit tests of TDD, are a form of executable documentation. Unlike unit tests, they focus on invariant conditions rather than on test cases. Some of the benefits of using assertions can be enjoyed whether or not DbC is applied strictly, such as the use of assertions to identify how and where problems arise at run-time.

Question 5

Define appropriate quality

Answer 5

Software that is fit for its purpose and of sufficiently high quality. One way of characterising the
notion of appropriate quality is to say that it is in conformance to the requirements and expectations of the customer.

Question 6

What can we check to establish if quality is appropriate?

Answer 6

. how the product operates against what was both explicitly required and implicitly expected by the customer

. that quality and development standards – whether in-house rules or ISO 9000 mandated practices – have been followed

. that best practice of software engineers, as professionals operating to professional standards, has been followed during development.

Question 7

Suggest three expectations that a customer might have of a software product without perhaps being aware of them.

Answer 7

Three possibilities are:
. the product will not conflict with other software that they use
. the product will boost productivity
. the product will be simple to use.

Question 8

Software Quality Factors (SQFs) can be grouped into six classes. What are they?

Answer 8

Functionality, reliability, usability, efficiency, maintainability and portability.

Question 9

SQFs affected by product operation requirements include?

Answer 9

SQFs affected by product operation requirements include:

. correctness: how well a system fulfils the customer’s overall objectives – how well the software does what the customer wants

. reliability: the likelihood with which a system can be expected to perform its intended function – how well the software does what it is supposed to do

. efficiency: the level of computing resources (including time) required by a system to perform its function – how well it runs on the customer’s hardware

. integrity: the strength of measures to ensure that modification or deletion of data by unauthorised persons (or by any other unintended means) does
not occur

. usability: the effort required to learn about, operate, prepare input for and interpret the output of a system – how easy the system is to use.

Question 10

SQFs affected by product revision requirements include?

Answer 10

. maintainability: the effort required to deal with expected changes and to understand, locate and fix errors in a system

. flexibility: the effort required to modify an operational system – how easily the system can be changed while in service

. testability: the effort required to test a system to ensure that it performs its intended function. Different kinds of testing may vary according to how easy they are to adapt when requirements are revised.

Question 11

SQFs affected by product transition requirements include?

Answer 11

. portability: the effort required to transfer the system from one hardware platform and/or software environment to another – how easily the system
can be used on another machine should the customer change their platform or should other customers require it

. reusability: the extent to which a system (or system component) can be reused in other applications – how easy it is to reuse some of the software to make future developments more cost effective

. interoperability: the effort required to couple one system to another – how easy it is to interface the system with another, should the customer require it.

Question 12

Explain how increasing integrity within a system could affect efficiency.

Answer 12

Increasing integrity within a system means strengthening measures to ensure that modification or deletion of data by unauthorised persons, or by any other unintended means, does not occur. This might
involve the use of passwords to access certain data and an authentication server to check a user’s identity, or it might mean that network traffic needs to be encrypted and decrypted. Each of these factors adds an overhead to processing, so efficiency is likely to be
reduced

Question 13

Identify one other pair of SQFs that are not independent

Answer 13

Another pair of SQFs that are not independent is usability and portability. For example, many of the features of the Apple Macintosh that contribute to its reputation for usability are built into its operating system. Applications that take advantage of these features are less portable to other systems, such as Windows or Linux.

Question 14

What are the four SQFs of primary importance?

Answer 14

. Correctness. A popular measure for assessing correctness is defects per thousand lines of code (defects per KLOC), where a defect may be defined
as a verified lack of conformance to requirements. Defects may be identified by testers or reported by users of a software product after the product has been released for general use. In the case of a large project
they may be counted over a standard period of time, for example one month.

. Integrity. This is measured by considering the proportion of ‘attacks’ on a product as opposed to bona fide uses. In cases where integrity is of great
importance, and different kinds of attack can be identified, it can be useful to measure or estimate quantities such as the likelihood that an attack of a
given type will occur within a given time and the likelihood that an attack of a given type will be repelled. If historical data is available that allows
these probabilities to be calculated with some accuracy, for example using information from a log file, then it becomes possible to assign numerical
values in a simple way to the integrity of a system.

. Maintainability. Unfortunately there is no way to measure maintainability directly, and so we must measure it indirectly. A simple measure is mean
time to change (MTTC), which is the average of the times it takes to analyse a bug report, design an appropriate modification, implement the change, test it and distribute the change to all users. In general, the lower the MTTC (for equivalent types of changes), the more maintainable the software product is.

. Usability: any system with a user interface and that will be used by people other than the developers should be usability tested. Usability testing involves users systematically trying out the user interface and the
system behind it – although for some purposes the system may be simulated. There are also forms of evaluation such as heuristic review that can be used to make substantial improvements to user interfaces without involving users. For any system that depends on how users will interact with it, user interface design and testing should be continual engineering
focuses.

Question 15

What is validation?

Answer 15

Validation is concerned with whether the right product is being built.

Validation does not assume that either the specifications or development processes are adequate. Validation focuses on ensuring that outputs meet the needs, including implicit needs and expectations of
customers and stakeholders. These may never have been written down. For this reason validation is generally harder than verification. Validation can
have far-reaching effects and can result in changes to specifications or organisational processes, or both. Validation ultimately depends on acceptance by customers.

Question 16

What is verification?

Answer 16

Verification concerns whether the product is built in the right way.

Verification assumes that the current specifications are
correct and focuses on ensuring that development processes produce outputs that meet the specifications. More specifically, verification tests the extent to which the product conforms with the various, often evolving, system descriptions designed to help produce it.

Question 17

Verification of a system involves which two tasks?

Answer 17

. ensuring that all system descriptions are self-consistent

. ensuring that all system descriptions are consistent and complete with respect to those from which they were derived.

Question 18

Validation comprises?

Answer 18

. ensuring that all system descriptions are consistent with the customer’s requirements, including implicit requirements.

Question 19

Give a simple example of two system descriptions that might contradict each other.

Answer 19

There are numerous possibilities. One elementary example would be if a structural model for a hotel reservation system indicated that a reservation
could be made for more that one room but the implementation only allowed one room per reservation.

Question 20

Why is it important for the customer’s requirements statement to be self consistent?

Answer 20

If the customer’s requirement statement lacks self consistency, then either the resulting system will be inconsistent or it will not satisfy the customer’s requirements. The system builders can decide (implicitly or explicitly) how to resolve the inconsistencies or, if the inconsistent
requirements affect different parts of the system and are not picked up by the developers, the developers could inadvertently build the inconsistencies into the product.

Question 21

List some approaches that can be used to promote testing throughout development

Answer 21

. prototyping (the rapid creation of exploratory software artefacts that are discarded after evaluation)

. iterative approaches (where early software artefacts are built on rather than discarded)

. frameworks such as the dynamic systems development method (DSDM) – promoted by the not-for-profit, vendor-independent DSDM Consortium –
which documents best practice processes for iterative and incremental development.

Question 22

What is unit testing usually performed on?

Answer 22

Unit testing is usually performed on classes, with the focus on the systematic testing of methods using test cases.

Question 23

Explain Test-driven development in nutshell?

Answer 23

In TDD, code is written in very small increments – of the order of no more than five or ten lines of code.

But crucially, no code is written without first
devising and writing an automated test case for each code increment.

The essence of the cycle is as follows:
1 Decide on a code increment.
2 Decide on a test.
3 Write the test.
4 Run all tests, expecting the new test to fail (so that you know that the test has ‘teeth’).
5 Write the code.
6 Run all the tests and succeed.

Question 24

What are the benefits of TDD?

Answer 24

TDD is claimed to bring many benefits, including the following:

. Test coverage: Test coverage is in some respects comprehensive, with virtually all code having associated tests. These tests should all have been run successfully throughout development. Finished code therefore already has an extensive test suite.

. Regression testing and early discovery of errors: Many kinds of error are discovered and corrected at the earliest opportunity. Changes that break tests can be quickly identified and rectified.

. Executable documentation: The tests both show how the code should be used and indicate what it should do by means of test cases.

. Beneficial effect on design: The writing of tests contributes to design as well as to testing. TDD encourages clarity, modularity and a focus on
interface over implementation.

. Complementarity with DbC: TDD tests by test cases, whereas DbC tests by invariants. These perspectives are usefully complementary.

. Promotes good code quality: TDD promotes sustained focus on quality, design for testability and early and frequent refactoring.

. Inhibition of ‘featuritis’: Because addition of extraneous code is discouraged during TDD cycles, creeping featuritis – the addition of unnecessary features – is inhibited.

Question 25

What are the limitations of TDD?

Answer 25

TDD is not a panacea and has many limitations and possible pitfalls, including the following:

. User interface: TDD does not readily apply to user interface testing, for which it is better to apply techniques such as usability testing.

. Testing of applications integrated with databases: TDD alone is not adequate for the comprehensive testing of databases.

. Multithreaded systems: TDD is not generally suitable for the testing of multithreaded systems, as results may depend on the vagaries of timing.

. Customer acceptance: TDD cannot take the place of
customer testing.

. Legacy systems and systems reusing large code components: If large amounts of code are being reused and TDD was not applied when they were being coded, then at best unit tests can be added retrospectively.
Such a retrofit would not constitute TDD, since the testing would not guide the evolution of the code. In many cases, retrofitting unit tests to large bodies of existing code may be impractical.

. Management support: If management chooses to penalise time spent on writing tests during development, TDD is unlikely to flourish.

. Code whose requirements are liable to change greatly: Sometimes, for example during scientific research or product development research, the
purpose of developing software may be to find out what the requirements should be in some little-understood area or what happens in some poorly
understood situation. This is known as exploratory programming. In these and other situations, requirements may change so extensively and
frequently that there is repeated wholesale invalidation of existing unit tests. This would render TDD a hindrance rather than a help.

. Integration testing: TDD cannot take the place of integration testing

Question 26

Give two reasons why it is useful to run a unit test before the relevant code increment has been written?

Answer 26

(a) If the test unexpectedly already passes at this point, this demonstrates that it is not a good test of the next increment.
(b) If it fails in an unexpected way, this demonstrates a faulty or incomplete understanding of the test that needs to be addressed in order to have a good grip on the code and test.

Question 27

Suppose the following constructor is included in Account.
public Account (int aBalance, int anOverdraftLimit)
{
// initialise an account with a given balance
// and overdraft limit
assert aBalance >= 0 && anOverdraftLimit >= 0;
// body of constructor goes here
assert getBalance() == aBalance &&
getOverdraftLimit() == anOverdraftLimit;
}

(a) Explain the meaning of the pre- and postconditions using natural language.

(b) If assertions are enabled and the following statement is executed what will happen?
Account acc1 = new Account(200, -50);

(c) If assertions are enabled and the following statement is executed what will happen?
Account acc1 = new Account(0, 200);

Answer 27

(a) The precondition verifies that the arguments aBalance and anOverdraftLimit are both greater than or equal to zero. The postcondition verifies that the variables balance and overdraftLimit have been correctly initialised with the values of the corresponding arguments.
(b) A value of −50 for the overdraft limit will mean the boolean expression in the precondition assertion evaluates to false and an assertion error will be thrown
(c) The precondition is met and so a new Account will be created with a zero balance and an overdraft limit of 200.

Question 28

What are other general categories of testing?

Answer 28

. Requirements-based testing draws on previously gathered or formulated testable requirements to check that a system meets the customer’s requirements. The final stage in this form of testing is acceptance testing.

. Usability testing refers to testing of the user interface.

. Developmental testing is a term that refers to all of the testing carried out by the team developing the software. It is useful to distinguish between developmental testing at three different levels of scope – unit testing, integration or component testing and system testing.

. Regression testing is any form of testing during development or system maintenance that systematically checks that fixing one bug has not
introduced others.

Question 29

Given n classes A1, …, An, each of which uses the method foo implemented in m classes C1, …, Cm, all of which are subclasses of parent class C, calculate how many tests will be required using the following
approaches:
(a) the safe approach (Figure 5)
(b) the minimal approach (Figure 6)
(c) the balanced approach (Figure 7) to integration testing.

For this case which would be the most appropriate choice?

Answer 29

(a) m × n
(b) max(m, n)
(c) m + n − 1

m + n − 1 is not much bigger than max(m, n), so the balanced approach might as well be used in preference to the minimal approach. However
m × n is generally much bigger than both m + n − 1 and max(m, n). So if the safe approach were used in preference to the balanced approach the testing load could increase dramatically.

Question 30

System testing should comprise of what?

Answer 30

. user-command testing (or operator testing) tests all user commands from the point of view of tolerance of syntax errors and data input errors

. interface and protocol testing if the system communicates with other systems in the outside world, tests its interaction with the communication system

. start-up and initialisation testing tests the system’s ability to be started in a working hardware/software configuration – in the case where there are many combinations of hardware/software, all configurations should be system tested individually

. restart testing tests the ability of the system to recover from errors of internal state

. performance testing tests that the system meets all specified operating requirements for speed, number of concurrent users permitted, and so on

. stress testing tests that the system can operate reliably at the limits of each of its resources – for example to make web server simulate the accesses of hundreds or thousands of users all at the same time to see if it can cope with the load

. security testing tests that the system does not offer opportunities to breach security

. acceptance testing is performed by the customer and after which, all being well, the system is accepted.

Question 31

Are there any situations in which system testing should be carried out by the implementers of a system?

Answer 31

Probably the only situation where this is appropriate is when the project team is small. In small teams, one person might play the part of requirements engineer, designer, implementer, tester and maintenance
engineer.

Question 32

What do you think is the relationship between system testing and acceptance testing?

Answer 32

In general, the same tests will be carried out during acceptance testing and system testing. System testing is an in-house activity and a customer need never know how system testing went – any bugs can be dealt with before the customer sees them. Acceptance testing, on the other hand, is conducted with much more at stake – the customer can accept or reject a
system based on its performance at acceptance testing.

Question 33

Why should regression testing be necessary even after the customer has accepted the product after acceptance testing?

Answer 33

Acceptance testing is the process of showing that the software meets the customer’s requirements, not that there aren’t bugs in the code. In fact, given that a system is put into use, bugs that require fixing are almost certain to be found after acceptance testing. In addition, the system will be maintained, with functionality added and changed, leading to a requirement for regression testing.

Question 34

Use the following phrases, which describe four kinds of testing, to fill the gaps in the following three sentences.

usability testing, requirements testing, security testing, regression testing

. TDD and DbC are valuable but not comprehensive tools for _________________.

. TDD has ________________ built into it.

. DbC and TDD cannot substitute for thorough _______________ or _________.

Answer 34

. TDD and DbC are valuable but not comprehensive tools for requirements testing.

. TDD has regression testing built into it.

. DbC and TDD cannot substitute for thorough usability testing or security testing.

Question 35

What is black-box testing?

Answer 35

In black-box testing, test cases are designed by looking at the specification (that is, requirements, high-level design and external interfaces) of the system to be tested

Question 36

What is white box testing?

Answer 36

In white-box testing, test cases are designed by looking at the detail of the implementation of the system to be tested.

Question 37

What are some common black box testing techniques?

Answer 37

The techniques that can be used in black-box testing are characterised by the fact that they can use only information available from the specification in order to develop test cases. This means that to produce test data for a system or subsystem only the defined relationships between inputs and outputs can be scrutinised

A quintessential black-box technique is partitioning (also known as equivalence partitioning) combined with boundary testing (also known as fence testing), which focuses on producing test data at the boundaries
between partitions of the input data space (or input domain). The input data space is partitioned into subdomains, where a subdomain is a set of input
values that require the same type of processing to be performed. Subdomains are obtained by the technique of case analysis, which determines, for each
user-perceived function of the (sub)system, the subdomain that results in that function being performed. Boundary testing is based on the observation that common errors are often caused by a data item being just one out, or, for example, a loop being executed one too many or one too few times – such errors are most visible at the boundaries of the input data space

Question 38

What is the best combination for testing

Answer 38

Combining black box and white box testing together.

Question 39

Give an example where black-box testing will test something that white-box testing would miss, and one where white-box testing will test something that
black-box testing would miss.

Answer 39

Because black-box testing takes its test cases from the specification, it is likely to pick up the following sorts of errors that white-box testing would miss (this is not an exhaustive list):

. operations required by the specification but not provided for by the implementation

. errors between the interfaces of two classes

. errors in the transformations between internal states of a class, if these affect input/output behaviour

. performance errors, in which system performance is
found to be wanting

. system initialisation and termination errors.

On the other hand, in looking inside the implementation, white-box testing will pick up the following sorts of errors that black-box testing would miss:

. the sequences of method calls in the body of a method that fail to perform the correct function

. boolean conditions in if statements, while loops, etc. incorrectly expressed

. loops that terminate incorrectly

. relationships between methods and attributes of classes that are incorrect.

Question 40

A weather-recording system records wind-speed data, rainfall data and barometric pressure, and sends summary data into a computer network. If the
wind-speed data is represented as an integer between 0 and 110, the rainfall data is represented as a floating point number between 0.0 and 150.0 (significant to 1 decimal place) and barometric pressure is represented as an integer between 800 and 1200, what is the input data space?

Answer 40

The input data space is the set of triples of values taken from the sets {0, 1, …, 110}, {0.0, 0.1, …, 150.0} and {800, 801, …, 1200}.

Question 41

Should the cyclomatic-complexity metric be used to measure the complexity of an object-oriented software system?

Answer 41

Because the cyclomatic-complexity metric is based on decision points, which are present only in methods, it is ‘blind’ to the class-structuring mechanisms that are available in object-oriented system descriptions. As
much of the complexity of an object-oriented system is held in the class structure, applying the cyclomatic-complexity metric to a whole system would not therefore be appropriate.

Question 42

Look again at the definition of the DIT metric. When calculating the metric do you think that classes from the Java API should be included? Do you think that API classes should be counted in Chidamber and Kemerer’s other metrics?

Answer 42

Chidamber and Kemerer’s metrics measure complexity, and API classes add to complexity. Hence they should be counted in all metric calculations.
For this reason, the Java documentation will be very useful when calculating these metrics.

Question 43

The WMPC metric measures the complexity of a class in terms of the sum of the cyclomatic complexities of its methods.
(a) Consider a class with ten methods, whose WMPC value is 40. How confident can you be that this class is not too complex?

(b) If the WMPC value of a class with ten methods is over 100, how certain can you be that this class is too complex?

Answer 43

For individual methods, a cyclomatic complexity of 10 or more should be regarded as a hint that the method is too complex. In the case of classes, a little more thought is needed.

(a) For a class with ten methods, a value for the WMPC metric of 40 would typically suggest acceptably low class complexity. However although nearly all of the ten methods might be acceptably simple, one or two might be unacceptably complex.
(b) By contrast, a complexity of greater than 10 × 10 = 100 is a fair indication that the behaviour of the class is too complex.