Unit 11 - Product quality, metrics, validation, verification and testing

Question 1

Suggest three expectations that a customer might have of a software product without, perhaps, being aware of them.

Answer 1

Three possibilities are:

• the product will not conflict with other software that they use;
• the product will boost productivity;
• the product will be simple to use.

Question 2

Explain how increasing integrity within a system could affect efficiency. Identify one other pair of SQFs which are not independent.

Answer 2

Increasing integrity within a system means making the system more secure. This might involve the use of passwords to access certain data and an authentication server to check a user’s identity, or it might mean that network traffic needs to be encrypted and decrypted. Each of these factors adds overhead to processing, so efficiency is likely to be reduced.

Question 3

Log entries in a web server indicate that, over the previous month, 113 attacks were made on the web server, but none were successful. Is the web server necessarily secure?

Answer 3

The web server repelled all attacks, so security is 1. From the formula, integrity is 1 (perfect). We get this from working out the formula:
integrityattack =1 – threatattack (1 – securityattack) This would seem to indicate that the web server is entirely secure. However, all we really know is that the web server was secure against these particular attacks. If the culprits were to use some other kind of attack, or change the parameters on the attack they used, they might well succeed. It is also possible that successful attacks have been made using some other method, but there were no log entries of sufficiently suspicious activity to attract our attention, perhaps because the attacker deleted them.

Question 4

What does ‘quality’ mean in contexts other than software?

Answer 4

You may have chosen to discuss a number of things here: a quality car service, a comparison of picture quality between DVD and video tape, the quality of services provided by local government, even the way in which light is depicted (the quality of the light) in Monet’s paintings of London. Whatever you have chosen, consider whether there are measurable attributes relating to quality, or whether quality refers to something less tangible.

Question 5

What are verification and validation?

Answer 5

verification means ‘are we building the product right, validation means ‘are we building the right product.

Question 6

Why is it important for the customer’s requirements statement to be self-consistent?

Answer 6

If the customer’s requirement statement lacks self-consistency, then either the resulting system will be inconsistent or it will not satisfy the customer’s requirements. The system builders can decide (implicitly or explicitly) how to resolve the inconsistencies, or, if the inconsistent requirements affect different parts of the system, the developers could inadvertently build the inconsistencies into the product.

Question 7

Give an example of a product you know where the user manual or documentation is not complete.

Answer 7

Most user manuals are not complete, in the sense that they will not describe all the features of a product. For example, a word processor might contain many features that are not fully described anywhere. For example, two features might interact in unexpected ways.

Question 8

Continue the trend of Figure 1 by extending the curve that appears there. Can you make any predictions about the number of errors that were originally present in the system?

Answer 8

The curve you have drawn may become near vertical at around 140–45 errors. If this is the case, then this indicates that the time between failures becomes very large in that range, meaning there are few errors left to be found. Consequently, the number of errors that were originally present was somewhere around 140–145.

Question 9

What do you think is the relationship between system and acceptance testing?

Answer 9

In general, the same tests will carried out during acceptance testing and system testing. System testing is an in-house activity and a customer need never know how system testing went; any bugs can be dealt with before the customer sees them. Acceptance testing, on the other hand, is conducted with much more at stake; the customer can accept or reject a system based on its performance at acceptance testing.

Question 10

A getter is a method that returns the value of one of an object’s attributes. A setter sets the value of one of an object’s attributes. What use could getters and setters be to unit testing?

Answer 10

Getters and setters allow the state of an object to be interrogated. Once the getters and setters have been tested (a process that may be as simple as verifying that the getter returns what was just set, or much more complex if the getters and setters do calculations), they can be used in more complex tests, setting initial state and verifying final state. Being able to set the state before a testing a unit, and to interrogate it afterwards, are indispensable aspects of testing.

Question 11

Given n classes A1, …, An, each of which uses the method foo implemented in m classes C1, …, Cm all of which are derived from parent class C, calculate how many tests will be required using the following approaches:

(a) the safe approach (as illustrated in Figure 3);
(b) the minimal approach (as illustrated in Figure 4);
(c) the balanced approach (as illustrated in Figure 5) to integration testing. What does this imply for the testing load?

Answer 11

(a) mn;
(b) max(m, n);
(c) m+n -1. m+n -1 is not much bigger than max(m, n), so the balanced approach might as well be used in preference to the minimal approach. However, mn is, in general, much bigger than both m+n -1 and max(m, n). So, if the safe approach were used in preference to the balanced approach, the testing load could increase dramatically.

Question 12

Are there any situations in which system testing should be carried out by the implementers of a system?

Answer 12

Probably the only situation where this is appropriate is when the project team is very small. In small teams, one person might play multiple roles, perhaps all of requirements engineer, designer, implementer, tester and maintenance engineer.

Question 13

Why should regression testing be necessary, since, after all, the customer has accepted the product after acceptance testing?

Answer 13

Acceptance testing is the process of showing that the software meets the customer’s requirements, not that there aren’t bugs in the code. In fact, given that a system is put into use, bugs that require fixing are almost certain to be found after acceptance testing. In addition, the system will be maintained, adding and changing functionality as needed. Regression testing is therefore necessary.

Question 14

Which of unit, integration, system and acceptance testing are parts of validation and which are parts of verification?

Answer 14

Unit and integration testing concentrate on whether parts of the system perform according to their specifications, answering the verification question (have we built the system correctly?). System and, therefore, acceptance testing concentrate on showing that the customer’s requirements have indeed been met, answering the validation question (have we built the right system?). Note, however, that there are no hard and fast distinctions. Unit testing can be used to demonstrate that a component satisfies a customer’s requirements – thus it can be viewed as validation, and system testing can be used to demonstrate that a system operates according to specification – in which case it can be viewed as verification.

Question 15

From Table 4 you can see that complexity is related to each of the SQFs reliability, maintainability, flexibility and testability. Suggest why this should be the case.

Answer 15

Reliability is directly related to the number of bugs in a system, and the number of bugs in a system, as we have seen, is related to its complexity. Modifying complex code is likely to more difficult than modifying simple code; maintainability and flexibility are therefore related to complexity. Determining test data for a complex system is likely to be more difficult than for a simple system; testability is therefore also related to complexity.

Question 16

Look again at the definition of the DIT metric. Do you think that, in calculating the metric, classes from the Java API should be included? Do you think that API classes should be counted in Chidamber and Kemerer’s other metrics?

Answer 16

Chidamber and Kemerer’s metrics measure complexity, and API classes add to complexity. Hence they should be counted in all metric calculations. For this reason, the Java documentation will be very useful when calculating these metrics.

Question 17

The WMPC metric measures the complexity of a class in terms of the sum of the cyclomatic complexities of its methods. For a class with 10 methods, suggest a WMPC value above which the behaviour of the class should be considered too complex.

Answer 17

For individual methods, a cyclomatic complexity of greater than 10 should be regarded as a hint that the method is too complex. For a class with 10 methods, therefore, a value for the WMPC metric of greater than 10610 = 100 should be regarded as indicating that the behaviour of the class is too complex.

Question 18

Should the cyclomatic-complexity metric be used to measure the complexity of a whole software system?

Answer 18

Because the cyclomatic-complexity metric is based on decision points, which are present only in methods, it is ‘blind’ to the class-structuring mechanisms that are available in object-oriented system descriptions. As much of the complexity of an object-oriented system is held in the class structure, applying the cyclomatic complexity metric to a whole system would not, therefore, be appropriate.

Question 19

Use the strategy for black box testing described above to choose a good set of test data for the cancel command discussed in Example 5 (part 2).

Answer 19

The subdomain for the cancel command is shown in Table 9. We need to test for each flightNumber in Table 8, and for each such flightNumber we need to test extreme, near-extreme, and central values of the seatNumber data. Five suitable test values for the seatNumber data are 1, 2, 200, 449 and 450. Thus a suitable set of test data for the cancel command is shown in Table 9.

Question 20

What is the point of testing the case n + 1 for a loop that is intended to execute no more than n times?

Answer 20

To ensure that the termination condition of the loop is correct.

Question 21

Give a situation in which black box testing will test something that white box testing would miss, and one in which white box testing will test something that black box testing would miss.

Answer 21

Because black box testing takes its test cases from the specification, it is likely to pick up the following sorts of errors that white box testing would miss (not an exhaustive list):

• operations required by the specification but not provided for by the implementation;
• errors between the interfaces of two classes;
• errors in the transformations between internal states of a class, if these affect input/output behaviour;
• performance errors, in which system performance is found to be wanting;
• system initialisation and termination errors.

On the other hand, in looking inside the implementation, white box testing will pick up the following sorts of errors that black box testing would miss (again, not an exhaustive list):

• that the sequences of method calls in the body of a method perform the correct function;
• that Boolean conditions in if statements, while loops, etc. are correctly expressed;
• that loops terminate correctly;
• that the relationships between methods and attributes of classes are correct.

Question 22

A weather-recording system records wind-speed data, rainfall data and barometric pressure, and sends summary data into a computer network. If the wind-speed data is represented as an integer between 0 and 110, the rainfall data is represented as a floating point number between 0.0 and 150.0 (significant to 1 decimal place) and barometric pressures is represented as an integer between 800 and 1200, what is the input data space?

Answer 22

The input data space is the set of triples of values taken from the sets {0, 1, …, 110}, {0.0, 0.1, …, 150.0} and {800, 801, …, 1200}.

Question 23

Do the test data for the TherapyServer (in Example 6) reveal any deficiencies in the code?

Answer 23

Whether an answer from response is a deficiency depends upon what the requirements for the TherapyServer are, and we were not given these. Thus we cannot say if it is correct or not.

Question 24

Develop test data for the simple while loop that appears at line 7 of the response method of TherapyServer code (see Table 10).

Answer 24

” “ (the empty string) will drop straight through the loop.
“hello?” will force the loop to execute once.
“hello there” will force the loop to execute twice.
As there is no a priori bound on the number of times that the loop can execute, only the
first three test cases for simple loops apply.

Question 25

What are the benefits of conducting inspections for:

(a) the product under inspection?
(b) the development process for that product?
(c) the inspection process?

Answer 25

(a) The quality of the product under inspection will benefit by having defects in the product found and removed. The product itself may well also be ready for delivery earlier as far fewer defects are found downstream, for example during testing.
(b) Keeping statistics on the types of defects found by inspections could pin-point problems in the development process, by indicating common defects in all products of a particular phase, for instance.
(c) The development of checklists for each of the uses of inspection is one way in which the inspection process could benefit; the on-the-job training of inspection personnel is another.

Question 26

What would the entry and exit criteria be, when conducting a review of a class diagram?

Answer 26

A suitable entry criterion for a class diagram might be that it is syntactically correct, as determined by a modelling tool. However, other criteria will also be needed depending on the stage of development process: at an early stage, we may simply require that enough classes are present to cover all the use cases we are going to implement; at the implementation stage, we may require that all operations are present. Our exit criteria is always the same, that is, the removal of defects, although the precise meaning of this may change.