Changing your testing as your world changes

Question 1

What are the 5 worlds of Software

Answer 1

Shrinkwrap (open-source, configurable, web-based)
Internal
Embedded
Games
Throwaway

Question 2

Exploratory Testing

Answer 2

Simultaneous learning, test design and test execution. Learn about the software as you test.

Question 3

Limitations of exploratory testing

Answer 3

• no concept of completeness
• not systematic
• expensive in terms of tester skill, may want more explicit process.

Question 4

Specification Based Testing

Answer 4

Evaluating software based on explicit claims in specification.

Question 5

Benefits of specification based testing

Answer 5

• Systemic (follow SRS)
• Spec is usually simpler than software
• Captures what matters most about the purpose of the software

Question 6

Limitation of informal specs

Answer 6

Hard to verify. Perhaps no mechanical way to convert it into tests

Question 7

Model-based testing

Answer 7

Using models of expected behaviour to produce test case specs that can reveal discrepancies between actual program behaviour and the model.

Goal: test a program specified by a model to find faults.

Question 8

Fuzz testing

Answer 8

Feeding random inputs until program crash or assertion violations.

Question 9

Give an example of model-based testing

Answer 9

Creating a control flow graph and testing inputs to ensure the program follows the same behaviour of the model.

Question 10

What is a test oracle?

Answer 10

Tells you whether a test result is good or not. Not always able to do this, however! you can use an oracle to determine if an answer may be invalid.

a, b > 0
oracle can make sure sum of a, b is above zero.
a + b > 0

Question 11

Partition testing

Answer 11

Partition the input domain based on their significance for the program. Then select values within those partitions to test. Recall Parnas Tables!

Question 12

Boundary Testing

Answer 12

Choosing inputs on semantically significant boundaries.

int increment(int i) {
    return i + 1;
}
You may want to set i to MAX_INT? this is a boundary since + 1 would set an error!

Question 13

Coverage Based Testing

Answer 13

Testing until you meet some pre-agreed code coverage goal.

Could be: All modules, All classes, All methods. Maybe every statement? or True/False decision?

Question 14

What if coverage criteria shows you do not hit every part of the code?

Answer 14

You could have dead branches of code that are never able to be run.

Question 15

Fault Injection

Answer 15

Introducing faults into the software to test.

Question 16

Give an example of partition testing/Boundary Testing/ Coverage based Testing and Fault Injection for an Instagram like app.

Answer 16

P: Partition state of file types and upload to see what can happen.
B: Different files sizes to see how the app reacts. Very large and very small. Try an empty file. Does it break?
C: If app implements different ways to search a target, ensure they all work.
F: Try to upload invalid files, see how the app reacts.

Question 17

Partition, Boundary, Coverage, Fault, Exploratory, Fuzz, Model based. Give an example for Blackbox, Whitebox and Greybox

Answer 17

B: Exploratory, Fuzz,
W: Coverage based, Fault injection
G: Model-based, Spec based

Mostly, these tests can be pretty fluid between the categories.

Question 18

Partition, Boundary, Coverage, Fault, Exploratory, Fuzz, Model based. Functional or Non-functional?

Answer 18

Combination of both. You can look at all attributes whether functional or non-functional within the context of these tests.

Question 19

How does testing change between Agile and Waterfall?

Answer 19

Agile: Might test frequently but often without a specification.
Waterfall: May define tests before any coding is even done. Pure waterfall may not allow for testing to be much use.

Question 20

How do you know the right way to do testing?

Answer 20

You have to examine:

• Software “World”
• Stage of development
• Skill of test team
• Novelty of product
• availability of reference models
• Availability of explicit specifications.

Question 21

Exhaustive testing

Answer 21

Almost never possible and extremely costly.

Question 22

When to stop testing?

Answer 22

• meet coverage goals
• Meet defect discovery rate
• Marginal cost (cost of shipping with a defect? Cost per defect discovered?)
• team consensus
• Management wants you to stop!