Changing your testing as your world changes Flashcards
What are the 5 worlds of Software
Shrinkwrap (open-source, configurable, web-based) Internal Embedded Games Throwaway
Exploratory Testing
Simultaneous learning, test design and test execution. Learn about the software as you test.
Limitations of exploratory testing
- no concept of completeness
- not systematic
- expensive in terms of tester skill, may want more explicit process.
Specification Based Testing
Evaluating software based on explicit claims in specification.
Benefits of specification based testing
- Systemic (follow SRS)
- Spec is usually simpler than software
- Captures what matters most about the purpose of the software
Limitation of informal specs
Hard to verify. Perhaps no mechanical way to convert it into tests
Model-based testing
Using models of expected behaviour to produce test case specs that can reveal discrepancies between actual program behaviour and the model.
Goal: test a program specified by a model to find faults.
Fuzz testing
Feeding random inputs until program crash or assertion violations.
Give an example of model-based testing
Creating a control flow graph and testing inputs to ensure the program follows the same behaviour of the model.
What is a test oracle?
Tells you whether a test result is good or not. Not always able to do this, however! you can use an oracle to determine if an answer may be invalid.
a, b > 0
oracle can make sure sum of a, b is above zero.
a + b > 0
Partition testing
Partition the input domain based on their significance for the program. Then select values within those partitions to test. Recall Parnas Tables!
Boundary Testing
Choosing inputs on semantically significant boundaries.
int increment(int i) {
return i + 1;
}
You may want to set i to MAX_INT? this is a boundary since + 1 would set an error!
Coverage Based Testing
Testing until you meet some pre-agreed code coverage goal.
Could be: All modules, All classes, All methods. Maybe every statement? or True/False decision?
What if coverage criteria shows you do not hit every part of the code?
You could have dead branches of code that are never able to be run.
Fault Injection
Introducing faults into the software to test.
Give an example of partition testing/Boundary Testing/ Coverage based Testing and Fault Injection for an Instagram like app.
P: Partition state of file types and upload to see what can happen.
B: Different files sizes to see how the app reacts. Very large and very small. Try an empty file. Does it break?
C: If app implements different ways to search a target, ensure they all work.
F: Try to upload invalid files, see how the app reacts.
Partition, Boundary, Coverage, Fault, Exploratory, Fuzz, Model based. Give an example for Blackbox, Whitebox and Greybox
B: Exploratory, Fuzz,
W: Coverage based, Fault injection
G: Model-based, Spec based
Mostly, these tests can be pretty fluid between the categories.
Partition, Boundary, Coverage, Fault, Exploratory, Fuzz, Model based. Functional or Non-functional?
Combination of both. You can look at all attributes whether functional or non-functional within the context of these tests.
How does testing change between Agile and Waterfall?
Agile: Might test frequently but often without a specification.
Waterfall: May define tests before any coding is even done. Pure waterfall may not allow for testing to be much use.
How do you know the right way to do testing?
You have to examine:
- Software “World”
- Stage of development
- Skill of test team
- Novelty of product
- availability of reference models
- Availability of explicit specifications.
Exhaustive testing
Almost never possible and extremely costly.
When to stop testing?
- meet coverage goals
- Meet defect discovery rate
- Marginal cost (cost of shipping with a defect? Cost per defect discovered?)
- team consensus
- Management wants you to stop!
