FX Flashcards
What is software verification?
ensure that the actual implementation meets the requirements set forth at the beginning
The main thrust of Software Engineering II is on Testing. If you had to hazard a guess, which of the five phases above do you think Testing falls into?
If you answered Verification you would be correct. But if you think Verification is something that happens after all the other phases are complete, you would be incorrect. Other than the Requirements phase, testing can play a crucial role throughout the software development process.
What is unit testing?
Unit Testing is when individual units of the software are tested…
What is a failure?
Failure - a deviation from the expected behavior
These failures occur because there exists a “bug” in the code (a fault).
What is the a fault
Fault: - an instance of incorrect code that can lead to a failure
A fault is introduced to the program when a programmer makes a mistake (an error).
What is a error?
Error - a mistake that introduces a fault (e.g. typo and conceptual misunderstanding)
Functional Testing
These types of tests are used to verify that the software meets the requirement specifications when it comes to functionality; does the software do the things it is expected to do?
Examples:
Does clicking “save” actually save the changes to the hard drive?
Does the search algorithm actually find the shortest path?
Does the isItPrime function correctly identify prime numbers?
Does the database correctly supply responses to the client?
Non-functional Testing
These types of tests are used to verify that the software performs at the required levels. This can literally mean performance, but also usability, reliability, and robustness.
Examples
Does the webpage load in less than 2 seconds?
Does the interface provide enough clues for easy navigation?
Does the system successfully recover from a catastrophic failure?
Does the service support 10,000+ active users at a time?
You are interested in learning how user friendly your software is.
Manual - This type of task really requires a human to interact with the software.
Scenarios 2: You want to make sure your new changes didn’t break your existing features
Automated or Manual
Automated - Using pre-written tests that were known to pass, you can quickly verify that your changes didn’t cause any of them to fail. This is called Regression Testing.
Scenarios 3: You want to test for game breaking bugs in a massively online multiplayer game.
Manual While automated tests could be used to test components of the game, testing the combined product really requires humans actively trying to break the game while it is running.
Scenarios 4: You want to verify your system can handle a large range of possible inputs.
Automated - Instead of having a human try to input thousands of possible inputs, automated tests can randomly generate inputs in a fraction of the time. This is called Random Testing.
When is black box testing useful?
Black Box Testing is when you write tests based purely on the description provided for the software (a.k.a. the specification).
What are the limitations of black box testing?
It isn’t possible to test every possible input, so tests may miss logic branches/program paths untested (more on this in the White Box module)
There is no way to know why the failure occurs, just that the failure indicates a fault
Poorly written specifications can lead to inaccurate tests
Random Testing - Advantages
Advantages
Quick to write
Can cover large portions of the input domain with very little code
Tests have no bias
Can potentially generate an input that no one considered
Random Testing Disadvantages
Many random inputs could fall under the same “test case”, so become redundant and a waste of resources
May not test the tricky parts of the input domain like edge cases
Without any targeting, random inputs could easily miss glaring errors
Randomly selecting inputs for each test run could see some runs pass while others fail
How does Partition Testing improve a testing suite?
partition Method is a way of reducing the potentially thousands of different combinations of testable features (i.e. test cases) down to a number that can be realistically implemented.
When is white box testing useful?
White box testing focuses on verifying that all the code works as intended by trying to have as much of it run as possible during testing. This is done through something called coverage.
Definition Time
Code coverage is the extent to which a given test suite executes the source code of the software.
Why is 100% statement coverage not enough?
All the non-executed statements are inside conditionals that aren’t triggered:
Do you think 100% branch coverage means 100% statement coverage? Why?
100% branch coverage guarantees 100% statement coverage. In this way we can say branch coverage subsumes statement coverage.
What is the input domain for my_abs?
def my_abs(val):
if(val > 0):
return val
else
return -val
Answer-The input domain consists of all real numbers!
What is an oracle?
The oracle’s sole job is to watch for error states in the software and save the random inputs that generated those states for later inspection. . An oracle can be as simple as displaying error generating input to the screen or it could be an elaborate piece of software that generates formal bug reports.
How can we make our random testers “smarter”?
With guided random testing, we use what is called a heuristic to help us make our random generation smarter
How does random testing help with large input domains?
Can cover large portions of the input domain with very little code
value in dynamically adding tests to a testing suite
rror recovery, an accurate number of tests run, and the ability to use asserts with random tests
What software design approach is most directly associated with TDD?
Agile
What does it mean to write the “bare minimum” of code?
Write the bare minimum of code to make the test pass
When do we know we are done?
stop development after adding “enough” tests without triggering a new failure
can you identify where in the process we refactored?
We refactored everytime we added test cases that broke our implementation. So when we changed return ‘1’ to return str(num), that was refactoring. We did this repeatedly as we went along and fixed our failing tests.
Define Continuous Integration
The primary goal of CI is to have many developers working on the same codebase at the same time. CI as a set of guiding principles
Continuous Integration Principles
Use a VCS like Git to maintain a central codebase.
Building the software should be automated and easily triggered. Once built, the software should be able to test itself against a provided test suite. Everyone needs to commit work to the shared codebase at least once a day. Every commit to master should be built and tested. Mandatory Code Review when requesting changes to be merged into the shared codebase
What is a Fagan Inspection?
A process of trying to find defects in documents (such as source code or formal specifications) during various phases of the software development process
Planning - gathering participants and resources needed for the inspection process
Overview - meeting to discuss the important aspects of the project
Preparation - participants must review the material before the meeting
Inspection meeting - the actual moment the code is physically inspected for defects
Rework - any defects that are discovered are “fixed”
Follow-up - someone follows up with the developer to ensure the reworks have been done correctly
How can I be a good Code Reviewer?
Read every line you are assigned
Put effort into your feedback
Think about how you would do it
How can I be a good Code Reviewee?
Proofread
Make it worth it
Shine a light on it
What is the difference between a Stub and a Mock?
Stubs
Stubs contain predefined data that is returned when called, but do not imitate behavior.
Mocks
Mocks simulate the behavior of a service and its actions can be verified.
What does Unittest’s patch do?
which looks up an object in a given module and replaces that object with a Mock . Usually, you use patch() as a decorator or a context manager to provide a scope in which you will mock the target object.
When is it appropriate to employ MC/DC?
Modified Condition/Decision Coverage’s purpose is to only test the important conditions to limit the number of test cases required.
Which type of testing requires the input of the client/user?
Acceptance Testing
Which type of testing requires the input of the client/user?
Acceptance Testing
What are the differences between a Mutation-Based and Generation-Based Fuzzer?
Mutation-Based - This type of fuzzer starts by selecting a valid input. It then mutates the input in some random way and throws it at the software under test. The tester can achieve decent coverage by curating a varied list of valid inputs that would trigger different states in the software.
Generation-Based -
As you may have guessed, generation fuzzers do not start with valid inputs. These fuzzers use some “knowledge” of the input domain to create the random inputs. This is similar to how we used “rules” to generate random inputs to test a password validator. The difference here is that in Exploration: Random Testing we were unit testing and now we are more concerned about how these random inputs affect the system.
What are the differences between Load and Stress Testing?
Load Testing is where software is tested for performance under expected operating conditions.
Stress Testing is where software is tested to ensure it functions under extreme operating conditions.
Fuzzing
aims to discover errors in programs by feeding them random inputs
Fuzzer
is the tool/software used to perform Fuzzing
What happens during the Requirements phase?
Interviews with stakeholders
Craft user stories and use cases
What happens during the Design phase?
Layout the architecture of the program
Produce a design document
Which of the following are examples of Functional Testing?
Regression Testing
Unit Testing
Identify the 4 elements to a testing framework.
Test runner
Test case
Test fixture
Test suite
Each unittest test case must include at least one ________.
Assertion
Context: Mocks
Given the following code snippet, what goes in the blank?
from unittest.mock import Mock
mock = Mock()
# Set return_value mock.abs.\_\_\_\_\_\_ = "7"
return_value
Given the following code snippet, what is the output of the print statement?
mock = Mock()
mock. func(“Hello World”)
mock. func(“Greetings Planet”)
print(mock.func.call_args)
call(‘Greetings Planet’)
Given the following snippet. If we wanted to use a mock to trigger an exception, what would we use in the blank?
Mock file reader to control its behavior
open = Mock()
def load_file():
# Does absolutely nothing other than raise the desired exception
content = open('temp_file.txt')
if content.length != 0:
return content
return None
class TestCase(unittest.TestCase):
def test_read_file(self):
# Test for IOError
open.\_\_\_\_\_\_ = IOError
# This is a context manager that allows us to test for exceptions
with self.assertRaises(IOError):
load_file()
side_effect
Which of the following is called before each test case in a TestCase object?
setUp()
Which of the following is called after all the test cases in a TestCase Object are run (not after each test)?
tearDownClass()
