Black box Flashcards
Software testing is an important technique for assessing the quality of a software product. In this chapter, we will explain the following: 1 • • • • • the basics of software testing, a verification and validation practice, throughout the entire software development lifecycle the two basic techniques of software testing, black-box testing and white-box testing six types of testing that involve both black- and white-box techniques. strategies for writing fewer test cases and still finding a
What is black box testing?
Testing in which the tester does not know or have access to source code itself. It is derived from the box being ‘black’ or ‘opaque’. Tester knows inputs and expected output and tests.
Black box testing (also called functional testing) is testing that ignores the internal mechanism of a system or component and focuses solely on the outputs generated in response to selected inputs and execution conditions.
What is white box testing?
Testing where the tester knows or has access to source code, this would be the software developer. This is in reference that the box is invisible e.g.: white.
White box testing (also called structural testing and glass box testing) is testing that takes into account the internal mechanism of a system or component.
Software testing is part of what two processes?
Verification and Validation.
What is an informal definition of verification?
Verification: Are we building the product right?
Through verification, we make sure the product behaves the way we want it to. For example, on the left in Figure 1, there was a problem because the specification said that players should collect $200 if they land on or pass Go. Apparently a programmer implemented this requirement as if the player had to pass Go to collect. A test case in which the player landed on Go revealed this error.
Verification (the first V) is the process of evaluating a system or component to determine whether the products of a given development phase satisfy the conditions imposed at the start of that phase [11]. Verification activities include testing and reviews. For example, in the software for the Monopoly game, we can verify that
two players cannot own the same house.
What is an informal definition of validation?
Validation: Are we building the right product?
Through validation, we check to make sure that somewhere in the process a mistake hasn’t been made such that the product build is not what the customer asked for; validation always involves comparison against requirements. For example, on the right in Figure 1, the customer specified requirements for the Monopoly game – but the programmer delivered the game of Life. Maybe the programmer thought he or she “knew better” than the customer that the game of Life was more fun than Monopoly and wanted to “delight” the customer with something more fun than the specifications stated. This example may seem exaggerated – but as programmers we can miss the mark by that much if we don’t listen well enough or don’t pay attention to details – or if we second guess what the customer says and think we know better how to solve the customer’s problems.
What is software testing?
Software testing is the process of analyzing a software item to detect the differences between existing and required conditions (that is, bugs) and to evaluate the features of the software item [9, 12].
Verification vs Validation.
Verification
Are we building the product right?
“I landed on “Go” but didn’t get my $200!”
Validation
Are we building the right product?
“I know this game has money and players and “Go” – but this is not the game I wanted.”
What are the 5 ways software may not be incorrect?
Mistake – a human action that produces an incorrect result
Fault [or Defect] – an incorrect step, process, or data definition in a program
Failure – the inability of a system or component to perform its required function within the specified performance requirement.
Error – the difference between a computed, observed, or measured value or condition and the true, specified, or theoretically correct value or condition
Specification – a document that specifies in a complete, precise, verifiable manner, the requirements, design, behavior, or other characteristic of a system or component, and often the procedures for determining whether these provisions have been satisfied.
Define ‘mistake’
a human action that produces an incorrect result.
Define ‘fault’ (aka ‘defect’)
an incorrect step, process, or data definition in a program.
Define ‘failure’
the inability of a system or component to perform its required function
within the specified performance requirement.
Define ‘error’
the difference between a computed, observed, or measured value or
condition and the true, specified, or theoretically correct value or condition.
Define ‘specification’
a document that specifies in a complete, precise, verifiable
manner, the requirements, design, behavior, or other characteristic of a system or component, and often the procedures for determining whether these provisions have been satisfied.
What is the order that errors, faults, and mistakes occur?
mistake > fault > failure > error
A mistake committed by a person becomes a fault (or defect) in a software artifact, such as the specification, design, or code.
This fault, unless caught, propagates as a defect in the executable code. When a defective piece of code is executed, the fault may become a visible anomaly (a variance from the specification or desired behavior) and a failure is observed
Otherwise, the fault remains latent. Testing can reveal failures, but it is the faults that must be found and removed [3]; finding a fault (the cause of a failure) can be time consuming and unpredictable. Error is a measure of just how incorrect the results are.
What do we try to cause during testing?
A purpose of testing is to cause failures in order to make faults visible [10] so that the faults can be fixed and not be delivered in the code that goes to customers. Another purpose of testing is to assess the overall quality level of the code. For example, a test team may determine a project with too many high-severity defects should be sent back to development for additional work to improve the quality before the testing effort should continue. Or, the management may have a policy that no product can ship if testing is continuing to reveal high-severity defects.
What are the two expense cases of testing?
To test, or not to test.
To minimize costs of testing, what is one goal of testing?
To minimize the costs associated with testing and with software failures, a goal of testing must be to uncover as many defects as possible with as little testing as possible
We want to write test cases that what likelyhood of uncovering how many faults?
In other words, we want to write test cases that have a high likelihood of uncovering the faults that are the most likely to be observed as a failure in normal use
All software testing is performed with what type of code?
Executable code, that is code that runs so we can perform tests against it.
What type of minimal code might need to be written to perform a test against?
Scaffolding code (or just scaffolding)
How is scaffolding defined?
Scaffolding is defined as computer programs and data files built to support software development and testing but not intended to be included in the final product [11].
What does scaffolding do?
Scaffolding code is code that simulates the functions of components that don’t exist yet and allow the program to execute [16]
What are the two components of scaffolding?
Scaffolding code involves the creation of stubs and test drivers.
What’s a stub?
Stubs are modules that simulate components that aren’t written yet, formally defined as a computer program statement substituting for the body of a software module that is or will be defined elsewhere [11]. For example, you might write a skeleton of a method with just the method signature and a hard-coded but valid return value.
What is a test driver?
Test drivers are defined as a software module used to involve a module under test and often, provide test inputs, controls, and monitor execution and report test results [11]. Test drivers simulate the calling components (e.g. hard-coded method calls) and perhaps the entire environment under which the component is to be tested [1].
What is a mock object?
Mock objects are temporary substitutes for domain code that emulates the real code. For example, if the program is to interface with a database, you might not want to wait for the database to be fully designed and created before you write and test a partial program. You can create a mock object of the database that the program can use temporarily. The interface of the mock object and the real object would be the same. The implementation of the object would mature from a dummy implementation to an actual database.
Software testing should be performed against what type of software.
All software! But especially large software systems.
How many types of software testing are there?
6
Each type of testing has what that defines the correct behavior the test is examining?
A specification
The specification’s goal is what?
Identify the failure.
To test the correct (expected) behavior against the incorrect behavior (observed failure).
How often should Software Testing be performed?
Software testing is an activity that should be done throughout the whole development process [3].
What are other V&V processes?
inspections and pair programming.
What is the formal definition of Validation?
Validation is the process of evaluating a system or component during or at the end of the development process to determine whether it satisfies specified requirements [11]. At the end of development validation (the second V) activities are used to evaluate whether the features that have been built into the software satisfy the customer requirements and are traceable to customer requirements. For example, we validate that when a player lands on “Free Parking,” they get all the money that was collected
What is the formal definition of Verification?
Verification (the first V) is the process of evaluating a system or component to determine whether the products of a given development phase satisfy the conditions imposed at the start of that phase [11]. Verification activities include testing and reviews. For example, in the software for the Monopoly game, we can verify that
two players cannot own the same house.
Name the types of software testing techniques?
Unit testing, Integration testing, Functional and System testing, Acceptance testing, Regression testing, Beta testing
What is the opacity of: Unit testing?
White box
What is the opacity of: Integration testing?
Black and White box
What is the opacity of: Functional and system testing?
Black box
What is the opacity of: Acceptance testing?
Black box
What is the opacity of: Regression testing?
Black and White box
What is the opacity of: Beta testing?
Black Box
What is the specification of: Unit testing?
Low-level design and/or code structure
What is the specification of: Integration testing?
Low- and high-level design
What is the specification of: Functional and System testing?
high-level design, requirements specification and analysis
What is the specification of: Acceptance testing?
requirements specification
What is the specification of: Regression testing?
Any changed documentation, high-level design
What is the specification of: Beta testing?
None.
What is the general scope of: Unit testing?
Small unit of code, no larger than a class
What is the general scope of: Integration testing?
Multiple classes
What is the general scope of: Functional testing?
Functional = the whole product; System = whole product in representative environment
What is the general scope of: Acceptance testing?
Whole product in customer’s environment
What is the general scope of: Beta testing?
Whole product in customer’s environment
What is the general scope of: Regression testing?
Any of above
Who performs: Unit testing?
Programmer(s) who wrote code
Who performs: Integration testing?
Programmer(s) who wrote code
Who performs: Functional and System testing?
Independent tester
Who performs: Acceptance testing?
Customer
Who performs: Beta testing?
Customer
Who performs: Regressino testing?
Progammer(s) OR Independent tester
What do you see when a test case fails?
You see the ‘failure’ but need to find the ‘fault still.
When is the best time to find a ‘failure’?
As early in the development process as possible.
When is it harder to find ‘fault’ that caused a ‘failure’?
The further you go into the development process.
What are you likely to find if you ‘Unit’ test often enough?
The new failure was in the code you just wrote or tested, and will be easier to find than later on.
If you wait until system or acceptance testing, a failure could be anywhere in the system – you will have to be an astute detective to find the fault now.
What is the formal definition of a Test Plan?
A test plan is a document describing the scope, approach, resources, and schedule of intended test activities. It identifies test items, the features to be tested, the testing tasks, who will do each task, and any risks requiring contingency plans [11]. An important component of the test plan is the individual test cases. A test case is a set of test inputs, execution conditions, and expected results developed for a particular objective, such as to exercise a particular program path or to verify compliance with a specific requirement [11].
When should a Test Plan be established?
Early in the development cycle.
Why is writing a Test Plan early helpful?
This allows you to think through a thorough set of test cases. If you wait until the end of the cycle to write and execute test cases, you might be in a very chaotic, hurried time period. Often good test cases are not written in this hurried environment, and ad hoc testing takes place.
What is Ad Hoc testing?
With ad hoc testing, people just start trying anything they can think of without any rational roadmap through the customer requirements. The tests done in this manner are not repeatable.
Name an example time when the Test Plan can be written?
as soon as customer requirements analysis has completed, the test team should start writing black box test cases against that requirements document.
What’s one example of when a Test Plan should be written early?
By doing so this early, the testers might realize the requirements are not complete. The team may ask questions of the customer to clarify the requirements so a specific test case can be written. The answer to the question is helpful to the code developer as well. Additionally, the tester may request (of the programmer) that the code is designed and developed to allow some automated test execution to be done. To summarize, the earlier testing is planned at all levels, the better.
Is the Test Planning and Test Execution a singular or iterative process?
Iterative (and essential)
What tests should you write when the ‘requirements documentation’ is available?
Functional and System
When the ‘requirements documentation’ changes which tests should change?
Functional and System
Black box testing is also called what?
Functional testing or Behavioral testing
What does black box testing focus on?
Determining whether or not a program does what it is supposed to do based on the ‘functional requirements’ based from the ‘requirements documentation’ provided by the customer of the code.
Where do the ‘functional requirements’ come from?
the ‘requirements documentation’
Where does the ‘requirements documentation’ come from?
provided from the ‘customer requirements analysis’ performed against the customer.
How many categories does Black box testing attempt to find errors from?
5
Black Box testing attempts to find errors from where?
The external behavior of the code as used by a customer.
What are the categories Black box testing runs against?
(1) incorrect or missing functionality; (2) interface errors; (3) errors in data structures used by interfaces; (4) behavior or performance errors; and (5) initialization and termination errors.
Black box testing determines if functions [blank] to work, according to the [blank]
APPEAR and SPECIFICATION
Through this testing, we can determine if the functions appear to work according to specifications. However, it is important to note that no amount of testing can unequivocally demonstrate the absence of errors and defects in your code.
Who should perform the Black Box testing?
Independent tester, NOT the programmer who wrote code.
Why should this person perform Black Box testing?
t is best if the person who plans and executes black box tests is not the programmer of the code and does not know anything about the structure of the code. The programmers
of the code are innately biased and are likely to test that the program does what they programmed it to do. What are needed are tests to make sure that the program does what the customer wants it to do. As a result, most organizations have independent testing groups to perform black box testing. These testers are not the developers and are often referred to as third-party testers. Testers should just be able to understand and specify what the desired output should be for a given input into the program, as shown in Figure 3.
What is the flow of a Black box test?
Input > [program] (black box test) > output
What part of the program and test does Black Box take into account?
Input and Output, without regard to internal code.
What should the anatomy of a test case look like?
Test ID > Description (input) > Expected Results > Actual Results (output)
First, you give each test case a unique identifier. When you are tracking large projects, you might need to itemize those test cases that have not yet passed. This identifier is recorded in the first column. For example, you might need to say something like, “All my test cases are running except playerMovement1. I’m working on that one today.” Next
in the second column of the table, you specifically describe the set of steps and/or input for the particular condition you want to test (including what needs to be done to prepare for the test case to be run). The third column is the expected results for an input/output oracle – what is expected to come out of the “black box” based upon the input (as described in the “description”).
What is an ‘oracle’?
An oracle is any program, process, or body of data that specified the expected outcome of a set of tests as applied to a tested object [1];
What is one type of oracle for Black Box?
input/output oracle
What does this Black Box oracle do?
This IOoracle is an oracle that specifies the expected output for a specified input [1]. In the last column, the actual results are recorded after the tests are run. If a test passes, the actual results will indicate “Pass.” If a test fails, it is helpful to record “Fail” and a description of the failure (“what came out”) in the actual results column.
