Black box

Question 1

What is black box testing?

Answer 1

Testing in which the tester does not know or have access to source code itself. It is derived from the box being ‘black’ or ‘opaque’. Tester knows inputs and expected output and tests.

Black box testing (also called functional testing) is testing that ignores the internal mechanism of a system or component and focuses solely on the outputs generated in response to selected inputs and execution conditions.

Question 2

What is white box testing?

Answer 2

Testing where the tester knows or has access to source code, this would be the software developer. This is in reference that the box is invisible e.g.: white.

White box testing (also called structural testing and glass box testing) is testing that takes into account the internal mechanism of a system or component.

Question 3

Software testing is part of what two processes?

Answer 3

Verification and Validation.

Question 4

What is an informal definition of verification?

Answer 4

Verification: Are we building the product right?

Through verification, we make sure the product behaves the way we want it to. For example, on the left in Figure 1, there was a problem because the specification said that players should collect $200 if they land on or pass Go. Apparently a programmer implemented this requirement as if the player had to pass Go to collect. A test case in which the player landed on Go revealed this error.

Verification (the first V) is the process of evaluating a system or component to determine whether the products of a given development phase satisfy the conditions imposed at the start of that phase [11]. Verification activities include testing and reviews. For example, in the software for the Monopoly game, we can verify that
two players cannot own the same house.

Question 5

What is an informal definition of validation?

Answer 5

Validation: Are we building the right product?

Through validation, we check to make sure that somewhere in the process a mistake hasn’t been made such that the product build is not what the customer asked for; validation always involves comparison against requirements. For example, on the right in Figure 1, the customer specified requirements for the Monopoly game – but the programmer delivered the game of Life. Maybe the programmer thought he or she “knew better” than the customer that the game of Life was more fun than Monopoly and wanted to “delight” the customer with something more fun than the specifications stated. This example may seem exaggerated – but as programmers we can miss the mark by that much if we don’t listen well enough or don’t pay attention to details – or if we second guess what the customer says and think we know better how to solve the customer’s problems.

Question 6

What is software testing?

Answer 6

Software testing is the process of analyzing a software item to detect the differences between existing and required conditions (that is, bugs) and to evaluate the features of the software item [9, 12].

Question 7

Verification vs Validation.

Answer 7

Verification
Are we building the product right?
“I landed on “Go” but didn’t get my $200!”

Validation
Are we building the right product?
“I know this game has money and players and “Go” – but this is not the game I wanted.”

Question 8

What are the 5 ways software may not be incorrect?

Answer 8

Mistake – a human action that produces an incorrect result

Fault [or Defect] – an incorrect step, process, or data definition in a program

Failure – the inability of a system or component to perform its required function within the specified performance requirement.

Error – the difference between a computed, observed, or measured value or condition and the true, specified, or theoretically correct value or condition

Specification – a document that specifies in a complete, precise, verifiable manner, the requirements, design, behavior, or other characteristic of a system or component, and often the procedures for determining whether these provisions have been satisfied.

Question 9

Define ‘mistake’

Answer 9

a human action that produces an incorrect result.

Question 10

Define ‘fault’ (aka ‘defect’)

Answer 10

an incorrect step, process, or data definition in a program.

Question 11

Define ‘failure’

Answer 11

the inability of a system or component to perform its required function
within the specified performance requirement.

Question 12

Define ‘error’

Answer 12

the difference between a computed, observed, or measured value or
condition and the true, specified, or theoretically correct value or condition.

Question 13

Define ‘specification’

Answer 13

a document that specifies in a complete, precise, verifiable
manner, the requirements, design, behavior, or other characteristic of a system or component, and often the procedures for determining whether these provisions have been satisfied.

Question 14

What is the order that errors, faults, and mistakes occur?

Answer 14

mistake > fault > failure > error

A mistake committed by a person becomes a fault (or defect) in a software artifact, such as the specification, design, or code.

This fault, unless caught, propagates as a defect in the executable code. When a defective piece of code is executed, the fault may become a visible anomaly (a variance from the specification or desired behavior) and a failure is observed

Otherwise, the fault remains latent. Testing can reveal failures, but it is the faults that must be found and removed [3]; finding a fault (the cause of a failure) can be time consuming and unpredictable. Error is a measure of just how incorrect the results are.

Question 15

What do we try to cause during testing?

Answer 15

A purpose of testing is to cause failures in order to make faults visible [10] so that the faults can be fixed and not be delivered in the code that goes to customers. Another purpose of testing is to assess the overall quality level of the code. For example, a test team may determine a project with too many high-severity defects should be sent back to development for additional work to improve the quality before the testing effort should continue. Or, the management may have a policy that no product can ship if testing is continuing to reveal high-severity defects.

Question 16

What are the two expense cases of testing?

Answer 16

To test, or not to test.

Question 17

To minimize costs of testing, what is one goal of testing?

Answer 17

To minimize the costs associated with testing and with software failures, a goal of testing must be to uncover as many defects as possible with as little testing as possible

Question 18

We want to write test cases that what likelyhood of uncovering how many faults?

Answer 18

In other words, we want to write test cases that have a high likelihood of uncovering the faults that are the most likely to be observed as a failure in normal use

Question 19

All software testing is performed with what type of code?

Answer 19

Executable code, that is code that runs so we can perform tests against it.

Question 20

What type of minimal code might need to be written to perform a test against?

Answer 20

Scaffolding code (or just scaffolding)

Question 21

How is scaffolding defined?

Answer 21

Scaffolding is defined as computer programs and data files built to support software development and testing but not intended to be included in the final product [11].

Question 22

What does scaffolding do?

Answer 22

Scaffolding code is code that simulates the functions of components that don’t exist yet and allow the program to execute [16]

Question 23

What are the two components of scaffolding?

Answer 23

Scaffolding code involves the creation of stubs and test drivers.

Question 24

What’s a stub?

Answer 24

Stubs are modules that simulate components that aren’t written yet, formally defined as a computer program statement substituting for the body of a software module that is or will be defined elsewhere [11]. For example, you might write a skeleton of a method with just the method signature and a hard-coded but valid return value.

Question 25

What is a test driver?

Answer 25

Test drivers are defined as a software module used to involve a module under test and often, provide test inputs, controls, and monitor execution and report test results [11]. Test drivers simulate the calling components (e.g. hard-coded method calls) and perhaps the entire environment under which the component is to be tested [1].

Question 26

What is a mock object?

Answer 26

Mock objects are temporary substitutes for domain code that emulates the real code. For example, if the program is to interface with a database, you might not want to wait for the database to be fully designed and created before you write and test a partial program. You can create a mock object of the database that the program can use temporarily. The interface of the mock object and the real object would be the same. The implementation of the object would mature from a dummy implementation to an actual database.

Question 27

Software testing should be performed against what type of software.

Answer 27

All software! But especially large software systems.

Question 28

How many types of software testing are there?

Answer 28

6

Question 29

Each type of testing has what that defines the correct behavior the test is examining?

Answer 29

A specification

Question 30

The specification’s goal is what?

Answer 30

Identify the failure.

To test the correct (expected) behavior against the incorrect behavior (observed failure).

Question 31

How often should Software Testing be performed?

Answer 31

Software testing is an activity that should be done throughout the whole development process [3].

Question 32

What are other V&V processes?

Answer 32

inspections and pair programming.

Question 33

What is the formal definition of Validation?

Answer 33

Validation is the process of evaluating a system or component during or at the end of the development process to determine whether it satisfies specified requirements [11]. At the end of development validation (the second V) activities are used to evaluate whether the features that have been built into the software satisfy the customer requirements and are traceable to customer requirements. For example, we validate that when a player lands on “Free Parking,” they get all the money that was collected

Question 34

What is the formal definition of Verification?

Answer 34

Verification (the first V) is the process of evaluating a system or component to determine whether the products of a given development phase satisfy the conditions imposed at the start of that phase [11]. Verification activities include testing and reviews. For example, in the software for the Monopoly game, we can verify that
two players cannot own the same house.

Question 35

Name the types of software testing techniques?

Answer 35

Unit testing, Integration testing, Functional and System testing, Acceptance testing, Regression testing, Beta testing

Question 36

What is the opacity of: Unit testing?

Answer 36

White box

Question 37

What is the opacity of: Integration testing?

Answer 37

Black and White box

Question 38

What is the opacity of: Functional and system testing?

Answer 38

Black box

Question 39

What is the opacity of: Acceptance testing?

Answer 39

Black box

Question 40

What is the opacity of: Regression testing?

Answer 40

Black and White box

Question 41

What is the opacity of: Beta testing?

Answer 41

Black Box

Question 42

What is the specification of: Unit testing?

Answer 42

Low-level design and/or code structure

Question 43

What is the specification of: Integration testing?

Answer 43

Low- and high-level design

Question 44

What is the specification of: Functional and System testing?

Answer 44

high-level design, requirements specification and analysis

Question 45

What is the specification of: Acceptance testing?

Answer 45

requirements specification

Question 46

What is the specification of: Regression testing?

Answer 46

Any changed documentation, high-level design

Question 47

What is the specification of: Beta testing?

Answer 47

None.

Question 48

What is the general scope of: Unit testing?

Answer 48

Small unit of code, no larger than a class

Question 49

What is the general scope of: Integration testing?

Answer 49

Multiple classes

Question 50

What is the general scope of: Functional testing?

Answer 50

Functional = the whole product; System = whole product in representative environment

Question 51

What is the general scope of: Acceptance testing?

Answer 51

Whole product in customer’s environment

Question 52

What is the general scope of: Beta testing?

Answer 52

Whole product in customer’s environment

Question 53

What is the general scope of: Regression testing?

Answer 53

Any of above

Question 54

Who performs: Unit testing?

Answer 54

Programmer(s) who wrote code

Question 55

Who performs: Integration testing?

Answer 55

Programmer(s) who wrote code

Question 56

Who performs: Functional and System testing?

Answer 56

Independent tester

Question 57

Who performs: Acceptance testing?

Answer 57

Customer

Question 58

Who performs: Beta testing?

Answer 58

Customer

Question 59

Who performs: Regressino testing?

Answer 59

Progammer(s) OR Independent tester

Question 60

What do you see when a test case fails?

Answer 60

You see the ‘failure’ but need to find the ‘fault still.

Question 61

When is the best time to find a ‘failure’?

Answer 61

As early in the development process as possible.

Question 62

When is it harder to find ‘fault’ that caused a ‘failure’?

Answer 62

The further you go into the development process.

Question 63

What are you likely to find if you ‘Unit’ test often enough?

Answer 63

The new failure was in the code you just wrote or tested, and will be easier to find than later on.

If you wait until system or acceptance testing, a failure could be anywhere in the system – you will have to be an astute detective to find the fault now.

Question 64

What is the formal definition of a Test Plan?

Answer 64

A test plan is a document describing the scope, approach, resources, and schedule of intended test activities. It identifies test items, the features to be tested, the testing tasks, who will do each task, and any risks requiring contingency plans [11]. An important component of the test plan is the individual test cases. A test case is a set of test inputs, execution conditions, and expected results developed for a particular objective, such as to exercise a particular program path or to verify compliance with a specific requirement [11].

Question 65

When should a Test Plan be established?

Answer 65

Early in the development cycle.

Question 66

Why is writing a Test Plan early helpful?

Answer 66

This allows you to think through a thorough set of test cases. If you wait until the end of the cycle to write and execute test cases, you might be in a very chaotic, hurried time period. Often good test cases are not written in this hurried environment, and ad hoc testing takes place.

Question 67

What is Ad Hoc testing?

Answer 67

With ad hoc testing, people just start trying anything they can think of without any rational roadmap through the customer requirements. The tests done in this manner are not repeatable.

Question 68

Name an example time when the Test Plan can be written?

Answer 68

as soon as customer requirements analysis has completed, the test team should start writing black box test cases against that requirements document.

Question 69

What’s one example of when a Test Plan should be written early?

Answer 69

By doing so this early, the testers might realize the requirements are not complete. The team may ask questions of the customer to clarify the requirements so a specific test case can be written. The answer to the question is helpful to the code developer as well. Additionally, the tester may request (of the programmer) that the code is designed and developed to allow some automated test execution to be done. To summarize, the earlier testing is planned at all levels, the better.

Question 70

Is the Test Planning and Test Execution a singular or iterative process?

Answer 70

Iterative (and essential)

Question 71

What tests should you write when the ‘requirements documentation’ is available?

Answer 71

Functional and System

Question 72

When the ‘requirements documentation’ changes which tests should change?

Answer 72

Functional and System

Question 73

Black box testing is also called what?

Answer 73

Functional testing or Behavioral testing

Question 74

What does black box testing focus on?

Answer 74

Determining whether or not a program does what it is supposed to do based on the ‘functional requirements’ based from the ‘requirements documentation’ provided by the customer of the code.

Question 75

Where do the ‘functional requirements’ come from?

Answer 75

the ‘requirements documentation’

Question 76

Where does the ‘requirements documentation’ come from?

Answer 76

provided from the ‘customer requirements analysis’ performed against the customer.

Question 77

How many categories does Black box testing attempt to find errors from?

Answer 77

5

Question 78

Black Box testing attempts to find errors from where?

Answer 78

The external behavior of the code as used by a customer.

Question 79

What are the categories Black box testing runs against?

Answer 79

(1) incorrect or missing functionality; (2) interface errors; (3) errors in data structures used by interfaces; (4) behavior or performance errors; and (5) initialization and termination errors.

Question 80

Black box testing determines if functions [blank] to work, according to the [blank]

Answer 80

APPEAR and SPECIFICATION

Through this testing, we can determine if the functions appear to work according to specifications. However, it is important to note that no amount of testing can unequivocally demonstrate the absence of errors and defects in your code.

Question 81

Who should perform the Black Box testing?

Answer 81

Independent tester, NOT the programmer who wrote code.

Question 82

Why should this person perform Black Box testing?

Answer 82

t is best if the person who plans and executes black box tests is not the programmer of the code and does not know anything about the structure of the code. The programmers
of the code are innately biased and are likely to test that the program does what they programmed it to do. What are needed are tests to make sure that the program does what the customer wants it to do. As a result, most organizations have independent testing groups to perform black box testing. These testers are not the developers and are often referred to as third-party testers. Testers should just be able to understand and specify what the desired output should be for a given input into the program, as shown in Figure 3.

Question 83

What is the flow of a Black box test?

Answer 83

Input > [program] (black box test) > output

Question 84

What part of the program and test does Black Box take into account?

Answer 84

Input and Output, without regard to internal code.

Question 85

What should the anatomy of a test case look like?

Answer 85

Test ID > Description (input) > Expected Results > Actual Results (output)

First, you give each test case a unique identifier. When you are tracking large projects, you might need to itemize those test cases that have not yet passed. This identifier is recorded in the first column. For example, you might need to say something like, “All my test cases are running except playerMovement1. I’m working on that one today.” Next
in the second column of the table, you specifically describe the set of steps and/or input for the particular condition you want to test (including what needs to be done to prepare for the test case to be run). The third column is the expected results for an input/output oracle – what is expected to come out of the “black box” based upon the input (as described in the “description”).

Question 86

What is an ‘oracle’?

Answer 86

An oracle is any program, process, or body of data that specified the expected outcome of a set of tests as applied to a tested object [1];

Question 87

What is one type of oracle for Black Box?

Answer 87

input/output oracle

Question 88

What does this Black Box oracle do?

Answer 88

This IOoracle is an oracle that specifies the expected output for a specified input [1]. In the last column, the actual results are recorded after the tests are run. If a test passes, the actual results will indicate “Pass.” If a test fails, it is helpful to record “Fail” and a description of the failure (“what came out”) in the actual results column.