Fundamentals of testing

Question 1

What is testing? (simple)

Answer 1

when we are testing something we are checking whether it is OK

Question 2

Why testing is needed?

Answer 2

Testing is necessary because we all make mis-
takes. Some of those mistakes are unimportant, but some of them are expensive
or dangerous.

Question 3

What is “testing is context dependent” principle?

Answer 3

Testing is done differently in different contexts. For example, safety-critical software is
tested differently from an e-commerce site.

Question 4

What is risk?

Answer 4

A risk is something that has not hap-pened yet and it may never happen; it is a potential problem. When we discuss risks, we need to consider how likely it is that
the problem would occur and the impact if it happens.

Question 5

What is likelihood and impact of risk

Answer 5

For example, whenever
we cross the road, there is some risk that we’ll be injured by a car. The likeli-
hood depends on factors such as how much traffic is on the road, whether there
is a safe crossing place, how well we can see, and how fast we can cross. The
impact depends on how fast the car is going, whether we are wearing protective
gear, our age and our health. The risk for a particular person can be worked out
and therefore the best road-crossing strategy.

Question 6

What is error or mistake?

Answer 6

If someone makes an error or mistake in using the software, this may lead
directly to a problem - the software is used incorrectly and so does not behave
as we expected.

Question 7

What is deffect (bug/fault)?

Answer 7

people also design and build the software and they
can make mistakes during the design and build. These mistakes mean that there
are flaws in the software itself. These are called defects or sometimes bugs or faults.

Question 8

What is a failure?

Answer 8

When the software code has been built, it is executed and then any defects may
cause the system to fail to do what it should do (or do something it shouldn’t),
causing a failure. Not all defects result in failures; some stay dormant in the code
and we may never notice them.

Question 9

What failures and defects may arise from?

Answer 9

• errors in the specification, design and implementation of the software and
system;
• errors in use of the system;
• environmental conditions;
• intentional damage;
• potential consequences of earlier errors, intentional damage, defects and
failures

Question 10

When do defects arise?

Answer 10

stages:
requirements
design
build
product work

Question 11

What is a role of testing in software development, maintenance and
operations?

Answer 11

Rigorous
testing is necessary during development and maintenance to identify defects, in
order to reduce failures in the operational environment and increase the quality
of the operational system.

Question 12

In which terms testing can measure quality of software?

Answer 12

In terms of number of

defects found, the tests run, and the system covered by the tests.

Question 13

Quality of what attributes of software can be tested?

Answer 13

We can do this
for both the functional attributes of the software (for example, printing a report
correctly) and for the non-functional software requirements and characteristics
(for example, printing a report quickly enough).

Question 14

What is a specification validation and verification?

Answer 14

validation (‘is
this the right specification?’) and verification (‘is the system correct to spec-
ification?’).

Question 15

What is root cause analysis?

Answer 15

When we detect failures, we might try to track them back to their root cause,
the real reason that they happened.

Question 16

What is “exhaustive testing is impossible” principle?

Answer 16

Testing everything (all combinations of inputs and preconditions) is not feasible except for 
trivial cases. Instead of exhaustive testing, we use risks and priorities to focus testing efforts.

Question 17

ISTQB definition of software

testing as a process

Answer 17

• Process - Testing is a process rather than a single activity - there are a series
of activities involved
• All life cycle activities - testing as a process that takes
place throughout the software development life cycle.
• Both static and dynamic
• Planning - activities take place before and after test execution. We need to
manage the testing; for example, we plan what we want to do; we control the
test activities; we report on testing progress and the status of the software
under test; and we finalize or close testing when a phase completes.
• Preparation - We need to choose what testing we’ll do, by selecting test con
ditions and designing test cases.
• Evaluation - As well as executing the tests, we must check the results and
evaluate the software under test and the completion criteria, which help us
decide whether we have finished testing and whether the software product
has passed the tests.
• Software products and related work products - we don’t just test code. We test
the requirements and design specifications, and we test related documents
such as operation, user and training material.

Question 18

What is test basis?

Answer 18

the thought process of designing tests early in the life cycle can help to prevent defects from being introduced into code. We sometimes refer to this as ‘verifying the test basis via the test design’. The test basis includes documents such as the requirements and design specifications.

Question 19

ISTQB definition of software

testing regarding objectives of testing

Answer 19

• Determine that (software products) satisfy specified requirements.
• Demonstrate that (software products) are fit for purpose.
• Detect defects - We most often think of software testing as a means of
detecting faults or defects that in operational use will cause failures.

Question 20

What is “Early testing”principle?

Answer 20

Testing activities should start as early as possible in the software or system development life
cycle and should be focused on defined objectives.

Question 21

What testing objectives can be?

Answer 21

• finding defects
• gaining confidence in and providing information about the level of quality;
• preventing defects.

Question 22

What is a “development testing” and its main objective?

Answer 22

Once the code is written, programmers
and testers often run a set of tests so that they can identify and fix defects in
the software. In this ‘development testing’ (which includes component, inte-
gration and system testing), the main objective may be to cause as many fail-
ures as possible so that defects in the software are identified and can be fixed.

Question 23

What is regression testing?

Answer 23

It is testing to ensure
nothing has changed that should not have changed.

Testing of a previously tested program following modification to ensure that defects have not been introduced or uncovered in unchanged areas of the software, as a result of the changes made. It is performed when the software or its environment is changed.

Question 24

What is “defect clustering” testing principle?

Answer 24

A small number of modules contain most of the defects discovered during pre-release
testing or show the most operational failures.

Question 25

Why defects tend to cluster?

Answer 25

This can happen because an area of the code is particularly
complex and tricky, or because changing software and other products tends
to cause knock-on defects.

Question 26

What is “pesticide paradox” testing principle?

Answer 26

If the same tests are repeated over and over again, eventually the same set of test cases will no longer find any new bugs. To overcome this ‘pesticide paradox’, the test cases need to be regularly reviewed and revised, and new and different tests need to be written to exercise
different parts of the software or system to potentially find more defects.

Question 27

What is debugging?

Answer 27

When a test finds a defect that must be fixed, a programmer must do some work
to locate the defect in the code and make the fix. In this process a programmer will examine the code for the immediate cause of the
problem, repair the code and check that the code now executes as expected.

Question 28

What is the difference beteen testing and debugging

Answer 28

Developers
may test their own fixes, in which case the very tight cycle of identifying faults, debugging, and retesting is often loosely referred to as debugging. However,
often following the debugging cycle the fixed code is tested independently both
to retest the fix itself and to apply regression testing to the surrounding
unchanged software.

Question 29

What is “Testing shows presence of defects “ principle?

Answer 29

Testing can show that defects are present, but cannot prove that there are no defects.
Testing reduces the probability of undiscovered defects remaining in the software but, even if
no defects are found, it is not a proof of correctness.

Question 30

What is “Absence of errors fallacy” testing principle?

Answer 30

Finding and fixing defects does not help if the system built is unusable and does not fulfill the
users’ needs and expectations.

Question 31

Please, list the testing principles

Answer 31

Testing shows presence of defects
Exhaustive testing is impossible
Early testing
Defect clustering
Pesticide paradox
Testing is context dependent
Absence-of-errors fallacy

Question 32

What are the basic steps in the activities within the fundamental test process

Answer 32

• planning and control;
• analysis and design;
• implementation and execution;
• evaluating exit criteria and reporting;
• test closure activities.

Question 33

What is test planning?

Answer 33

During test planning, we make sure we understand the goals and objectives of
the customers, stakeholders, and the project, and the risks which testing is
intended to address. This will give us what is sometimes called the mission of
testing or the test assignment.

Question 34

What is test policy?

Answer 34

Test policy gives rules for testing, e.g.

‘we always review the design documents’.

Question 35

What is test strategy?

Answer 35

est strategy is the overall high-level
approach, e.g. ‘system testing is carried out by an independent team reporting
to the program quality manager.

Question 36

What are the major tasks in test planning which hel us build a test plan?

Answer 36

Determine the test approach (techniques, test items, coverage, identifying and interfacing with the teams involved in testing, testware)

Implement the test policy and/or the test strategy

Determine the required test resources

Schedule test analysis and design tasks

Determine the exit criteria

Question 37

Tell more about “Determine the scope and risks and identify the objectives of testing” task in test planning activity

Answer 37

we con
sider what software, components, systems or other products are in scope for
testing; the business, product, project and technical risks which need to be
addressed; and whether we are testing primarily to uncover defects, to show
that the software meets requirements, to demonstrate that the system is fit
for purpose or to measure the qualities and attributes of the software.

Question 38

Tell more about “Determine the test approach” task in test planning activity

Answer 38

we consider
how we will carry out the testing, the techniques to use, what needs testing
and how extensively (i.e. what extent of coverage). We’ll look at who needs
to get involved and when (this could include developers, users, IT infrastruc
ture teams); we’ll decide what we are going to produce as part of the testing
(e.g. testware such as test procedures and test data). This will be related to
the requirements of the test strategy.

Question 39

Tell more about “Implement the test policy and/or the test strategy” task in test planning activity

Answer 39

we mentioned that there
may be an organization or program policy and strategy for testing. If this is
the case, during our planning we must ensure that what we plan to do
adheres to the policy and strategy or we must have agreed with stakeholders,
and documented, a good reason for diverging from it.

Question 40

Tell more about “Determine the required test resources” task in test planning activity

Answer 40

from the planning we have already done we can now go into detail; we decide
on our team make-up and we also set up all the supporting hardware and
software we require for the test environment.

Question 41

Tell more about “Schedule test analysis and design tasks, test implementation, execution and
evaluation” task in test planning activity

Answer 41

we will need a schedule of all the tasks and activities, so that we
can track them and make sure we can complete the testing on time.

Question 42

Tell more about “Determine the exit criteria” task in test planning activity

Answer 42

we need to set criteria such as coverage criteria (for
example, the percentage of statements in the software that must be executed
during testing) that will help us track whether we are completing the test activ
ities correctly. They will show us which tasks and checks we must complete for
a particular level of testing before we can say that testing is finished.

Question 43

What is test control?

Answer 43

test control is an ongoing
activity. We need to compare actual progress against the planned progress, and
report to the project manager and customer on the current status of testing,
including any changes or deviations from the plan. We’ll need to take actions
where necessary to meet the objectives of the project.

Question 44

What are the major tasks in test control activity?

Answer 44

Measure and analyze the results of reviews and testing
Monitor and document progress, test coverage and exit criteria
Provide information on testing
Initiate corrective actions
Make decisions

Question 45

Tell more about “Measure and analyze the results of reviews and testing” task in test control activity

Answer 45

We need to know
how many reviews and tests we have done. We need to track how many
tests have passed and how many failed, along with the number, type and
importance of the defects reported.

Question 46

Tell more about “Monitor and document progress, test coverage and exit criteria” task in test control activity

Answer 46

It is important
that we inform the project team how much testing has been done, what the
results are, and what conclusions and risk assessment we have made. We must
make the test outcome visible and useful to the whole team.

Question 47

Tell more about “Provide information on testing” task in test control activity

Answer 47

We should expect to make regular and
exceptional reports to the project manager, project sponsor, customer and
other key stakeholders to help them make informed decisions about
project status. We should also use the information we have to analyze the
testing itself.

Question 48

Tell more about “Initiate corrective actions” task in test control activity

Answer 48

For example, tighten exit criteria for defects fixed,
ask for more effort to be put into debugging or prioritize defects for fixing
test blockers.

Question 49

Tell more about “Make decisions” task in test control activity

Answer 49

Based on the measures and information gathered during
testing and any changes to business and project risks or our increased under
standing of technical and product risks, we’ll make decisions or enable others
to make decisions: to continue testing, to stop testing, to release the software
or to retain it for further work for example.

Question 50

What is test analysis and design?

Answer 50

Test analysis and design is the activity where general testing objectives are trans-
formed into tangible test conditions and test designs. During test analysis and
design, we take general testing objectives identified during planning and build
test designs and test procedures (scripts).

Question 51

What are the major tasks in test analysis and design activity?

Answer 51

Review the test basis (such as the product risk analysis, requirements, architecture, design specifications, and interfaces), examining the specifications
for the software we are testing

Identify test conditions based on analysis of test items, their specifications,
and what we know about their behavior and structure

Design the tests (you’ll see how to do this in Chapter 4), using techniques to
help select representative tests that relate to particular aspects of the soft
ware which carry risks or which are of particular interest, based on the test
conditions and going into more detail

Evaluate testability of the requirements and system

Design the test environment set-up and identify any required infrastructure
and tools

Question 52

What is test implementation and execution activity?

Answer 52

During test implementation and execution, we take the test conditions and
make them into test cases and testware and set up the test environment. This
means that, having put together a high-level design for our tests, we now start
to build them. We transform our test conditions into test cases and procedures,
other testware such as scripts for automation. We also need to set up an envi-
ronment where we will run the tests and build our test data.

Question 53

What are the major tasks in test iplementation and execution activity?

Answer 53

Implementation:
Develop and prioritize our test cases, using the techniques you’ll see in Chapter 4, and create test data for those tests
Create test suites from the test cases for efficient test execution
Implement and verify the environment

Execution:
Execute the test suites and individual test cases, following our test procedures
Log the outcome of test execution and record the identities and versions of the software under test, test tools and testware
Log the outcome of test execution and record the identities and versions of the software under test, test tools and testware.
Where there are differences between actual and expected results, report discrepancies as incidents.
Repeat test activities as a result of action taken for each discrepancy.

Question 54

What is evaluating exit criteria and reporting activity?

Answer 54

Evaluating exit criteria is the activity where test execution is assessed against
the defined objectives. This should be done for each test level, as for each we
need to know whether we have done enough testing. Based on our risk assess-
ment, we’ll have set criteria against which we’ll measure ‘enough’. These crite-
ria vary for each project and are known as exit criteria.

Question 55

What are the major tasks in Evaluating exit criteria activity?

Answer 55

Check test logs against the exit criteria specified in test planning
Assess if more tests are needed or if the exit criteria specified should be
changed
Write a test summary report for stakeholders

Question 56

What is test closure activities?

Answer 56

During test closure activities, we collect data from completed test activities to
consolidate experience, including checking and filing testware, and analyzing
facts and numbers.

Question 57

What are the major tasks in test closure activities?

Answer 57

Check which planned deliverables we actually delivered and ensure all
incident reports have been resolved through defect repair or deferral.
Finalize and archive testware, such as scripts, the test environment, and any
other test infrastructure, for later reuse.
Hand over testware to the maintenance organization who will support the
software and make any bug fixes or maintenance changes, for use in con
firmation testing and regression testing.
Evaluate how the testing went and analyze lessons learned for future
releases and projects.

Question 58

What psychoogical factors influence success of testing?

Answer 58

• clear objectives;
• a balance of self-testing and independent testing;
• recognition of courteous communication and feedback on defects.

Question 59

What are the advantages of independent testing?

Answer 59

Early in the life cycle, reviews of requirements and design
documents by someone other than the author helps find defects before coding
starts and helps us build the right software. Following coding, the software can
be tested independently. This degree of independence avoids author bias and is
often more effective at finding defects and failures.

Question 60

What are the levels of indepence?

Answer 60

• tests by the person who wrote the item under test
• tests by another person within the same team, such as another programmer;
• tests by a person from a different organizational group, such as an independ
ent test team
• tests designed by a person from a different-organization or company, such as
outsourced testing or certification by an external body.

Question 61

What are the testing principles from psychological perspective?

Answer 61

• Communicate findings on the product in a neutral, fact-focused way without
criticizing the person who created it. For example, write objective and factual
incident reports and review findings.
- Don’t gloat - you are not perfect either!
- Don’t blame - any mistakes are probably by the group rather than an
individual.
- Be constructively critical and discuss the defect and how you are going to
log it.
• Explain that by knowing about this now we can work round it or fix it so the
delivered system is better for the customer.
- Say what you liked and what worked, as well as what didn’t work.
- Show what the risk is honestly - not everything is high priority.
- Don’t just see the pessimistic side - give praise as well as criticism.
- Show what risks have been uncovered and the benefits of the review or
test.
• Start with collaboration rather than battles. Remind everyone of the
common goal of better quality systems.
- Be polite and helpful, collaborate with your colleagues.
- Try to understand how the other person feels and why they react as
they do.
- Confirm that the other person has understood what you have said and
vice versa.
- Explain how the test or review helps the author - what’s in it for him
or her.
- Offer your work to be reviewed, too.

Question 62

What is requirement?

Answer 62

A condition or capability needed by a user to solve a problem or achieve an objective
that must be met or possessed by a system or system component to satisfy a contract, standard,
specification, or other formally imposed document. [After IEEE 610

Question 63

What is review?

Answer 63

An evaluation of a product or project status to ascertain discrepancies from planned
results and to recommend improvements. Examples include management review, informal review,
technical review, inspection, and walkthrough. [After IEEE 1028]

Question 64

What is test case?

Answer 64

A set of input values, execution preconditions, expected results and execution postconditions, developed for a particular objective or test condition, such as to exercise a
particular program path or to verify compliance with a specific requirement.

Question 65

What is confirmation testing?

Answer 65

Confirmation testing or re-testing: When a test fails because of the defect then that defect is reported and a new version of the software is expected that has had the defect fixed. In this case we need to execute the test again to confirm that whether the defect got actually fixed or not. This is known as confirmation testing and also known as re-testing. It is important to ensure that the test is executed in exactly the same way it was the first time using the same inputs, data and environments.

Hence, when the change is made to the defect in order to fix it then confirmation testing or re-testing is helpful.

Question 66

What is incident?

Answer 66

While executing a test, you might observe that the actual results vary from expected results. When the actual result is different from the expected result then it is called as incidents, bugs, defects, problems or issues.
To be specific, we sometimes make difference between incidents and the defects or bugs. An incident is basically any situation where the system exhibits questionable behavior, but often we refer to an incident as a defect only when the root cause is some problem in the item we are testing.
Other causes of incidents include misconfiguration or failure of the test environment, corrupted test data, bad tests, invalid expected results and tester mistakes.

Question 67

What is test coverage?

Answer 67

The degree, expressed as a percentage, to which a specified coverage item has been
exercised by a test suite.

Question 68

What is software?

Answer 68

Computer programs, procedures, and possibly associated documentation and data
pertaining to the operation of a computer system.

Question 69

What is test data?

Answer 69

Data that exists (for example, in a database) before a test is executed, and that affects or is
affected by the component or system under test.

Question 70

What is test execution?

Answer 70

The process of running a test on the component or system under test, producing actual result(s).

Question 71

What is test log?

Answer 71

A chronological record of relevant details about the execution of tests. [IEEE 829]

Question 72

What is test plan?

Answer 72

A document describing the scope, approach, resources and schedule of intended test activities. It identifies amongst others test items, the features to be tested, the testing tasks, who will do each task, degree of tester independence, the test environment, the test design
techniques and entry and exit criteria to be used, and the rationale for their choice, and any risks requiring contingency planning. It is a record of the test planning process. [After IEEE 829]

Question 73

What is test summary report?

Answer 73

A document summarizing testing activities and results. It also contains an evaluation of the corresponding test items against exit criteria. [After IEEE 829]

Question 74

What is testware?

Answer 74

Artifacts produced during the test process required to plan, design, and execute tests, such as documentation, scripts, inputs, expected results, set-up and clear-up procedures, files, databases, environment, and any additional software or utilities used in testing. [After Fewster
and Graham]

Question 75

What is independence of testing?

Answer 75

Separation of responsibilities, which encourages the accomplishment of objective testing. [After DO-178b]

Question 76

What is quality?

Answer 76

The degree to which a component, system or process meets specified requirements and/or user/customer needs and expectations. [After IEEE 610]

Question 77

What is exhaustive testing?

Answer 77

A test approach in which the test suite comprises all combinations of input values
and preconditions.