Chapter 5 - Managing the Test Activities

Question 1

What is a test plan ?

Answer 1

A test plan describes the test objectives, resources and processes for a test project.

Question 2

What does the test plan :

Answer 2

• Documents the means and schedule for achieving test objectives
• Helps to ensure that the performed test activities will meet the established criteria
• Serves as a means of communication with team members and other stakeholders
• Demonstrates that testing will adhere to the existing test policy and test strategy (or explains why
  the testing will deviate from them)

Question 3

The typical content of a test plan includes :

Answer 3

• Context of testing (e.g., test scope, test objectives, test basis)
• Assumptions and constraints of the test project
• Stakeholders (e.g., roles, responsibilities, relevance to testing, hiring and training needs)
• Communication (e.g., forms and frequency of communication, documentation templates)
• Risk register (e.g., product risks, project risks)
• Test approach (e.g., test levels, test types, test techniques, test deliverables, entry criteria and
  exit criteria, independence of testing, metrics to be collected, test data requirements, test
  environment requirements, deviations from the test policy and test strategy)
• Budget and schedule

Question 4

What is release planning ? (K1)

Answer 4

Release planning looks ahead to the release of a product, defines and re-defines the product backlog,
and may involve refining larger user stories into a set of smaller user stories. It also serves as the basis for the test approach and test plan across all iterations.

Question 5

What is iteration planning ? (K1)

Answer 5

Iteration planning looks ahead to the end of a single iteration and is concerned with the iteration backlog.
Testers involved in iteration planning participate in the detailed risk analysis of user stories, determine the testability of user stories, break down user stories into tasks (particularly testing tasks), estimate test effort
for all testing tasks, and identify and refine functional and non-functional aspects of the test object.

Question 6

What are the entry criteria ?

Answer 6

Entry criteria define the preconditions for undertaking a given activity. If entry criteria are not met, it is likely that the activity will prove to be more difficult, time-consuming, costly, and riskier.
Entry criteria and exit criteria should be defined for each test level, and will differ based on the test objectives.
Entry criteria that a user story must fulfill to start the development and/or testing activities are called Definition of Ready.

Question 7

What are the exit criteria ?

Answer 7

Exit criteria define what must be achieved to declare an activity completed.
Entry criteria and exit criteria should be defined for each test level, and will differ based on the test objectives.
In Agile software development, exit criteria are often called Definition of Done.

Question 8

What does typical entry criteria include ?

Answer 8

• availability of resources (e.g., people, tools, environments, test data, budget, time)
• availability of testware (e.g., test basis, testable requirements, user stories, test cases)
• initial quality level of a test object (e.g., all smoke tests have passed)

Question 9

What does typical exit criteria include ?

Answer 9

• measures of thoroughness (e.g., achieved level of coverage, number of unresolved defects, defect density, number of failed test cases)
• binary “yes/no” criteria (e.g., planned tests have been executed, static testing has been performed, all defects found are reported, all
  regression tests are automated).
  Running out of time or budget can also be viewed as valid exit criteria.

Question 10

What are the four estimation techniques ? (descirbed in syllabus) (K3)

Answer 10

• Estimation based on ratios
• Extrapolation
• Wideband Delphi
• Three-point estimation

Question 11

Most commonly used test case prioritization strategies ? (K3)

Answer 11

• Risk-based prioritization, where the order of test execution is based on the results of risk analysis. Most risky test case -> less risky.
• Coverage-based prioritization, where the order of test execution is based on coverage. Highest coverage executed first.
• Requirements-based prioritization, where the order of test execution is based on the priorities of
  the requirements traced back to the corresponding test cases. Priorities defined by stakeholders.

Question 12

What is a Test Pyramid ? (K1)

Answer 12

The test pyramid is a model showing that different tests may have different granularity. The test pyramid model supports the team in test automation and in test effort allocation by showing that different test objectives are supported by different levels of test automation. The higher the layer, the lower the test granularity, the lower the test isolation and the higher the test execution time.
Fom bottom to top: Unit tests / service tests / UI tests.

Question 13

Testing Quadrants ?

Answer 13

• Q1 (technology facing, supports the team) = component tests/component integration tests (automated)
• Q2 (business facing, support the team) functional tests/examples/user story tests/UX prototypes/API simulations/prototypes (manual or automated)
  -Q3 (business facing, critique the product) exploratory testing/usability testing/ user acceptance testing (user-oriented tests and often manual)
  -Q4 (technology-facing, critique the product) smoke tests/non-functional tests. (often automated)

Question 14

Main risk management activities :

Answer 14

• Risk analysis (consisting of risk identification and risk assessment)
• Risk control (consisting of risk mitigation and risk monitoring)

Question 15

A risk can be characterized by two factors :

Answer 15

• Risk likelihood – the probability of the risk occurrence (greater than zero and less than one)
• Risk impact (harm) – the consequences of this occurrence

Question 16

What are project risks :

Answer 16

Related to management and control of the project.
- Organizational issues (e.g., delays in work products deliveries, inaccurate estimates, cost cutting)
- People issues (e.g., insufficient skills, conflicts, communication problems, shortage of staff)
- Technical issues (e.g., scope creep, poor tool support)
- Supplier issues (e.g., third-party delivery failure, bankruptcy of the supporting company)

When they occur, may have an impact on the project schedule, budget or scope, which affects the project’s ability to achieve its objectives.

Question 17

What are product risks :

Answer 17

Related to the product quality characteristics.
When they occur, may result in various negative consequences, including:
- User dissatisfaction
- Loss of revenue, trust, reputation
- Damage to third parties
- High maintenance costs, overload of the help desk
- Criminal penalties
- In extreme cases, physical damage, injuries or even death

Question 18

What does Product risk analysis consist of ?

Answer 18

It consists of risk identification and risk assessment. Risk identification is about
generating a comprehensive list of risks.
Stakeholders can identify risks by using various techniques and
tools, e.g., brainstorming, workshops, interviews, or cause-effect diagrams.

Question 19

Product risk analysis may influence the thoroughness and test scope. Its results are used to:

Answer 19

• Determine the test scope to be carried out
• Determine the particular test levels and propose test types to be performed
• Determine the test techniques to be employed and the coverage to be achieved
• Estimate the test effort required for each task
• Prioritize testing in an attempt to find the critical defects as early as possible
• Determine whether any activities in addition to testing could be employed to reduce risk

Question 20

Product Risk Control - Actions that can be taken to mitigate the product risks by testing are as follows:

Answer 20

• Select the testers with the right level of experience and skills, suitable for a given risk type
• Apply an appropriate level of independence of testing
• Perform reviews and static analysis
• Apply the appropriate test techniques and coverage levels
• Apply the appropriate test types addressing the affected quality characteristics
• Perform dynamic testing, including regression testing

Question 21

What is/does Test Monitoring ?

Answer 21

Test monitoring is concerned with gathering information about testing. This information is used to assess
test progress and to measure whether the exit criteria or the test tasks associated with the exit criteria are
satisfied, such as meeting the targets for coverage of product risks, requirements, or acceptance criteria.

Question 22

What is/does Test Control ?

Answer 22

Test control uses the information from test monitoring to provide, in a form of the control directives,
guidance and the necessary corrective actions to achieve the most effective and efficient testing.

Question 23

Control directives in Test Control ?

Answer 23

• Reprioritizing tests when an identified risk becomes an issue
• Re-evaluating whether a test item meets entry criteria or exit criteria due to rework
• Adjusting the test schedule to address a delay in the delivery of the test environment
• Adding new resources when and where needed

Question 24

What is/does Test Completion ?

Answer 24

Test completion collects data from completed test activities to consolidate experience, testware, and any
other relevant information. Test completion activities occur at project milestones such as when a test level
is completed, an agile iteration is finished, a test project is completed (or cancelled), a software system is
released, or a maintenance release is completed.

Question 25

What is the purpose in test metrics ? (K1)

Answer 25

Test metrics are gathered to show progress against the planned test schedule and budget, the current quality of the test object, and the effectiveness of the test activities with respect to the test objectives or an iteration goal. Test monitoring gathers a variety of metrics to support the test control and test completion.

Question 26

What are Metrics used in Testing ? (K1)

Answer 26

• Project progress metrics (e.g., task completion, resource usage, test effort)
• Test progress metrics (e.g., test case implementation progress, test environment preparation
  progress, number of test cases run/not run, passed/failed, test execution time)
• Product quality metrics (e.g., availability, response time, mean time to failure)
• Defect metrics (e.g., number and priorities of defects found/fixed, defect density, defect detection
  percentage)
• Risk metrics (e.g., residual risk level)
• Coverage metrics (e.g., requirements coverage, code coverage)
• Cost metrics (e.g., cost of testing, organizational cost of quality)

Question 27

What is the puprose of Test Reporting ?

Answer 27

Test reporting summarizes and communicates test information during and after testing.

Question 28

What is the puprose of Test Progress Reports ?

Answer 28

Test progress reports support the ongoing test control and must provide enough information to make modifications to the test schedule, resources, or test plan, when such changes are needed due to deviation from the plan
or changed circumstances.

Question 29

What is the puprose of Test Completion Reports ?

Answer 29

Test completion reports summarize a specific test activity (e.g., test level, test
cycle, iteration) and can give information for subsequent testing.

Question 30

Test Progress Reports are usually generated on a reguilar basis and include :

Answer 30

• Testing period
• Test progress (e.g., ahead or behind schedule), including any notable deviations
• Impediments for testing, and their workarounds
• Test metrics (see section 5.3.1 for examples)
• New and changed risks within testing period
• Testing planned for the next period

Question 31

Typical test completion reports include:

Answer 31

• Test summary
• Testing and product quality evaluation based on the original test plan (i.e., test objectives and exit
  criteria)
• Deviations from the test plan (e.g., differences from the planned test schedule, duration, and
  effort).
• Testing impediments and workarounds
• Test metrics based on test progress reports
• Unmitigated risks, defects not fixed
• Lessons learned that are relevant to the testing

Question 32

How to communicate the Status of Testing ?

Answer 32

• Verbal communication with team members and other stakeholders
• Dashboards (e.g., CI/CD dashboards, task boards, and burn-down charts)
• Electronic communication channels (e.g., email, chat)
• Online documentation
• Formal test reports

Question 33

What does Configuration Management (CM) provides in testing ?

Answer 33

CM provides a discipline for identifying, controlling, and tracking
work products such as test plans, test strategies, test conditions, test cases, test scripts, test results, test logs, and test reports as configuration items.

Question 34

To properly support testing, Configuration Management (CM) ensures the following :

Answer 34

• All configuration items, including test items (individual parts of the test object), are uniquely identified, version controlled, tracked for changes, and related to other configuration items so that traceability can be maintained throughout the test process
• All identified documentation and software items are referenced unambiguously in testware

Question 35

What are typical objectives of defect reports :

Answer 35

• Provide those responsible for handling and resolving reported defects with sufficient information
  to resolve the issue
• Provide a means of tracking the quality of the work product
• Provide ideas for improvement of the development and test process

Question 36

A defect report logged during dynamic testing typically includes:

Answer 36

• Title with a short summary of the anomaly being reported
• Date when the anomaly was observed, issuing organization, and author, including their role
• Identification of the test object and test environment
• Context of the defect (e.g., test case being run, test activity being performed, SDLC phase, and
  other relevant information such as the test technique, checklist or test data being used)
• Description of the failure to enable reproduction and resolution including the test steps that
  detected the anomaly, and any relevant test logs, database dumps, screenshots, or recordings
• Expected results and actual results
• Severity of the defect (degree of impact) on the interests of stakeholders or requirements
• Priority to fix
• Status of the defect (e.g., open, deferred, duplicate, waiting to be fixed, awaiting confirmation htesting, re-opened, closed, rejected)
• References (e.g., to the test case)