Chapter 5: test monitoring and control

Question 1

What is the purpose of test monitoring

Answer 1

to gather information and provide feedback and visibility about test activities

Question 2

How is information collected for test monitoring?

Answer 2

Information can be collected manually or automatically

Question 3

What should test monitoring assess?

Answer 3

• test progress
• measure whether the test exit criteria, or the testing tasks associated with an Agile project’s definition of done, are satisfied, such as meeting the targets for coverage of product risks, requirements, or acceptance criteria.

Question 4

What is testing control?

Answer 4

Test control describes any guiding or corrective actions taken as a result of information and metrics gathered and (possibly) reported. Actions may cover any test activity and may affect any other software lifecycle activity.

Question 5

Give examples of test control

Answer 5

• re-priortizing test due to identified risks
• changing the test schedule due to availability or unavailability of a test environment or other resources
• Re-evaluating whether a test item meets an entry or exit criterion due to rework

Question 6

What can test metrics assess?

Answer 6

• (plan) progress against the the planned schedule or budget
• (quality of testing) current quality of the test project
• (adequacy of testing) adequacy of the test approach
• (effectiveness of testing) effectiveness of the test activities with respect to the objectives

Question 7

List common test metrics

Answer 7

• Percentage of planned work done in test case preparation (or percentage of planned test cases implemented)
  
  • Percentage of planned work done in test environment preparation
  • Test case execution (e.g., number of test cases run/not run, test cases passed/failed, and/or test conditions passed/failed)
  • Defect information (e.g., defect density, defects found and fixed, failure rate, and confirmation test results)
  • Test coverage of requirements, user stories, acceptance criteria, risks, or code
  • Task completion, resource allocation and usage, and effort
  • Cost of testing, including the cost compared to the benefit of finding the next defect or the cost compared to the benefit of running the next test

Question 8

What is the purpose of a test report?

Answer 8

The purpose of test reporting is to summarize and communicate test activity information, both during and at the end of a test activity (e.g., a test level).

Question 9

What is the test report called when it is prepared during a test activity v when it is created at the end of a test activity?

Answer 9

The test report prepared during a test activity may be referred to as a test progress report, while a test report prepared at the end of a test activity may be referred to as a test summary report

Question 10

What might be included in the test progress report?

Answer 10

• (current status against test plan) The status of the test activities and progress against the test plan
  
  • (blockers) Factors impeding progress
  • (future) Testing planned for the next reporting period
  • (quality) The quality of the test object

Question 11

What is typically included in a test summary report (sometimes known as the test completion report)?

Answer 11

• (what happened) Information on what occurred during a test period
  
  • (what changed/ deviations from the plan) Deviations from plan, including deviations in schedule, duration, or effort of test activities
  • (status of testing and product qual) Status of testing and product quality with respect to the exit criteria or definition of done
  • (blockers) Factors that have blocked or continue to block progress
  • (metrics) Metrics of defects, test cases, test coverage, activity progress, and resource consumption. (e.g., as described in 5.3.1)
  • Residual risks (see section 5.5)
    Reusable test work products produced

Question 12

Test reports can change depending on the project, what kind of report might a complex project have?

Answer 12

For example, a complex project with many stakeholders or a regulated project may require more detailed and rigorous reporting than a quick software update

Question 13

Test reports can change depending on the project, what kind of report might an agile dev project have?

Answer 13

Agile development, test progress reporting may be incorporated into task boards, defect summaries, and burndown charts, which may be discussed during a daily stand-up meeting (see ISTQBCTFL-AT).

Question 14

Test reports can change depending on the project, how might the audience effect the report?

Answer 14

The type and amount of information that should be included for a technical audience or a test team may be different from what would be included in an executive summary report. In the first case, detailed information on defect types and trends may be important. In the latter case, a high-level report (e.g., a status summary of defects by priority, budget, schedule, and test conditions passed/failed/not tested) may be more appropriate. ISO standard (ISO/IEC/IEEE 29119-3) refers to two types of test reports, test progress reports and test completion reports (called test summary reports in this syllabus), and contains structures and examples for each type

Question 15

What can effect the contents of a test report?

Answer 15

The contents of a test report will vary depending on the project, the organizational requirements, the software development lifecycle and the audience