V - Test Management

Question 1

Risk

Answer 1

A factor that could result in future negative consequences.

Question 2

Risk likelihood

Answer 2

The probability that a risk will become an actual outcome or event.

Question 3

Risk impact

Answer 3

The damage that will be caused if the risk becomes an actual outcome or event.

Question 4

Product Risks

Answer 4

Product risks involve the possibility that a work product (e.g., a specification, component, system, or test) may fail to satisfy the legitimate needs of its users and/or stakeholders.

AKA Quality Risks, when associated with specific quality characteristics of a product (ex. functional suitability, reliability, performance, security, etc.)

Question 5

What are some examples of Product Risks?

Answer 5

Think of anything related to software for this!

• Software might not perform its intended functions according to the specification.
• A system architecture may not adequately support some non-functional requirement(s).
• A loop control structure may be coded incorrectly.
• Response-times may be inadequate for a high-performance transaction
  processing system.

Question 6

What are Project Risks?

Answer 6

Situations that may have a negative effect on a project’s ability to achieve its objectives.

Any risk that can delay your project from delivering on time/budget.

Test Managers are responsible for these.

Question 7

What are some examples of Project Risks?

Answer 7

Project Issues: Delays in task completion, inaccurate estimates, reallocation of funds, late changes, etc.

Organizational Issues: Lack of skills, training, or staff. Personnel issues leading to conflict, user/staff unavailable due to business priorities.

Political Issues: Testers not communication needs/results effectively, Developers/Testers not following up on results.

Supplier Issues: A third party failing to deliver, contractual issues.

Technical Issues: Requirements not clearly defined, test environment isn’t ready, data conversion/migration planning is late, poor defect management.

Question 8

What is risk-based testing?

Answer 8

Testing the software, based on the risk(s) that have been identified. Proactive measure to reduce product risks.

• The most critical, and therefore high-priority test cases are executed first.
• This can mean that lower-priority risks are not covered if time and budget
  are insufficient.

Question 9

What are the 3 phases to Risk Management?

Answer 9

1 - Risk Identification: What type of risk is this?

2 - Risk Assessment/Prioritization: What kind of impact will this have? Based on the impact, what is the priority?

3 - Risk Mitigation

Question 10

What is a Risk Matrix?

Answer 10

A visual representation used to help identify which risks are high priority and which are low priority.

AKA heat map

Question 11

What factors can influence the Test Planning?

Answer 11

• the test policy and the test strategy of the organization
• the development lifecycles and methods being used
• the scope of testing and objectives
• testability
• risks, criticality and constraints
• the availability of resources

Question 12

What are the contents of a Test Plan?

Answer 12

• Test objectives and risks
• Entry and exit criteria
• Test effort
• Test activities
• Timetable
• Metrics
• Resources and Budgets

Question 13

What is the difference between a Test Strategy and a Test Approach?

Answer 13

Test Strategy is the generic requirements for testing for one or more projects within an organization. It outlines how testing is to be performed and is aligned with the Test Policy.

Test Approach is the implementation of the Test Strategy for a specific project.

Question 14

What are the different types of Test Strategies?

Answer 14

Analytical: Based on known facts (requirements/risks).
Ex. risk-based testing, where tests are designed and prioritized based on the level of risk.

Model-based: Based on any diagram or model of some required aspect of the product (state models/reliability growth).
Ex. Model-based performance test of the car configurator app for mobile devices. Based on models of network traffic.

Methodical: Following pre-defined rules or tests, already set up by the organization (common failures/important quality characteristics).
Ex. Maintenance testing of a simple, stable e-commerce website using a checklist.

Process-Compliant/Standard-Compliant: Following industry-standards (government/banking).
Ex. Test of the usability of the Car Configurator. According to ISO 9241-11:2020 Ergonomics of human-system interaction – Part 11: Usability: Definitions and concepts.

Directed/Consultative: Advice, guidance, or instructions of stakeholders, business domain experts, or technology experts, who may be outside the test team or outside the organization itself.
Ex. Needing to perform security testing, but do not have any internal experts. You hire an external consultant/expert to help.

Regression-averse: Re-use existing resources, data, environments, use-cases, etc.
Ex. The work results, in particular the regression tests from the project “Boat Configurator“, will be reused in the project “Car Configurator“.

Reactive: You don’t have a Test Plan. You test based on the reaction from the software. Very common in Agile.
Ex. Because there is no exhaustive requirements specification,
the Car Configurator is tested exploratively in 2-hour test sessions.

Combination of strategies listed.
Ex. Risk-based testing (an analytical strategy) can be combined with exploratory testing (a reactive strategy).

Question 15

What is Entry Criteria/Definition of Ready?

Answer 15

The preconditions for undertaking a given test activity (BEFORE you start testing).

This includes the available of:
- Testable requirement, user stories, and/or models.
- Test items that have met the exit criteria for any prior test levels.
- Test environment
- Test tools
- Test data and/or other resources

Question 15

What is Exist Criteria/Definition of Done?

Answer 15

Exit criteria define what conditions must be achieved in order to declare a test level or a set of tests completed (what to check AFTER testing).

• Planned tests have been executed.
• A defined level of coverage (requirements, user stories, acceptance criteria, risks, code) has been achieved.
• The number of unresolved defects is within an agreed limit.
• The number of estimated remaining defects is sufficiently low.
• The assessed levels of relevant quality characteristics are sufficient
  (reliability, performance efficiency, usability, security).

Question 15

What are some test estimation techniques?

Answer 15

Metrics-based technique: Test manager bases estimation on metrics of former similar projects or based on typical values.

Expert-based technique: Based on the experience of the owners of the testing tasks or by experts.

Question 16

What factors can influence the test effort?

Answer 16

Product Characteristics: Product risks, quality of test basis, size of the product, complexity, legal/compliance requirements, etc.

Development Process Characteristics: Stability/maturity of organization, development model, test approach, test tools, time pressure, etc.

People Characteristics: Skills/experience involved, team cohesion/leadership.

Test Results: Number/severity of defects found, amount of rework required.

Question 17

What is the best way to prioritize a Test Plan?

Answer 17

Always want to prioritize functional testing first, to ensure the software actually worked. THEN, you can prioritize non-functional testing (load/performance).

Question 18

What is the purpose of Test Monitoring?

Answer 18

The Test Manager is gathering information to make sure things are running according to plan.

Question 19

What is the purpose of Test Control?

Answer 19

Any guiding or corrective actions taken as a result of information and metrics gathered and (possibly) reported.

Question 20

What are some examples of Test Control actions?

Answer 20

Re-prioritizing tests based on identified risks.

Changing the test schedule due to resource availability (person/environment, etc.)

Re-evaluating whether a test item meets an entry or exit criterion due to rework.

Question 21

Why are metrics used in testing?

Answer 21

progress against
the planned schedule and budget

adequacy of
the test approach

current quality of the test object

effectiveness of the test activities with respect to the objectives

Question 22

What are some common Test Metrics used?

Answer 22

Burn-down chart (common in Agile).

Test Case Execution (common in waterfall)

Defect Information

Slides 351 - 353

Question 23

What is the difference between a Test Progress Report and a Test Summary Report?

Answer 23

Test Manager prepares both.

Test Progress Report is meant for your internal team, showing day-to-day progress. Very detailed. Prepared during a test activity.

Test Summary Report meant for the management team, high-level information. Prepared at the end of a test activity.

Question 24

What are the properties of a good defect report?

Answer 24

• Only 1 defect is described.
• Test procedure is concise/comprehensible.
• The failure is clearly defined, reproducible, and localizable.
• Test environment used is clearly defined.
• Test data used is documented.

Question 25

What is included in version control?

Answer 25

• Statuses of all test items and items of the testware.
• Make sure that each version can be retrieved (under an object name).
• Make differences between versions of the test items and testware traceable.
• All identified documents and software items are unambiguously named in the test documentation.