Test Planning, Strategies and the Delights of Monitoring Flashcards

1
Q

Define Planning

A
  • Scheduling of Activities (steps and in what order)
  • Allocating resources
  • Devising unambiguous milestones.
How well did you know this?
1
Not at all
2
3
4
5
Perfectly
2
Q

Define Monitoring

A

Judging progress against a defined plan (how are we doing?)

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
3
Q

What must a good plan have?

A

visibility! The ability to monitor each step and make judgements

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
4
Q

What is a Quality Process

A

Set of activities and responsibilities.

Focused on ensuring adequate dependability.
Concerned with project schedule or product usability.

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
5
Q

What is the key principle of Quality planning

A

Cost of detecting and repairing fault increases as time passes between committing the errors and detecting resultant faults.

Quality plan includes matched set of validation and verification to detect faults quickly after the error is committed.

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
6
Q

Quality planning leads to a quality end product

A

If our intermediate artifacts (early design, prototypes, test plans) are of poor quality, our end product will likely be of poorer quality.

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
7
Q

How can we verify intermediate steps?

A
  • Internal consistency checks (compliance with rules for that artifact type).
  • External consistency checks (consistency with related artifacts, conformance to prior specification).
  • Generation of correctness conjectures ( Lay groundwork of external consistency checks of other work products. Motivate refinement of current product)
How well did you know this?
1
Not at all
2
3
4
5
Perfectly
8
Q

Difference between a strategy and a plan

A

Strategy describes organization and structure over several projects. A plan is a standard structure prescribed in the strategy.

Strategy evolves slowly with organization and policy changes. Plan evolves quickly to project needs.

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
9
Q

What do you consider when making a test strategy?

A
  • You take past experiences in consideration.
  • Use body of explicit knowledge instead of individual experiences. This is amenable to improvement, less vulnerable to organization change (losing people)
  • Helps you avoid common recurring errors, increases development efficiency, maintain consistent process.
How well did you know this?
1
Not at all
2
3
4
5
Perfectly
10
Q

How to determine which strategy is good for a Organization?

A
  • Consider structure and size
  • Consider the process of the strategy
  • Consider the domain where the strategy is good.
How well did you know this?
1
Not at all
2
3
4
5
Perfectly
11
Q

What are the elements of a test strategy

A
  • Common quality requirements (unambiguous definitions and measures.)
  • Activities prescribed by overall process.
  • Guidelines for project staffing and assignment of roles.
How well did you know this?
1
Not at all
2
3
4
5
Perfectly
12
Q

Define a test strategy

A

Addresses risk and presents a process to reduce those.

Composed of test factors (risks/issues that need to be addressed) and Test phase (the phase of development life cycle where testing occurs.)

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
13
Q

What are some test factors?

A

Correctness, file integrity, authorization, audit trail, access control, compliance, reliability, coupling, performance… etc

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
14
Q

How to build a test strategy?

A
  1. select and rank test factors: specific to the system being developed.
  2. Identify system development phases: Obtained from the development methodology (waterfall, agile)
  3. Identify business risks with system: brainstorm including developers, users, customers and test personel.
  4. Place risks in test factor/test phase: risk team determine test phase in which each risk needs to be addressed by test team and which test factor risk is associated with.
How well did you know this?
1
Not at all
2
3
4
5
Perfectly
15
Q

What are the main elements of a test plan

A
  • Items to be verified.
  • Activities and resources
  • approaches to be followed
  • Criteria for evaluating results.
How well did you know this?
1
Not at all
2
3
4
5
Perfectly
16
Q

Define Quality Goals

A

Expressed as properties satisfied by the product
- Include metrics monitored during the project
Ex: “before entering acceptance testing, product must pass comprehensive systems testing”

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
17
Q

What is a task schedule

A
  • Based on quality strategy and past experience
  • Breaks large tasks into subtasks
  • includes dependencies
  • Guidelines and objectives (process visibility, schedule for steady effort).
18
Q

Define a critical path

A

chain of activities to be completed sequentially and have the maximum overall duration

19
Q

Define critical dependence

A

task on a critical path scheduled immediately after some other task on critical path.

20
Q

How do you eliminate risk in plan building?

A

You cannot, but you can make steps to assess, control and monitor them.

21
Q

What are the different types of risks?

A

Generic (personnel, technology, schedule)

Quality (development, execution, requirements)

22
Q

What are personnel risks

A
  • Loss of staff member,
  • Staff member under qualified.

Control:

  • cross training individuals (reduce dependence)
  • Continuous education
  • identify skill gaps
  • include training in schedule
23
Q

What are technology risks

A
  • High fault rate because unfamiliar with off the shelf solution
  • test and analysis tool doesn’t meet expectation.

Control:

  • schedule extra time for testing unfamiliar interfaces
  • invest training in off the shelf component
  • Monitor, document and publicize common errors
24
Q

What are Schedule risks

A
  • Inadequate unit testing leads to unexpected delays and expenses.
  • difficulty in scheduling meetings makes inspection a bottleneck

Control:

  • track and reward quality unit testing.
  • set aside time in weekly schedule for inspections, inspections take precedence
  • distributed and asynchronous inspection techniques.
25
Q

What are test execution risks

A
  • Execution costs higher than planned.
  • Scarce resources available

Control:

  • minimize parts that require full system execution
  • inspect architecture to assess and improve testing
  • increase intermediate feedback.
26
Q

What is a Contingency test plan

A

Part of initial test plan (what could go wrong? how will we know?, how to recover?)

  • evolves with test plan
  • Derived from risk analysis.
  • Defines actions
27
Q

What are test plan types?

A

System Test plan: How testing will occur,
information on software, test objectives and risks, business functions
creates a roadmap for specific tests.

Unit test plan: each component have its own plan
Determine when unit testing complete.

28
Q

What are components of the System Test Plan.

A
  1. General Information
  2. Plan
  3. Specifications and Evaluation
  4. Test descriptions
29
Q

What are the components of the Unit Test Plan

A
  1. Plan
  2. Business and structural function testing
  3. Interface Test Description
  4. Test Progression
30
Q

How do you improve current and next processes.

A

Identifying weak aspects of a process can be difficult.

But analysis of fault history can help engineers build a feedback mechanism to track relevant faults and their root causes.

  • Sometimes information can be fed to current product development.
  • More often, helps engineers improve future development of products.
31
Q

What is Root Cause Analysis (RCA)

A

Technique for identifying and eliminating process faults.

- First developed in the nuclear power industry, used in many fields.

32
Q

What are the four main steps of RCA

A

What are the faults

When did the faults occur? when were they found.

Why did the faults occur.

How could the faults have been prevented.

33
Q

Identifying what are the faults (RCA)

A

Identify a class of important faults.

Faults are categorized by
- Severity = impact of fault
- Kind:
no fixed set of categories, they evolve and adapt.
Goal: - Identify few important classes of faults and remove their causes.
- Not trying to compare trends for different classes of faults, but focusing on a few important classes.

34
Q

What are the different types of fault severities?

A

Critical, severe, moderate, cosmetic.

35
Q

Explain the Pareto Distribution

A

Pareto Rule (80/20). A few (20) are vital, many are trivial (80).

20% of the code is responsible for 80% of faults.

  • Faults accumulate in few components,
  • Some classes of faults dominate.
36
Q

Identifying why faults occured (RCA)?

A

Trace representative faults back to causes.
Identify root cause.

Iterative analysis:
- explain error led to fault
- explain cause of error
- explain cause of cause
...
37
Q

How can faults be prevented?

A

Kinda a paradox. You want cost effective improvement, not perfection.

There are many approaches:
-conditions in design inspection.
to
- making explicit consideration to conditions in all requirements analysis and design steps.

38
Q

Explain team organization in test plans?

A

You have a quality team. with defined roles and responsibilities.

Occurs at:
strategic level:
- test strategy, structure, external requirements (certification agency etc..)
tactical level:
- test plan.
39
Q

What are some pros and cons to assigning roles and responsibilities?

A

Same individuals play dev and tester
- potential interests conflict.

Roles assigned to different individuals.
- Conflict between individuals. Two individuals may not share same motivation to deliver quality product on time.

40
Q

Explain how an independent testing team can help, but also worsen a situation.

A

Minimize conflict between roles played by same individual. Not same deadline pressures.

However, Increases risk of conflict between different individuals.
- Can minimize this by creating checks before completion of an activity.

41
Q

Identify communication in testing.

A

Testing and dev teams must share goal of shipping quality product on shcedule.
- Testers should not be seen as taking responsibility for quality. Should also understand the schedule too.

independent quality teams require mature dev process.

  • Test designers must:
    1. work on precise specifications
    2. execute tests in a control environment.

Version and configuration must be defined.

Failures and faults must be tracked and monitored.

42
Q

Motivations to outsource test and analysis

A

Wrong one:
because testing is less demanding so lower skill levels can do it.

why?
confuses test execution with analysis and design (which is demanding)

Better motivations:
To maximize independence.

Planning includes:

  • Milestones and delivery for these activities.
  • Checks on quality of delivery in both directions.