Metrics Collection Flashcards

1
Q

There is little to no positive return on investment in the short run;

A

value comes over the long run.

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
2
Q

if we want the automation project to survive the bean counters

A

We should collect meaningful measurements where we can, that show the value of the automation

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
3
Q

For a large, complex automation project

A

refer to TAE syllabus.

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
4
Q

while test automation execution time tends to be smaller than the equivalent manual test time

A

the analysis, design, development, troubleshooting and maintenance of automated tests tends to take much longer than the same work for manual tests.

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
5
Q

The manual test, being abstract, likely does not need to be changed when the GUI of the SUT changes. In other side, the automated script likely…

A

would need to be modified for even minor changes to the GUI.

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
6
Q

An automation failure,

A

might take more time to troubleshoot

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
7
Q

up front investment requirements for automation:

A
  • purchase of one or more tools
  • suitable environments to run the tools
  • creation of the architecture and framework
  • training (or hiring) automators
How well did you know this?
1
Not at all
2
3
4
5
Perfectly
8
Q

achieving positive ROI likely means that

A

we need to automate and run large numbers of tests.

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
9
Q

A common problem with many automation metrics:

A

is that they often must be estimated rather than measured directly

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
10
Q

The requirement to automate and run large numbers of tests:

A

a higher level of scalability

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
11
Q

metrics that might be useful for a small, startup project:

A
  • Fixed costs to get the automation up and running
  • Regression test effort that has been saved by the automation
  • Effort expended by automation team supporting the automation
  • statement/decision coverage
  • level, interface or data flow coverage
  • level, requirement, feature, or identified risk coverage
  • Configurations tested coverage
  • User stories covered (in Agile)
  • Use cases covered
  • Number of successful runs between failures
  • Patterns of automation failures
  • Number of automation failures found as compared to SUT failures found by automation
How well did you know this?
1
Not at all
2
3
4
5
Perfectly
12
Q

Steps to explain the project validity and importance and prove that Automation provides value to the organization:

A
  1. Try to find meaningful metrics to collect about your project
  2. Make sure that management, understand the provided value
  3. Avoid argue using rosy scenarios and a creative imagination
How well did you know this?
1
Not at all
2
3
4
5
Perfectly