Metrics Collection

Question 1

There is little to no positive return on investment in the short run;

Answer 1

value comes over the long run.

Question 2

if we want the automation project to survive the bean counters

Answer 2

We should collect meaningful measurements where we can, that show the value of the automation

Question 3

For a large, complex automation project

Answer 3

refer to TAE syllabus.

Question 4

while test automation execution time tends to be smaller than the equivalent manual test time

Answer 4

the analysis, design, development, troubleshooting and maintenance of automated tests tends to take much longer than the same work for manual tests.

Question 5

The manual test, being abstract, likely does not need to be changed when the GUI of the SUT changes. In other side, the automated script likely…

Answer 5

would need to be modified for even minor changes to the GUI.

Question 6

An automation failure,

Answer 6

might take more time to troubleshoot

Question 7

up front investment requirements for automation:

Answer 7

• purchase of one or more tools
• suitable environments to run the tools
• creation of the architecture and framework
• training (or hiring) automators

Question 8

achieving positive ROI likely means that

Answer 8

we need to automate and run large numbers of tests.

Question 9

A common problem with many automation metrics:

Answer 9

is that they often must be estimated rather than measured directly

Question 10

The requirement to automate and run large numbers of tests:

Answer 10

a higher level of scalability

Question 11

metrics that might be useful for a small, startup project:

Answer 11

• Fixed costs to get the automation up and running
• Regression test effort that has been saved by the automation
• Effort expended by automation team supporting the automation
• statement/decision coverage
• level, interface or data flow coverage
• level, requirement, feature, or identified risk coverage
• Configurations tested coverage
• User stories covered (in Agile)
• Use cases covered
• Number of successful runs between failures
• Patterns of automation failures
• Number of automation failures found as compared to SUT failures found by automation

Question 12

Steps to explain the project validity and importance and prove that Automation provides value to the organization:

Answer 12

1. Try to find meaningful metrics to collect about your project
2. Make sure that management, understand the provided value
3. Avoid argue using rosy scenarios and a creative imagination