Metrics Collection Flashcards
There is little to no positive return on investment in the short run;
value comes over the long run.
if we want the automation project to survive the bean counters
We should collect meaningful measurements where we can, that show the value of the automation
For a large, complex automation project
refer to TAE syllabus.
while test automation execution time tends to be smaller than the equivalent manual test time
the analysis, design, development, troubleshooting and maintenance of automated tests tends to take much longer than the same work for manual tests.
The manual test, being abstract, likely does not need to be changed when the GUI of the SUT changes. In other side, the automated script likely…
would need to be modified for even minor changes to the GUI.
An automation failure,
might take more time to troubleshoot
up front investment requirements for automation:
- purchase of one or more tools
- suitable environments to run the tools
- creation of the architecture and framework
- training (or hiring) automators
achieving positive ROI likely means that
we need to automate and run large numbers of tests.
A common problem with many automation metrics:
is that they often must be estimated rather than measured directly
The requirement to automate and run large numbers of tests:
a higher level of scalability
metrics that might be useful for a small, startup project:
- Fixed costs to get the automation up and running
- Regression test effort that has been saved by the automation
- Effort expended by automation team supporting the automation
- statement/decision coverage
- level, interface or data flow coverage
- level, requirement, feature, or identified risk coverage
- Configurations tested coverage
- User stories covered (in Agile)
- Use cases covered
- Number of successful runs between failures
- Patterns of automation failures
- Number of automation failures found as compared to SUT failures found by automation
Steps to explain the project validity and importance and prove that Automation provides value to the organization:
- Try to find meaningful metrics to collect about your project
- Make sure that management, understand the provided value
- Avoid argue using rosy scenarios and a creative imagination