Concepts Overview Flashcards
Test Automation Definition
automatic execution of functional tests, designed in a way that simulates a human being executing the manual tests.
largely automated
test execution
usually still manually performed
test analysis, test design, and test implementation
often done manually
- Creation and deployment of the data
- Evaluation of the pass/fail status for a test
Automation requires the design, creation, and maintenance of different levels of testware. What is Testware?
- tools used
- code libraries
- test scripts
- logging and reporting structures
Test Automation objectives
- Reducing the amount of time needed for running tests
- Improve the efficiency of testing by reducing the cost of each test run
- Test more and other things than we might be able to manually test
- Being able to push more testing earlier in the SDLC
- Increase the frequency with which tests can be run
Automation Advantages
- Running automated tests may be more efficient than running them manually
- Performing tests that are not at all (or easily) manually (such as reliability or efficiency tests).
- Reducing the time needed for test execution
- Increasing the frequency that some tests can be run
- Freeing up manual testers to run more interesting and complex manual tests
- Reducing mistakes made by bored or distracted manual testers,
- Executed tests earlier in the process
- Executing tests outside of normal business hours
- Increasing confidence in the build
Automation Disadvantages
- Increases in costs
- Delays, costs, and mistakes associated with testers as they learn new technologies
- complexity may become overwhelming
- Unacceptable growth in the size of the automated tests
- Software development skills needed on test team
- Considerable maintenance of tools, environments, and test assets required
- Technical debt is easy to add (extra programming is added to improve context and reasonableness of the automated tests)
- The pesticide paradox is increased
- False positives occur when automation failures are due to defects in the automation itself
- Tools tend to be single threaded when an event happen. Humans can figure out what happened on the fly
Automation limitations
- Unrealistic expectations by management
- Short term thinking
- Organizational maturity is required
- Automated test oracles may be different than manual test oracles
- Not all tests can or should be automated
- Manual testing will still be required
- Human beings find most bugs; automation can only find what it is programmed to find and is limited by the pesticide paradox
- False sense of security due to large numbers of automated tests running without finding many bugs
- Technical problems for the project when using bleeding edge tools or techniques
- Requires cooperation with development, which can create organizational issues
Three Columns of a Manual Test Script Information
a) Abstract (purpose)
b) Data (to perform the test)
c expected actions or results
Value of manual test
Context and Reasonableness
To figure out about the cost of the Automation project
We must ask ourself: Will the cost become a meaninful benefit after a specific time?
Success Factors
- Long term plan aligned with the business goals
- stark attention to detail
- process maturity
- a formalized architecture and framework
- proper training to the automation team
- documentation maturity
- realistic managerial expectations
- a development team that understands automation and is willing to collaborate with manual testers
- automation at the right level
Why Not all manual tests should be automated?
- An automated script, if recorded, has limited understanding of what it can do if something other than the expected occurs
- we must figure in the cost of creating an automated script Because requires more analysis, more design, more engineering and more maintenance than a manual script
- Some manual tests simply cannot be automated because the thought process of the manual tester is essential to the success of the test
