Logging and Reporting Mechanisms Flashcards
(Within the TAS) One way of implementing the Test Execution Layer:
Writing test automation scripts from scratch.
Mention an example of test execution library in Python:
pytest
can run tests and report their results.
Unit test libraries
Pytest default tests selection and execution:
Looks for all files (in the current directory or its subdirectories.) which matches the specified pattern and then executes them.
Main pytest flags that may modify its behavior:
- Verbose mode: pytest -v
- Quiet mode: pytest -q
- Generate a report in HTML format: pytest –html=report.html
Tests marks that can be put before a test definition so that pytest give it a special treatment.
- @pytest.mark.skip: pytest will not run the test.
- @pytest.mark.xfail: the test is expected to fail.
When a automated script test fails, what limitations does it have an error message?
- is wholly insufficient.
- frequently has no context
- may have nothing to do with the actual failure which stopped the execution of the test.
What is Logging?
a way to track events that occur during execution.
How the Logging gets done?
- may be done before a step is taken and after a step is taken
- may include the data that was used in the step
- may include the behavior that occurred as a result of the step.
- can help a test analyst determine quickly if the failure was caused by the SUT or not.
If eventually, the project could reach a big size then the logging will be required.
It is more effective and efficient to start logging robustly from the start.
At any point in an automated script, the automator may add logging calls to report out any information like:
- general information for later tracing the test
- warnings about something that happened that does not rise to the level of a failure
- actual errors
Five different levels of messages in Python (DIWEC: From lowest to highest levels):
- DEBUG
- INFO
- WARNING
- ERROR
- CRITICAL
DEBUG level purpose:
for diagnosing problems
INFO level purpose:
for confirmation that things worked
WARNING level purpose:
something unexpected occurred, a potential problem
ERROR level purpose:
a serious problem occurred
CRITICAL level purpose:
a critical problem occurred
if the console is set to level WARNING, the user will not see:
DEBUG or INFO messages, but will see WARNING, ERROR, and CRITICAL logging messages.
Essentials steps to set the loggin level in Python:
- import sentence: import Logging
- level setting: logging.basicConfig(level=logging.WARNING)
- write the desired logging message: logging.info(“my info message”)
What is assertion?
Python built in device to check whether the correct data or behavior occurred:
Assertion definition:
a statement that is expected to be true at a particular point in the script.
assertion sintax:
assert sumVariable==4, “sumVariable should equal 4.”
What is reporting?
supplying that information, and likely other contextual information to the various stakeholders, outward and upward, who want or need it.
Two important advice about reporting:
- determine who wants or needs reports and what information they are interested in.
- in the way that it is possible, automate report creation and dissemination
Report content:
- a summary with an overview of the system being tested
- the environment(s) the tests were run in
- the results that occurred during the run.
two ways to deal with reporting:
- create them from the logs and other information and make them available to the stakeholders such they can download them when they want.
- create and send the reports as soon as they are ready to the stakeholders who want them.
