Logging and Reporting Mechanisms

Question 1

(Within the TAS) One way of implementing the Test Execution Layer:

Answer 1

Writing test automation scripts from scratch.

Question 2

Mention an example of test execution library in Python:

Answer 2

pytest

Question 3

can run tests and report their results.

Answer 3

Unit test libraries

Question 4

Pytest default tests selection and execution:

Answer 4

Looks for all files (in the current directory or its subdirectories.) which matches the specified pattern and then executes them.

Question 5

Main pytest flags that may modify its behavior:

Answer 5

• Verbose mode: pytest -v
• Quiet mode: pytest -q
• Generate a report in HTML format: pytest –html=report.html

Question 6

Tests marks that can be put before a test definition so that pytest give it a special treatment.

Answer 6

• @pytest.mark.skip: pytest will not run the test.

- @pytest.mark.xfail: the test is expected to fail.

Question 7

When a automated script test fails, what limitations does it have an error message?

Answer 7

• is wholly insufficient.
• frequently has no context
• may have nothing to do with the actual failure which stopped the execution of the test.

Question 8

What is Logging?

Answer 8

a way to track events that occur during execution.

Question 9

How the Logging gets done?

Answer 9

• may be done before a step is taken and after a step is taken
• may include the data that was used in the step
• may include the behavior that occurred as a result of the step.
• can help a test analyst determine quickly if the failure was caused by the SUT or not.

Question 10

If eventually, the project could reach a big size then the logging will be required.

Answer 10

It is more effective and efficient to start logging robustly from the start.

Question 11

At any point in an automated script, the automator may add logging calls to report out any information like:

Answer 11

• general information for later tracing the test
• warnings about something that happened that does not rise to the level of a failure
• actual errors

Question 12

Five different levels of messages in Python (DIWEC: From lowest to highest levels):

Answer 12

• DEBUG
• INFO
• WARNING
• ERROR
• CRITICAL

Question 13

DEBUG level purpose:

Answer 13

for diagnosing problems

Question 14

INFO level purpose:

Answer 14

for confirmation that things worked

Question 15

WARNING level purpose:

Answer 15

something unexpected occurred, a potential problem

Question 16

ERROR level purpose:

Answer 16

a serious problem occurred

Question 17

CRITICAL level purpose:

Answer 17

a critical problem occurred

Question 18

if the console is set to level WARNING, the user will not see:

Answer 18

DEBUG or INFO messages, but will see WARNING, ERROR, and CRITICAL logging messages.

Question 19

Essentials steps to set the loggin level in Python:

Answer 19

• import sentence: import Logging
• level setting: logging.basicConfig(level=logging.WARNING)
• write the desired logging message: logging.info(“my info message”)

Question 20

What is assertion?

Answer 20

Python built in device to check whether the correct data or behavior occurred:

Question 21

Assertion definition:

Answer 21

a statement that is expected to be true at a particular point in the script.

Question 22

assertion sintax:

Answer 22

assert sumVariable==4, “sumVariable should equal 4.”

Question 23

What is reporting?

Answer 23

supplying that information, and likely other contextual information to the various stakeholders, outward and upward, who want or need it.

Question 24

Two important advice about reporting:

Answer 24

• determine who wants or needs reports and what information they are interested in.
• in the way that it is possible, automate report creation and dissemination

Question 25

Report content:

Answer 25

• a summary with an overview of the system being tested
• the environment(s) the tests were run in
• the results that occurred during the run.

Question 26

two ways to deal with reporting:

Answer 26

• create them from the logs and other information and make them available to the stakeholders such they can download them when they want.
• create and send the reports as soon as they are ready to the stakeholders who want them.