Performance Testing

Question 1

User-based objectives

Answer 1

1. focus primarily on end-user satisfaction and business goals.
2. are less concerned about feature types or how a product gets delivered.

Question 2

Technical objectives

Answer 2

focus on operational aspects and providing answers to questions regarding a system’s ability to scale, or under what conditions degraded performance may become apparent.

Question 3

Key objectives of performance testing include

Answer 3

1. identify potential risks
2. find opportunities for improvement
3. identify necessary changes

Question 4

Questions to stakeholders about the tes

Answer 4

1. What transactions will be executed in the performance test and what average response time is expected?
2. What system metrics are to be captured (e.g., memory usage, network throughput) and what values are expected?
3. What performance improvements are expected from these tests compared to previous test cycles?

Question 5

The Performance Test Plan Content

Answer 5

1. Objective
2. Test Objectives
3. System Overview
4. Types of Performance Testing to be Conducted
5. Acceptance Criteria
6. Test Data
7. System Configuration
8. Test Environment
9. Test Tools
10. Profiles
11. Relevant Metrics
12. Risks

Question 6

Objective from PTP

Answer 6

1. describes the goals, strategies and methods for the performance test
2. enables a quantifiable answer to the central question of the adequacy and the readiness of the system to perform under load.

Question 7

Acceptance Criteria (PTP)

Answer 7

1. response time is a user concern, throughput is a business concern, and resource utilization is a system concern
2. AC should be set for all relevant measures and related back to the following as applicable:
   a. overall objectives
   b. SLAs
   c. Baseline values

Question 8

Baseline values

Answer 8

a set of metrics used to compare current and
previously achieved performance measurements. This enables particular performance improvements to be demonstrated and/or the achievement of test acceptance criteria to be confirmed.

Question 9

Test Data can include:

Answer 9

1. User account data
2. User input data
3. Database (e.g., the pre-populated database that is populated with data for use
   in testing)

Question 10

User Data creation process should address the following aspects:

Answer 10

1. data extraction from production data
2. importing data into the SUT
3. creation of new data
4. creation of backups that can be used to restore the data when new cycles of testing are performed
5. data masking or anonymizing. Adds risk to the performance tests as it may not have the same data characteristics as seen in real-world use.

Question 11

System Configuration (PTP)

Answer 11

1. A description of the specific system architecture, including servers (e.g., web, database, load balancer)
2. Definition of multiple tiers
3. Specific details of computing hardware (e.g., CPU cores, RAM, Solid State
   Disks (SSD), Hard Drive Disks (HDD) ) including versions
4. Specific details of software (e.g., applications, operating systems, databases,
   services used to support the enterprise) including versions
5. External systems that operates with the SUT and their configuration and
   version (e.g., Ecommerce system with integration to NetSuite)
6. SUT build / version identifier

Question 12

Test Environment

Answer 12

The test environment is often a separate environment that mimics production, but at a smaller scale. How will the examples be extrapolated?

Question 13

Operational profiles

Answer 13

1. provide a repeatable step-by-step flow through the application for a particular usage of the system.
2. Aggregating these operational profiles results in a load profile (commonly referred to as a scenario)

Question 14

Risks (PTP)

Answer 14

1. areas not measured as part of the performance testing
2. limitations to the performance testing
3. limitations of the test environment

Question 15

Examples of Recommended Actions

Answer 15

1. change physical components (hardware, routers)
2. change software (e.g., optimizing applications and database calls),
3. altering network (e.g., load balancing, routing)

Question 16

Test Analysis Elements

Answer 16

1. Status of simulated (e.g., virtual) users.
2. Transaction response time.
3. Transactions per second.
4. Transaction failures.
5. Hits (or requests) per second.
6. Network throughput.
7. HTTP responses.

Question 17

A typical performance testing report may include:

Answer 17

1. Executive Summary
2. Test Results
3. Test Logs/Information Recorded
4. Recommendations

Question 18

Executive Summary

Answer 18

The goal is to present concise and understandable conclusions, findings, and recommendations for management with the goal of an actionable outcome.

Question 19

Test Results may include

Answer 19

1. A summary providing an explanation and elaboration of the results.
2. Results of a baseline test that serves as “snapshot” of system performance at
   a given time and forms the basis of comparison with subsequent tests.
3. A high-level diagram showing any architectural components that could (or did) impact test objectives.
4. A detailed analysis (tables and charts) of the test results showing response times, transaction rates, error rates and performance analysis.

Question 20

Recommendations may include:

Answer 20

1. Technical changes recommended, such as reconfiguring hardware or software
   or network infrastructure
2. Areas identified for further analysis (e.g., analysis of web server logs to help
   identify root causes of issues and/or errors)
3. Additional monitoring required of gateways, servers, and networks so that
   more detailed data can be obtained for measuring performance characteristics and trends (e.g., degradation)

Question 21

Transient state monitoring

Answer 21

1. some standard approaches—such as monitoring averages—may be very misleading.

Question 22

Test Execution

Answer 22

1. generation of a load against the SUT according to a load profile
2. monitoring all parts of the environment
3. collecting and keeping all results and information related to the test.

Question 23

Execution: important points

Answer 23

1. specify how failures should be handled to make sure that no system issues are introduced. (logouts)
2. One technique for verifying load tests which are communicating directly on the protocol level is to run several GUI-level (functional) scripts or even to execute similar operational profiles manually in parallel to the running load test.

Question 24

Communicate to stakeholders with buisness focus

Answer 24

1. are less interested in the distinctions between functional and non-functional quality characteristics.
2. The connection between product risks and performance test objectives must be clearly stated.
3. Stakeholders must be made aware of the balance between the cost of planned performance tests and how representative the performance testing results will be, compared to production conditions.
4. Project risks must be communicated. These include constraints and dependencies concerning the setup of the tests, infrastructure requirements (e.g., hardware, tools, data, bandwidth, test environment, resources) and dependencies on key staff.
5. The high-level activities must be communicated (see Sections 4.2 and 4.3) together with a broad plan containing costs, time schedule and milestones.

Question 25

Communicating to a Stakeholders with a Technology Focus

Answer 25

1. The planned approach to generating required load profiles must be explained and the expected involvement of technical stakeholders made clear.
2. Detailed steps in the setup and execution of the performance tests must be explained to show the relation of the testing to the architectural risks.
3. Steps required to make performance tests repeatable must be communicated. (organizational and technical)
4. Where test environments are to be shared, the scheduling of performance tests must be communicated to ensure the test results will not be adversely impacted.
5. Mitigations of the potential impact on actual users if performance testing needs to be executed in the production environment must be communicated and
   accepted.
6. Technical stakeholders must be clear about their tasks and when they are
   scheduled.

Question 26

Transactions

Answer 26

Transactions describe the set of activities performed by a system from the point of initiation to when one or more processes (requests, operations, or operational processes) have been completed.

Question 27

Elapsed time for that transaction

Answer 27

transaction response time + the think time

Question 28

Ramp-up

Answer 28

By ramping up load and measuring the underlying transaction times, it is possible to correlate the cause of degradation with the response times of one or more transactions.

Question 29

Operational profiles

Answer 29

1. Specify distinct patterns of interaction with an application such as from users or other system components.
2. May be combined to create a desired load profile for achieving particular performance test objectives

Question 30

Principal steps for identifying operational profiles

Answer 30

1. identify data to be gathered
2. gather the data using one or more sources
3. evaluate the data to construct the operational profiles