Performance Testing

Question 1

Performance testing should start at the ________ level and continue until _________ __________.

Answer 1

component (unit) level; system deployment

Question 2

What is the risk of waiting until the end of a project to implement performance testing?

Answer 2

Performance problems may not be solvable due to time, money, and technological constraints.

Question 3

List three goals/purposes of performance testing.

Answer 3

1. Measuring system performance under given conditions.
2. Determining max concurrent user load or transactional volume a system can handle before it fails.
3. Providing info to assist in capacity planning for a system.

Question 4

How are performance requirements normally recorded? Provide an example.

Answer 4

Normally recorded in over-arching requirements (ex. all system responses to users that require longer than 3 seconds must display a wait notification).

Question 5

How might performance requirements be defined in Agile?

Answer 5

May be documented in non-functional epics.

Question 6

What is the role of stakeholders when it comes to determining performance requirements? (i.e., what happens without it?)

Answer 6

Lack of stakeholder performance requirements make it difficult for testers to assess test results and determine acceptable performance.

Question 7

What are the four key tasks in aligning performance testing with the SDLC?

Answer 7

1. During requirements definition: specific requirements for performance clearly defined and agreed upon by all stakeholders
2. During design and coding: performance factors considered and built into design and subsequent code
3. During integration and system testing: any new integration introduces new performance testing opportunities
4. During acceptance testing: assess performance within the business or operational context

Question 8

List some of the challenges with performance testing.

Answer 8

• obtaining definition and agreement from stakeholders
• obtaining funding for testing tools
• obtaining best fit solution for performance testing tool
• accurately profiling load levels
• acquiring skilled test engineers to plan, design, and conduct a realistic performance test
• building a representative performance test environment

Question 9

What are some of the risks of performance testing? (Long list)

Answer 9

• inadequate performance test design
• incomplete coverage of protocols and connectivity
• inadequate test environments
• inability to apply performance tool correctly
• inadequate coverage of functions
• inadequate amount of user and load data
• lack of defined stakeholder requirements for performance
• lack of agreement on defined stakeholder performance requirements
• lack of appropriate performance metrics

Question 10

What are some of the benefits of performance testing?

Answer 10

• opportunity to “right-size” the system to handle expected load
• opportunity to plan mitigation steps if loads exceed expected levels
• early test results can help define acceptable levels of performance
• expectations of future growth can be tested to see if system can support it
• system performance weaknesses can be identified and fixed before occurring in prod
• long lead time items can be addressed

Question 11

What are the six areas of defining a performance test plan?

Answer 11

1. Scope
2. Strategy and approach
3. Risk assessment
4. Definition of test objectives
5. Responsibilities
6. Reporting metrics

Question 12

When defining a performance test plan, how is scope framed? (List and define five items.)

Answer 12

1. Functionality: what features will be included/excluded from performance testing?
2. Architecture: which aspects of the system architecture will be included?
3. Transactions: what typical user transactions will be included?
4. Users: which classes of users should be included on selection factors?
5. Data: how much data should be used and how should it be accessed?

Question 13

When defining a performance test plan, what does strategy and approach define? What might it include?

Answer 13

Defines the way performance testing is to be conducted (may include outsourcing).

Question 14

When defining a performance test plan, what purpose does risk assessment serve?

Answer 14

Performance testing can be risk-driven to focus on areas that carry the highest risk.

Question 15

When defining a performance test plan, what is the purpose of defining test objectives?

Answer 15

At a higher level, objectives describe what the performance test is intended to achieve.

Question 16

When defining a performance test plan, what is the purpose of defining responsibilities?

Answer 16

Both internal and external team roles must be defined; performance testing often requires support from system architects, database analysts, network engineers, etc.

Question 17

When defining a performance test plan, what is the purpose of defining reporting metrics?

Answer 17

Important to define which metrics are most meaningful to stakeholders to ensure those metrics are tracked.

Question 18

When defining a specific performance test within an overall test plan, what four areas must be considered? List and define.

Answer 18

1. Environments: full-scale environments often difficult to build due to cost and complexity; can use a cloud-based test environment that is rapidly scaled to simulate prod
2. Load and throughput profiles: current and predicted profiles must be understood for both load levels and data throughout (peak load times and load levels)
3. Operational profiles: describes the functions users are expected to perform on the system and the frequency; profiles are implemented as scripts to be executed by virtual users
4. Test data: need representative amounts and types of test data

Question 19

What are the four areas of conducting performance testing? (Test ________)

Answer 19

1. Test preparation
2. Test execution
3. Test evaluation
4. Test reporting

Question 20

In the performance test preparation stage, what components should be verified as ready for testing? List and define.

Answer 20

1. Test environment: run preliminary tests to make sure basic test environments are in place; validate co-existing applications, verify test environment capacities
2. Data: all databases and files should be populated with the designed volumes of test data to ensure test are not blocked
3. Testware: test scripts, procedures, etc. should be in place and executing correctly

Question 21

What needs to occur during the performance test execution phase? When are performance anomalies tested?

Answer 21

Test tools are run and execution is monitored. Interpretation of results requires human intelligence. Performance anomalies can be tested in real time or after test is complete.

Question 22

During the performance test evaluation phase, are test results more definitive or comprehensive? What is the main question performance testing seeks to answer?

Answer 22

Performance testing is more comprehensive and informative, providing a snapshot view.

“Does the system or application meet stated performance goals or requirements?”

Question 23

What are the two types of reporting in the performance test reporting phase? List and define.

Answer 23

• Risk-based reporting: helps identify which aspects of system performance may carry the most relative risk; helps stakeholders understand where to focus efforts.
• Objectives-based reporting: performance test results are reported with traceability to performance test.

Question 24

Why are tools essential for performance testing? List four.

Answer 24

1. Creating simulated concurrent user load (used for concurrency testing)
2. Sustaining high levels of load
3. Accurately measuring load levels and system response
4. Repeating performance testing whenever needed

Question 25

What are five challenges that exist when it comes to using performance test tools?

Answer 25

1. Performance tools do not know what to test (determined by the performance tester)
2. Environment has to be representative
3. High volumes of test data needed
4. Expertise needed; this is not commonly found within organization, usually involved outsourcing
5. Tools can be expensive