Chapter 2 Flashcards
Performance Measurement Fundamentals
Vocab
Measurement
The process of quantifying and evaluating the attributes of a software product or process to assess its quality, performance, and progress.
Used to assess key-performance-indicators (KPIs) and metrics
Examples include:
* Time To First Byte (TTFB)
* Peak Response Time
* Throughput
* Error rates
* Resource utilization (e.g., CPU, memory, disk I/O)
Vocab
Metric
Derived from measurements
A standardized unit of measurement that provides a snapshot of the system’s behavior under various loads, scenarios, or conditions.
Why are measurements and metrics necessary?
- Defines goals for
- Evaluates the results of
performance testing
What are the 6 project risks that can occur without well defined measurements and metrics?
Note project vs product risks from the CTFL material
- Unknown if performance levels are acceptable for operational objectives
- Performance requirements aren’t defined in measurable terms
- May not be possible to ID trends that can predict lower levels of performance
- Cannot compare actual to expected test results due to lack of baseline
- Results are subjective rather than objective
- Results are not understandable, thereby breaking one of the principles
For Fun
Define SMART goals
Helpful for testing and even personal/professional growth
- Specific - Clearly define what you want to achieve, avoiding vague or general statements.
- Measurable - Quantify your goal, allowing you to track progress and measure success.
- Achievable - Ensure the goal is realistic and attainable based on your resources and capabilities.
- Realistic - Align your goal with your values and long-term objectives, making it meaningful and relevant.
- Time-bound - Set a specific deadline or timeframe for achieving the goal, providing focus and motivation.
What are the three contextual environments of perf testing?
- Business
- Operational
- Technical
What are some examples of technical environments?
- Web-based
- Mobile
- IOT
- Desktop client devices
- Server-side processing
- Mainframe
- Databases
- Networks
- Embedded
What are some examples of technical metrics?
- Response time
- Resource utilization
- Throughput rate of key transaction
- Batch processing time
- Numbers of errors that would impact performance
- Completion time
- Background load on shared resources
- Software metrics (such as complexity)
See 2.1.2
What are some examples of business metrics?
- Business process efficiency
- Throughput of data, transactions, and other units of work
- Service level agreements (SLAs) and any violations
- Scope of usage
- Concurrency of usage
- Timing of usage
See 2.1.2
What are some examples of operational metrics?
- Operational processes
- System restoration
- Alerts and warnings
Things that aren’t user-facing in nature
See 2.1.2
What is the goal-question-metric (GQM) approach to metric selection?
- Establish the goals
- Ask questions to know when the goals have been achieved
Metrics are associated with each question to ensure the answer is mesurable.
The process may be iterative, as more or better metrics might need to be derived.
Define the “probe effect”
When the metrics collection effort impacts the performance of the system. The effort should no more than minimally impact.
Is it ok to use more than one tool to collect metrics?
Multiple tool usage is common enough, but care must be taken. Redundant metrics might be collected and other problems introduced.
What are 3 sources of performance metrics?
- Performance Test tools
- Performance Monitoring tools
- Log analysis tools
Describe Performance Test Tools
- They all provide measures and metrics.
- Some report in a text format, others use graphics.
- Many allow the ability to export results for reporting.
