Software Measurement and Instrumentation Flashcards
Static vs Dynamic Measurements
Static Measurements are made without measuring software.
Dynamic measurements require that the software be executing during measurement.
What type of measurements does SPE use?
SPE uses dynamic measurements
What is a state?
Reflects information about the systems or hardware or software within a period of time when computer is running
What is an event?
An event is when the CPU changes state
What is a performance monitor?
Observes the activities of a computer system and records performance information.
They sample the systems state by activating and recording the current state.
What are hot spots?
Portions of code that take up most of CPU time
What is event recording?
Records the occurrence of particular events along with the appropriate performance data
Types of event recording?
- Program Event Recording
2. System Event Recording
What is an internal measurement? What is an external monitor?
Requires code to be inserted into programs to detect events
External monitor can record data about software outside of it.
What is data granurality?
Measures how indepth the measurement of data process is. E.x. the initiation and completion of a proces, how many times a method is called.
Three choices for data granularity
- Record all data
- Summarize data and record periodically
- Tally data inbetween events and record when event ocurs.
What is Instrumentation?
Instrumentation is built to enable the collecting of SPE data such as workload intensity or service requirements. Done through inserting probes at different parts of code.
Reasons for Instrumentation
- Conveniently gather exactly the data you need
- Match the data granularity you need at different level
- Control the measurement process by turning selected instrumentation on or off
Instrumentation Design Considerations
- Defining the events to be measured
- Choosing the correct granularity (most fine granularity is just measuring everything, much overhead though)
- Dynamically selecting the data to be recorded
Purpose of Software Measurements
- System understanding: Measurements of an existing system can provide performance information about similar system
- Model Specification: Measurements of experiments, prototype or other systems can provide SPE needed data.
- Model Updates: Measurements of evolving software can replace current measurements that can be outdated.
- Model Verification: Compare model results to performance objectives to ensure match up.
- Software Performance Evaluation: Figure out areas of improvement in software.
Four Types of Measurement Data
- Workload data: number/rate/pattern of requests from workload functions
- Data characteristics: amount, size, frequency for each data item. Locality of data references.
- Execution Characteristics: Path characteristics, Software resource usage, Processing overhead matrix
- Computer System Usage (Performance benchmarks)
What is path characterisitcs?
Number of times each significant path is executed to compute loop repetitions, execution probabilities and conditional paths.
What is scenario response time?
end-to-end elapsed time
What is scenario throughput?
Number of times scenario is executed per unit of time
What is computer system resource usage?
wait time + service time
What is resource utilization?
The percentage of time busy
What is Resource throughput?
Rate at which resource completes requests
What is Resource queue length?
average number of jobs waiting for resource
Key considerations for performance measurement planning?
- Representative: measurement results must accurately reflect the factors that affect performance
- Reproducible: the workload, software and computer system must be controlled in an environment when running all tests to ensure results stay consistent
What are performance benchmarks?
Performance benchmarks are standardized performance measurements (e.x. throughput of service)
What is a load-driver?
Used to simulate user behaviour under load conditions
What is system perbutation?
Instrumentation used can effect the system being measured which can effect the data collected.
How to control system perbutation?
- Deactivate/activate instrumentation at correct times
- Select correct data granularity
What are capture ratios?
Capture ratio characterizes the percent of time that the measurement technique account for either directly or by inference
What is system overhead?
Always include system overhead in SPE models.
What is measurement timing?
- Duration of events and states must match the resolution of the operating system clock
- Sampling rate of monitor must be enough to detect useful events but not too low to be causing system perbutation
Averages for typical behaviour
Run measurements long enough to get an average, minimum, maximum that are representative of the behavior
What are data collection techniques?
System monitors: observe system-level states, require detailed information on OS.
Program monitors: observe application/code level. Don’t need to know OS as much.
System event recorders: Records system events, provided by the OS.
What do we need to do before performing measurements?
1) Define Measurement Tools
2) Define Measurement Strategy (what data we need, how we get it, design experiments)
What is the input of a load driver?
Workload profiles i.e. number of users, arrival rate, frequency of transactions.
What is the output of a load driver?
Script that simulates user requests
What provides most realistic measurements?
Average or peak load environments
