7. Operations: Monitoring and Auditing Program Performance Flashcards
What should metrics reflect?
Currency and value to the organization
*Must add value by by accurately reflecting the state of business objectives and goals
What should currency and value metrics be like?
- Simple
- Quantifiable
- Easy to use
- Correlated to
- Business performance
- Operational goals
- Technical outcomes
- Regulatory guidelines
What is good practice when it comes to measurement systems?
- Easy to understand
- Repeatable
- Reflective of relevant indicators to the organization
What is a metric?
A unit of measurement that should be as objective as possible and provide data that helps to answer specific questions about the business operations
What is the difference between an objective and a goal?
Objective - broad based
Goal - measurable, easy to understand, relevant and useful to the organization
In what ways can organizations implement privacy objectives?
Metrics that
- Normalize the privacy concepts - allow for meaningful privacy regime conversations
- Eliminate terminology and jargon - allow for decisions at an operational level
- Not based on a specific technology or application
- Advance the maturity of the privacy program and operations
What should metrics demonstrate?
- Compliance
- Program success
- Program maturity
- Resource utilization
- Return on investment
- Process improvement
What do metrics highlight?
Trends, issues, and gaps.
What do the right metrics allow for?
The development of KPIs that assist the organization in setting and tracking multiple objectives and goals.
What are general industry guidelines for metrics?
- Identify the privacy goals critical to the organization (why and to whom)
- Develop the formal intent of the metric based on goals
- Apply practical measurement to qualify the output (success, failure, goal met etc.)
- Evaluate and categorize metric data
What should metrics reflect?
- Compliance
- Data-driven decision making
- Overall impact of the privacy program
*Practical privacy program management in creating and maintaining compliance factors
What is the role of the metric owner?
Evangelize the purpose and intent of the metric.
What should a metric owner know?
- What is critical about the metric
AND
- How it fits into the business objective
What is the responsibility of the metric owner?
Monitoring process performance, variance, and undertaking visualizations
Preforming regular reviews to determine if the metric is still effective and provides value
What is one of the easiest statistical methods for data reporting?
Trend analysis - spot patterns in the information as viewed over a period.
What are examples of trending methods?
- Simple data patterns
- Fitting a trend
- Trends in random data
- Goodness of fit
What is a time series analysis?
Shows trends in an upward or downward tendency
*E.g.: number of incidents over time, how many PIAs were completed, number of privacy training and awareness sessions
What is a cyclical component?
Shows data over a period focused on regular functions
*E.g.: measuring the number of incident responses in the month after an organization rolls out new privacy training - analysis focuses on explaining any changes in the number of reported incidents as the stance from the training increases
What is an irregular component?
Analysis that focuses on what is left over when the other components of the series (time and cyclical) have been accounted for
E.g.: Absence incidents over a long period
What is return on investment (ROI)?
An indicator to measure financial gain or loss or a project or program in relation to its cost.
What is an ROI analysis?
Provides the quantitative measurement of benefits and costs, strengths and weaknesses of the organization’s privacy controls
The data can be:
1. Fixed or variable
2. Represent the best attempt to form an economic risk assessment to determine the probability of a loss
3. Maximize the benefits of investments that do not generate revenue, rather prevent loss
What are the considerations when developing a ROI assessment?
- ROI of the function must be related to the reason for implementing the function
- The value of the asset must be defined (E.g., the cost of producing or reproducing the information, repercussions if the information is not available, harm to reputation, loss of confidence
What is the best approach to developing a privacy ROI?
Look at a risk that has been mitigated and how to track the risk in financial terms
How is business resiliency measured?
Through metrics associated with:
1. Data privacy
2. Incident response
3. Compliance
4. System outages
5. Other factors defined through business objectives