7. Operations: Monitoring and Auditing Program Performance Flashcards
What should metrics reflect?
Currency and value to the organization
*Must add value by by accurately reflecting the state of business objectives and goals
What should currency and value metrics be like?
- Simple
- Quantifiable
- Easy to use
- Correlated to
- Business performance
- Operational goals
- Technical outcomes
- Regulatory guidelines
What is good practice when it comes to measurement systems?
- Easy to understand
- Repeatable
- Reflective of relevant indicators to the organization
What is a metric?
A unit of measurement that should be as objective as possible and provide data that helps to answer specific questions about the business operations
What is the difference between an objective and a goal?
Objective - broad based
Goal - measurable, easy to understand, relevant and useful to the organization
In what ways can organizations implement privacy objectives?
Metrics that
- Normalize the privacy concepts - allow for meaningful privacy regime conversations
- Eliminate terminology and jargon - allow for decisions at an operational level
- Not based on a specific technology or application
- Advance the maturity of the privacy program and operations
What should metrics demonstrate?
- Compliance
- Program success
- Program maturity
- Resource utilization
- Return on investment
- Process improvement
What do metrics highlight?
Trends, issues, and gaps.
What do the right metrics allow for?
The development of KPIs that assist the organization in setting and tracking multiple objectives and goals.
What are general industry guidelines for metrics?
- Identify the privacy goals critical to the organization (why and to whom)
- Develop the formal intent of the metric based on goals
- Apply practical measurement to qualify the output (success, failure, goal met etc.)
- Evaluate and categorize metric data
What should metrics reflect?
- Compliance
- Data-driven decision making
- Overall impact of the privacy program
*Practical privacy program management in creating and maintaining compliance factors
What is the role of the metric owner?
Evangelize the purpose and intent of the metric.
What should a metric owner know?
- What is critical about the metric
AND
- How it fits into the business objective
What is the responsibility of the metric owner?
Monitoring process performance, variance, and undertaking visualizations
Preforming regular reviews to determine if the metric is still effective and provides value
What is one of the easiest statistical methods for data reporting?
Trend analysis - spot patterns in the information as viewed over a period.
What are examples of trending methods?
- Simple data patterns
- Fitting a trend
- Trends in random data
- Goodness of fit
What is a time series analysis?
Shows trends in an upward or downward tendency
*E.g.: number of incidents over time, how many PIAs were completed, number of privacy training and awareness sessions
What is a cyclical component?
Shows data over a period focused on regular functions
*E.g.: measuring the number of incident responses in the month after an organization rolls out new privacy training - analysis focuses on explaining any changes in the number of reported incidents as the stance from the training increases
What is an irregular component?
Analysis that focuses on what is left over when the other components of the series (time and cyclical) have been accounted for
E.g.: Absence incidents over a long period
What is return on investment (ROI)?
An indicator to measure financial gain or loss or a project or program in relation to its cost.
What is an ROI analysis?
Provides the quantitative measurement of benefits and costs, strengths and weaknesses of the organization’s privacy controls
The data can be:
1. Fixed or variable
2. Represent the best attempt to form an economic risk assessment to determine the probability of a loss
3. Maximize the benefits of investments that do not generate revenue, rather prevent loss
What are the considerations when developing a ROI assessment?
- ROI of the function must be related to the reason for implementing the function
- The value of the asset must be defined (E.g., the cost of producing or reproducing the information, repercussions if the information is not available, harm to reputation, loss of confidence
What is the best approach to developing a privacy ROI?
Look at a risk that has been mitigated and how to track the risk in financial terms
How is business resiliency measured?
Through metrics associated with:
1. Data privacy
2. Incident response
3. Compliance
4. System outages
5. Other factors defined through business objectives
What are the benefits to a strong business resilience program?
Help the organization prepare for audits and demonstrate compliance with regulatory requirements
What is a Privacy Maturity Model (PMM)?
A useful metric that focuses on a scale rather than an endpoint.
What are the five privacy maturity levels?
- Ad hoc - informal procedures and processes applied inconsistently
- Repeatable - procedures and processes are not fully documented and do not cover all relevant aspect
- Defined - fully documented procedures and processes and cover all relevant aspects
- Managed - reviews are conducted to assess the effectiveness of the control in place
- Optimized - regular review and feedback are used to ensure continual improvement toward optimization of a given process
What are the types of privacy monitoring?
- Compliance through audits - focuses on the collection, use, and retention of PI
- Regulation through monitoring changes in legislation and regulation requirements - focuses on updating policies accordingly
- Environment monitoring - focuses on vulnerabilities (E.g., physical concerns, programmatic concerns, insider threats, cybersecurity threat)
- Training data - focuses on ensuring that employees are aware of laws, regulations, and organization requirements for the handling of data
What are the common approaches to compliance monitoring through audits?
- Self-monitoring
- Audit management
- Security/system management
- Risk management
*Compliance monitoring is essential for detecting and correcting violations, supporting enforcement actions, and evaluating progress.
What are forms of monitoring?
- Scanning tools for network storage - used to identify risks to PI and monitor internal policy compliance
- Internal and external audits
- Breach management practices
- Compliant monitoring process
- Data retention and record management
- Privacy controls at the operational and program level
- Privacy protection across HR processes
- Supplier/third party monitoring
What is an audit?
An ongoing process of evaluating the effectiveness of controls throughout the organization’s operations, systems, and processes.
What can audits identify?
Risks posed by vulnerabilities and weaknesses and provide opportunities to strengthen the organization.
What is the purpose of a privacy audit?
Determine the degree to which technology, processes and people comply with privacy policies and practices.
*Based on the purpose, audits may be conducted regularly, ad hoc, or on demand.
What is the rationale for a privacy audit?
Provide evidence regarding whether privacy operations are doing what they were designed to do and whether privacy controls are correctly managed.
What are the phases of a privacy audit?
- Plan
- Prepare
- Audit
- Report
- Follow-up
What are the elements of an audit plan?
- Conducting a risk assessment
- Setting a schedule
- Selecting the auditor
- Preparing a pre-audit questionnaire
- Hosting an introductory meeting
- Completing a checklist
What are the elements of an audit preparation?
- Confirm the schedule
- Prepare additional checklists
- Sample criteria
- Finalize the audit plan
What are the elements of the audit?
- Meeting with stakeholders and business process owners
- Execute the functional goals of the audit
What are the elements of an audit report?
- Record and report noncompliance (categorizing instances as major or minor)
- Draft a formal audit report
- Host a case-out meeting
- Distribute copies of the audit report
*Audit report is a formal record of what was audited and when, insight into areas that comply (or do not), details to support the findings, and suggested corrective action and work estimates.
*Instances of non-compliance are documented to include facts, evidence, best practices, and standards that help assess the situation
*Findings should be communicated to stakeholders along with risk, remediation plans, and associated cots
What are the elements of an audit follow-up?
- Confirmation of the scope of remediation activities, scheduling those activities, and addressing any requirements around methodology
What are the different types of audits?
- First- party (internal) - reflect constant, standardized, and valid privacy management that aligns with to a particular standard, guideline, or policy
- Second-party (supplier)
- Third-party (independent) - identify weaknesses of internal controls, legitimize first-party audits, provide expert recommendations
