Module 8: Ongoing AI Issues and Concerns: Awareness of AI Auditing and Accountability Issues

Question 1

Which regulations/guidance documents call for first party auditing of AI systems?

Answer 1

• Algorithmic Accountability Act (US)
• GDPR
• The Federal Reserve and Office of the Comptroller of the Currency’s SR 11-7 guidance on model risk management

Question 2

What are some solutions to existing challenges facing AI auditors?

Answer 2

1. Leverage existing frameworks and codes of ethics ( such as COBIT Framework, Institute of Internal Auditors AI Auditing Framework, and COSO ERM Framework)
2. Internal auditors need to build a new way of auditing AI, including by incorporating data, models, outputs and processes to guarantee compliance, ethics and transparency.
3. Utilize third party audits (EU Digital Services Act calls for this)

Question 3

What are the main elements of ISACA’s COBIT 2019 framework?

Answer 3

• Process descriptions
• Desired outcomes
• Base practices
• Work products

Question 4

Name some examples of risks related to AI strategy.

Answer 4

• Lack of alignment between IT plans and business needs.
• IT plans that are inconsistent with the organization’s expectations or requirements.
• Improper translation of IT tactical plans from the IT strategic plans.
• Ineffective governance structures failing to ensure accountability and responsibility for IT processes related to the AI function.

Question 5

Name some AI auditing frameworks.

Answer 5

• COBIT 2019
• COSO
• GAO
• IIA
• PDPC

Question 6

Which framework provides human rights-related goals for businesses?

Answer 6

UN Guiding Principles Reporting Framework

Question 7

What set of AI objectives were published by the European Commission’s High-Level Expert Group on AI?

Answer 7

2019 Ethics Guidelines for Trustworthy AI

Question 8

What are the 2 different types of AI audits?

Answer 8

1. Bias/discrimination
2. Ethical/responsible

Question 9

What are 3 different means of AI audits identified by UK regulators?

Answer 9

1. Technical audits (data/code)
2. Empirical audits (inputs/outputs)
3. Governance audits (procedures/decisions)

Question 10

What does the Ada Lovelace Institute’s taxonomy of social media audit methods focus on?

Answer 10

• Scraping
• Accessing data through application programming interfaces
• Analyzing code

Question 11

What are the markers/indicators that determine when an AI system should be subject to enhanced accountability, such as third-party audits (e.g., automated decision-making or sensitive data)?

Answer 11

• Accountability mechanisms required by governments, companies and civil society
• EU AI Act - requires providers of high risk AI systems to conduct a prior conformity assessment before placing them in the market
• Adequate and appropriate human review measures

Question 12

What is AI Verify?

Answer 12

An AI governance testing framework and toolkit to ensure systems meet declared performance benchmarks launched by the Singapore government.