Module 3: Risk and Risk Factors

Question 1

What is algorithmic bias?

Answer 1

Algorithmic bias is a systemic deviation of an algorithm’s output, performance, or impact relative to some norm, aim, standard, or baseline.

Question 2

What is explicit vs. implicit bias?

Answer 2

In the former case, the algorithm’s developer intentionally projects its own biases into the algorithm, leading to biased outputs. In the latter case, bias creeps in undetected through the algorithm’s design or via the data used to train the algorithm. In those cases, bias is implicit and unintended.

Question 3

What two main approaches to fairness are applied in the context of AI?

Answer 3

Individual fairness relates to the Aristotelian doctrine of “treating like cases alike.” In other words, it demands similarly situated individuals be treated equally. We avoid direct discrimination on the basis of protected characteristics, such as race, gender, or religion.

Group fairness, on the other hand, does not consider individuals, but instead looks at statistical differences between groups.

Question 4

What is demographic parity? Does it need to exhibit correct predictions?

Answer 4

Demographic parity requires that the distribution of predictions is identical across subpopulations.

Demographic parity is entirely independent of the true value (Y). In other words, it does not consider if the predictions are correct, only if the predictions are equally distributed.

Question 5

What is a drawback of demographic parity?

Answer 5

Using demographic parity is inappropriate for a given task, such as when base rates significantly differ between groups, and those differences are relevant to the decision at hand.

Question 6

How is algorithmic accuracy calculated and is it always useful?

Answer 6

Accuracy denotes the fraction of correct predictions.

Especially in cases of unbalanced data, accuracy is not a good measure.

E.g. There are 100 job applicants, 10 of whom are suitable, 90 of whom are not. Imagine now the algorithm correctly classifies the 90 applicants as unsuitable (TN), but also classifies the 10 suitable candidates as unsuitable (FN). In this case the algorithm would have an accuracy of 90% even though it would only provide the user with a single output: unsuitable.

Question 7

What is Predictive Rate Parity ?

Answer 7

In some contexts, such as medical diagnostics or credit lending, the cost of false positives is high, and thus accuracy of the predictions for the positive or negative class is important. In other words, we want to maximize the positive predictive value (PPV) or precision

Positive Predictive Value (PPV) or Precision = TP / (TP+FP)

PPV is an important concept for algorithmic performance evaluation. It denotes the proportion of correctly predicted positive outcomes.

We can now define another fairness measure, predictive rate parity, which is satisfied when PPV is equal across subpopulations.

Question 8

What is a downside of Predictive Rate Parity?

Answer 8

A downside of predictive rate parity is that it does not account for differences in base rates across groups. If base rates differ significantly, then achieving predictive rate parity becomes extremely challenging and can have adverse effects on algorithmic performance.

Question 9

What is sensitivity and how is it calculated?

Answer 9

Sensitivity is important to know in case of rare events how well the prediction performs. It is also known as True Positive Rate.

Sensitivity = TP / (TP + FN)

Maximizing sensitivity requires us to minimize false negatives.

Question 10

What is the difference between PPV and Sensitivity?

Answer 10

Whereas PPV measured the proportion of correctly predicted positives to all positive predictions, sensitivity measures the proportion of correctly predicted positives to all actual positives.

Question 11

What is equal opportunity and how is it measured?

Answer 11

If sensitivity is the same across groups, we have achieved equal opportunity, which is the last important fairness measure we discuss here. Equal opportunity ensures that individuals of each group have an equal chance of being correctly identified as positive.

In the case of equal opportunity, the equality constraint is only applied to the true positive rate, so that each group has the same opportunity of being granted a positive outcome. A more-restrictive version of equal opportunity is equalized odds, where we not only require equality of sensitivity across groups, but also equality of specificity

Question 12

What is specificity and how is it calculated?

Answer 12

Specificity captures the proportion of actual negatives that are identified as such

Specificity = TN / (TN+FP)

Question 13

Can all three notions of fairness be achieved?

Answer 13

Unfortunately, a mathematical theorem demonstrates that doing so is impossible. Specifically, when base rates between populations are different, which is almost always the case, then it is impossible to satisfy demographic parity, predictive rate parity, and equal opportunity simultaneously.

Question 14

Assign the following labels to the statements

Accuracy ; PPV / Precision ; Sensitivity ; Demographic Parity ; Predictive Rate Parity ; Equal Opportunity

“Of all cases, how many did we correctly identify as either positive or negative?”

Of all positively predicted cases, how many were actually positive?”

“Of all actual positive cases, how many did we correctly identify as such?”

“Is the likelihood of a positive prediction the same across groups?”

“Is the likelihood of a true positive prediction the same across groups?”

“Is the likelihood of being positive when predicted to be positive the same across groups?”

Answer 14

Accuracy “Of all cases, how many did we correctly identify as either positive or negative?”

Positive predictive value/Precision “Of all positively predicted cases, how many were actually positive?”

Sensitivity “Of all actual positive cases, how many did we correctly identify as such?”

Demographic parity “Is the likelihood of a positive prediction the same across groups?”

Predictive rate parity “Is the likelihood of a true positive prediction the same across groups?”

Equal opportunity “Is the likelihood of being positive when predicted to be positive the same across groups?”

Question 15

What is required for problem specification?

Answer 15

Problem specification always requires us to have some degree of familiarity with the domain and needs in question. In other words, domain expertise is a necessary (though not sufficient) condition for good problem specification.

Question 16

Which statistical property is causing an issue with the removal of protected characteristics from the estimation process?

Answer 16

Endogeneity refers to situations in which the error term in a model is correlated with the predictor variable. In simpler terms, it refers to situations in which there are hidden or overlooked correlations that are not captured by the model in question.

If protected criteria are correlated with features of a model, then the omission of the protected criteria can lead to bias in the estimated feature coefficients

Question 17

Why is proxying of concepts by features problematic?

Answer 17

1) Some concepts cannot be easily operationalized (e.g. team work) and their omission or misspecification can lead to bias

2) We need to be careful that the construction of proxies does not lead to bias (e.g. productivity measured as the total output while protected groups might have the same work per time rate but lower total output)

Question 18

What is a problem with learning from past data in supervised learning in the context of protected groups?

Answer 18

Past data can have biases that will be reproduced when the algorithm is trained on the data

Question 19

What are potential solutions (with their drawbacks) how to deal with the situation when protected group characteristics are correlated with features?

Answer 19

We could simply remove all correlated variables but much of the time, these variables are highly relevant for prediction, and the algorithm would perform much worse if we removed them.

Another solution could be to include the protected variable when we model our relationship but then not use it in the prediction step, and instead substitute it with an average. This would only slightly decrease accuracy but ensure that (a) people who are alike in all relevant respects but have different protected characteristics are treated the same, and (b) correlated variables are not contaminated by the protected variables. However, as previously mentioned, care needs to be taken to ensure legality

Question 20

What is representation bias?

Answer 20

The collected sample is not representative of the target population and can result in problematic model specifications. Performance can be particularly low for groups that are at the intersection of different, underrepresented subgroups.

Question 21

What is measurement bias?

Answer 21

When measurement methods are not consistent across the sample. If a doctor spends more time meeting with and evaluating wealthy patients, resulting data on the likelihood of early-stage disease discovery may be biased in favor of that group.

Question 22

Why does a representative sample might bias against minority groups?

Answer 22

Given the statistical nature of both supervised and unsupervised learning algorithms, these algorithms deliver the statistically ideal output. For example, consider a medical training set of patients, 3% of whom are pregnant women who would require different treatment. It is likely that the algorithm would significantly underperform for this group because there are many fewer data points.

Question 23

How does a balanced dataset look like?

Answer 23

This means that relevant minority subgroups occupy a similar proportion of the space as majority groups. This ensures that the algorithm recognizes the minority groups as statistically relevant. Notably, this is different from having proportional representation.

As becomes clear, balancing a dataset can mean that the dataset becomes less representative of the actual population. A balanced dataset for a hiring algorithm, for example, could mean that male and female applicants are represented equally (50%) within the dataset, even if only 20% of job applicants historically were female.

Question 24

What are the three methods bootstrapping can be used to achieve balanced datasets?

Answer 24

(1) over-sampling datapoints from the minority group
(2) under-sampling datapoints from the majority group
(3) a combination of these.

Question 25

What two methods can be used to balance a dataset?

Answer 25

It is also possible to create (generate) synthetic data points or to remove existing datapoints to adjust data composition as well.

Question 26

How can Ensemble Methods address balanced dataset issues?

Answer 26

Ensemble methods, for example, use multiple models. Here, one could ensure that the entirety of the minority class data from the training set is used to train each model, whereas the majority class data is randomly sampled in each case.

Question 27

How could a cost function be used to achieve an estimation where group minorities are taken into account?

Answer 27

One could design a cost function that has a higher penalty for misclassification of minority class cases than for misclassification of majority class cases.

Question 28

How can the objective function introduce bias?

Answer 28

The choice of objective function introduces value judgments into the development of algorithms. What the algorithm optimizes for can make an ethically significant difference. Is the algorithm trying to minimize prediction errors (focus on performance), or does it focus on how errors are distributed (focus on error distribution)?

Question 29

How can the use of benchmark datasets introduce bias?

Answer 29

Another source of bias can enter in the testing stage. It is currently common practice to test algorithms against publicly available benchmark datasets. These datasets themselves may not be representative of the intended use population and are not necessarily a good indicator of a model’s quality.

Question 30

Can algorithms be used in different contexts?

Answer 30

Algorithms that are used within different contexts than what they were designed for can lead to a loss of performance or other challenges. This applies both to different contexts as well as to the use of historical data that might be outdated. To give just one example: In the context of law enforcement, an algorithm to predict recidivism may not be adequate when trying to determine sentence length.

Question 31

How does the perception of value by model developers influence the performance of a model?

Answer 31

if the values of the user do not align with the values that were built into the algorithm, this might lead to harm. Going back to our example of a “good employee”: often, companies will not develop their algorithms in-house, but instead use algorithms developed by third parties. If their understanding of a good employee differs from the developers’ understanding, the algorithm will not deliver the desired results.

Question 32

Which twin concepts appear as a central concern when evaluating risks of AI?

Answer 32

Explainability and interpretability

Explainability refers to the capacity to explain in understandable terms how an AI system makes decisions or predictions, often after the fact (hence referred to as “ex-post explainability” in the literature). Interpretability, on the other hand, refers to the degree to which a human can comprehend and predict the model’s behavior with built-in mechanisms to understand inherently how the inputs in the model affect the outputs (hence referred to as “inherently interpretable models” in the literature).

Question 33

What is the black-box problem?

Answer 33

At the core of this challenge is the difficulty of understanding how certain algorithms make decisions. Here, there are vast differences among models. Some, such as decision-trees, can be interpreted so the algorithmic reasoning becomes clear to humans, but others are much more opaque. For example, as neural networks become more intricate with numerous layers and large amounts of data, understanding the decision-making process becomes an arduous if not impossible task.

Question 34

Why is the black-box problem meaninful?

Answer 34

When decisions made by AI systems have substantial impacts on individuals or groups of individuals, the inability to explain algorithmic decisions can lead to accountability gaps or the erosion of trust among users and stakeholders. If a healthcare algorithm makes a recommendation for medical treatment, for example, doctors and patients alike need to be able to understand the basis of this recommendation.

Question 35

How can the black-box problem be divided?

Answer 35

The black-box challenge can be divided into a technical and a philosophical part. The philosophical part consists of determining what counts as a good explanation. When we ask people for explanations, we often intuitively know whether an explanation is good or bad. However, finding a satisfying definition of the concept of explanation is an entirely different story. The technical part consists of finding ways to obtain good explanations.

Question 36

Why is secrecy a problem in the case of AI models?

Answer 36

A first form of opaqueness is secrecy, in which an individual subject to algorithmic profiling is not aware of having been subject to algorithmic decision making in the first place. An even more radical form of ignorance is when individuals are unaware of the very existence of the algorithm in question.

Question 37

Why is confidentiality a problem in the case of AI models?

Answer 37

Confidentiality refers to the opaqueness of the algorithmic decision-making process itself. That is, affected parties may be aware that they are subject to algorithmic decision making, but they do not have access to the workings of the algorithms or to the reasons for why a certain outcome was reached. This has been a main concern in the recent discourse on the use of algorithmic decision making.

Question 38

What are reasons why the inner workings of an algorithm might not have to be disclosed?

Answer 38

Security is one of them: If spam filter service providers were to make their code public, circumventing these filters would become much easier. Releasing too much detail about an algorithm therefore isn’t always desirable, as it might increase risks of fraud, scams, or cyber-attacks.

A second reason for not making the workings of algorithms public regards proprietary interests and trade secrets. Keeping their algorithm secret helps companies retain their competitive advantages. For companies using third-party algorithms, confidentiality can become a big challenge if it prevents them from assessing the impacts of the algorithm on their own customer base.

Question 39

Why is a lack of pre-existing knowledge w.r.t. the AI model a problem?

Answer 39

A third hurdle is that interrogating an algorithm often requires specialized knowledge. Even if subjects or users have access to source codes and training data, they may not be able to make sense of them, unless they have received specialized training.

Question 40

What is XAI?

Answer 40

Explainable AI aims to make AI systems more understandable. AI systems are becoming increasingly complex and therefore increasingly inscrutable for human interlocutors.

Question 41

What are feature importance scores?

Answer 41

It ranks the importance of input features for outcomes. Feature importance scores can not only help to scrutinize how a model makes decisions, but can help identify potential biases or errors in the model.

Alternatives:

• Shapley values, for example, calculate the contribution of each input feature to the model’s output by calculating the marginal effect of including each feature and doing so across all possible feature combinations
• LUCID (Locating Unfairness through Canonical Inverse Design), on the other hand, works backward from a particular algorithmic output (e.g., “loan granted”) to create a distribution over the input space conditional on the particular output, thereby revealing a model’s internal logic.

Question 42

What are surrogate models used for?

Answer 42

Surrogate models are what their name suggests: simpler, more interpretable models that approximate the behavior of a complex AI system. LIME (local interpretable model-agnostic explanations), for example, approximates a complex model’s prediction locally with an interpretable model (e.g., linear regression or decision-tree).

Question 43

What are challenges with XAI?

Answer 43

The first is that an oversimplification of a complex decision-making process can distort our understanding of the decision to the point of misinterpretation.

Second, several XAI techniques are computationally intensive, which makes developing and deploying them more costly.

Finally, the above techniques are all ex-post techniques, which means that they are used to understand the models after they have been trained.

Question 44

What is human autonomy?

Answer 44

The first dimension refers to our ability to have values, hold beliefs, and make decisions that are in some important sense our own and not the product of distorting external factors, such as manipulation.

The second dimension refers to our ability to execute these decisions and requires us to have the freedom and the opportunity to do so.

As long as users and subjects have adequate control over outsourcing (e.g. AI), as long as there is meaningful human oversight, we remain autonomous in these respects.

Question 45

How can AI be used to manipulate people?

Answer 45

AI systems shape our online experience. Search algorithms determine what search engine entries show up first, and recommendation algorithms determine what advertisement we see, or which social media posts show up on our timelines. Simultaneously, these algorithms draw on large amounts of information about us and our past (online) behavior, allowing them to make predictions about future actions and influence our behavior.

Question 46

What is automation bias?

Answer 46

It refers to the tendency of humans to over-rely on automated systems, leading to complacency and reduced vigilance. This in turn can lead to errors of oversight, which is especially problematic in critical situations.

Critical systems, such as transportation, healthcare, and public safety, increasingly rely on AI for efficiency and improved decision-making.

However, there is the risk of over-dependence on (often immature) AI systems in critical scenarios, but also highlights the need for robust testing and validation processes.

Question 47

What is skill atrophy in the context of automation bias?

Answer 47

Prolonged reliance on automation can lead to a decay in essential skills, as operators move from being active decision makers to becoming passive observers.

Question 48

What can be done against automation bias?

Answer 48

The first is to inform users and operators of the risks of automation bias, and to ensure they understand the limitations of AI.

The second step is to implement design mechanisms that require periodic human input and verification. This keeps operators and users engaged and alert.

Question 49

What are the golden rules of AI development?

Answer 49

(1) Just because we can does not mean we should.

(2) A further golden rule for AI design is to involve diverse stakeholders, which include domain experts, the wider public, ethicists, and policy makers for AI applications in critical situations.

(3) AI in critical systems requires both developers and users to be transparent and accountable. There needs to be clear lines of responsibility in case of failures or adverse outcomes. Such failures, finally, can be avoided by ensuring AI systems are regularly audited and updated.

Question 50

What are the three main reasons for reputational risks?

Answer 50

(a) privacy breaches
(b) algorithmic bias
(c) lack of explainability

Question 51

What type of criticisms do companies faces?

Answer 51

Competence and value alignment

Competence is criticized when there is a perception that the company lacks the relevant expertise and capacities to ensure the algorithms they deploy are safe and fair, and the data they collect or use is stored safely. Technical failings of algorithms, unfairness, or lack of explainability often fall into this category.

Value alignment is questioned when it becomes clear that the company has been fully aware of the risks and nevertheless went ahead.

Question 52

How can organizational governance mechanisms mitigate risks?

Answer 52

(1) Ensuring privacy and security: Data governance policies should comply with regulations such as GDPR and include secure data storage, handling, and processing practices, along with transparent data usage policies.

(2) Ensuring algorithmic fairness: Algorithms should be trained on diverse datasets and regularly audited for potential discriminatory patterns.

(3) Promoting transparency: Be as transparent as possible to your customer base. This includes ensuring that algorithms remain explainable using XAI techniques, but also ensuring that customers are aware of the use of algorithms. This builds both trust and accountability.

Question 53

What is the most immediate economic concern related to AI?

Answer 53

One of the most immediate economic concerns is the displacement of jobs by AI.

When looking at past technological breakthroughs, such as the ones that led to the industrial revolution, automation traditionally affected low-skill job sectors. This may not be the case with AI, as high-skill jobs, such as accounting equally seem to be at stake. Nevertheless, the gap between those who have the skills to work alongside AI systems and those without them could widen, further exacerbating existing socio-economic inequalities. For risk professionals, this underscores the need for strategies that address workforce re-skilling, and policies that mitigate the unequal distribution of AI’s economic benefits and burdens.

Question 54

How can the development of AI lead to global inequalities?

Answer 54

Currently, most large AI firms are based in the US, and technological adoption of AI systems varies widely between countries. At this point, there is a risk that AI advancement will exacerbate global inequalities with nations that are at the center of AI development leaping ahead in terms of economic growth and political influence, leaving behind less technologically developed countries.

Question 55

What is the most immediate concern to political institutions and democratic elections?

Answer 55

AI technologies enable sophisticated misinformation campaigns. Ranging from bots that create misleading, harmful social media posts to AI generated imagery and videos, AI has the potential to mislead and influence public opinion. This poses a serious risk to political institutions and democratic elections. Managing risks from misinformation campaigns requires strict governance measures that ensure information authenticity and reliability, as well as the deployment of AI detection tools to expose AI generated content.