lecture 8

Question 1

Transparency

Answer 1

Refers to the willingness and ability of
an organization developing or using an
ML system to be transparent and share
information with all its stakeholders
about the whole ML pipeline (data and
its provenance, annotation process,
algorithms used, purpose, risks)

Question 2

Explainability

Answer 2

Refers to the process of “providing
information” about how the ML model
reached a given output(s), often based
on input data and/or features.

Question 3

Examples of harms

Answer 3

Asymmetrical concentration of power
* think of google facebook amazon and apple

Social sorting and and discrimination

profiling and mass manipulation

minority erasure

Question 4

Reliability

Answer 4

When a good thing happens: The
“system” does what it is supposed
to do as it is “intended”

Question 5

Safety

Answer 5

When no bad things happen: The
“system” and its usages do not cause “unintended” harm (including metal-health discrimination, leading to bad decision-making, etc.) to users (non-users, society, the environment, etc.)

Question 6

Accountability

Answer 6

The “responsible(s)” for the failure should be held accountable and take the burdens e.g., repairing the consequences and damages

Question 7

Liability

Answer 7

The harmed parties are owed “duty of care”: indentities, medical care, mending reputation damage, etc. This often depends on the nature of the harm but also its causes

Question 9

What is the difference between ideal and the actual ML transparency?

Answer 9

For ideal Transparency:
* implies accountable
* ensures Fairness and Justice
* increases trust

Actual ML Transparency
- Seriously missing transparency
- Lots of papers and research projects are on this topic
- Makes the News Headlines often

Question 10

What is the difference between ideal explainability and actual explainability

Answer 10

ideal explainability:
* explains the outputs, like trust and uncertainty
* explains biases and help alleviating biases
* collaborates and augments human knowledge and intelligence –> helps humans improve and come together on knowledge and intelligence

Actual Explainability
* involves a lot of debugging
* improves the model accuracy wise

Question 11

What is the difference between interpretability and explainbility?

Answer 11

There is no clear answer but:

Interpretability
* is dependedent on the model architecture (model specific)
* is computationally fast
* can be inferred (deduced, concluded) from the model alone

wheras:
explainability
* Focus on model agnostic (where the structure of the model is not taken into account; black box model approach –> solely focus on observable elements)
* is computationally intensive
* is based on feature values and predictions

Question 12

Stakeholders
who needs explainability?
and
who are we explaining to?

Answer 12

Three groups are most often proposed:
* practioners: consisting of data scientists and ML engineers
* Observers: consisting of business stakeholders and regulators
* Users: consisting of the domain experts

Additional groups:
* decision subjects: persons who are directly affected by ML usage
* non-users: persons or groups directly or indirectly affected by the ML usage
* civil-society watch dogs

Question 13

What is desiderata

Answer 13

the desirable properties or characteristics that a model or algorithm should possess

Question 14

What will explainable AI = XAI bring?

Explainability desiderata

Answer 14

Todays age questions the understanding of the machine learning process is unclear. The machine learning process leads to a learned function that performes a task or makes a decision or recommendation which is unclear to the the users what happen and why choices were made.

In the future (with XAI) there is a new machine learning process that leads to an explainable model with an explanation interface. Ensuring that the user can understand the task performed or why not and what happened inside the model.

Question 15

Why is XAI needed?

Answer 15

Model will improve
* the quality of model will increase

Verify performance
* confirmation that model behaves as expected

Build trust
* Increase confidence in the reliability of the model

Remediation
* Understanding what actions to take to alter a prediction

Understand model behaviour:
* Can be used to construct a simplified model in the users mind, which can be used as a surrogate for understanding the model’s performance

Monitoring (and Accountability)
* There is an ongoing assessment that a model’s performance remains acceptable and compliant (to the eventual standards and regulations)

Question 16

Stakeholders desiderata. Who to involve and why?

Answer 16

Users affected by the model decisions
* to understand their situation and verify fair decisions

Domain experts and the user of the model
* to trust the model itself and gain scientific knowledge

Regulatory entities/agencies
* to certify model compliance with the legislation in force audits

Managers and executive board members
* to assess regulatory compliance, understand corporate AI applications

Data scientists, developers, product owners
* to ensure and improve product efficiency, research and new functionalities

Question 17

How to evaluate explainability, from the regulatory perspective?

Answer 17

The explanation should be …

• … meaningful; the system provides explanations that are understandable to the intended consumer(s)
• …accurate; the explanation corretly reflects the reason for generating the output and/or accurately reflects the system’s process
• …limited to the knowledge it has; system only operates under conditions for which it was designed and when it reaches sufficient confidence in its output

Question 18

How to evaluate explainability from the business perspective?

Answer 18

Explainability evaluated on the basis of:
* comprehensible: how much effort is needed for a human to interpret it?
* succinctness: how concise is the explainability?
* actionability: how actionable is the explanation? What can we do with it?
* reusability: could it be interpreted/reused by another AI system?
* accuracy: how accurate iis the explanation?
* completeness: does the ‘explanation’ explain the decision completely, or only partially?

Question 19

What is the taxonomy of explainability?

Answer 19

explainability is divided into:
* intrinsic explanation vs post-hoc explantion
* model specific vs model agnostic
* local vs cohort vs global
* different data types

Question 20

Explainability: the diffrence between intrinsic explanation vs post-hoc explantion

Answer 20

Intrinsic explanation
Some ML models are inherently interpretable, meaning simple enough in structure that we can understand how the model is making predictions by just looking at the model.

post-hoc explanation
Some ML models are inherently more complex and less intuitive. Post-hoc methods then involve using the trained model and data to understand why certain predictions are being made.
In some cases, post hoc methods can be applied to models that have intrinsic explainability as well.

Question 21

Explainability: the diffrence between model agnostic vs model specific

Answer 21

Model agnostic explainer or explainability method can be applied to any model

Model specific explainer or explainability method can only be used with certain model types.

Question 22

Explainability, difference between:
Local vs Cohort vs global

Answer 22

Local explanation focusses only on a single prediction.

Global explanation attempts to make claims about trends and behaviours for model predictions across an entire dataset

Many local explainers can be turned into global explainers by using aggregations of the local results. Thus many techniques are useful in providing both local and global explanations.

cohort explanation is a global explanation, performed on a slice of the full dataset. This slice can, for example, be a subset defined by a single feature value.
This helps understand issues related to a particular subset or cohort

Question 23

Explainability: the diffrence between data types

Answer 23

• Tabular data
• textual data
• images
• graphs

Question 24

What is the idea behind LIME (Local Interpretable Model-Agnostic Explanations) ?

Answer 24

Function of lime:
LIME tries to understand the features that influence the prediction of the area around a single instance of interest.

or in other words:
a technique that approximates any black box machine learning model with a local, interpretable model to explain each individual prediction.

The idea behind lime is:
complex ML models decisions boundaries are very complex, however, zooming in on a small enough area/neighbourhood, then decision bounderies are in a local neighbourhood which can be much simpler and even be linear (no matter how complex the model is at global level).

Question 25

LIME :simplified step by step

Answer 25

Consider a model **m** whose prediction **y ** for observation **x** is to be explained. LIME...: * ...generates *synthetic observations* * ...labels the *synthetic data* by passing it to the model **m** * weights the synthetic observations using the *Euclidean distance* from **x** *to the synthetic observating* (euclidean distance is just simply the shortest distance between two points). * produces a locally weighted *(linear) regression* * use the *regression coefficients* as a mean to explain prediction y ## Footnote https://paperswithcode.com/method/lime#:~:text=**LIME**%2C%20or%20**,locally%20with%20an%20interpretable%20model.

Question 26

LIME Visualization of tabular data (slide 40 advised to look at)

Answer 26

There is a model, from this model data is derived with a prediction. LIME explains this prediction based on certain data that is present. Then the human is able to make the decision based on the explanation that lime provides.

Question 27

LIME Visualizatoins of other data types (like pictures)

Answer 27

LIME helps understand how an image classification prediction is made, showing what elements of a picture are taken into account for a decision.

Question 28

LIME Pros and Cons

Answer 28

Pros: * LIME is a very popular technique for producing pixel-level attributions (= highlight the pixels that were relevant for a certain image classification by a neural network) * LIME is easy to implement and has a well-maintained pyhton implementation * LIME has an intuitive and easy-to-explain algorithm and implementation. * There are a wide variety of visualizatoin options Cons * The performance of LIME depends on the complexity of the model * The accuracy of LIME depends on the complexity of the model * LIME can have variation in explanations for the same prediction (this can cause inconsistencies)

Question 29

Shapely values & SHAP (SHapley Additive exPlanations)

Answer 29

**SHAP** is a widely used post-hoc explainability tool implementing **shapely values** Shapley values are based on coalition **game theory** and tell **how to fairly distribute a pay-out among players** **A coalition of players cooperate** and obtain a certain overall **gain** from that cooperation some players may **contribute more** to the coalition than others or may possess different bargaining power.

Question 30

SHAP and Shapley values in the ML Context

Answer 30

In ML, game theory is used to* determine a feature's responsibility for a change in the model output prediction* for this: **Shapley value provides** * the (weighted) average marginal *contribution of a feature value across all possible coalitions* * the *average change in the prediction* that a coalition (of features) receives when the feature value joins the coalition In other words, Shapley values consider all possible predictions for an instance using all possible combinations of inputs Shapley value does **not** provide: * the change in the prediction **performance** when feature is *removed* from the model

Question 31

See slide 47 for example with pics

Answer 31

Imagine a ML model trained to predict apartment prices, now we want to understand how did each feature contribute to the prediction? we can make different coalitions, which result in different apartment costs, e.g. coalition for cat-banned with: {park-nearby, 50m2, 2nd floor} for which the average apartment cost would be 310.000 euro. (so a **coalition** is just a combination of features) **shapley values** then *enumerate all possible coalitions* and *compute the weighted average change in the prediction when the feature calue [cat-banned] joins the coalition*

Question 32

Where can SHAP be implemented?

Answer 32

For ML model/architectures: Tree models, XGBoost, scikit-learn * has high perfomance when computing Shapley values DNNs, TensorFlow * Based on DeepLIFT, approximate Shapley values, difficult to configure Differentiable models TensorFlow * Slower than DeepExplainers also approcimates Shapley values Linear regression * computes the exact Shapley value; i.e. the weights multiplied by feature values Model agnostic * more difficult to configure, and slowest SHAP Explainer in terms of computation time.

Question 33

SHAP Pros and Cons

Answer 33

Pros * SHAP has strong theoretical foundation * There are commonly implemented explainability techniques available in several open-source packages as well as cloud-based option * Shapley values can be used for individual predictions, cohorts, and to globally explain the model * Values provide an intuitive understanding for stakeholders Cons * The feature influence is often combined with causality by practitioners, end users and stakeholders * SHAP is computationally intensive and difficult to use with models with more than 100 features, because it will go through every coalition possible. * choosing a good baseline can be difficult

Question 34

Pitfalls and issues for **explainability**

Answer 34

**assuming causality** Most common and most dangerous. Almost no explainability technique is able to definitely establish causality for an ML model operating in the real world. **Overfitting intent** overfitting can lead to a user to have false confidence in the model. Extrapolating from an explanation, assuming the model understands concepts that are familiar to users. Often that is unlikely and can lead to the user understanfing of the model and its actual behavior. **overreaching for additional explanations** can result in confirmation bias as other explanations are misused to confirm existing expectations.

Question 35

LIME vs SHAP General

Answer 35

While LIME excels in localized insights, SHAP provides a broader understanding, which is crucial for complex models. The choice hinges on task requirements. You can pick LIME for focused, instance-level clarity and SHAP for comprehensive global and local perspectives.

Question 36

If you have a simpler model, consider LIME, as it excels in providing clear insights. Choose SHAP for complex models, including deep neural networks or ensemble methods, to gain both local and global interpretability. Scope of task LIME or SHAP?

Answer 36

If you’re seeking **localized** insights for individual predictions in simpler models, opt for **LIME**. If your task demands a broader understanding, encompassing both **global and local perspectives**, and involves complex models, **SHAP** is more suitable.

Question 37

LIME vs SHAP model characteristics

Answer 37

If you have a **simpler** model, consider **LIME**, as it excels in providing clear insights. Choose **SHAP** for **complex models**, including deep neural networks or ensemble methods, to gain both local and global interpretability.

Question 38

LIME vs SHAP task complexity

Answer 38

For **simpler models** where localized interpretability suffices, **LIME** is suitable. The effectiveness of LIME provides clear insights for **individual predictions**. Use **SHAP** for **complex ML models** as it offers a **broader perspective** on feature contribution.

Question 39

LIME vs SHAP Stability and consistency

Answer 39

**LIME** might exhibit *instability due to random sampling*, which makes it less consistent across runs. If a *stable and consistent* interpretation is crucial, particularly in sensitive applications, **SHAP** is preferred.