Path7.Mod1.c - Responsible AI Dashboard - Evaluate the RAI Dashboard

Question 1

EA Exp Cfs CA

Depending on the Components selected, the Insights you’ll see on your dashboard

Answer 1

• Error Analysis
• Explanations
• Counterfactuals
• Causal Analysis

Question 2

Describe Error Analysis

Answer 2

You can review and understand how Errors are distributed in your dataset i.e. are there more errors with cohorts representing Women vs Men, Blacks vs Whites, when the person is disabled, etc.

Question 3

ETM EHM

Two visual representations for Error Analysis

Answer 3

Error Tree Map: Explore which combinations of cohorts results in more false predictions

Error Heat Map: Grid overview of a model’s errors over the scale of selected features (ie. the relationship of features and their values and the % error yield per combination)

Question 4

The dumb name she gave the entity from that Haunted episode…

• Describe Explanations w.r.t. Model Explainers
• Name the most common explainer used and what it does

Answer 4

• When you want to understand and calculate Feature Importance ie how much a feature influenced the final prediction. You can use various Model Explainers to calculate it.
• The most common explainer is the mimic explainer. It trains an interpretable model using the same data and task.

Question 5

AFI IFI

Two types of Feature Importance available in the Feature Importance Bar Chart

Answer 5

Aggregate Feature Importance: How each feature in the test data influences the model’s prediction overall

Individual Feature Importance: How each feature in the test data impacts an individual prediction. When selected, you are presented a grid of all individual data points that were scored. Select one or many and you’ll see to what degree the features influenced those datapoint’s result

In other words, Overall influence vs Individual prediction influence

Question 6

Marvel’s….

Describe Counterfactuals

Answer 6

What-If scenarios i.e. how a model’s output changes based on a change in the input, that provide deeper insight into feature importance.

Select a datapoint and the desired model’s prediction for that point. Create a What-If Counterfactual in the panel to discover your desired prediction for differing input.

Question 7

“Mean Feature Importance”

Describe Causal Analysis

Answer 7

The use of statistical techniques to estimate the average effect of a feature on a desired prediction.

Question 8

ACE ICE TP

Three available Causal Analysis tabs in the RAI Dashboard

Describe what a Treatment Feature is

Answer 8

• Aggregate Causal Effects: Average Causal Effects for predefined Treatment Features
• Individual Causal Effects: Change individual Treatment Features to explore their influence on a prediction
• Treatment Policy: What parts of your data points benefit most from Treatment

Treatment Features - Features you want to change to optimize predictions