Domain 4

Question 1

Responsible AI

Answer 1

is a set of guidelines and principles to ensure that AI systems operate in a safe, trustworthy, and ethical manner

Question 2

Fairness

Answer 2

aims to ensure that models treat everyone equitably and impartially, regardless of their age, where they live, their gender, or their ethnicity

Question 3

It’s important to be able to explain in human terms why a model made a particular decision,

Answer 3

Explainability

Question 4

Transparency

Answer 4

is about providing clear information about model capabilities, limitations, and potential risks to stakeholders. Transparency includes making sure that users know when they are interacting with AI.

Question 5

Fairness of a model is measured by the bias and variance of outcomes across different groups.

Answer 5

the bias and variance of outcomes across different groups.

Question 6

Overfitting becomes a problem when

Answer 6

the training dataset is not representative of the real world. As a result, the model only performs well on inputs that resemble the training data.

Question 6

Underfitting can occur for some groups when

Answer 6

there wasn’t enough training data that matched their characteristics, so the model doesn’t perform well for them.

Question 7

Class imbalance occurs when

Answer 7

a feature value has fewer training samples when compared with another value in the dataset. In this example, the feature for sex shows that women constitute 32.4% of the training data, whereas men constitute 67.6%.

Question 8

These are crucial for conducting periodic reviews of datasets to identify and address potential issues or biases

Answer 8

Regular Audits

Question 9

Consider using these as a starting point to reduce the amount of training that your model needs, reducing your environmental impact and sustainability

Answer 9

already-trained model. Reuse of existing work is the key principle of sustainability

Question 10

Transparency is

Answer 10

about providing clear information about model capabilities, limitations, and potential risks. It also means making sure that users know when they are using AI.

Question 11

Accountability

Answer 11

means establishing clear lines of responsibility for AI model outcomes and decision making.

Question 12

Biases

Answer 12

are imbalances in data, or disparities in the performance of a model across different groups.

Question 13

SageMaker Clarify helps you mitigate bias by

Answer 13

detecting potential bias during the data preparation, after model training, and in your deployed model, by examining specific attributes.

Question 14

SageMaker Clarify can improve explainability by

Answer 14

looking at the inputs and outputs for your model, treating the model itself as a black box. By making these observations, it determines the relative importance of each feature.

Question 15

How does SageMaker Clarify evaluate bias, etc?

Answer 15

SageMaker Clarify examines your dataset and model by using processing jobs. A SageMaker Clarify processing job uses the SageMaker Clarify processing container to interact with an Amazon S3 bucket. The S3 bucket would contain your input datasets, and a model that is deployed to a SageMaker inference endpoint. The SageMaker Clarify processing container obtains the input data set and configuration for analysis from an S3 bucket. For feature analysis, the SageMaker Clarify processing container sends requests to the model container, and retrieves model predictions from the response from the model container. After that step, the processing container computes and saves analysis results to the S3 bucket. These results include a JSON file with bias metrics, and global feature attributions, a visual report, and additional files for local feature attributions. You can download the results from the output location and view them.

Question 16

SageMaker Clarify: The difference in positive proportions in predictions metric indicates whether

Answer 16

the model predicts positive outcomes differently for each class.

Question 17

specificity measures

Answer 17

how often the model correctly predicts a negative outcome

Question 18

The recall difference metric is

Answer 18

the difference in recall of the model between two classes. Any difference in these recalls is a potential form of bias.

Question 19

The accuracy difference metric

Answer 19

is the difference between the prediction accuracies for different classes.

Question 20

The treatment equality is

Answer 20

the difference in the ratio of false negatives to false positives. Even if the accuracy of the model is the same for two classes, this ratio could have differences. A difference in the type of errors that occur for different classes can constitute bias

Question 21

guardrails in Amazon Bedrock

Answer 21

You can use guardrails to define threshold for content filters for hate, insults, sexual content, or violence. You can also block topics altogether. For these topics, you can use plain text to describe the topics that should be denied.

Question 22

Guardrails can be set on both

Answer 22

the prompt and the model response, so if a prompt passes the guardrail, the response can still be blocked

Question 23

Another feature in SageMaker Clarify is the ability to run evaluation jobs of large language models so that

Answer 23

you can compare models

Question 24

SageMaker Clarify evaluation jobs can run four different types of tasks, including

Answer 24

text generation, text classification, question and answering, and text summarization.

Question 25

The five different dimensions of the four different types of SageMaker Clarify evaluation jobs

Answer 25

Prompt stereotyping measures the probability of your model including biases in its response. It includes biases for race, gender, sexual orientation, religion, age, nationality, disability, physical appearance, and socioeconomic status. Toxicity checks your model for sexual references, rude, unreasonable, hateful, or aggressive comments, profanity, insults, flirtations, attacks on identities and threats. Factual knowledge checks the veracity of the model responses. Semantic robustness checks whether your model output changes because of keyword typos, random changes to uppercase, and random additions or deletions of white spaces. Accuracy compares the model output to the expected responses, such as classifying and summarizing the data correctly.

Question 26

A model’s transparency measures

Answer 26

the degree to which ML owners and stakeholders can understand how a model works and why it produces its outputs. A model that is highly transparent uses an algorithm that is straightforward to interpret, such as linear regression.

Question 27

Transparency has two measures

Answer 27

interpretability and explainability.

Question 28

Explainability is

Answer 28

being able to describe what a model is doing without knowing exactly how. It treats the model as a black box, so every model can be observed and explained

Question 29

With interpretability

Answer 29

you can document how the inner mechanisms of the model impact the output,

Question 30

tradeoffs when choosing a model with high transparency.

Answer 30

These tradeoffs are performance and security.

Question 31

Transparent AI models are more susceptible to attacks because

Answer 31

hackers have more information about the inner mechanisms and can find vulnerabilities in the model.

Question 32

AI service cards are

Answer 32

a form of responsible AI documentation. They provide customers with a single place to learn about the intended use cases, limitations, responsible AI design choices, and deployment and performance optimization best practices,

Question 33

AI service cards currently exist for several AWS AI service APIs.

Answer 33

These APIs include matching faces with Amazon Rekognition, analyzing IDs with Amazon Textract, detecting PII with Amazon Comprehend and more. There is also an AI service card for Amazon’s foundation model in Amazon Bedrock, Amazon Titan Text

Question 34

Is there a tool that helps you create your own AI service cards build on AWS?

Answer 34

for models that you create, you can use SageMaker Model Cards to help document the lifecycle of a model from designing, building, training, and evaluation.

Question 35

Discuss two capabilities of SageMaker Clarify around responsible AI

Answer 35

SageMaker Clarify model processing jobs can also report on explainability. SageMaker Clarify provides feature attributions based on the concept of Shapley values. You can use Shapley values to determine the contribution that each feature made to the model predictions. Another type of analysis available in SageMaker Clarify is a partial dependence plot. This plot shows you how a model’s predictions changes for different values of a feature. In this case, it looks at age.

Question 36

Human-centered AI refers to

Answer 36

designing AI systems that prioritize the needs and values of humans. In human-centered AI, designers and developers engage in interdisciplinary collaboration and often involve psychologists, ethicists, and domain experts to collect diverse perspectives and expertise. Users are involved in the development process to make sure that the AI will be genuinely beneficial and user-friendly.

Question 37

Amazon Augmented AI or Amazon A2I, incorporates human review for samples of the inferences made by an AWS AI service or a custom model.

Answer 37

You can configure Amazon A2I to send inferences with low-confidence scores to human reviewers before sending them to the client. Their feedback can then be added to training data to re-train the model. Besides reviewing low-confidence inferences, you can have the human reviewers review random predictions as a way to audit the model. With Amazon A2I, you can use a pool of reviewers in your own organization or use Mechanical Turk. You can figure how many reviewers need to review each prediction.

Question 38

Reinforcement learning from human feedback, or RLHF,

Answer 38

is an industry standard technique for ensuring that large language models produce content that is truthful, harmless, and helpful.

Question 39

To use RLHF, you train a separate model which serves as a reward model.

Answer 39

The reward model is trained by humans who review multiple responses from the large language model for the same prompt and indicate their preferred response. Their preferences become the training data for the reward model, which when trained, can predict how high a human would score a prompt response. The large language model then uses the reward model to refine its responses for maximum reward.

Question 40

Collecting the preferences from humans for RLHF can be accomplished most readily with

Answer 40

SageMaker Ground Truth