Domain 4: Responsible AI Guidelines 14%

Question 1

_____ is a set of guidelines and principles to ensure that AI systems operate in a safe, trustworthy, and ethical manner.

Answer 1

Responsible AI

Question 2

_____ aims to ensure that models treat everyone equitably and impartially, regardless of their age, where they live, their gender, or their ethnicity.

Answer 2

Fairness

Question 3

To explain in human terms why a model made a particular decision, which is known as _____.

Answer 3

Explanibility

Question 4

_____ makes sure that AI systems are tolerant of failures and minimize errors.

Answer 4

robustness

Question 5

_____ are primarily about protecting user privacy, and not exposing personal identifiable information, or PII.

Answer 5

Privacy and security

Question 6

_____ is about meeting and auditing compliance with industry standards and best practices, including estimating properly and mitigating risk.

Answer 6

Governance

Question 7

_____ is about providing clear information about model capabilities, limitations, and potential risks to stakeholders; it includes making sure that users know when they are interacting with AI.

Answer 7

Transparency

Question 8

Fairness avoids perpetuating or amplifying _____ and _____ through AI systems.

Answer 8

societal biases and discrimination

Question 9

Fairness of a model is measured by _____.

Answer 9

the bias and variance of outcomes across different groups

Question 10

_____ becomes a problem when the training dataset is not representative of the real world. As a result, the model only performs well on inputs that resemble the training data.

Answer 10

Overfitting

Question 11

_____ can occur for some groups when there wasn’t enough training data that matched their characteristics, so the model doesn’t perform well for them.

Answer 11

Underfitting

Question 12

One of the main reasons for model bias is _____, which occurs when a feature value has fewer training samples when compared with another value in the dataset.

Answer 12

class imbalance

Question 13

What is the foundation of responsible AI?

Answer 13

Ethical datasets

Question 14

_____ is about representing diverse populations, perspectives, and experiences in our training data.

Answer 14

Inclusivity

Question 15

_____ means incorporating a wide range of attributes, features, and variables to avoid bias.

Answer 15

Diversity

Question 16

_____ involve obtaining informed consent from data subjects and providing clear information about data usage.

Answer 16

Consent and transparency

Question 17

What ethical practices/factors must be considered when choosing a model?

Answer 17

1. Environmental impact
2. Sustainability
3. Transparency
4. Accountability
5. Stakeholder engagement

Question 18

_____ means establishing clear lines of responsibility for AI model outcomes and decision making.

Answer 18

Accountability

Question 19

_____ are imbalances in data, or disparities in the performance of a model across different groups.

Answer 19

Biases

Question 20

_____ helps you mitigate bias by detecting potential bias during the data preparation, after model training, and in your deployed model, by examining specific attributes. It also improves explainability by looking at the inputs and outputs for your model, treating the model itself as a black box.

Answer 20

SageMaker Clarify

Question 21

T/F: SageMaker Clarify can understand the basis for how deep learning models are making predictions without understanding the inner workings.

Answer 21

True

Question 22

SageMaker Clarify examines your dataset and model by using processing jobs, which uses the SageMaker Clarify processing container to interact with an Amazon S3 bucket, which contains your input datasets and a model that is deployed to _____.

Answer 22

a SageMaker inference endpoint

Question 23

The SageMaker Clarify processing container obtains _____ and _____ from an S3 bucket.

Answer 23

the input data set and configuration for analysis

Question 24

For _____, the SageMaker Clarify processing container sends requests to the model container, and retrieves model predictions from the response from the model container.

Answer 24

feature analysis

Question 25

What analysis results are saved to the S3 bucket by the processing container by SageMaker Clarify?

Answer 25

1. JSON file w/ bias metrics
2. visual report
3. additional files for local feature attributions

Question 26

The _____ metric indicates whether a particular class has a larger proportion of the rejected outcomes in the dataset than the accepted outcomes.

Answer 26

demographic disparity

Question 27

The _____ metric indicates whether the model predicts positive outcomes differently for each class. This metric can be compared with the label imbalance in the training data. The goal is to see whether the bias in positive proportions changes after the training, or whether the bias is also present in the data.

Answer 27

difference in positive proportions in predictions

Question 28

_____ measures how often the model correctly predicts a negative outcome.

Answer 28

specificity

Question 29

The _____ metric is the difference in recall of the model between two classes. Any difference in these recalls is a potential form of bias.

Answer 29

recall difference

Question 30

The _____ metric is the difference between the prediction accuracies for different classes. This result can occur when the data contains class imbalance.

Answer 30

accuracy difference

Question 31

The _____ is the difference in the ratio of false negatives to false positives.

Answer 31

treatment equality

Question 32

_____ is the result of the AI model attempting to fill in the gaps when something is missing in its training data.

Answer 32

Hallucination

Question 33

T/F: Data privacy is a risk because sensitive data that makes its way into a large language model can leak and be incorporated into its output.

Answer 33

True

Question 34

T/F: Fortunately, for foundation models in Amazon Bedrock, you can configure guardrails to filter and block inappropriate content using plaintext to describe topics that should be denied.

Answer 34

True

Question 35

What are the 4 types of tasks that can be run in SageMaker Clarify? These tasks can also be run in what?

Answer 35

1. text generation
2. text classification
3. question/answer
4. text summarization

Bedrock

Question 36

_____ measures the probability of your model including biases in its response. It includes biases for race, gender, sexual orientation, religion, age, nationality, disability, physical appearance, and socioeconomic status.

Answer 36

Prompt stereotyping

Question 37

_____ checks your model for sexual references, rude, unreasonable, hateful, or aggressive comments, profanity, insults, flirtations, attacks on identities and threats.

Answer 37

Toxicity

Question 38

_____ checks the veracity of the model responses.

Answer 38

Factual knowledge

Question 39

_____ checks whether your model output changes because of keyword typos, random changes to uppercase, and random additions or deletions of white spaces.

Answer 39

Semantic robustness

Question 40

_____ compares the model output to the expected responses, such as classifying and summarizing the data correctly.

Answer 40

Accuracy

Question 41

A model’s _____measures the degree to which ML owners and stakeholders can understand how a model works and why it produces its outputs. How much of this that’s required often depends upon regulatory requirements so that consumers are protected against bias and unfairness.

Answer 41

transparency

Question 42

What are the two measures of transparency?

Answer 42

interpretability and explainability

Question 43

T/F: Linear regression would be on the simple end and neural network on the complex end.

Answer 43

True

Question 44

_____ is being able to describe what a model is doing without knowing exactly how. It treats the model as a black box, so every model can be observed and explained.

Answer 44

Explainability

Question 45

T/F: When starting a new AI or ML project, we need to know whether interpretability is a hard business requirement.

Answer 45

True

Question 46

T/F: With interpretability, you can document how the inner mechanisms of the model impact the output, but explainability does not consider the inner mechanisms.

Answer 46

True

Question 47

What are two tradeoffs you should consider when choosing a model w/ high transparency?

Answer 47

performance and security

Question 48

T/F: Transparent AI models are more susceptible to attacks because hackers have more information about the inner mechanisms and can find vulnerabilities in the model.

Answer 48

True

Question 49

Transparency might require sharing details about the data that is used to train the model, which raises concerns about _____.

Answer 49

data privacy

Question 50

Who services open source AI projects?

Answer 50

GitHub

Question 51

T/F: When using a fully trained model that’s hosted by AWS, you only interact with APIs and have no direct access to the model.

Answer 51

True

Question 52

_____ are a form of responsible AI documentation. They provide customers with a single place to learn about the intended use cases, limitations, responsible AI design choices, and deployment and performance optimization best practice.

Answer 52

AI service cards

Question 53

Match faces withm_____ , analyze IDs with _____ , and detect PII with _____.

Answer 53

Amazon Rekognition
Amazon Textract
Amazon Comprehend

Question 54

For models that you create, you can use _____ to help document the lifecycle of a model from designing, building, training, and evaluation.

Answer 54

SageMaker Model Cards

Question 55

T/F: When you create a model card, SageMaker autopopulates details about your SageMaker trained model in the card.

Answer 55

True

Question 56

You can use _____ to determine the contribution that each feature made to the model predictions.

Answer 56

Shapley values

Question 57

This plot shows you how a model’s predictions changes for different values of a feature.

Answer 57

partial dependence plot

Question 58

In _____, designers and developers engage in interdisciplinary collaboration and often involve psychologists, ethicists, and domain experts to collect diverse perspectives and expertise.

Answer 58

human-centered AI

Question 59

_____ incorporates human review for samples of the inferences made by an AWS AI service or a custom model. You can configure it to send inferences with low-confidence scores to human reviewers before sending them to the client. Their feedback can then be added to training data to re-train the model.

Answer 59

Amazon Augmented AI or Amazon A2I

Question 60

_____ is an industry standard technique for ensuring that large language models produce content that is truthful, harmless, and helpful.

Answer 60

Reinforcement learning from human feedback, or RLHF

Question 61

To use this technique, you train a separate model which serves as a reward model. The reward model is trained by humans who review multiple responses from the large language model for the same prompt and indicate their preferred response. Their preferences become the training data for the reward model, which when trained, can predict how high a human would score a prompt response. The large language model then uses the reward model to refine its responses for maximum reward.

Answer 61

RLHF

Question 62

Collecting the preferences from humans for RLHF can be accomplished most readily with _____.

Answer 62

SageMaker Ground Truth