12. Explainable AI - AI II Flashcards
Can a Multi-Layer Perceptron (MLP) with one hidden layer solve the following logic functions?
a) AND
b) OR
c) XOR
Answer: a, b, and c
MLPs can solve all Boolean logic functions, including AND, OR, and XOR, which are important for non-linearly separable problems.
How many weights (excluding biases) does a fully connected feedforward neural network have with:
4 input nodes
4 hidden nodes
2 output nodes
Options:
a) 32
b) 9
c) 18
d) 24
Answer: d) 24
Formula: (4×4) + (4×2) = 24
The first term (4×4) represents connections between input and hidden layer.
The second term (4×2) represents connections between hidden and output layer.
Why do we need an MLP (Multi-Layer Perceptron) to solve the XOR problem?
A single-layer perceptron (SLP) cannot solve XOR because XOR is not linearly separable.
An MLP with a hidden layer can learn non-linear decision boundaries using activation functions and hidden neurons.
The hidden layer transforms input space into a linearly separable feature space.
Why does an MLP need a hidden layer for solving the XOR function?
Complete the missing values in this truth table.
Why is an MLP needed for XOR?
A Single-Layer Perceptron (SLP) can only separate inputs linearly.
XOR is not linearly separable, meaning you cannot draw a straight line to separate the outputs.
An MLP adds a hidden layer, which allows it to transform the data into a space where XOR becomes linearly separable.
How does the hidden layer work?
The hidden layer has two neurons (h1 and h2) that act as intermediate steps.
The network learns weights to transform the XOR problem into a solvable format.
Step-by-Step: How It Works
Inputs (x1, x2) go into the hidden layer (h1, h2).
The hidden layer learns patterns:
If one input is 1 → Activates one hidden neuron.
If both inputs are the same (0,0 or 1,1) → Activates both or none.
The output layer takes h1 & h2 and calculates y:
If only one hidden neuron is active, the output is 1 (which matches XOR).
If both or none are active, the output is 0.
For a Convolutional Neural Network (CNN) consisting of 5 feature maps with a filter size of 3×3, how many weights are there in the first layer?
Each feature map requires a 3×3 = 9 weight filter. Since there are 5 feature maps, the total number of weights is:
5 × 9 = 45
Correct answer: b) 45
How many neurons are in a feature map in the first layer of a Convolutional Neural Network (CNN) when using a 3×3 filter on a 5×5 image, with stride = 1?
Step-by-Step Explanation:
Understanding the filter movement:
A filter (or kernel) is a small window (3×3 in this case) that moves over the input image.
The filter starts at the top-left corner and slides across the image.
Stride = 1 means the filter moves one pixel at a time in both directions.
How to calculate output size (feature map size):
The formula to calculate the output feature map size is:
Outputsize=(Imagesize−Filtersize)+1 Applying it to our case: (5−3)+1=3
This means the output feature map is 3×3.
Total neurons in the feature map:
Since each position in the 3×3 feature map corresponds to a neuron, the total number of neurons is:
3×3=9
Correct answer: a) 9 neurons
How many neurons per layer and in total does a CNN have if given the following specifications?
Input image size: 7×7
5 input filters of 2×2
Feature map stride: 1
Pooling dimension: 2×2
Pooling stride: 2
Classes: 3
Feature map calculation:
Filter size 2×2, stride 1, input 7×7
Output feature map size: (7−2)/1 + 1 = 6 × 6
Each feature map has 6×6 = 36 neurons
5 feature maps, so total neurons in this layer: 5 × 36 = 180
Pooling layer calculation:
Pooling reduces 6×6 → 3×3
Total pooling layer neurons: 5 × 9 = 45
Fully connected layer:
One neuron per class → 3 neurons
Total neurons:
180 (conv) + 45 (pooling) + 3 (fc) = 228 neurons
Final Answer: 228 neurons
Which of the following non-derivative functions can only produce outputs in the range 0 to 1?
a) Linear activation
b) ReLU activation
c) Sigmoid function
d) Softmax function
Correct answers: c) Sigmoid function & d) Softmax function
Explanation:
The sigmoid function squeezes any input into a range between 0 and 1, making it useful for probability-based tasks.
The softmax function does something similar but for multiple outputs—ensuring that all output values sum to 1, which is perfect for classification problems.
Linear activation can produce values outside the 0-1 range.
ReLU activation outputs 0 for negative inputs and the same value for positive inputs, meaning it can go beyond 1.
How many weights (excluding biases) does a feedforward fully connected neural network with:
5 input nodes
3 hidden nodes
1 output node
No biases
have?
Correct answer: b) 18
Explanation:
Input to Hidden Layer:
Each of the 5 input nodes connects to 3 hidden nodes.
That means 5 × 3 = 15 weights.
Hidden to Output Layer:
Each of the 3 hidden nodes connects to 1 output node.
That means 3 × 1 = 3 weights.
Total weights:
15 (input to hidden) + 3 (hidden to output) = 18.
Which of the following are considered activation functions in neural networks?
a) Step function (Hard limiter)
b) Softmax function
c) Boolean logic function
d) Dropout
Correct answers: a) Step function & b) Softmax function
Explanation:
Step function (Hard limiter) is a basic activation function that outputs either 0 or 1 depending on whether the input passes a certain threshold.
Softmax function converts values into probabilities (between 0 and 1) so that they sum to 1, making it useful in classification problems.
Boolean logic functions (like AND, OR, XOR) are not activation functions; they are logic gates used in digital circuits.
Dropout is not an activation function; it’s a regularization technique that prevents overfitting by randomly ignoring some neurons during training.
Why are deep learning models difficult to trust?
Deep learning models act as black boxes, meaning we can’t see exactly how they make decisions.
They use millions of parameters that are hard to interpret.
Lack of transparency makes it difficult to understand, debug, or justify their predictions.
This is critical for high-stakes fields like healthcare and finance.
In what applications is XAI especially important?
Healthcare → Doctors need to trust AI diagnoses.
Finance → Loan approvals must be fair and transparent.
Legal System → AI predictions must be explainable in court.
Autonomous Vehicles → Understanding why a car made a decision is crucial for safety.
What are data explainers?
AI models rely on data, so understanding the dataset is crucial.
Poorly understood data can lead to bias and errors.
Example:
An imbalanced dataset can cause AI to favor one class over another.
Checking feature distributions (e.g., histograms) helps improve performance.
How do we explain AI models?
Deep learning is often a black box.
Model explainability techniques help visualize how AI makes decisions.
Example: Heatmaps show which parts of an image influence classification.
Helps make AI models interpretable and trustworthy.
What does Explainable AI (XAI) mean?
XAI aims to make AI model decisions understandable to humans.
Focuses on:
Comprehensibility → Can stakeholders understand it?
Interpretability → Does the model make logical sense?
Helps build trust, fairness, and transparency in AI systems.
How can dataset issues impact AI performance?
AI models perform poorly if trained on biased or unbalanced data.
Example: If certain classes are underrepresented, the model may struggle to predict them.
Solution: Analyze dataset distributions and apply balancing techniques.
What is the difference between intrinsic and post hoc model explainability?
Intrinsic Explainability: The model is designed to be both high-performing and explainable from the start (e.g., decision trees, interpretable deep learning).
Post Hoc Explainability: The model is first trained, and then explainability techniques (XAI) are applied to interpret its predictions.
Key idea: Intrinsic models are built for transparency, while post hoc techniques explain already-trained models.
What is the difference between global and local explainability in AI models?
Global Explainability: Describes how the entire model makes decisions across all inputs.
Example: Feature importance analysis across the whole dataset.
Local Explainability: Explains individual predictions for specific data points.
Example: Why did the model classify a specific image as a cat?
Key idea: Global gives an overview of model behavior, while local focuses on single decisions.
What are different ways AI models can explain their decisions?
Visual explanations - Highlight important areas in images (e.g., heatmaps).
Mathematical/Computational explanations - Use logic functions and probability for developers.
Language-based explanations - Use natural language to describe model reasoning for end-users.
Key idea: Different users need different types of explanations.
How do we visualize what a Convolutional Neural Network (CNN) has learned?
CNN filters detect patterns (edges, textures, objects).
Feature maps highlight important parts of an image.
Pooling layers reduce dimensions but keep key features.
Fully connected layers combine learned features for final classification.
Key idea: CNNs break images into patterns, layer by layer, to recognize objects.
How can we visualize what an Artificial Neural Network (ANN) is representing?
In the first layer, visualizing the values of the weights is straightforward.
Deeper layers require more advanced techniques, as simply showing weights between layers may not be meaningful.
One approach is to visualize the entire receptive field of neurons in later layers.
Techniques like DeepDream can generate images that show what a neuron is “looking for.”
What is DeepDream, and how does it explain neural networks?
Post hoc method: Applied after training to understand what neurons have learned.
Global explainability: Shows representations for entire neurons/layers.
Model-specific: Mostly used for CNNs.
It generates images that highlight patterns a neuron is detecting in data.
Each square in DeepDream output represents what different feature maps in a deep layer are focusing on.
What is SHAP (Shapley Additive Explanations), and how does it work?
Post hoc method: Analyzes a trained model.
Local & Global: SHAP can explain individual predictions (local) and feature importance across datasets (global).
Model-agnostic: Works with any AI model, not dependent on architecture.
Shows how much each feature contributes to a decision using heatmaps (e.g., highlighting pixels in an image that influenced classification).
How does SHAP help in interpreting AI predictions?
Highlights which parts of an image (or other data) were important for a prediction.
Helps in understanding why a model classified an object a certain way.
Provides transparency in model decision-making.
Useful in debugging and improving AI models.
What is LRP (Layer-wise Relevance Propagation), and how does it explain AI models?
Post hoc: Applied after training to evaluate feature relevance.
Local explainability: Shows how individual data points affect predictions.
Model-specific: Traces relevance from output back to input, assigning importance to features.
Uses heatmaps to visualize relevant parts of an image that influenced predictions.
How does the DeepDream algorithm function?
Developed by Google to visualize what neurons are looking for.
Creates images based on neuron activation.
Learns to generate images that maximally activate specific neurons.
Helps in understanding what features a network prioritizes.
Why train a CNN on faces?
Each face has slight variations (position, shading).
CNN learns to recognize patterns despite these variations.
Deeper networks become more invariant to small changes.
What does each convolution feature map do?
Detects the presence of a specific feature in different parts of an image.
Example: A filter may highlight edges, textures, or facial features.
Feature maps encode where and how strongly a feature appears.
How do CNNs process images?
Convolution Layers - Extract patterns (edges, textures).
Pooling Layers - Reduce data size while keeping important info.
Fully Connected Layers - Use extracted features to classify the image.
What do CNN Filters Look For?
Each filter detects a unique feature.
Early layers detect simple shapes (lines, edges).
Middle layers identify more complex patterns (eyes, noses).
Deep layers recognize full objects (faces, cars).
CNNs stack these layers to build an understanding of images!
What is the Receptive Field in CNNs?
The receptive field is the area of the image a neuron responds to.
Early layers: Small receptive field (detects local features).
Deep layers: Large receptive field (detects entire objects).
Some deep layers already recognize abstract face-like features!
What happens when we use DeepDream?
DeepDream exaggerates the features a neuron is looking for.
The algorithm maximally activates a neuron using an input image.
This creates highly detailed, sometimes surreal images!
How does DeepDream work?
Starts with a random image.
Finds the neuron of interest.
Modifies the image to maximize neuron activation.
Produces an exaggerated, artistic version of the features that the neuron recognizes.
What does the Softmax layer do?
The final layer in classification networks.
Assigns a probability score to each possible class.
Example: A CNN trained on faces assigns high activation to neurons representing different face classes.
What happens when DeepDream is applied to an unfamiliar image?
If a model is trained to recognize dog breeds and DeepDream is applied to a human face, the output may resemble a hybrid of human and dog-like features.
The model amplifies the features it has learned, even when they don’t naturally exist in the input image.
Key takeaway: DeepDream doesn’t create realistic images but exaggerates learned features, sometimes leading to surreal or bizarre outputs.
Can you train on DeepDream outputs?
Yes! You can input an image into DeepDream and then train a model on the generated patterns.
This might help the model better recognize specific features or artistic styles.
What Happens if We Use DeepDream on a New Image?
If you apply DeepDream to a face or object:
The model amplifies existing patterns it has learned.
Example: If trained on dog breeds and applied to a human face, it might blend dog-like patterns into the face!
What is the key principle behind SHAP?
SHAP is based on cooperative game theory, evaluating how each feature contributes to a model’s prediction by considering all possible feature combinations.
Why can SHAP be computationally expensive?
It evaluates all feature combinations, which can be demanding, especially for large images.
How can SHAP be used to determine which pixels matter most?
By using super pixels, grouping similar pixels together to determine their collective importance in the model’s prediction.
What is LRP and how does it work?
LRP is a local explanation method that backpropagates relevance from the output to the input layer, generating a heatmap showing which parts of an image contributed most to the model’s decision.
When is LRP useful?
LRP is useful for single-image explanations, helping interpret CNN decisions.
What are some limitations of LRP?
Needs interpretation—it finds relevant pixels but doesn’t explain why.
Can detect misleading patterns (e.g., “Clever Hans” effects where models focus on unintended cues).
What is disentanglement in CNN-based models?
Disentanglement methods clarify what concepts feature maps encode in a trained CNN-based model.
What is Concept Relevance Propagation (CRP)?
CRP is a method that builds on Layer-wise Relevance Propagation (LRP) to find the most relevant channels for a model’s prediction.
How does CRP refine feature relevance compared to LRP?
CRP backpropagates relevance values from specific layers (conditioned on the predicted class) and generates heatmaps for the most relevant channels.
What are the two main types of CRP analysis?
Local analysis: Identifies where relevant features appear in a specific image.
Global analysis: Identifies what features are encoded in the trained network and their relevance to the prediction.
What is ShapGAP, and why is it useful?
ShapGAP is a metric that quantifies the difference between explanations from multiple models, helping compare interpretability across models.
Why do different models give different explanations?
Different architectures (e.g., YOLO vs. SSD) or different training versions of the same model can result in varying feature importance and explanations.
What are some alternative XAI (Explainable AI) approaches?
LIME: Local interpretability using perturbed samples.
Prototypes: Uses reference examples to explain model predictions.
Surrogate models: Simplifies complex models for better interpretability.
What are the key takeaways from Explainable AI (XAI)?
XAI makes deep learning models more interpretable.
It helps build trust in model predictions.
Different approaches exist:
Model-specific vs. model-agnostic
Intrinsic vs. post-hoc
Visual vs. non-visual
XAI itself can still make mistakes.
