lecture 1 Flashcards
What is the main objective of a deep reinforcement learning agent?
to learn a sequential decision-making task from experience in an environment to achieve specific goals
transitions in reinforcement learning
usually stochastic
- because the next state in the environment may depend on various factors beyond the agent’s control
observations (w) and actions (a)
- may be high-dimensional
- observations may not provide full knowledge of the underlying state because the agent only receives partial information about the environment ( w =/= x )
experience
- may be constrained
- no access to accurate simulator
- limited data
What is the ‘reality gap’ in reinforcement learning, and why is it a challenge?
- refers to the difference between the agent’s experience in a simulator and the real-world environment
- is a challenge because if the simulator is not accurate enough, the policies learned in simulation may not perform well in the real world
Why might an agent have limited access to data in reinforcement learning?
- Safety constraints: Real-world exploration can be risky in fields like robotics and healthcare, limiting data collection.
- Compute constraints: High computational costs may limit the ability to run simulations in some environments.
- Exogenous factors: In areas like weather prediction or financial markets, data is inherently limited due to reliance on external conditions beyond control.
How can the reality gap and limited data challenges be addressed in reinforcement learning?
- Develop an accurate simulator: A more accurate simulator reduces the reality gap, enabling better policy transfer to real-world environments. (reduces reality gap)
- Design the learning algorithm for generalization: Algorithms should be designed to improve generalization, allowing agents to perform well in unseen states or environments, even with limited training data.
What does generalization refer to in reinforcement learning?
- The capacity to achieve good performance in an environment where limited data has been gathered.
- The capacity to obtain good performance in a related but different environment by transferring learned knowledge.
How can an agent achieve generalization with limited data?
- Regularization: Prevents overfitting to specific training scenarios.
- Experience replay: Reuses past experiences during training.
- Exploration strategies: Ensures the agent gathers diverse experiences, improving its ability to generalize to unseen states.
What is transfer learning in reinforcement learning, and why is it important?
- Transfer learning in reinforcement learning involves training an agent in one environment and adapting it to a related but different environment.
- It is important because it allows the agent to reuse knowledge, improving its performance in new tasks.
What are common methods for transfer learning in reinforcement learning?
- Fine-tuning: Retrains the agent on a new environment while retaining previously learned knowledge.
- Multi-task learning: Trains the agent on multiple tasks simultaneously to encourage it to learn generalizable features.
- Domain adaptation: Modifies the agent’s policy or input representation to handle differences between the source and target environments.
What is a supervised learning algorithm?
- maps a dataset of learning examples into a predictive model
- assuming the dataset is independently and identically distributed (i.i.d.) and representative of the true underlying data distribution.
What are bias and variance in supervised learning?
- Bias: Represents how much the model’s predictions differ from the true function. High bias leads to underfitting, where the model is too simple to capture patterns.
- Variance: Represents how much the model’s predictions vary when using different subsets of the training data. High variance leads to overfitting, where the model fits the noise in the data.
What is the ideal model in terms of bias and variance?
- low bias and low variance.
- The goal is to balance bias and variance by tuning model complexity and using techniques like regularization and data augmentation.
How can variance (overfitting) be reduced in supervised learning?
Increasing the size of the dataset can help reduce variance by improving the model’s generalization to new data.
What is bias-variance decomposition?
Bias-variance decomposition describes how the total error of a model can be broken down into:
- Bias: The error due to incorrect assumptions in the model.
- Variance: The error due to sensitivity to the specific training data set used. (parametric variance = overfitting)
- Irreducible error: The inherent noise in the output that no model can eliminate. (internal variance)
What does the bias-variance decomposition highlight in reinforcement learning?
highlights a tradeoff between:
- Bias: Error directly introduced by the learning algorithm, leading to underfitting.
- Parametric variance: Error due to the limited amount of data available, leading to overfitting.
Why is direct bias-variance decomposition less straightforward for loss functions other than L2 loss in reinforcement learning?
- Overfitting arises from the sensitivity of the model’s predictions to variations in the training data, which manifests as variance in the loss function.
- With non-L2 loss functions, variance in the loss may not directly correspond to statistical variance due to the non-linear transformation introduced by the loss function.
How can prediction error be decomposed when using non-L2 loss functions?
prediction error can be decomposed into:
- A bias-related term representing the lack of expressivity of the model.
- A variance-related term representing the sensitivity to the limited amount of data.
Replacement for the bias-variance tradeoff in reinforcement learning
tradeoff between:
- A sufficiently rich learning algorithm to reduce model bias.
- A learning algorithm that is not too complex, to avoid overfitting to the limited amount of data.
What is a batch or offline reinforcement learning algorithm?
- Mapping a dataset D into a policy π_D
- This can be done independently of whether the policy is derived from a model-based or model-free approach.
How can the suboptimality of the expected return in an MDP be decomposed?
- Asymptotic bias: Bias in the policy due to limitations of the learning algorithm, even when given an infinite amount of data.
- Overfitting error: Error due to the finite size of the dataset, leading to overfitting.
What is the bias-overfitting tradeoff in reinforcement learning?
- On one side of the scale is data, which represents the percentage of error due to overfitting. This decreases with more data.
- On the other side is policy class complexity, which represents the percentage of error due to asymptotic bias. Increasing policy class complexity reduces bias but increases the risk of overfitting when data is limited.
How can the best policy be obtained in reinforcement learning?
The best policy can be obtained by balancing bias and overfitting through:
- Using a sufficiently expressive policy class to reduce bias.
- Ensuring the dataset is large and diverse enough to prevent overfitting.
What are the key strategies for improving generalization in reinforcement learning?
- Using an abstract representation: Create a compact and informative state representation by discarding irrelevant or redundant features.
- Optimizing the objective function: Properly shape the reward function and tune key parameters (e.g., reward shaping, tuning the discount factor 𝛾).
- Choosing the appropriate learning algorithm: Select the right type of function approximator (e.g., neural networks) and decide between model-based and model-free approaches.
- Improving the dataset: Enhance the diversity of the dataset using better exploration strategies to improve generalization and address the exploration/exploitation dilemma.
How does using an abstract representation help in reinforcement learning?
reduces overfitting and improves generalization by focusing only on essential features, avoiding spurious correlations caused by irrelevant features
Why is the appropriate level of abstraction important in the bias-overfitting tradeoff?
- helps balance bias and overfitting
- A small but rich abstract representation allows for improved generalization by focusing on essential information and reducing unnecessary complexity.
What are the potential issues of including too many irrelevant features in the state representation?
- Overfitting: The RL algorithm may pick up spurious correlations.
- Increased variance: Adds complexity without meaningful improvement in performance.
How does removing irrelevant or redundant features affect bias and variance?
- Removing irrelevant or redundant features helps reduce overfitting (variance) but may introduce some bias.
- This is because removing features that differentiate states with different roles in the dynamics can reduce the model’s expressivity.
How can modifying the objective function improve the policy learned by a deep RL algorithm?
- by introducing a bias that helps the agent generalize better
- This involves optimizing an objective function that may diverge from the true objective but accelerates learning and improves performance in practice.
What are the main approaches for modifying the objective function in reinforcement learning?
-
Reward shaping:
- Adds an auxiliary reward signal to help the agent learn faster.
- Can accelerate learning but requires careful design to avoid misleading the agent with rewards that deviate too far from the true objective.
- Reward normalization is important for deep RL. -
Tuning the discount factor 𝛾:
- Determines how much the agent values future rewards relative to immediate rewards.
- Proper tuning balances short-term and long-term objectives, improving the agent’s generalization ability across tasks.
How does the discount factor 𝛾 influence reinforcement learning?
- Lower 𝛾: The agent focuses more on immediate rewards, which is useful for tasks where quick feedback is essential or long-term planning is less critical.
- Higher 𝛾: The agent prioritizes long-term cumulative rewards, which is important in environments where future rewards are more significant.
Proper tuning helps balance the tradeoff between short-term and long-term objectives.
What is the recommended approach to tuning the discount factor 𝛾 during training?
- to start with a low value of 𝛾 and gradually tune it during training.
- In the given case, a discount factor of 1 is used with a finite horizon at test time.
What are the three main components that an RL agent may include?
- Value function representation: Predicts the expected return for each state or state-action pair, indicating how good a state or action is.
- Policy representation: Directly maps states to actions.
- Model of the environment: Predicts the next state and reward, which can be used for planning in model-based reinforcement learning.
What is the difference between model-free and model-based reinforcement learning approaches?
- Model-free approaches: Directly learn the policy or value function without building a model of the environment.
- Model-based approaches: Use a learned model of the environment to plan actions, typically leading to more sample-efficient learning.
What role do function approximators play in deep reinforcement learning?
- Function approximators, such as neural networks, map inputs (states) to outputs (values or policies).
- i.e., used to estimate value functions or policies in deep reinforcement learning.
- They are crucial for enabling the agent to generalize and determine how features are treated at higher levels of abstraction.
How does the choice of function approximator affect generalization in reinforcement learning?
- The choice of function approximator determines the level of abstraction introduced by deep learning, which depends on the network architecture and feature selection mechanisms (e.g., attention mechanisms).
- Depending on the task, choosing between model-free or model-based approaches is critical, as model-based methods are more sample-efficient but require accurate models, while model-free methods are often more robust in complex environments without a model.
What is the parallel between RL algorithms and human cognition as described by Daniel Kahneman?
- System 1: Fast, instinctive, and automatic decision-making, similar to model-free RL, where decisions are based on learned patterns.
- System 2: Slow, deliberate, and logical decision-making, similar to model-based RL, where decisions involve planning based on a model of the environment.
What is transfer learning in the context of reinforcement learning?
- leveraging knowledge acquired in one task or environment to improve performance in a different but related task or environment
- involves using a pre-trained policy or value function in a different setting that may share similar dynamics but differ in visual representation, reward structure, or specific states
Why is exploration critical in reinforcement learning, and what is the main challenge associated with it?
- Exploration is critical because the agent needs to gather sufficient information about the environment to find an optimal policy.
- The main challenge is balancing exploration (trying new actions) and exploitation (choosing known best actions)
- Too much exploitation can lead to suboptimal solutions.
- Too much exploration can delay convergence to an optimal policy.
What is undirected exploration in reinforcement learning?
- methods that do not prioritize specific areas of the environment based on uncertainty or potential rewards
- ex: epsilon-greedy
What is directed exploration in reinforcement learning, and when is it useful?
- aims to explore areas with high novelty or uncertainty
- useful when:
- When rewards are not sparse, uncertainty in the value function can guide exploration.
- When rewards are sparse, explicit exploration rewards may be used.
- ex: UCB, thompson sampling
What is the role of learning algorithms in reinforcement learning?
- govern how the agent updates its function approximators using data from interactions with the environment.
- Examples include Q-learning and policy gradients.
What is replay memory, and why is it important in reinforcement learning?
stores past experiences (state, action, reward, next state) to break temporal correlations in training data, thereby improving sample efficiency and stability during training.
What are controllers in reinforcement learning?
Controllers handle meta-level operations such as training, validation, and testing phases, as well as hyperparameter tuning.
How do policies influence an RL agent’s behavior?
- define how the agent chooses actions, balancing exploration and exploitation
- e.g., e-greedy policy
What role does the environment play in reinforcement learning?
The environment is where the agent interacts to gather experiences and learn optimal behavior through trial and error.
