Week 5

Question 1

What is supervised learning in machine learning?

Answer 1

Supervised learning is a machine learning technique where models learn from labeled data (training set) and apply this knowledge to new, unseen data (test set) with the goal of function approximation and classification.

Question 2

What types of methods are commonly used in supervised learning?

Answer 2

Common methods in supervised learning include linear regression, logistic regression, and Support Vector Machine (SVM).

Question 3

What is unsupervised learning in machine learning?

Answer 3

Unsupervised learning involves learning from unlabeled data to find hidden structures within the dataset.

Question 4

What is the goal of unsupervised learning?

Answer 4

The goal of unsupervised learning is data description and pattern recognition.

Question 5

What methods are utilized in unsupervised learning?

Answer 5

Methods used in unsupervised learning include K-Means clustering, neural networks, and principal component analysis (PCA).

Question 6

How does feedback differ between supervised and unsupervised learning?

Answer 6

In supervised learning, instructive feedback is used to guide the learning process, whereas unsupervised learning typically does not use feedback.

Question 7

What does reinforcement learning focus on?

Answer 7

Reinforcement learning focuses on the interaction between an agent and its environment.

Question 8

How does an agent interact with the environment in RL?

Answer 8

The agent selects actions, and the environment provides evaluative feedback based on those actions.

Question 9

What is the sequence of events in a reinforcement learning process?

Answer 9

The sequence in RL involves state, action, reward, new state, and so on, as exemplified by the SARSA algorithm.

Question 10

What does SARSA stand for in reinforcement learning?

Answer 10

SARSA stands for State-Action-Reward-State-Action, which is a sequence that describes how an agent learns from the consequences of its actions in an environment.

Question 11

What is the Markov property in the context of Markov Decision Processes?

Answer 11

The Markov property states that the conditional probability distribution for the system at the next time step depends only on the current state, not on the sequence of events that preceded it.

Question 12

How do the effects of actions in a Markov Decision Process relate to the state and history?

Answer 12

In an MDP, the effects of actions depend only on the current state of the system and not on the prior history of how the agent arrived at that state.

Question 13

What do deterministic transitions mean in a Markov Decision Process?

Answer 13

Deterministic transitions in an MDP mean that the next state of the system is determined with certainty given the current state and action taken.

Question 14

What are stochastic transitions in a Markov Decision Process?

Answer 14

Stochastic transitions in an MDP involve probability, indicating that the next state is not determined with certainty and can be one of several possible states.

Question 15

what are the elements of the environment en agent of reinforced learning?

Answer 15

Question 16

What is the Law of Effect according to Thorndike (1944)?

Answer 16

The Law of Effect states that actions followed by satisfaction will be strengthened, while those that produce discomfort will be weakened.

Question 17

What is learning defined as in the provided material?

Answer 17

Learning is defined as updating your expectations based on new information or experiences.

Question 18

What happens if you can perfectly predict the future according to the learning model?

Answer 18

If the actual outcome matches the expected outcome, no adjustment of expectations is necessary.

Question 19

How should expectations be adjusted when predictions are not perfect?

Answer 19

When the actual outcome does not match the expected outcome (prediction error exists), expectations should be adjusted accordingly to improve future predictions.

Question 20

explain why the right calculations are more efficient.

Answer 20

the left calculation requires more memory capacity over time, while the right calculation is minimal and constant.

Question 21

why would you use **weighted-average ** (learning rate) instead of a normal average?

Answer 21

Question 22

What is the exploration-exploitation dilemma?

Answer 22

The exploration-exploitation dilemma is a decision-making problem where one must choose between taking the best known action (exploitation) or trying new actions to discover better alternatives (exploration).

Question 23

What is exploitation in the context of decision-making?

Answer 23

Exploitation involves always taking the action with the highest estimated value (greedy), which is considered optimal in stable environments that are fully known.

Question 24

What is the risk associated with exploitation?

Answer 24

The risk with exploitation is the possibility of missing out on better alternatives that have not been explored yet.

Question 25

What does exploration mean in decision-making strategies?

Answer 25

Exploration refers to taking different actions to gather more information about the environment, which can potentially lead to discovering more valuable actions.

Question 26

What is the problem with an exploration strategy?

Answer 26

The issue with exploration is that it does not necessarily maximize immediate reward, which is the goal of reinforcement learning (RL).

Question 27

Why is a balance between exploitation and exploration necessary?

Answer 27

A balance between exploitation and exploration is necessary to achieve optimal decision-making, ensuring both the use of the best-known options and the discovery of potentially better choices.

Question 28

using the greedy option doesn’t always lead to the most optimal outcome. How do we use the function in the picture to create better chance to an optimal outcome?

Answer 28

every x trial (epsilon) we choose a random action. (exploration).

Question 29

what is the diffence between random exploration and directed exploration

Answer 29

exploration= completly random

directed exploration tries to pick actions that have not been chosen yet or have not been chosen a much recent.

Question 30

explain the Bellman optimality equation

Answer 30

a fundamental concept in reinforcement learning. The equation is used to find the optimal policy, which tells an agent the best action to take in every state.

Question 31

Value Iteration (learning from knowledge) is an algorithm used in reinforcement learning. Explain the algorithm.

Answer 31

used to find the optimal policy by iteratively improving the value function of each state. It is a form of dynamic programming that solves the Bellman Optimality Equation.

1. Initialization: You start by initializing the value V for all states s in your state space to arbitrary values, except for the terminal states which might be initialized to the final reward or zero.
2. Iteration: For each state s, you update the value V (s)
   V(s) by using the Bellman Optimality Equation.

Question 32

simulate a sweep for state 14 using the bellman equation.

Discount rate= follows the idea that close reward is better than no reward.

r=reward
yV=discount factor
s’=value of the subsequent state under the optimal policy.

Answer 32

Question 33

What are the shortcomings of Dynamic Programming?

Answer 33

DP is computationally expensive, especially as the size of the state space grows, which can make it impractical for large or complex environments.

Question 34

What assumptions does Dynamic Programming make?

Answer 34

DP assumes a perfect model of the environment, meaning it requires perfect knowledge of all state transitions and rewards.

Question 35

Why can DP not be used without a perfect model of the environment?

Answer 35

Without perfect knowledge, DP cannot accurately predict the state transitions and rewards, which are necessary for finding the optimal policy.

Question 36

What characterizes Monte Carlo (MC) Reinforcement Learning?

Answer 36

Monte Carlo RL is characterized by learning directly from experience, without requiring a model of the environment’s dynamics.

It has to finish.
It does not compare with other states.

Question 37

using the same grid as before how would the following trajectory path look like using the MC method.

Answer 37

important to notice that you start at the reward (15) and go backwards to the first state (0).

Question 38

What are some problems or shortcomings of Monte Carlo methods?

Answer 38

The convergence to the optimum can be slow, requiring a lot of sampling, and sufficient exploration must be maintained throughout the learning process.

Question 39

within SARSA what is difference between On-policy and Off-policy?

Answer 39

This means that the learning process follows the policy that is currently being improved upon. In other words, the agent learns about the policy it is using to make decisions, as opposed to “off-policy” methods like Q-learning, where the agent learns about a potentially different policy from the one it follows.

Question 40

What is a pro of on-policy methods?

Answer 40

On-policy methods, such as SARSA, ensure that the policy being evaluated and improved is the same as the policy being used to make decisions, which often leads to more stable and consistent learning.

Question 41

What is a con of on-policy methods?

Answer 41

On-policy methods can be less efficient than off-policy methods because they can only learn from the current policy, potentially leading to slower convergence to the optimal policy.

Question 42

What is a pro of off-policy methods?

Answer 42

Off-policy methods, like Q-learning, can learn from data generated by any policy (exploratory or even suboptimal), making them more flexible and often faster at finding the optimal policy.

Question 43

What is a con of off-policy methods?

Answer 43

Off-policy methods can be less stable and more complex to implement because they must correctly account for the difference between the policy being evaluated and the policy used to generate the data.

Question 44

What is the curse of dimensionality?

Answer 44

the curse of dimensionality refers to the phenomenon where the volume of the state space increases exponentially with each additional dimension, making computational problems much more complex.

Question 45

What is the credit assignment problem in the context of the curse of dimensionality?

Answer 45

The credit assignment problem is the challenge of determining which actions or decisions led to a particular outcome, especially when many decisions are involved over time.

Question 46

What is the first solution to temporal credit assignment?

Answer 46

The first solution is Standard Temporal Difference (TD) Learning, which updates the value of a state-action pair based on the difference between the expected future rewards and the actual rewards received

Question 47

What is the second solution to temporal credit assignment?

Answer 47

he second solution is the Monte Carlo method, which uses the “long-term” memory of actions and rewards to update values, relying on complete episodes to make updates.

Question 48

What is the third solution to temporal credit assignment?

Answer 48

The third solution is Eligibility Traces, which provide a “short-term” memory of actions to bridge the temporal gap, allowing for credit to be assigned more accurately to actions that lead to a reward.

Question 49

What are the advantages of model-free RL?

Answer 49

Model-free RL is simple and efficient, making it accessible and straightforward to implement without the need for a model of the environment.

Question 50

What are the disadvantages of model-free RL?

Answer 50

Model-free RL can be slow and rigid. It often leads to outcome insensitivity, where the learning process doesn’t adjust adequately to changes in the environment.

Question 51

What are the advantages of model-based RL?

Answer 51

Model-based RL is fast and flexible. It allows for behavioral adjustments through planning, as it involves a model of the environment which can simulate future states.

Question 52

What are the disadvantages of model-based RL?

Answer 52

Model-based RL is complex and costly. It requires a significant amount of computational resources to model the environment and update the model based on new information.

Question 53

How is human behavior described in terms of RL strategies?

Answer 53

Human behavior is often seen as a mixture of both model-free and model-based RL, utilizing the simplicity of model-free methods and the strategic planning of model-based methods.