markov models

Question 1

when we do not have control over the state transitions but the states are completely observable then we have a:

Question 2

how do you solve a POMDP?

Answer 2

the model has to find, for any action/observation history, the action that maximizes the expected discounted reward. the model has to find, for any action/observation history, the action that maximizes the expected discounted reward:

Question 3

a POMDP model contains: (3)

Answer 3

• A state transition (probability) function: P(s_{t+1}|s_t, a_t) (probability of the next state, given the current state and the action)
• An observation function: P(o_t|s_t,a_t) (probability of the observation, given the current state and the action)
• A reward function: E(r_t|s_t,a_t) (expected reward, given the current state and the action)

Question 4

Partially Observable Markov Decision Processes are

Answer 4

MDPs that have partially observable states instead of the agent knowing exactly what the current state is.

Question 5

what is the difference between POMDPs and MPDs?

Answer 5

in MDPs, an agent knows exactly what the current state is.

Question 6

what are 2 methods to find the optimal policy of an MDP

Answer 6

value iteration

policy iteration

Question 7

what is the utility in a MPD?

Answer 7

the utility of a state is the expected sum of discounted rewards if the agent executes the policy pi. The true utility of a state corresponds to the optimal policy pi*.

Question 8

what is a markov decision process?

Answer 8

MPD’s are functions to choose actions, given the state of the world which are generally called policies. They associate an optimal decision with every state that the agent might reach in uncertain environments.

Question 9

why / when are HMM’s used?

Answer 9

because an experimenter can not always observe the states, only measure them by the observable outcome.

Question 10

how is a hidden markov model different form a markov model?

Answer 10

hidden markov models do not have observations given but rather represent the states by the probability distribution of several possible observations that can be expected to occur in this state.: the probability that we observe k given we’re in state i.

Question 11

how do markov chains work?

Answer 11

1) starts in a state decided by starting distribution probilities.
2) visit states based on the probability to go from one state to the next
3) is a sttochastic process

Question 12

when we have control over the state transitions but the states aren’t completely observable then we have a:

Answer 12

partially observable markov decision process: POMDP

Question 13

when we do not have control over the state transitions and the states are not completely observable then we have a:

Answer 13

Hidden markov model (HMM)

Question 14

when we do have control over the state transitions but the states are completely observable then we have a:

Answer 14

markov decision process (MDP)

Question 15

when we do not have control over the state transitions but the states are completely observable then we have a:

Answer 15

markov chain

Question 16

when the observations are given, the Markov model is also called a:

Answer 16

markov chain model

Question 17

what is the markov property?

Answer 17

assuming that future states depend only on the current state, not on the events that occurred before it.

Question 18

what is a markov model?

Answer 18

A Markov model has observable states of the world, and an observed sequence and it assumes the Markov property.

Question 19

where are markov models applied?

Answer 19

in areas where there is recognition of temporal patterns with variable duration, such as speech and handwriting recognition. This has the problem that the patterns can be very short- or very long- lasting.