Discrete Markov Process Flashcards
What type of processes are modelled by discrete-time discretestate
Markov processes?
Discrete-Time Markov processes are processes that are memoryless and can only take one of mutually exclusive states. The transition rates are constant
They can be discretisized with a delta t that is small enough so that only one transition can occur.
What is the fundamental quantity to be
computed?
The fundamental quantity comuted is the state probabilty vector which describes the probability of a system to be in a certain state.
What are the assumptions of this model?
What are
the properties of the transition probabilities?
The transition probabilites are below or equal to one and non-negative
How can you compute the multi-step transition probabilities
using the one-step transition probability matrix A?
With a inital state vector C we simply multiply the transition matrix A n times.c
How can you compute the unconditional state probability vector
P[X(n)=j] from the knowledge of the one-step transition
probability matrix A?
We can use raise A and raise it to the power of n.
For steady state we use the approach:
Π = Π * A
Derive the fundamental equation for homogeneous discrete-time
discrete-state Markov processes.
Assuming a homogenous process with constant transition rates.
Pj(1) = P[X(1) = j]
Pj(1) = ΣP[X(1) = j | X(0) = i] * P[X(0) = i]
Pj(1) = pij *ci
P(1) = X * A
Repeat this for P(2)
-> Fundamental Equation
P(n) = C * A^n
What are the steps to compute the general solution to the
fundamental equation of a discrete-time discrete-state Markov
process?
We now that there is a vector V so that
V * A = w * V -> eigenvalue Problem
Put into homogenous form and use determinant to find non-trivial solution.
Now find alphas and c so that P(n) and C can be displayed using the eigenvector basis
How can you compute point availability at time k and steady state
availability in discrete-time discrete-state Markov processes?
Point availability is is simply 1- Probability of failure states at time k.
Steady state can be determined through 1) limit approach 2) Π = Π * A
What are steady-state probabilities in discrete-time discrete-state
Markov processes. How can they be computed? Do they depend
on the initial condition C? Motivate your answer.
The steady-state probailites are the probabilites of finding the system in a certain state after many time steps.
They do not depend on the initial condition, but the initial condition determines how long it takes for the system to move close to the steady state.
What is the definition of first-passage probabilities and how do
you compute them?
The first-passage probability is the probability that the system reaches a state for the first time.
We can simply calculate them by taking the probability of the system of a system to be in state j, and then remove from that all the probabilites of eralier arrival.
How can you compute reliability at time k using discrete-time
discrete-state Markov processes?
We can compute reliability by excluding all the repair processes of the failure states and then simply determine the state probability vector.
Define recurrent, transient and absorbing states of a Markov
process. How to compute the average occupation time of a state?
Recurrent states are visited repeatedly over time.
Their average occupation time is simply the inverse of the transition probability out of the state.
Transient states are not visited again,
Absorbing states cannot be left by the system and have an infinite occupation time
