Chapter 4

Question 1

Define a finite markov chain

Answer 1

The process X is said to be a finite Markov chain if it satisfies the Markov property and the state space is finite. We will usually take
X = {1, …, N} or X = {0, 1, …, N}.

Question 2

what does p(i,j) denote

Answer 2

Probability of moving from state i to state j in 1 step. Probability of moving to state j given we are in state i

Question 3

What is pn(i)

Answer 3

pn(i) = P(Xn = i)

Question 4

Define a stochastic matrix

Answer 4

A stochastic matrix is a square matrix whose columns are probability vectors. A probability vector is a numerical vector whose entries are real numbers between 0 and 1 whose sum is 1

Question 5

What is the distribution of a markov chain dependent on?

Answer 5

determined by its initial distribution and transition
probabilities

Question 6

How to interpret matrix P to power of n

Answer 6

The i,j th entry of the matrix P ^N is probability of going from state i to state j in n steps

Question 7

What are other names for the limiting distirbution

Answer 7

equilibrium, invariance, stationary distirbution

Question 8

What is two states both lead to each other - meaning

Answer 8

The relation ↔ is an equivalence relation, it partitions the
state space into disjoint sets called communication classes

Question 9

Define irreducible and reducible

Answer 9

If there is only one communication class, the chain is said to
be irreducible. Otherwise, it is said to be reducible.

Question 10

Is a random walk with reflecting/ absorbing boundaries irreducible or reducible

Answer 10

Reflecting- irreducible
Absorbing - reducible

Question 11

Define transient state

Answer 11

A state i is transient if there is non-zero probability that
the chain may never return to state i again - will only be visited a finite number of times. ie: A state i is transient iff there exists a state j for which i → j but j doesn’t lead to i.

Question 12

What is the opposite of a transient state?>

Answer 12

Recurrent state or erogodic

Question 13

Define a recurrent state

Answer 13

The chain will definitely return to state i again.
Hence, recurrent states will be visited infinitely often.
A special type of recurrent state is an absorbing state

Question 14

What can be told about the period of a state if states are in the same communication class

Answer 14

Their periods are the same if i communicates with j

Question 15

Define a aperiodic state

Answer 15

It has a period of 1

Question 16

How do we know when a markov chain has a unique existing stationary distribution

Answer 16

If the Markov chain {Xt } is aperiodic and irreducible, then it
has a unique stationary distribution π and Pn has a limit - perron frobenius theorem

Question 17

What does the ergodic theorem tell us?

Answer 17

Proportion of the number of visits to a state i in n steps will converge to the limiting distribution pi(i)

Question 18

Define in words the return times of an irreducible aperiodic markov chain

Answer 18

How long do we wait before returning to a state j - return time is the shortest time before returning