14 stochastic processes in continuous time

Question 1

Stochastic processes in continuous time:intro

Answer 1

we consider two simple continuous-time stochastic (Markov) processes: first the simple death process, and then the simple birth process. These will pave the way to the
study of continuous-time birth-and-death processes in the next chapter.

Question 2

The SIMPLE death process: assumptions
“linear death process”

some probabilities

S(t)

Answer 2

The death process is a Markov process in continuous time in which the number of individuals in a population can only decrease in time.

Assume: initially N cells in the pop
cells may die only, new ones not made
Sample paths of death process Xᵢ decreases with t step pattern

each cell independently has a constant death rateµ > 0

if a cell is alive at time t, probability it dies in the
interval (t, t + ∆t) is µ∆t as ∆t → 0.

probability tat the cell survives in the interval is 1 − µ∆t

S(t)~ P(a cell, alive at t = 0, survives to time t)
then
S(t + ∆t) = S(t)(1 − µ∆t),
[S(t + ∆t) -S(t)]/∆t = −µS(t), with S(0) = 1
which in the continuous time limit∆t → 0 gives
**dS(t)/dt = −µS(t), with S(0) = 1 **
yielding S(t) = exp(−µt)

the time that the cell dies is a continuous RV with exponential distribution an mean 1/µ

Question 3

The death process:
probability distribution
pₖ(t)

Answer 3

Xₜ ~#cells alive at time t

probability distribution {pₖ(t)}ₖ₌₀ ᴺ
pₖ(t) = P{Xₜ=k}

X₀ =N
p_N(0)=1 and p_{k>N}(0)=0 cells can only die and ~never exceed N
ALways reduction in #: sample paths like steps on stairs

Question 4

The death process:
SIMPLE DEATH PROCESS
deriving ODE

Answer 4

each cell independently constant death rate µ > 0

*If a cell is alive at time t the probability that it dies in the interval (t, t + ∆t) is µ∆t as ∆t → 0

*probability that the cell survives in the interval (t, t + ∆t) is 1 − µ∆t

*S(t) probabilty a cell alive at t=0 survives to time t then S(t + ∆t) = S(t)(1 − µ∆t)
**
or (S(t + ∆t) − S(t))/∆t
= −µS(t), with S(0) = 1
**
which in the continuous-time limit ∆t → 0 gives

dS(t)/dt = −µS(t) with S(0) = 1, yielding S(t) = exp(−µt)

** time that the cell dies is a continuous random variable with an exponential distribution and mean 1/µ**

Question 5

The death process:
looking at whole population of cells

probability that n cells survive up to time t

Answer 5

p_N (t) = exp(−Nµt)

comes from n products
p_1(t) = P {one cell survives to time t} = S(t) = exp(−µt)

P{1 cell is dead at time t} = 1 − S(t)
= 1 − p1(t) = 1 − exp(−µt)

Question 6

The death process:
looking at whole population of cells

With X_0 = N, the probability that one cell has died and N − 1 are still alive at time t is

Answer 6

pN−1(t) =
Nexp−(N−1)µt) (1 − exp(−µt))

N ways to pick one cell for death

Question 7

The death process:
looking at whole population of cells

With X_0 = N, the probability that one cell has died and N − 2 are still alive at time t is

Answer 7

p_{N−2}(t) =
N(N − 1)/2
** exp(−(N−2)µt)** (1 − exp(−µt)^2

Question 8

The death process:
looking at whole population of cells
Assuming that X_0 = N, the probability that j ≤ N cells have died and N − j are
still alive at time t is

Answer 8

pN−j (t) = N!/j!(N − j)! x
exp(−(N−j)µt) (1 − exp(−µt))^j

We can say that the number of cells alive at time t has a binomial distribution Xt ∼Bin(N, q) with parameters N and q ≡ e−µt,

clearly probability conservation:
sum j=0 to N P_{N-j} = 1

Question 9

The death process:
looking at whole population of cells

extinction probability

Answer 9

setting j = N that the probability that N cells
have died at time t is

p_0(t) = (1 − e−µt)^N

which is the extinction probability at time t in the death process with N initial cells (X0 = N).
Clearly limt→∞ p_0(t) = 1 simply says that in the death process the eventual extinction of the
population is certain.
An alternative way to find the pk(t) is by using the differential equations

Question 10

The death process:
looking at whole population of cells

An alternative way to find the pk(t) is by using the differential equations

Answer 10

for 0 ≤ k < N,
dpₖ(t)/dt = -kµpₖ(t) + (k+1)µpₖ₊₁(t)
loss term-gain term (master
equation)
and

dp_N/dt=
-Nµp_N (t),

with p_k=N (0) = 0 and p_N (0) = 1

Question 11

master
equation

The death process:
looking at whole population of cells

Answer 11

for 0 ≤ k < N,
dpₖ(t)/dt = -kµpₖ(t) + (k+1)µpₖ₊₁(t)
loss term-gain term

The loss term −kµpₖ says that a population with k cells (state k) can be left at rate kµ at which one of the k cells dies (leading to a state
k −1).

This is balanced by the gain term (k + 1)µpₖ₊₁ saying that the state k can be reached from state k + 1 after the death of a cell at rate (k + 1)µ.

Since we are dealing with a death process the number of cells can never exceed N, and hence
there is only a loss term contributing to dp_N/dt

Question 12

death process

dp_N/dt=
-Nµp_N (t),

Answer 12

can be solved to obtain
p_N(t)=exp(-Nµt)
using k=N-1 obtain dp_{N-1}/dt solved to get p_{N-1}(t)
repeatedly
we could focus on mean behaviour instead

x(t)=E(X_t) = sum k=1 to infinity of [kpₖ(t)]

dx(t)/dt =sum k=1 to N of
[ k dpₖ(t)/dt]
=
=−µx(t)

since x(0)=E(X_0)=N we have
x(t)=Nexp(−µt)

Question 13

death process:
extinction time

extinction times

probability of extinction at time t

Answer 13

Let T be the extinction time, defined as
T = {first time that X_t is 0}.

T is a continuous RV on the state space S=[0,∞] s.t t T ≤ t if
and only if X_t = 0

The probability of extinction at time t is: p_0(t) = (1−exp(−µt))ᴺ

cumulative distribution function of T:
P {T ≤ t} = P {X_t = 0}
= p_0(t) = (1 − e(−µt))ᴺ

says how probability to go extinct up to time t accumulates
thus
P {T ≤ 0} = p_0(0) = 0,
P{T ∈ [0,∞)} = p_0(∞) = 1,

the probability that
the population goes extinct between t_1 and t_2 ≥ t1 is
P{t1 ≤ T ≤ t2} = P{T ≤ t_2} − P{T ≤ t_1} = p_0(t_2) − p_0(t_1)

Question 14

death process

Thus, by definition, the probability density of the extinction time, T, is

Answer 14

ρ(t) =dP {T ≤ t}/dt

dp₀(t)/dt
= µNe{⁻µᵗ}(1 − exp(−µt))ᴺ⁻¹

This clearly implies
P{T≤ t} = p₀(t)
= ∫₀ᵗ ρ(τ )dτ

and P {t₁ ≤ T ≤ t₂} =
∫ₜ_₁ ᵗ-² ρ(t)dt =
p₀ (t₂)−p₀(t₁)

graph of cumulative distribution p₀(t) upper bound at 1

Question 15

simple death process calculating the mean extinction time

Answer 15

The mean extinction time can be computed from the density function
ρ(t) =dP {T ≤ t}/dt
=
dp₀(t)/dt
= µNe{⁻µᵗ}(1 − exp(−µt))ᴺ⁻¹

E(T)=
∫₀∞ tρ(t)dt =
µN ∫₀∞ t exp(−µt) (1− exp(−µt) ᴺ⁻¹ dt

by integrating by parts: compute µE(T) for the first few N: µE(T) grows logarithmically with N
E(T) ≈1/µ ln N when N → ∞
t against E(T) graph peaks then decreases

Question 16

intuitive way of simple death process mean extinction time

Answer 16

Extinction from transition of a state with N cells to N-1 cells with µ_N = Nµ followed by transitions to N-1, N-2,,.. with rate µ_(N-1)= (N-1)µ

The time to jump from N to N −1 is an exponentially-distributed random variable with mean1/(Nµ)

N-1 to N-2 ….mean 1/{(N-1)µ}

….

Jump from X_t=2 to X_t=1 …. mean 1/(2µ)

X_t=1 to X_t=0 ….mean 1/µ

E(T)= 1/µ+ 1/2µ+….+ 1/Nµ
= (1/µ)(1+1/2+1/3+…+1/N)

E(T) =approx 1/µ(γ + ln N),

(limit as n tends to infinity of sum 1+1/2+..+1/n is γ + ln N, where γ = 0.577 . . . is the Euler Mascheroni constant)

confirms that the mean extinction time grows logarithmically with N.

Question 17

“simple birth process” or the “Yule process”.

Answer 17

continuous-time stochastic process in which each individual (cell) of a population can reproduce without ever dying

In the birth process,** each cell alive at time t has a probability β∆t of dividing into two cells
before t + ∆t, as ∆t → 0**.

Assume each cell, independently, has a constant birth rate, β > 0.

cells do not die
sample path upward stairs

Question 18

“simple birth process” or the “Yule process”.

pₙ(t) = P {Xₜ = n},

MASTER EQUATION derivation

Answer 18

Let X_t be the number of cells alive at time t, and the probability that there are n cells in the population at t be pₙ(t) = P {Xₜ = n},
assuming that there are initially X₀= n₀ > 0 cells,

Since X₀= n₀>0 and cells cant die pₙ(t)=0 for all t ≥ 0 if n< n₀

each of the n₀ can divide with a rate β the state is left with rate βn₀:
dp_n₀(t)/dt= −βn₀ p_ₙ₀(t)

which with the initial condition p_(n_0) (0)=1 gives
p_(n_0)(t) = exp(−βn₀t)

n₀ + 1 cells: any reproduce at rate β to new state n₀ + 2 (loss term −β(n₀+1)p ₙ_₀₊₁ for rate of change of pₙ-₀₊₁
n₀ cells: one reproduce at rate β to new state n₀ + 1 (gain term β(n₀)p ₙ_₀ for rate of change of pₙ-₀₊₁

With p_ₙ-₀(t) =exp(−βn₀t) we have thus

dpₙ-₀₊₁(t)/dt
= −β(n₀+1)pₙ-₀₊₁(t)+βn₀pₙ-₀(t)
= −β(n₀+1)pₙ-₀₊₁+1(t) + βn₀exp(−βn₀t)

with pₙ-₀₊₁(0)=0

,

Question 19

“simple birth process” or the “Yule process”.

pₙ(t) = P {Xₜ = n},

MASTER EQUATION

Answer 19

With p_ₙ-₀(t) =exp(−βn₀t) we have thus

dpₙ-₀₊₁(t)/dt
= −β(n₀+1)pₙ-₀₊₁(t)+βn₀pₙ-₀(t)
= −β(n₀+1)pₙ-₀₊₁+1(t) + βn₀exp(−βn₀t)

with pₙ-₀₊₁(0)=0

Equation for the rate of change of p_n when n>n_0

dp_n/dt
= −βnpₙ(t) + β(n−1)pₙ₋₁(t)

loss term + gain term
with pₙ(0) = 0,

Question 20

“simple birth process” or the “Yule process”.

pₙ(t) = P {Xₜ = n},

MASTER EQUATION

dp_n/dt
= −βnpₙ(t) + β(n−1)pₙ₋₁(t)

loss term + gain term
with pₙ(0) = 0,

Answer 20

The linear ordinary differential equation for p_{n_0+1) can readily be solved using standard
methods:

p_(n_0+1)(t) =
n₀( exp(−βn₀t) − exp(−β(n₀+1)t)
= n₀exp(−n₀βt) (1 − exp(−βt))

which has a max:
t_max =
(1/β) log(1 + (1/n₀))

graph of p_{n_0+1} (t)
peaks at t_max decreases as t goes to infinity

we could continue and use to solve ODE for n_0+2 etc but instead,…

Question 21

“simple birth process” or the “Yule process”.

PGF

∂G(z, t)/∂t

Answer 21

pgf: p_k(t) ≡ 0 for k < n_0
sum from k=n_0 to infinity only needed p_k(t)*z^k

∂G(z, t)/∂t=−βz(1 − z)∂G(z, t)/∂z

, with G(z, 0) = z^{n_0}

The solution of the pde
G(z,t) =
[(zexp(-βt))/(1-z+zexp(-βt))]^n_0

G(1, t) = 1 (probability conservation
G(0, t) =p_0(t) = 0 when n_0 > 0

Question 22

∂G(z, t)/∂t

satisfies….

Answer 22

birth process?

∂G(z, t)/∂t=−βz(1 − z)∂G(z, t)/∂z

, with G(z, 0) = z^{n_0}

Question 23

“simple birth process” or the “Yule process”.

PGF compare
G(z,t) =
[(zexp(-βt))/(1-z+zexp(-βt))]^n_0

with powers

Answer 23

zexp(−βt)[1+z^2(1- exp(−βt))+z^3(1-exp(−βt))^2+…]

case n_0=1
1/(1-x)= 1 +x + x^2+…. when |x|<1

since |z|≤ 1
|z(1- exp(−βt)|<1

G(z,t) = zexp(−βt)[1+z(1- exp(−βt))+z^2(1-exp(−βt))^2+…]

p_1(t)z + p_2(t) z^2 +….

Question 24

“simple birth process” or the “Yule process”.

Comparing each term with the same power of z in the last two lines, we readily find

Answer 24

Comparing each term with the same power of z in the last two lines, we readily find
p1(t) = e^{−βt}
p2(t) = exp(−βt)(1 − exp(−βt)), p3(t) = exp(−βt)(1 − exp(−βt))^2

and more generally, when
n_0 = 1, we have

p_n(t) = exp(−βt)[(1 − exp(−βt))]ⁿ−¹
for n ≥ 1
)
which is a geometric distribution

Question 25

“simple birth process” or the “Yule process”.

Comparing each term with the same power of z in the last two lines, we readily find

G(z, t)

Answer 25

G(z,t)=
z^n_0 exp(-n_0βt)) [(1+z(1-exp(-βt)) + z^2(1-exp(-βt))^2 +…..]^n_0

= p_(n_0)(t)z^{n_0} + p_(n_0+1)(t)z^(n_0+1)+,,,,,,

thus we recover

p_n_0(t) = exp(–n_0βt)
and
p_(n_0+1)(t)=
n_0exp(-n_0βt)(1- exp(-βt))
.
In general, we have
p_k(t)
=
(k-1)
(k-n_0) exp(-n_0βt) (1- exp(-βt))^(k-n_0)

for k ≥ n_0

,
“negative binomial distribution”
p_k(t)=0 for k<n_0
p_0(t)=0 pop extinction impossible in birth process!

Question 26

Appendix A: Cumulative distribution and probability density
functions (reminder & summary)

Answer 26

cumulative distribution function P {T ≤ t} : S → [0, 1] of a continuous random variable T (e.g. the extinction time) on the state space S describes how probabilities accumulate

P {T ≤ t} = P {T ∈ [0, t]}

ρ(t) ≡dP {T ≤ t}/dt
is the probability density function of T.

In the case of the extinction time T for a non-zero initial population, S = [0,∞) and
P {T ≤ t} = p0(t) = integral_[0,t] ρ(τ )dτ.
Therefore, the probability that extinction occurs between t1 and t2 ≥ t1 is

P {t1 ≤ T ≤ t2} = P {T ≤ t2} − P {T ≤ t1} = p0(t2) − p0(t1) = integral_[t_1,t_2] ρ(τ )dτ,

with P {T = 0} = 0 and P {T ∈ S} = 1.

We can use the probability density function to
compute the mean of T^n
(nth moment of T) for n ≥ 0 according to the general formula:

E(T^n) = integral_S τ^n ρ(τ )dτ

mean extinction time in the death process is thus IE(T) = integral_[0,∞] tρ(t)dt =integral_[0,∞] t dp_0/dt .dt

Question 27

Appendix B: integrating factor method to solve first-order differential equations (reminder)

Answer 27

dp_{n_0+1}(t)/dt
= −β(n0 + 1)pn0+1(t) + βn0e
−βn0t

is a first-order linear differential equation of the form

dy/dt + P(t)y = Q(t) with y(0) = 0,

IF I(t) = exp (integral P(t).dt)

p_{n_0+1}(t) =
βn_0*
[([exp(−βn_0t)]/β + K exp(−β(n0+1)t)]

,
where the constant K is found from the initial condition: p_{n_0+1}(0) = 0 = βn_0(β^−1 + K) ⇒
K = −β^{−1}

yielding p_{n_0+1}(t) =
n_0[exp(−βn_0t) − exp(−β(n_0+1)t]