Markov Chains

Published by Cambridge University Press. Follow the link for publication details.

Some sections may be previewed below. Click on the section number for a ps-file or on the section title for a pdf-file. This material is copyright of Cambridge University Press and is available by permission for personal use only.

Preface

Intro duction

1. Discrete-time Markov chains

1.1 Definition and basic properties

1.2 Class structure

1.3 Hitting times and absorption probabilities

1.4 Strong Markov property

1.5 Recurrence and transience

1.6 Recurrence and transience of random walks

1.7 Invariant distributions

1.8 Convergence to equilibrium

1.9 Time reversal

1.10 Ergodic theorem

1.11 Appendix: recurrence relations

1.12 Appendix: asymptotics for n!

2. Continuous-time Markov chains I

2.1 Q-matrices and their exponentials

2.2 Continuous-time random processes

2.3 Some properties of the exponential distribution

2.4 Poisson processes

2.5 Birth processes

2.6 Jump chain and holding times

2.7 Explosion

2.8 Forward and backward equations

2.9 Non-minimal chains

2.10 Appendix: matrix exponentials

3. Continuous-time Markov chains II

3.1 Basic properties

3.2 Class structure

3.3 Hitting times and absorption probabilities

3.4 Recurrence and transience

3.5 Invariant distributions

3.6 Convergence to equilibrium

3.7 Time reversal

3.8 Ergodic theorem

4. Further theory

4.1 Martingales

4.2 Potential theory

4.3 Electrical networks

4.4 Brownian motion

5. Applications

5.1 Markov chains in biology

5.2 Queues and queueing networks

5.3 Markov chains in resource management

5.4 Markov decision processes

5.5 Markov chain Monte Carlo

6. Appendix: probability and measure

6.1 Countable sets and countable sums

6.2 Basic facts of measure theory

6.3 Probability spaces and expectation

6.4 Monotone convergence and Fubini's theorem

6.5 Stopping times and the strong Markov property

6.6 Uniqueness of probabilities and independence of sigma-algebras

Markov Chains

Contents