Stationary Distribution

A stationary distribution is a probability distribution that remains unchanged under the evolution of a Markov chain or a more general Markov process. If the initial state has this distribution, the state at every later time has the same distribution, although individual trajectories may continue moving between states. Also called an invariant distribution, it describes a probabilistic equilibrium rather than a state in which movement stops. Stationarity is distinct from convergence: a stationary distribution can exist even when distributions from other initial conditions do not approach it. (ocw.mit.edu)

Mathematical definition

Consider a time-homogeneous, discrete-time chain $(X_n)_{n\geq0}$ on a finite or countable state space $S$ . Its transition matrix $P$ has entries

P_{ij}=\Pr(X_{n+1}=j\mid X_n=i),

which are nonnegative and sum to one across each row. Writing distributions as row vectors, a stationary distribution $\pi$ satisfies

\pi P=\pi,\qquad \pi_i\geq0,\qquad \sum_{i\in S}\pi_i=1.

Equivalently, $\pi_j=\sum_i\pi_iP_{ij}$ for every state $j$ . These global balance equations express the preservation of probability at each state. (ocw.mit.edu)

In linear algebra, $\pi$ is a normalized, nonnegative left eigenvector of $P$ with eigenvalue $1$ . It is also a fixed point of the distribution update map $\mu\mapsto\mu P$ . For a finite chain, its entries can be found by solving a system of linear equations together with the normalization constraint. An eigenvector alone is insufficient unless it represents a probability distribution. (people.csail.mit.edu)

Existence and uniqueness

Every finite-state Markov chain has at least one stationary distribution. If it is irreducible, meaning every state can be reached from every other state with positive probability in some number of steps, its stationary distribution is unique and strictly positive. Irreducibility is sufficient, but not necessary, for uniqueness. (lancaster.ac.uk)

For a finite reducible chain, each closed communicating class has its own stationary distribution, extended by zero outside that class. All stationary distributions of the chain are convex combinations of these class distributions. Thus uniqueness holds precisely when there is only one closed communicating class; transient states receive zero stationary probability. (en.wikipedia.org)

On a countably infinite state space, irreducibility alone does not guarantee existence. An irreducible chain has a stationary probability distribution exactly when it is positive recurrent: the expected time to return to a state is finite. In that case,

\pi_i=\frac{1}{\mathbb E_i[T_i^+]}, \qquad T_i^+=\inf\{n\geq1:X_n=i\},

where $\mathbb E_i$ denotes expectation starting at $i$ . An invariant measure with infinite total mass cannot be normalized into a stationary probability distribution. (statslab.cam.ac.uk)

Stationarity and long-run behavior

For an irreducible, positive recurrent, aperiodic chain,

\lim_{n\to\infty}(P^n)_{ij}=\pi_j.

Aperiodicity excludes systematic restrictions on possible return times. It is needed for this convergence theorem, not for the existence or uniqueness of a stationary distribution. (statslab.cam.ac.uk)

For example,

P=\begin{pmatrix}0&1\\1&0\end{pmatrix}

has the unique stationary distribution $(1/2,1/2)$ . Starting from either individual state, however, the chain alternates deterministically, so its distribution does not converge. Starting with the stationary distribution preserves equal probabilities at every time. (people.csail.mit.edu)

Time averages behave differently. For an irreducible positive recurrent chain, the fraction of steps spent in state $i$ converges almost surely to $\pi_i$ , even if the chain is periodic. This is an ergodic theorem, related to the law of large numbers. Moreover, initializing a time-homogeneous chain with $\pi$ produces a strictly stationary process: shifting all observation times equally leaves its finite-dimensional distributions unchanged. (statslab.cam.ac.uk)

Detailed balance and examples

A useful sufficient condition for stationarity is detailed balance:

\pi_iP_{ij}=\pi_jP_{ji} \quad\text{for all }i,j.

Summing over $i$ yields global balance. Detailed balance equates probability flow in both directions between every pair of states and characterizes reversibility in equilibrium. It is stronger than stationarity: a stationary chain can sustain directional circulation without satisfying detailed balance. (lancaster.ac.uk)

For the two-state chain

P=\begin{pmatrix}1-a&a\\b&1-b\end{pmatrix}, \qquad 0<a,b<1,

balance gives $\pi_1a=\pi_2b$ , hence

\pi=\left(\frac{b}{a+b},\frac{a}{a+b}\right).

This illustrates how occupancy depends on transition probabilities in both directions. Another example is a random walk around a finite circle with identical clockwise, counterclockwise, and holding probabilities at every state. Its stationary distribution is uniform, even when clockwise and counterclockwise probabilities differ and detailed balance fails. (lancaster.ac.uk)

Continuous time and applications

For a finite-state continuous-time Markov chain with generator matrix $Q$ , the stationary equations become

\pi Q=0,\qquad \pi_i\geq0,\qquad \sum_i\pi_i=1.

Equivalently, $\pi P(t)=\pi$ for every elapsed time $t$ . A finite irreducible continuous-time chain has a unique stationary distribution, and its transition probabilities converge to that distribution without a separate aperiodicity requirement. (ocw.mit.edu)

In Markov chain Monte Carlo, the transition mechanism is constructed to preserve a chosen target distribution. Under appropriate recurrence and convergence conditions, simulated trajectories provide samples and time averages associated with that target. Detailed balance offers one construction method, but invariant distributions can also be preserved by nonreversible dynamics. (people.csail.mit.edu)