Markov Property · aiwiki.page

The Markov property is a condition in probability theory under which the present state of a stochastic process contains all the information from its past needed to determine the conditional distribution of its future. Informally, the future and the past are independent once the present is known. Named after Andrey Markov, it is the defining principle of Markov chains and more general Markov processes. The property concerns conditional distributions, not whether successive states are unrelated or whether the future is predictable with certainty. (statslab.cam.ac.uk)

Mathematical definition

For a discrete-time process $X_0,X_1,\ldots$ , each $X_n$ is a random variable representing the state at time $n$ . On a countable state space, the Markov property can be written

\Pr(X_{n+1}=x\mid X_0=x_0,\ldots,X_n=x_n) = \Pr(X_{n+1}=x\mid X_n=x_n),

for histories having positive probability. Thus, knowing earlier states does not change the distribution of the next state once the current state is specified. Repeated application extends this statement from the next step to any finite sequence of future states. This is a form of conditional independence, rather than unconditional independence. (galton.uchicago.edu)

For general state spaces and continuous time, the definition uses a filtration $(\mathcal F_t)$ , an increasing family of sigma-algebras representing available information. An adapted process is Markov relative to this filtration if, for every bounded measurable function $f$ and $s<t$ ,

\mathbb E[f(X_t)\mid\mathcal F_s] = \mathbb E[f(X_t)\mid X_s] \quad\text{almost surely}.

This formulation uses conditional expectation and avoids conditioning directly on individual states that may have probability zero. The filtration matters: information beyond the process’s own observed history can affect whether the equality holds. (ethz.ch)

Transition probabilities and time homogeneity

A Markov process can be described through transition probabilities $P_{s,t}(x,A)$ , giving the probability of occupying a measurable set $A$ at time $t$ , conditional on state $x$ at time $s$ . The Markov property does not require these probabilities to remain unchanged over time. Time homogeneity is the additional condition that they depend on $t-s$ , rather than separately on $s$ and $t$ . (web.stanford.edu)

For a time-homogeneous chain on a finite or countable state space, one-step probabilities form a transition matrix $P$ , with nonnegative entries and rows summing to one. The $n$ -step probabilities are entries of $P^n$ . More generally, transitions satisfy the Chapman–Kolmogorov equations:

P_{s,t}(x,A) = \int P_{u,t}(y,A)\,P_{s,u}(x,dy), \qquad s<u<t.

These equations combine transitions through an intermediate state. (galton.uchicago.edu)

Time homogeneity also differs from being a stationary process, whose finite-dimensional distributions are invariant under time shifts. A homogeneous chain may have changing marginal distributions because of its initial condition. Starting it in a stationary distribution makes the resulting process stationary. Neither stationarity nor homogeneity should be substituted for the Markov condition. (web.stanford.edu)

Examples and the meaning of “memory”

A simple random walk illustrates the property. At each step, an independent coin toss determines whether the walker moves one unit left or right. Given the present position, previous positions provide no additional information about subsequent positions. Nevertheless, the current position records the accumulated effects of previous steps, so describing the process as “memoryless” does not mean that the past has left no trace. (stat.uchicago.edu)

Brownian motion is a continuous-time example. At a fixed time $s$ , the future increments $B_{s+t}-B_s$ form a Brownian motion independent of the history through $s$ . Consequently, the conditional distribution of future positions depends on that history only through $B_s$ . Independent increments establish the Markov property here, but independent increments are not required of Markov processes generally. (stat.uchicago.edu)

Whether a model is Markov depends on its state representation. Position alone may omit relevant information about a moving system, whereas position together with velocity can specify its evolution in an idealized mechanical model. A state may therefore incorporate information about the past while still satisfying the Markov condition. (web.stanford.edu)

Strong Markov property

The ordinary Markov property concerns fixed, deterministic times. The strong Markov property extends the restart principle to stopping times: random times whose occurrence can be determined from information already available. A first hitting time is a standard example. (stat.uchicago.edu)

For a time-homogeneous process with transition operators $P_t$ , the strong property takes the form

\mathbb E[f(X_{\tau+t})\mid\mathcal F_\tau] = (P_t f)(X_\tau)

on $\{\tau<\infty\}$ , subject to the usual measurability conditions. Conditional on the state at $\tau$ , the subsequent process follows the same transition law as a process started there. Discrete-time Markov chains have this property; in continuous time, it is not automatic for every Markov process. Brownian motion and standard continuous-time Markov chains satisfy it. Arbitrary random times involving knowledge of the future cannot replace stopping times. (statslab.cam.ac.uk)

Hidden states and decision models

In a hidden Markov model, the latent state sequence is Markov, but the observation sequence generally is not. Earlier observations can help infer the current hidden state and thereby improve prediction beyond what the latest observation provides. Thus, observing only a function or noisy measurement of a Markov state need not preserve the property. (stat.cmu.edu)

In reinforcement learning, a Markov decision process uses a controlled version: conditional on the current state and action, the distribution of the next state and reward is independent of earlier interaction history. This permits dynamic programming and Bellman equations to express future values recursively using current states. When observations omit relevant state information, a partially observable Markov decision process distinguishes the underlying Markov dynamics from the agent’s incomplete information. The distinction is between the dynamics of the modeled state and the predictive adequacy of the observations supplied to the agent. (web.stanford.edu)