Bayes' Theorem

Bayes’ theorem is an identity in probability theory that expresses the probability of an event given another event in terms of the reverse conditional probability and their individual probabilities. It provides the mathematical foundation for Bayesian inference, in which probabilities assigned to hypotheses or unknown quantities are updated using observed evidence. Although closely associated with Bayesian statistics, the theorem itself follows from ordinary probability rules and does not require a particular interpretation of probability. (online.stat.psu.edu)

Mathematical statement and derivation

For events $A$ and $B$, with $P(B)>0$, the theorem states

Here, $P(A\mid B)$ is the conditional probability of $A$ given that $B$ has occurred. The expression on the right additionally requires $P(A)>0$ for $P(B\mid A)$ to be defined in the elementary event-based formulation. The theorem reverses the direction of conditioning; it does not assert that $P(A\mid B)$ and $P(B\mid A)$ are equal. (ma.imperial.ac.uk)

Its proof follows by writing the probability of the intersection in two ways:

Dividing by $P(B)$ yields the formula. Thus, the theorem describes two compatible factorizations of the same joint probability rather than introducing an additional assumption about how events are related. (ma.imperial.ac.uk)

If $H_1,\ldots,H_n$ form a mutually exclusive and exhaustive partition of the sample space, the law of total probability supplies the denominator:

This version makes explicit that the probability of evidence must account for every alternative in the partition, not merely the hypothesis under examination. (online.stat.psu.edu)

Prior, likelihood, and posterior

When $H$ represents a hypothesis and $E$ represents evidence, the formula is commonly interpreted through three components. The prior probability $P(H)$ describes uncertainty before incorporating $E$. The likelihood $P(E\mid H)$ describes how probable the evidence is under $H$. The posterior probability $P(H\mid E)$ describes uncertainty after conditioning on the evidence. The denominator normalizes the result. A likelihood is not, by itself, a probability assigned to the hypothesis. (plato.stanford.edu)

For an unknown continuous parameter $\theta$, the corresponding density formula is

The prior distribution $p(\theta)$ and likelihood function $p(D\mid\theta)$ determine the posterior distribution. The denominator is the marginal likelihood of the data. These expressions use a probability density function where appropriate: for a continuous random variable, a density value is not the probability of an exact point. Integration, rather than pointwise evaluation, gives probabilities over parameter intervals. (pmc.ncbi.nlm.nih.gov)

Illustrative calculation

Consider a hypothetical production system in which factory A supplies 30% of all items and factory B supplies 70%. Suppose their defect rates are 4% and 1%, respectively. These are stipulated values, not observations about actual factories.

For a randomly selected item known to be defective, the probability that it came from A is

In a hypothetical batch of 10,000 items with these exact proportions, A would contribute 120 defective items and B would contribute 70. Thus, 120 of the 190 defective items would originate from A. The calculation illustrates the same reverse-conditioning structure used in manufacturing examples of Bayes’ theorem. (online.stat.psu.edu)

The 4% defect rate at A is therefore distinct from the approximately 63.2% probability that a defective item originated there. Both factories’ production shares and defect rates enter the latter calculation.

Odds and sequential updating

For a hypothesis $H$ and its complement $\neg H$, the theorem can be expressed as

In words, posterior odds equal prior odds multiplied by a likelihood ratio. In model comparison, the corresponding ratio of marginal likelihoods is called a Bayes factor. Evidence favors one alternative relative to another when it is more probable under the first. (plato.stanford.edu)

Updating can proceed sequentially: a posterior from earlier observations becomes the prior for subsequent observations. The next likelihood must condition on information already incorporated. Multiplying separate likelihoods without this conditioning requires an appropriate conditional independence assumption; sequential updating does not itself establish independence. (pmc.ncbi.nlm.nih.gov)

Statistical and computational applications

In machine learning, a naive Bayes classifier uses the theorem to derive class probabilities from class priors and feature likelihoods. Its defining simplification is that features are mutually conditionally independent given the class. That assumption belongs to the classifier’s model, not to Bayes’ theorem. The resulting classifier can select the class with greatest posterior probability. (scikit-learn.org)

Bayesian analysis may retain the full posterior rather than only a point estimate. A maximum a posteriori estimate selects a posterior mode. When direct calculations are difficult, Markov chain Monte Carlo methods generate dependent draws targeting the posterior, while variational inference constructs an approximation. Numerical accuracy and convergence are separate questions from the validity of the theorem. (mc-stan.org)

Historical origin

The theorem bears the name of Thomas Bayes. His posthumous essay, An Essay towards Solving a Problem in the Doctrine of Chances, appeared in volume 53 of the Philosophical Transactions of the Royal Society, dated 1763, and was communicated by Richard Price. The essay investigated inverse probability: how observed successes and failures could inform uncertainty about an unknown probability of success. Its treatment concerned a particular statistical problem rather than the full range of modern applications of the general identity. (sites.socsci.uci.edu)