Automatic Differentiation

Automatic differentiation (AD) is a family of computational techniques for evaluating derivatives of numerical functions expressed as computer programs. It applies the chain rule to the elementary operations performed by a program, combining their local derivatives into derivatives of the overall computation. Unlike finite-difference approximation, AD does not introduce a differentiation step size. It underpins gradient-based machine learning and mathematical optimization, while also supporting scientific simulation and engineering design. (jmlr.org)

Mathematical foundation

A numerical program evaluates a function through a sequence of operations such as addition, multiplication, and elementary functions. Each operation has a known derivative rule. AD combines these rules according to the dependencies between intermediate values, effectively differentiating a composition of functions. The result is generally a numerical derivative at specified inputs, rather than a simplified symbolic formula. (jmlr.org)

These dependencies can be represented by a computational graph. Nodes represent inputs, intermediate values, and outputs; edges indicate which values an operation consumes. A recorded finite execution can form a directed acyclic graph, even when the source program contains loops. Derivative propagation follows this graph in either the forward or reverse direction. PyTorch, for example, constructs its autograd graph during execution and rebuilds it for subsequent executions, allowing the recorded structure to reflect changing control flow. (docs.pytorch.org)

AD differs from symbolic differentiation, which manipulates expressions, and numerical differentiation, which estimates derivatives from nearby function values. AD avoids finite-difference truncation error, but its computations remain subject to floating-point arithmetic and roundoff. Consequently, “exact derivatives” means application of analytic derivative rules without finite-difference approximation, not unlimited numerical precision. (jmlr.org)

Forward mode

For $f:\mathbb{R}^{n}\rightarrow\mathbb{R}^{m}$ , let $J_f(x)$ denote its Jacobian matrix. Forward-mode AD propagates an input perturbation $v$ alongside the ordinary computation, producing the Jacobian–vector product

\dot y=J_f(x)v.

This is the directional derivative of the output along $v$ . Each intermediate variable carries both its ordinary value and a tangent value. For multiplication $z=ab$ , the tangent rule is

\dot z=\dot a\,b+a\,\dot b.

Seeding successive input-coordinate directions yields successive columns of the Jacobian. Forward mode is therefore typically favorable when there are relatively few inputs and many outputs. (docs.jax.dev)

One interpretation uses dual numbers, written $a+b\varepsilon$ , where $\varepsilon^2=0$ . Arithmetic on these pairs propagates first-order derivative information together with function values. This algebraic interpretation explains why implementing suitable arithmetic rules can turn ordinary numerical evaluation into derivative evaluation. (jmlr.org)

Reverse mode

Reverse-mode AD first evaluates the function, then propagates output sensitivities backward. Given an output seed $u$ , it computes

\bar x=J_f(x)^{\mathsf T}u,

equivalently a vector–Jacobian product when sensitivities are written as row vectors. For scalar output, seeding its sensitivity with $1$ produces the full gradient with respect to the inputs. Reverse mode is usually preferable when there are many inputs but relatively few outputs. (docs.jax.dev)

Consider $f(x,y)=xy+\sin x$ , evaluated through $a=xy$ , $b=\sin x$ , and $f=a+b$ . Applying the reverse rules gives

\bar a=\bar b=1,\qquad \bar x=y+\cos x,\qquad \bar y=x.

The two contributions to $\bar x$ must be added because $x$ influences the output through two paths. This example illustrates the accumulation of sensitivities from dependent operations. (docs.pytorch.org)

Reverse propagation often needs intermediate values from the forward evaluation. Saving these values increases memory consumption. Gradient checkpointing, also called rematerialization, reduces storage by retaining selected intermediates and recomputing others during the backward pass. It trades additional computation for lower memory use; recomputation must preserve the relevant behavior of the original evaluation. (docs.pytorch.org)

Implementations and higher-order derivatives

AD systems commonly use operator overloading or program transformation. Operator overloading equips numerical operations with derivative-tracking behavior. Program transformation generates derivative computations from source code or an intermediate representation. These approaches can coexist with graph tracing and compilation. Their practical differences concern supported language features, optimization opportunities, runtime overhead, and the handling of array operations. (arxiv.org)

TensorFlow records relevant operations within a gradient tape. PyTorch records dependencies among tensor operations for reverse propagation. Such systems differentiate only computations visible to their derivative machinery: moving a calculation into an untracked external library can disconnect the derivative path. A disconnected input and an input whose mathematical derivative is zero are therefore distinct cases, even if an interface offers options to represent both as zero. (tensorflow.org)

Derivative computations can themselves be differentiated. Nested forward and reverse transformations produce higher-order derivatives, including a Hessian matrix. Often only a Hessian–vector product is required. For a twice continuously differentiable scalar function and constant $v$ ,

H_f(x)v=\nabla_x\!\left(\nabla f(x)^{\mathsf T}v\right).

This avoids explicitly storing the full Hessian and provides curvature information for numerical optimization. (docs.jax.dev)

Applications and limitations

In artificial neural networks, backpropagation is a specialized application of reverse-mode AD. It computes derivatives of a loss function with respect to model parameters; an optimization procedure then uses those derivatives to update the parameters. AD supplies sensitivity information rather than specifying the update rule itself. TensorFlow’s gradient-tape interface makes this separation explicit by returning gradients for subsequent optimizer use. (tensorflow.org)

AD does not make a nondifferentiable function differentiable. At a kink, a library may apply a documented convention, choose a subgradient where appropriate, or return an undefined result. Data-dependent branches differentiate the executed computation rather than the discrete decision selecting it. Invalid operations can also contaminate gradients: masking a division-by-zero result afterward does not necessarily remove that operation from the recorded derivative graph. Correct derivative evaluation therefore depends on both mathematical assumptions and implementation behavior. (docs.pytorch.org)