Function Approximation · aiwiki.page

Function approximation is the representation of a mathematical function by another function chosen to reproduce its values or behavior with controlled error. The approximating function usually belongs to a restricted family, such as polynomials, piecewise polynomials, rational functions, or neural networks. The subject connects approximation theory, which studies the possibilities and limits of such representations, with numerical analysis, which develops computational methods for constructing and evaluating them. A target may be known analytically or accessible only through sampled values. (dlmf.nist.gov)

Mathematical formulation and error criteria

An approximation problem specifies a target function $f$ , a domain $D$ , an admissible family $\mathcal A$ , and a measure of discrepancy. A best approximation minimizes that discrepancy within the family. Different criteria generally produce different approximants; therefore, “best” has meaning only relative to the stated domain, family, and error measure. (dlmf.nist.gov)

For bounded scalar-valued functions, the uniform error is

\|f-g\|_\infty=\sup_{x\in D}|f(x)-g(x)|.

This norm controls the largest absolute deviation. An alternative, associated with $L^p$ spaces, is

\|f-g\|_{L^p(\mu)} =\left(\int_D |f(x)-g(x)|^p\,d\mu(x)\right)^{1/p}, \qquad 1\le p<\infty.

The integral aggregates error according to a measure $\mu$ ; it need not control the error at every individual point. Least-squares approximation uses squared discrepancies, either integrated over a domain or summed over samples. (dlmf.nist.gov)

The degree of a polynomial, number of spline intervals, or number of network units supplies a complexity parameter. Approximation theory asks how the smallest attainable error changes as this parameter increases, rather than merely whether one particular fit appears accurate. (chebfun.org)

Principal approximation families

A polynomial approximant has the form

p_n(x)=\sum_{k=0}^{n}a_kx^k.

Other polynomial bases can represent the same space. In particular, Chebyshev polynomials provide useful representations on bounded intervals, with coefficients that reveal how rapidly an expansion can be truncated. For periodic functions, truncated Fourier series use sine and cosine terms instead. (chebfun.org)

Splines are piecewise polynomials joined subject to specified continuity conditions. They permit local variation without requiring a single polynomial of high degree over the entire domain. Rational approximants, written $r(x)=p(x)/q(x)$ , use ratios of polynomials; their denominators must be considered when determining where the approximation is valid. These families support different compromises between representation size, smoothness, and computational cost. (dlmf.nist.gov)

An artificial neural network gives another parameterized family. A one-hidden-layer scalar-output network can be written

g(x)=c+\sum_{j=1}^{m}a_j\,\sigma(w_j^\top x+b_j),

where $\sigma$ is an activation function. Unlike a fixed polynomial expansion, both the output coefficients and the parameters defining the component functions may be adjusted. (sciencedirect.com)

Existence and convergence

The Weierstrass approximation theorem states that every continuous real-valued function on a closed bounded interval can be approximated arbitrarily closely by polynomials in the uniform norm. Explicitly, for every $\varepsilon>0$ , some polynomial $p$ satisfies

\sup_{x\in[a,b]}|f(x)-p(x)|<\varepsilon.

This is an existence statement: it does not prescribe an optimal degree, a numerical algorithm, or a rate of convergence. (ocw.mit.edu)

Rates depend strongly on the target’s regularity and the approximation method. For Chebyshev approximation, suitable differentiability assumptions yield algebraic error decay, whereas analyticity in a sufficiently large complex neighborhood yields geometric decay. Corners and discontinuities can slow convergence substantially. Uniform convergence must also be distinguished from weaker notions that allow errors concentrated in small regions. (chebfun.org)

Universal approximation theorems establish related density properties for neural-network families under specified assumptions on activations, target functions, and error criteria. Such results concern representational capacity; they do not themselves guarantee that a training procedure will locate the required parameters. (sciencedirect.com)

Constructing an approximant

Interpolation requires an approximant to match prescribed sample values exactly. Approximation more broadly need not impose exact agreement. In ordinary least squares, coefficients minimize the sum of squared residuals. This distinction matters when observations contain noise: reproducing every observation and approximating an underlying function are different objectives. (dlmf.nist.gov)

Best uniform polynomial approximation minimizes the maximum error. For a continuous target on an interval, the best polynomial of degree at most $n$ is unique. Its nonzero error is characterized by an alternating pattern of equal-magnitude extreme deviations at at least $n+2$ ordered points. The Remez algorithm uses this structure to compute minimax approximations. (dlmf.nist.gov)

Exact interpolation alone does not ensure convergence as more nodes are added. The Runge phenomenon demonstrates that high-degree polynomial interpolation at equally spaced nodes can develop severe endpoint oscillations even for an analytic target. Chebyshev-distributed nodes provide much better control. Consequently, node placement and numerical stability are distinct concerns from the existence of an accurate approximant. (chebfun.org)

Function approximation in machine learning

In machine learning, a target is commonly inferred from training data rather than supplied as an explicit formula. A model family and loss function define the fitting problem. Squared-error loss, for example, measures discrepancies between predicted and observed numerical outputs. Accuracy on the fitted samples is not equivalent to generalization to previously unseen inputs. (deeplearningbook.org)

Error analysis distinguishes approximation error, arising from limitations of the model family; estimation or generalization error, associated with learning from finite data; and optimization error, arising when training does not achieve the desired optimum. A more expressive family can reduce approximation error without necessarily improving predictive performance. Regularization restricts or penalizes model complexity, while overfitting describes fitting behavior that does not transfer well beyond the observed data. These statistical issues are separate from whether the family can represent a target arbitrarily accurately in principle. (arxiv.org)

Computational uses

Function approximants support numerical evaluation, integration, differentiation, and root finding by replacing difficult functions with manageable representations. Polynomial and rational approximations also supply practical formulas for special functions, including error functions and elliptic integrals, with accuracy specified over particular ranges of their arguments. (chebfun.org)