Positional Encoding

Positional encoding is a family of techniques that represents the positions of elements in data processed by an artificial neural network. It is particularly important in the Transformer architecture, where position information complements content-based self-attention. Encodings may identify absolute locations, relative distances, or spatial relationships, using vectors, rotations, or modifications to attention scores. (arxiv.org)

Purpose and mathematical setting

In natural language processing, a sequence consists of tokens produced through tokenization. Their content representations, related to word embeddings, do not by themselves specify where they occur. Unlike a recurrent neural network, a Transformer does not inherently process tokens through an ordered chain of recurrent states. Position must therefore be represented through an additional mechanism. (arxiv.org)

Without positional signals or position-dependent masks, ordinary self-attention is permutation-equivariant: rearranging the inputs rearranges the outputs correspondingly, rather than introducing an understanding of sequence order. Relative position methods address this by making interactions depend on the separation between elements, not merely their content. The distinction concerns the attention operation itself; other architectural constraints can also introduce ordering information. (arxiv.org)

Absolute position representations

An absolute encoding associates position $p$ with a vector $e_p$. A common integration rule is

where $x_p$ is the content embedding and both vectors have the same dimension. The resulting representation contains both token identity and position information. (arxiv.org)

Learned absolute embeddings store a trainable vector for each supported position. In BERT, the input representation combines token, segment, and position embeddings. These components serve different purposes: position embeddings locate tokens, while segment embeddings distinguish sentence segments. Learned position vectors are model parameters rather than fixed mathematical formulas. (arxiv.org)

The original Transformer used fixed sinusoidal encodings:

Here $d$ is the encoding width and $i$ indexes coordinate pairs. Frequencies vary geometrically. Through identities from trigonometry, a fixed position shift can be represented by a linear map on each sine–cosine pair. (arxiv.org)

Relative position representations

Relative methods represent the displacement between a query position and a key position. The 2018 paper Self-Attention with Relative Position Representations introduced learned displacement vectors into attention calculations, including both key-related and value-related terms. Its implementation clips distances to a bounded interval, allowing sufficiently distant token pairs to share representations. (arxiv.org)

A simpler relative mechanism adds a position-dependent bias to an attention score:

The first term is a scaled inner product between query and key vectors; the second encodes displacement. Scores are converted to attention weights using the softmax function. Such biases need not be vectors added to input embeddings. (arxiv.org)

Relative representations emphasize relationships such as “three tokens earlier” rather than “token number twenty.” They are not inherently parameter-free: displacement vectors or bias values can be learned. Different formulations also differ in whether they modify attention scores alone or alter the representations being aggregated. (aclanthology.org)

Rotary position embeddings

Rotary position embedding, usually abbreviated RoPE, was introduced in the 2021 RoFormer paper. It applies position-dependent rotations to pairs of coordinates in query and key vectors, rather than simply adding a position vector to the input. These rotations can be expressed through a block-diagonal matrix $R_p$. (arxiv.org)

If the transformed query and key are $R_pq_p$ and $R_qk_q$, their interaction satisfies

Thus, although each rotation uses an absolute index, the positional part of their interaction depends on relative displacement. The underlying token content still matters; the full attention score is not solely a function of distance. (arxiv.org)

Rotations preserve vector lengths, and different coordinate pairs use different angular frequencies. RoPE consequently combines absolute position-dependent transformations with relative-position structure inside attention. Its mathematical definition accommodates varying sequence lengths, but that flexibility alone does not establish reliable behavior beyond the lengths encountered during training. (arxiv.org)

Linear biases and length generalization

Attention with Linear Biases, or ALiBi, introduces a distance-proportional penalty directly into attention scores. In causal attention, an earlier key at position $q\leq p$ receives a bias commonly written as $-m_h(p-q)$, with a fixed slope $m_h$ for attention head $h$. Different slopes give multiple attention heads different distance preferences. (arxiv.org)

ALiBi requires no learned position-embedding table. Its authors demonstrated length extrapolation in specified language-model experiments, including training on 1,024-token sequences and evaluating on 2,048-token sequences. Such results describe tested configurations, not unrestricted length generalization. (arxiv.org)

For RoPE-based large language models, position interpolation extends the context window by rescaling position indices into the original positional range. The 2023 method combines this transformation with limited fine-tuning. It distinguishes the ability to calculate encodings at new positions from the ability to use those positions effectively. (arxiv.org)

Spatial applications

Position representations also appear in computer vision. A Vision Transformer converts image patches into a sequence and adds learned positional embeddings to patch representations. These embeddings identify patch locations, not the locations of individual pixels within each patch. When image resolution changes, the original ViT method interpolates the pretrained positional embeddings across the patch grid, adapting spatial representations to the new layout. (arxiv.org)