To process text with a Transformer, the text must first be divided into tokens, and each token must then be converted into a vector composed of numerical values.

However, converting tokens into vectors alone does not represent where each token occurs within the token sequence.

Therefore, in a Transformer, a positional encoding representing the position within the token sequence is added to the input token embedding representing the token, and the resulting vector is provided to the model.

This article explains how the vectors provided to a Transformer are constructed by examining input token embeddings and positional encodings separately.

Transformer Configurations and the Scope of This Article

Transformers may use an encoder-only architecture, an encoder-decoder architecture, or a decoder-only architecture.

Figure 2-1 illustrates how a token sequence provided to an encoder is converted into a sequence of input vectors using input token embeddings and positional encodings, and then processed by multiple encoder blocks.

In an encoder-decoder architecture, the source-side token sequence is provided to the encoder, while the target-side token sequence is provided to the decoder.

On both the encoder and decoder sides, the token embeddings are multiplied by $\sqrt{D}$, after which positional encodings are added.
However, the target-side token sequence provided to the decoder is shifted by one position so that the model can predict the next token at each position.

The decoder also receives the representations produced by the encoder and generates its output while referring to the source-side token sequence.

In a decoder-only architecture, no encoder is used.
The input token sequence is converted into token embeddings, positional information is added, and the decoder blocks then predict the next tokens sequentially.

Methods for providing positional information are not identical across all Transformers.
The sinusoidal positional encoding described in this article is one of the methods used in Transformer models.

Some models use other methods, such as learned positional embeddings whose values are updated during training.

Figure 2-1 Input Token Embeddings and Positional Encodings in a Transformer Encoder

Vectors and Dimensions

Both input token embeddings and positional encodings are represented as vectors consisting of multiple numerical values.

For example, the following vector consists of four numerical values and is therefore a four-dimensional vector.

A Transformer uses vectors such as this to represent individual tokens.

In this article, the number of values forming a vector is denoted by $D$. Therefore, a $D$-dimensional vector contains $D$ numerical values.

Input Token Embeddings

In a Transformer, a $D$-dimensional input token embedding is prepared for each token in the vocabulary $V$.

The input token embedding corresponding to token $\omega$ is expressed as follows.

$\mathbb{R}^{D}$ indicates a $D$-dimensional vector whose elements are real numbers.

An input token sequence of length $n$ is expressed as follows.

Here, $\omega_i$ represents the token at position $i$ in the token sequence.

Through the embedding layer, each token is converted into its corresponding $D$-dimensional input token embedding.

Therefore, the sequence of input token embeddings corresponding to the input token sequence is expressed as follows.

The numerical values forming the input token embeddings are not manually assigned to individual tokens.
In a Transformer, the weights of the embedding layer are also learned during model training.

At this stage, each input token embedding is a vector corresponding to the token itself, and information representing its position within the token sequence has not yet been added.

Input Token Embedding Matrix

All input token embeddings held by the model can be represented collectively as the input token embedding matrix $\boldsymbol{E}$.

Here, $|V|$ is the total number of tokens in the vocabulary $V$, and $D$ is the dimensionality of each input token embedding.

Each row of the input token embedding matrix corresponds to one token.

Therefore, if the vocabulary contains $|V|$ tokens and each token is represented by a $D$-dimensional vector, the input token embedding matrix has dimensions $|V|\times D$.

Why Positional Information Is Necessary

The input token embedding $\boldsymbol{e}_{\omega_i}$ is a vector corresponding to token $\omega_i$; it does not itself represent position $i$ in the token sequence.

A Transformer also processes relationships between tokens primarily through attention mechanisms, without using recurrent or convolutional neural networks.

Consequently, if only input token embeddings were used, the order of the tokens within the sequence could not be explicitly provided to the model.

Rearranging the tokens in the input token sequence shown in Equation (2-3) produces an ordering different from the original sequence.

However, the input token embeddings themselves contain no information indicating where their corresponding tokens occur within the token sequence.

Therefore, positional encodings are added to the respective input token embeddings to explicitly provide the token order to the Transformer.

Positional Encoding

Positional encoding is a method for representing the position of each token within a token sequence as a vector.

In sinusoidal positional encoding, the vector corresponding to each position is constructed by combining sine and cosine functions with different periods.

Sine and Cosine Functions

The sine function $\sin$ and cosine function $\cos$ vary repeatedly with a fixed period according to their input values.

For both functions, the output ranges from -1 to 1.

Positional encoding combines multiple sine and cosine functions with different periods to construct a $D$-dimensional vector corresponding to each position.

Calculating Positional Encodings

Let the $D$-dimensional positional encoding corresponding to position $i$ in the token sequence be $\boldsymbol{p}_i$.

Each element of $\boldsymbol{p}_i$ is calculated using the following equations.

where $k=0,1,\ldots,\frac{D}{2}-1$.

The symbols represent the following quantities.

$i$: Position within the token sequence

$k$: Integer specifying a pair of sine and cosine functions

$D$: Dimensionality of the positional encoding and input token embedding

In the positional encoding, sine functions are used for the even-numbered elements, while cosine functions are used for the odd-numbered elements.
Because the sine and cosine functions are used in pairs, $D$ is assumed to be even in the equations above.

The positional encoding $\boldsymbol{p}_i$ corresponding to position $i$ is expressed as follows.

Positional Encoding at Position 0

When the position is $i=0$, the input to every sine and cosine function is 0.

Therefore, the positional encoding corresponding to position 0 is a vector in which 0 and 1 alternate, as shown below.

As position $i$ changes, the values of the respective sine and cosine functions also change, so the positional encoding corresponding to each position changes as well.

Wavelengths of Positional Encodings

In positional encoding, the wavelengths of the sine and cosine functions differ according to the vector dimension.

The wavelengths of the sine and cosine functions used in Equations (2-8) and (2-9) increase geometrically from $2\pi$ to $2\pi\cdot10000^{(D-2)/D}$.

When $D$ is large, the maximum wavelength approaches $10000\cdot2\pi$.

Therefore, a single positional encoding contains values from multiple sine and cosine functions having different wavelengths.

Relative Positional Relationships

Sinusoidal positional encodings have the property that a fixed positional offset can be represented by a linear transformation.

For any fixed positional offset $m$, the positional encoding $\boldsymbol{p}_{i+m}$ at position $i+m$ can be expressed as a linear function of the positional encoding $\boldsymbol{p}_i$ at position $i$.

This property is expected to make it easier for the model to learn to attend based on relative positions.

Adding Input Token Embeddings and Positional Encodings

Both the input token embedding and positional encoding are $D$-dimensional vectors.

Therefore, the input token embedding and positional encoding can be added element by element.

Let the token at position $i$ be $\omega_i$, its input token embedding be $\boldsymbol{e}_{\omega_i}$, and the positional encoding corresponding to the same position be $\boldsymbol{p}_i$.

The vector $\boldsymbol{x}_i$ provided to the Transformer is calculated using the following equation.

The input token embedding can be considered to be multiplied by $\sqrt{D}$ to bring the scale of the input token embeddings into balance with that of the positional encodings.

Calculating this for every position in the token sequence produces the following sequence of input vectors.

Why Sinusoidal Positional Encoding Was Adopted

In the original Transformer research, fixed sinusoidal positional encodings were compared experimentally with learned positional embeddings whose values are updated during training.

In these experiments, the fixed positional encodings and learned positional embeddings produced nearly identical results.

However, sinusoidal positional encodings may also be applicable to token sequences longer than those encountered during training.

Taking this possibility into consideration, fixed positional encodings were adopted.

Summary

In a Transformer, each token in the vocabulary is converted into a $D$-dimensional input token embedding.

However, input token embeddings alone cannot represent the positions or order of tokens within a token sequence.

Therefore, positional encodings calculated using sine and cosine functions are added to the input token embeddings.

The input vector corresponding to token $\omega_i$ at position $i$ is expressed by the following equation.

By arranging the input vectors calculated for each position as rows, an $n\times D$ input matrix is constructed.

This input matrix is processed by the Transformer blocks and transformed through self-attention and other mechanisms into contextualized representations that reflect the relationships between tokens.

About This Article

References
・Introduction to Large Language Models, supervised and written by Ikuo Yamada; written by Masatoshi Suzuki, Kosuke Yamada, and Linghan Li
・Ashish Vaswani et al., “Attention Is All You Need,” Advances in Neural Information Processing Systems 30, 2017.
・Alec Radford et al., “Improving Language Understanding by Generative Pre-Training,” OpenAI, 2018.


※This article was prepared with reference to the sources listed above and organized based on the author’s understanding.

Large Language Models (LLM)