Natural language processing (NLP) is a field that studies methods for enabling computers to process and work with the languages used by humans in everyday life.

In recent years, methods using large language models (LLMs)—neural networks trained on large-scale datasets—have become increasingly standard in natural language processing.

One of the fundamental neural network architectures underlying large language models is the Transformer.

What Is a Transformer?

The Transformer is a neural network architecture proposed by Google researchers in 2017.

It was originally proposed as a model for machine translation. It was subsequently applied to a wide range of tasks, including large language models, and has become one of the standard neural network architectures used in natural language processing.

Transformer inputs are commonly represented using units smaller than words, known as subwords, or as individual characters.

An input sentence is divided into such units and provided to the Transformer as a sequence of tokens.

A Transformer consists of components such as token embeddings, positional encodings, self-attention mechanisms, and feed-forward neural networks.
These components and their processing methods will be explained in detail in separate articles.

Background to the Transformer

When the Transformer was proposed, the principal sequence transduction models used recurrent neural networks or convolutional neural networks. Some of the highest-performing models also used attention mechanisms to connect the encoder and decoder.

In contrast, the Transformer was proposed as a neural network architecture centered on attention mechanisms for modeling relationships between inputs and outputs, without using recurrent or convolutional neural networks.

Recurrent neural networks process sequences in order, which limits the extent to which their training computations can be parallelized.

By using attention mechanisms to model relationships among the positions in input and output sequences, the Transformer made training easier to parallelize than with conventional recurrent models.

In the paper that introduced the Transformer, its performance was evaluated on English-to-German and English-to-French machine translation tasks. The results demonstrated higher translation performance than previous models while also reducing the time required for training.

Types of Transformer Architecture

Transformer architectures can be broadly divided into the following three types:

・Encoder-only architecture
・Encoder-decoder architecture
・Decoder-only architecture

Each architecture processes input token sequences differently and produces different types of output.

Encoder-Only Architecture

A Transformer composed only of an encoder produces contextualized token embeddings corresponding to the input token sequence.

The encoder sequentially applies L blocks to the input embeddings, adding contextual information through multiple stages of processing.

Through this multilayer processing, lower layers are generally considered to capture more surface-level information, intermediate layers more grammatical information, and higher layers more semantic information. In this way, progressively higher layers tend to represent increasingly complex and abstract contextual information.

Encoder-Decoder Architecture

An encoder-decoder architecture uses an encoder and a decoder to process two different token sequences.

In machine translation, the encoder processes a sentence in the source language, while the decoder generates the corresponding sentence in the target language.

The decoder differs from the encoder in that each block contains two attention mechanisms and the decoder calculates a probability distribution over possible output tokens.

Decoder-Only Architecture

A decoder-only architecture is based on the decoder of an encoder-decoder architecture, with the cross-attention mechanism removed.

Decoder blocks also differ from encoder blocks in that masking is applied to the attention mechanism.

About This Article

References
・Introduction to Large Language Models, supervised and written by Ikuo Yamada, with Masatoshi Suzuki, Kosuke Yamada, and Linghan Li
・Ashish Vaswani et al., “Attention Is All You Need,” Advances in Neural Information Processing Systems 30, 2017.


※This article was prepared with reference to the sources listed above and organized based on the author’s understanding.

Large Language Models (LLM)