Unsupervised learning builds models using only input data that does not have corresponding correct output labels.

Because the data is unlabeled, there is no “teacher” to provide guidance during learning.

The objective of unsupervised learning is not to learn a mapping from inputs to outputs, but to identify the structure inherent in the data itself.

Characteristics of Unsupervised Learning

Like the data used in supervised learning, the data handled in unsupervised learning can have various properties.

For example, values may be discrete or continuous, dimensionality may be low or high, and the length of the data may be fixed or variable.

What all unsupervised learning models have in common is that they learn from a set of unlabeled observations $\{\boldsymbol{x}_i\}$.

However, the objective of learning varies depending on the model.

Common applications include the following:

• Generating plausible new samples that could belong to the dataset
• Processing data, removing noise, filling in missing information, or compressing data
• Revealing the internal structure of data, such as by dividing a dataset into coherent clusters
• Determining whether new data belongs to the same dataset or is an outlier
  
  

Classification of Unsupervised Learning Models Using Latent Variables

Many unsupervised learning models define a relationship between observed data $\boldsymbol{x}$ and a latent variable $\boldsymbol{z}$ that is not directly observed.

A latent variable represents the structure underlying a dataset and usually has a lower dimensionality than the original data.

The latent variable $\boldsymbol{z}$ can therefore be regarded as a compressed representation that preserves the essential features of the data $\boldsymbol{x}$.

The direction of the mapping between the data and latent variables differs depending on the model.

Mapping from Data to Latent Variables

A representative example of a model that obtains a latent variable $\boldsymbol{z}$ from data $\boldsymbol{x}$ is k-means clustering.

In k-means clustering, each data point $\boldsymbol{x}$ is assigned to one of K clusters.

In this case, the latent variable $z$ represents the assigned cluster number $z\in\{1,2,\ldots,K\}$.

Mapping from Latent Variables to Data

Conversely, some models obtain data $\boldsymbol{x}$ from a latent variable $\boldsymbol{z}$.

In such a model, a distribution over the latent variable $\Pr(\boldsymbol{z})$ is first defined.

A latent variable is then sampled from this distribution and mapped into the data space to produce new data.

A model capable of producing new data in this manner is called a generative model.

Types of Generative Models

A generative adversarial network (GAN) learns how to generate data from latent variables.

It uses a loss designed to make the generated samples indistinguishable from real data.

Normalizing flows, variational autoencoders (VAEs), and diffusion models are representative probabilistic generative models.

In addition to generating new data, probabilistic generative models assign a probability or probability density $\Pr(\boldsymbol{x}\mid\boldsymbol{\phi})$ to each data point $\boldsymbol{x}$.

This probability is determined by the model parameters $\boldsymbol{\phi}$.

Training Probabilistic Generative Models

When training a probabilistic generative model, the parameters are adjusted to make the probabilities assigned to the observed data $\{\boldsymbol{x}_i\}$ as large as possible.

The loss function therefore uses the sum of the negative logarithms of the probabilities assigned to the individual data points, known as the negative log-likelihood.

A probability distribution has the property that its total probability sums or integrates to 1.

Therefore, increasing the probability assigned to the observed data naturally decreases the probability assigned to data that lies far from those observations.

Benefits of Assigning Probabilities

The ability to assign probabilities to data is useful not only as a training criterion, but also in its own right.

For example, the performance of two models can be compared quantitatively by comparing the probabilities they assign to test data.

Furthermore, by determining whether the probability of a data point falls below a threshold, it is possible to assess whether that point belongs to the same dataset or is an outlier.

Evaluating the Performance of Generative Models

The following are the main methods used to evaluate the performance of generative models quantitatively.

Likelihood of Test Data

One method of comparing probabilistic generative models is to measure the likelihood of the test data.

The likelihood of the training data is not suitable for evaluation.

This is because a model may assign an extremely high probability only to individual training data points while assigning very low probabilities to the regions between them.

Such a model may have a high likelihood on the training data while being capable only of reproducing that data.

Using the likelihood of test data makes it possible to evaluate how well the model generalizes beyond the training data and how broadly it covers the overall data distribution.

This is because if the model assigns high probabilities to only part of the training data, probabilities in the remaining regions will be low, causing some test data points to receive low probabilities as well.

However, this method has the following limitations:

• Because GANs do not assign explicit probabilities, this method cannot be applied to them
• For VAEs and diffusion models, estimating the likelihood is computationally expensive, although a lower bound on the log-likelihood can be calculated
• For normalizing flows, the likelihood can be calculated exactly and efficiently
  
  

Inception Score (IS)

The Inception Score (IS) is an evaluation metric for images and is suitable for evaluating generative models trained on the ImageNet dataset.

The score is calculated using a pretrained classification model, usually an Inception model.

The name of the metric is derived from this model.

IS is based on the following two principles:

• Each generated image $\boldsymbol{x}^*_i$ should resemble only one of the ImageNet classes. In other words, the classification probability $\Pr(y\mid\boldsymbol{x}^*_i)$ should ideally have a sharp peak at that class.
• Across the complete set of generated images, each class should appear with equal probability. In other words, the probability $\Pr(y)$, averaged over all generated images, should ideally be uniform.
  
  

IS averages the distance between these two distributions across all generated images.

The value becomes larger when one distribution has a sharp peak and the other is uniform.

More specifically, the KL divergence between $\Pr(y\mid\boldsymbol{x}^*_i)$ and $\Pr(y)$ is averaged, and the exponential of that average is then taken.

Here, $I$ is the number of generated images, and $\Pr(y)$ is calculated using the following equation.

IS has the following limitations:

• It can be used only for generative models targeting the ImageNet dataset and depends on a specific classification model
• Retraining the classification model may produce completely different values
• It does not evaluate diversity within each class, so a model that generates only one realistic image for each class may still receive a high score
  
  

Fréchet Inception Distance (FID)

The Fréchet Inception Distance (FID) is also an image-based metric. It calculates a symmetric distance between the distribution of generated samples and the distribution of real data.

However, accurately representing either distribution is difficult.

This is because representing the distribution of real data is precisely the task that a generative model is intended to perform. FID is therefore an approximate metric.

FID approximates both distributions as multivariate Gaussian distributions and estimates the distance between them using the Fréchet distance.

The distance is measured not between the original images themselves, but between activations from the deepest layer of an Inception classification network.

Because the hidden units in this layer are strongly associated with object classes, fine image details are disregarded and the comparison is performed at a semantic level.

FID can also account for diversity within the same class.

However, the evaluation depends heavily on the information retained in the features of the Inception network, and information discarded by the network is not reflected in the result.

Some of the discarded information may be important for generating realistic samples.

Manifold Precision and Recall

FID is affected by both the realism and diversity of generated samples, but it cannot distinguish between the two.

To evaluate these two properties separately, the overlap between the data manifold and the model manifold is considered.

The data manifold is the subset of the data space in which real data lies, whereas the model manifold is the subset of the data space in which generated samples lie.

• Precision: The proportion of generated samples that lie within the data manifold. This represents the realism of the generated samples
• Recall: The proportion of real data that lies within the model manifold. This represents the proportion of real data that the model is capable of generating
  
  

To estimate a manifold, a hypersphere is placed around each data point, with a radius equal to the distance to its kth-nearest neighbor.

By treating the union of these hyperspheres as an approximation of the manifold, it becomes straightforward to determine whether a new point lies within it.

Manifolds are usually calculated in the feature space of a classifier, which brings both advantages and disadvantages.

Summary

Unsupervised learning learns the structure of a dataset from unlabeled data.

Some unsupervised learning models are generative models and can produce new data.

Some generative models are also probabilistic generative models, which can not only generate new data but also assign probabilities to observed data.

The performance of generative models can be evaluated using measures such as test-data likelihood, the Inception Score, the Fréchet Inception Distance, and manifold precision and recall.

About This Article

References
・Understanding Deep Learning I: Supervised Learning and the Foundations of Models, Simon J. D. Prince, translated by Toshiaki Kurokawa
・Understanding Deep Learning II: Unsupervised Learning, Reinforcement Learning, and Ethics, Simon J. D. Prince, translated by Toshiaki Kurokawa


Note: This article was organized based on the author’s understanding, with reference to the publications listed above.

Chapter 1: Foundations of Machine Learning and Deep Learning