In supervised learning, a model represents a mapping from one or more inputs to one or more outputs.

A model is represented as a function that includes inputs and parameters.

The process of providing an input to a model and calculating an output is called inference.

The mathematical expression of a model represents a family of possible relationships between inputs and outputs. Selecting particular parameter values determines a specific relationship from within that family.

Basic Mechanism of Supervised Learning

In supervised learning, a model is constructed to receive an input $\boldsymbol{x}$ and produce a prediction $\boldsymbol{y}$.

For simplicity, assume that the input $\boldsymbol{x}$ and output $\boldsymbol{y}$ are both fixed-length vectors whose elements always appear in the same order.

Data in which each data point consists of predefined fields arranged in a fixed order is called structured data or tabular data.

To obtain a prediction $\boldsymbol{y}$ from an input $\boldsymbol{x}$, a model $f[\cdot]$ is required that takes $\boldsymbol{x}$ as its input and returns $\boldsymbol{y}$.

The model’s prediction is therefore expressed by the following equation.

Calculating a prediction $\boldsymbol{y}$ from an input $\boldsymbol{x}$ in this manner is called inference.

In addition to the input, the model contains a parameter vector $\boldsymbol{\phi}$.

By selecting the parameter values, a specific input–output relationship is determined from the family of relationships represented by the model.

The actual model is therefore expressed as follows.

Learning or training a model means finding the parameters $\boldsymbol{\phi}$ that enable the model to make reasonable predictions from its inputs.

Training uses a training dataset consisting of $I$ input–output pairs, each combining an input with its corresponding correct output.

During training, the parameters are adjusted so that the model’s prediction for each training input $\boldsymbol{x}_i$ becomes as close as possible to the corresponding training output $\boldsymbol{y}_i$.

The discrepancy between the model’s prediction and the correct output is quantified by a loss $L$.

The loss is a scalar value that indicates how inaccurately the model with parameters $\boldsymbol{\phi}$ predicts the correct outputs corresponding to the training inputs.

Once the training dataset is fixed, the loss can be treated as a function $L[\boldsymbol{\phi}]$ of the parameters $\boldsymbol{\phi}$.

The objective of training the model is to find the parameters $\hat{\boldsymbol{\phi}}$ that minimize this loss function.

A small loss after training indicates that parameters have been found that allow the model to predict the training outputs $\boldsymbol{y}_i$ appropriately from the training inputs $\boldsymbol{x}_i$.

However, a small loss on the training data alone does not necessarily mean that the model can also make appropriate predictions for unseen data.

The model’s performance is therefore evaluated using separate test data that was not used to train the model.

If the model also makes appropriate predictions for the test data, it is considered capable of generalizing to data that was not observed during training.

A model that has demonstrated sufficient performance can be deployed for use in actual predictions.

Example of Linear Regression

As a basic example of supervised learning, consider a model $y=f[x,\boldsymbol{\phi}]$ that predicts a single output $y$ from a single input $x$.

One-Dimensional Linear Regression Model

A one-dimensional linear regression model represents the relationship between the input $x$ and the output $y$ as a straight line.

Here, $\phi_0$ represents the $y$-intercept and $\phi_1$ represents the slope. The model therefore has a vector of two parameters, $\boldsymbol{\phi}=[\phi_0,\phi_1]^{\mathrm{T}}$.

Changing the values of the $y$-intercept $\phi_0$ and the slope $\phi_1$ changes the straight line representing the relationship between the input and output.

Equation (1.2-4) defines a family of straight lines that represent possible relationships between the input and output. Selecting values for the parameters $\boldsymbol{\phi}$ determines one particular line from this family.

Loss

In one-dimensional linear regression, the training dataset consists of $I$ input–output pairs.

To train the model, a criterion is required for determining which parameters $\boldsymbol{\phi}$ provide the best fit to the data.

The discrepancy between the model’s predictions and the correct outputs is therefore expressed numerically.

This numerical value is the loss. A smaller loss indicates that the model fits the training data more closely.

The model’s predicted value for an input $x_i$ is $f[x_i,\boldsymbol{\phi}]$. The difference between this prediction and the correct output $y_i$ is regarded as the prediction error.

In one-dimensional linear regression, the sum of the squared errors across all training data is expressed by the following equation.

Because the errors are squared, their magnitudes can be evaluated as positive values regardless of whether the predicted line lies above or below the data points.

The method of finding the parameters that minimize this sum of squared errors is called the method of least squares.

The loss $L$ is a function of the parameters $\boldsymbol{\phi}$.

The loss is large when the model fits the training data poorly and small when the model fits the training data well.

A function of this kind is called a loss function or cost function.

Training the linear regression model therefore involves finding the parameters $\hat{\boldsymbol{\phi}}$ that satisfy the following equation.

This model has two parameters: the $y$-intercept $\phi_0$ and the slope $\phi_1$.

The loss can therefore be calculated for various combinations of parameter values, allowing the loss function to be visualized as a surface.

The minimum point on the loss-function surface corresponds to the parameters that best fit the training data.

Training

The process of finding the parameters that minimize the loss is called model fitting, model training, or model learning.

One basic method for reducing the loss is gradient descent.

In gradient descent, initial values are first assigned to the parameters. The parameters are then changed gradually in a direction that reduces the value of the loss function.

This corresponds to moving gradually downward across the surface formed by the loss function.

The gradient of the surface at the current position is calculated, and the parameters are updated on the basis of this gradient in the direction that produces the greatest decrease in the loss.

This operation is repeated with the aim of reaching a point where the gradient approaches zero and the loss can no longer be reduced.

In one-dimensional linear regression, the parameters that minimize the loss can be obtained as a closed-form solution, so iterative calculations are not always necessary.

A closed-form expression is one in which the solution can be stated explicitly using a finite number of arithmetic operations and elementary functions.

For more complex models, however, a closed-form solution may not be available, or the number of parameters may be too large to calculate the loss for every possible combination.

Gradient descent can also be used to find the parameters of such complex models.

Testing

After training a model, it is necessary to determine how well it performs on data that was not used during training.

For this purpose, a test dataset prepared separately from the training data is provided to the model, and the loss between its predictions and the correct outputs is calculated.

The extent to which a model can generalize to unseen data also depends on how well the training data represents the data encountered in practice.

A simple model, such as a straight line, may be unable to capture the true relationship between the inputs and outputs adequately.

A situation in which a model lacks sufficient expressive capacity and cannot fit even the training data adequately is called underfitting.

Conversely, if a highly expressive model fits even the atypical features contained in the training data too closely, it may make inappropriate predictions for unseen data despite having a small loss on the training data.

This situation is called overfitting.

Discriminative and Generative Models

The type of model considered so far, expressed as $\boldsymbol{y}=f[\boldsymbol{x},\boldsymbol{\phi}]$, directly predicts an output $\boldsymbol{y}$ from an input $\boldsymbol{x}$.

This type of model is called a discriminative model.

By contrast, a model such as $\boldsymbol{x}=g[\boldsymbol{y},\boldsymbol{\phi}]$, which represents how real-world measurements $\boldsymbol{x}$ are generated from an output $\boldsymbol{y}$, is called a generative model.

A discriminative model directly determines the output $\boldsymbol{y}$ from the observed input $\boldsymbol{x}$.

When performing inference with a generative model, however, it is necessary to determine the value of $\boldsymbol{y}$ from which the observed $\boldsymbol{x}$ was generated.

However, tracing the generative relationship in reverse to determine an output from an observation is not always straightforward.

Generative models offer the advantage of allowing prior knowledge about how real-world measurements are generated to be incorporated into the model.

However, when a highly expressive discriminative model can be trained using a sufficiently large amount of training data, the relative benefit of incorporating prior knowledge into a generative model is often reduced.

For this reason, discriminative models that directly learn the relationship from inputs to outputs are widely used in modern machine learning.

Summary

In supervised learning, training data consisting of inputs paired with their correct outputs is used to learn the parameters of a model that predicts outputs from inputs.

The discrepancy between a model’s predictions and the correct outputs is represented by a loss function. Model training involves finding the parameters that reduce this loss.

One-dimensional linear regression is a simple model that represents the relationship between an input and an output as a straight line. It may therefore be unable to capture complex input–output relationships that cannot be represented by a straight line.

To represent more complex relationships, models with greater expressive capacity, such as shallow neural networks and deep neural networks, are used.

Deep neural networks, in particular, can sometimes represent complex functions efficiently by combining multiple layers.

However, simply increasing a model’s expressive capacity is not sufficient.

The model’s ability to generalize appropriately to unseen data must be evaluated using test data that is separate from the training data.

About This Article

Reference
・Understanding Deep Learning I: Supervised Learning and the Foundations of Models, Simon J. D. Prince, translated by Toshiaki Kurokawa


Note: This article was organized based on the author’s understanding, with reference to the publication listed above.

Chapter 1: Foundations of Machine Learning and Deep Learning