Previously in the series: The laws of large numbers and Basics of Bayesian learning.

Reminders: formalizing learning in ML and Bayesian learning

Learning and inference in neural nets and Bayesian models

As a very basic sketch, in order to specify an ML algorithm one needs five pieces of data.

An architecture: i.e., a parametrized space of functions that associates to each weight vector $θ \in R^{w}$ a function $f_{θ} : R^{d_{i n}} \to R^{d_{o u t}},$ from some input space to an output space.
An initialization prior on weights. This is a (usually stochastic) algorithm to initialize a weight from which to begin learning. Generally this is some Gaussian distribution on the weight $θ \in R^{w} .$ While this is often ignored, in many contexts it is actually quite important to get right for learning to behave reasonably.
Training data. This is a collection $D = (x_{1}, y_{1}; x_{2}, y_{2}; \dots; x_{n}, y_{n})$ of “observation” pairs $(x_{i}, y_{i}),$ with $x_{i} \in R^{d_{i n}}$ and $y_{i} \in R^{d_{o u t}} .$
A loss function. This is a function on $L (D, θ)$ which operationalizes a measure of how well $f_{θ}$ agrees with the data D.
A learning algorithm/optimizer. (“Learning algorithm” is used in theoretical contexts and “optimizer” is frequently used in engineering contexts.) This is an algorithm (usually, a stochastic algorithm) for finding the “learned” weight parameter $θ$ , usually by some local minimization or approximate minimization, of the function $L (θ)$ .

In most theoretical analyses of models, one uses gradient descent to conceptualize the learning algorithm. Some more sophisticated pictures (such as the tensor programs series) more carefully match realistic stochastic gradient descent by assuming a discontinuous process with finite learning steps rather than continuous gradient descent. Learning algorithms used in industry tend to include more sophisticated control over the gradients by things like gradient decay, momentum, ADAM, etc.

All of these algorithms have in common the property of being sequential and local (i.e., there is some step-to-step learning that ends when it converges to an approximate local minimum.) However, when working theoretically, a learning algorithm doesn’t have to be local or sequential.

Bayesian learning is one such (non-local and non-sequential) learning algorithm. This algorithm converts the learning problem to a Bayesian inference problem. Here the dictionary from ML to statistics is as follows:

Architecture $⟹$ D

...