DL Algorithms: Backpropagation Algorithm


Backpropagation (used in “backpropagation algorithms“) is a method used in artificial neural networks to calculate a gradient that is needed in the calculation of the weights to be used in the network (while “learning”). Backpropagation is shorthand for the backward propagation of errors, since an error is computed at the output and distributed backwards throughout the neural network’s layers. It is commonly used to train deep neural networks (deep learning).

The goal of any supervised learning algorithm is to find a function that best maps a set of inputs to their correct output. The motivation for backpropagation is to train a multi-layered neural network such that it can learn the appropriate internal representations to allow it to learn any arbitrary mapping of input to output.

What’s actually happening to a neural network as it learns?

Backpropagation is a generalisation of the delta rule to multi-layered feedforward neural networks, made possible by using the chain rule to iteratively compute gradients for each layer. It is closely related to the Gauss–Newton algorithm and is part of continuing research in neural backpropagation.

Backpropagation is a special case of a more general technique called automatic differentiation. In the context of learning, backpropagation is commonly used by the gradient descent optimization algorithm to adjust the weight of neurons by calculating the gradient of the loss function.