SGD

SGD#

class seli.opt.SGD(lr: float = 0.001)[source]#

Bases: Optimizer

Stochastic Gradient Descent optimizer.

The gradient is the direction of steepest descent. The SGD update simply scaled the gradient by the learning rate and takes a step in that direction. It does not account for information from previous gradients.

There has been some evidence that SGD has a regularization effect, which leads to better generalization performance, at the cost of slower convergence.

Methods Summary

call_param(grad, **_)

Process the gradients of a single parameter.

Methods Documentation

call_param(grad: Float[Array, '*s'], **_) → Float[Array, '*s'][source]#

Process the gradients of a single parameter. This function is useful for implementing custom optimizers that essentially run the same function for all parameters. This is the case for most well known optimizers.

Parameters:

  • loss (Float[Array, ""]) – The absolute loss value.

  • key (str) – The key of the parameter.

  • grad (Float[Array]) – The gradients of the parameter.

  • param (Float[Array]) – The parameter values.

Returns:

grad – The processed gradients of the parameter.

Return type:

Float[Array]

Contents