Stochastic gradient descent

A gradient descent algorithm in which the batch size is one. In other words, SGD trains on a single example chosen uniformly at random from a training set.¹

Footnotes

developers.google.com/machine-learning/glossary#SGD

Nested Learning: The Illusion of Deep Learning Architectures 이 글을 인용: “In addition to its neuroscientifically plausible and mathematically white-box nature, we advocate for its impor…”
Convex function 이 글을 인용: “Stochastic gradient descent: Many variations of stochastic gradient descent have a high probability of finding …”
ML crash course - Linear regression “Define and describe how gradient descent finds the optimal model parameters.”
On the Dangers of Stochastic Parrots: Can Language Models Be Too Big? “BERT, its variants, GPT-2/3, and others, most recently Switch-C, have pushed the boundaries of the possible bot…”
역전파 “The algorithm that implements gradient descent in neural networks.”
Clipping (machine learning) “Gradient clipping forces gradient values within a designated range during training.”
Stochastic Neighbor Embedding (paper) “A natural cost function is a sum of Kullback-Leibler divergences, one per object, which leads to a simple gradi…”
확률적 앵무새 논문의 오리진 스토리 관련 절: Chapter 12. Myth Busters
Vanishing gradient problem “Models suffering from the vanishing gradient problem become difficult or impossible to train.”
Exploding gradient problem “The tendency for gradient in DNNs (especially RNNs) to become surprisingly steep (high).”
mini-batch “For example, suppose the entire training set (the full batch) consists of 1,000 examples.”
Can In-Context Learning Support Intrinsic Curiosity? “However, evaluating these rewards traditionally requires expensive inner loops of gradient descent updates with…”
Learning rate “If you set the learning rate too high, gradient descent often has trouble reaching convergence.”
Gradient boosting “Gradient boosting is a powerful ensemble machine learning technique used for both regression and classification…”
Gradient (machine learning) “In machine learning, the gradient is the vector of partial derivatives of the model function.”
Gradient clipping “A commonly used mechanism to mitigate the exploding gradient problem by artificially limiting (clipping) the ma…”
Convex optimization “The process of using mathematical techniques such as gradient descent to find the minimum of a convex function.”
t-SNE “It is based on Stochastic Neighbor Embedding (paper) originally developed by Geoffrey Hinton and Sam Roweis, wh…”
Epoch (machine learning) “An epoch represents N/batch size training iterations, where N is the total number of examples.”
Optimizer (machine learning) “A specific implementation of the gradient descent algorithm.”