What Is Stochastic Gradient Descent (SGD)?
The original, simplest version of the algorithm that trains neural networks — updating parameters based on small random batches instead of the entire dataset at once.
Updating from small batches, not the whole dataset
Stochastic gradient descent updates a model's parameters using the error measured on a small, randomly sampled batch of training examples at a time, rather than computing the exact error across the entire dataset before making a single update — "stochastic" referring to that element of randomness in which examples get used when.
Why using small batches actually works well
Computing an update from the full dataset every single step would be prohibitively slow for large datasets. Using small random batches instead gives a noisier but much faster and more frequent stream of updates — and that noise turns out to help avoid getting stuck in certain poor solutions during training, rather than being purely a downside.
SGD as the ancestor of modern optimizers
Plain SGD is rarely used directly for training large models today — more advanced optimizers like Adam build on the same core idea (updating from small batches) while adding adaptive, per-parameter step sizing that plain SGD lacks.
Frequently Asked Questions
What is stochastic gradient descent?
An optimization approach that updates a model's parameters using the error measured on a small, randomly sampled batch of training examples, rather than computing an exact update from the entire dataset at once.
Is stochastic gradient descent still used for training large models?
Rarely in its plain form — modern optimizers like Adam build on the same core idea of batch-based updates while adding adaptive, per-parameter step sizing that plain SGD doesn't have.