What Is Next Sentence Prediction? An Older Auxiliary Training Task
A secondary training task from earlier language models, aimed at teaching a model to judge whether two sentences actually belong together.
Judging whether two sentences belong together
Next sentence prediction is a training task where a model is given two sentences and has to judge whether the second one genuinely follows the first in the original text, or whether it was randomly paired from somewhere unrelated — a binary judgment about sentence-level coherence.
Why this was added alongside masked language modeling
Masked language modeling alone teaches good word-level and sentence-level representations, but doesn't explicitly teach a model about relationships between separate sentences. Next sentence prediction was added as a secondary task specifically to build that sentence-to-sentence coherence understanding.
Why it fell out of favor
Later research found next sentence prediction contributed less than originally believed, and that other training approaches achieved comparable or better results without it — it's mostly a historical footnote in language model training today rather than a technique in active current use.
Frequently Asked Questions
What is next sentence prediction?
A training task where a model judges whether a second sentence genuinely follows a first sentence in the original text, or was randomly paired from somewhere unrelated — used to build sentence-level coherence understanding.
Is next sentence prediction still used in modern language models?
Not widely — later research found it contributed less than originally believed, and it has mostly fallen out of use in favor of other training approaches that achieve comparable results without it.