What Is Greedy Decoding? The Simplest Way an AI Model Generates Text
The simplest possible generation strategy — always pick the single most likely next token — and why it isn't always the best choice.
Always take the top choice
Greedy decoding generates text by picking the single highest-probability token at every single step, with no randomness and no consideration of how that choice affects what comes several steps later. It's the most straightforward possible way to turn a model's output probabilities into actual text.
The problem: local best isn't global best
Always taking the top choice at each individual step can lead to a worse overall sequence than briefly accepting a slightly lower-probability token that opens up a much better continuation later. Greedy decoding has no way to look ahead or reconsider, so it can get stuck in locally-good-but-globally-mediocre output.
Alternatives that look further ahead
Beam search keeps multiple candidate sequences in play simultaneously rather than committing greedily at each step, while sampling-based approaches deliberately introduce randomness — both aim to avoid the specific trap greedy decoding is prone to, at the cost of more compute or more variability.
Frequently Asked Questions
What is greedy decoding?
A text generation strategy that always picks the single highest-probability next token at each step, with no randomness and no ability to reconsider earlier choices based on what comes later.
What's the downside of greedy decoding?
It can get stuck in a locally-good-but-globally-mediocre sequence, because always taking the top choice at each step doesn't guarantee the best overall result — it can't look ahead or backtrack.