What Is Greedy Decoding in AI? A Worked Example

What is greedy decoding in AI? It is the simplest way for a language model to choose its next word: at every step, take the single most likely token and move on. No randomness, no looking ahead, no second thoughts. Because of that, the same prompt gives the same answer, and the output tends to be safe, plain and, over long stretches, repetitive.

To see why, you need two pieces: how a model scores words, and what "most likely at each step" misses.

How a model picks the next token

A model does not write a sentence in one go. It produces a score for every token in its vocabulary, turns those into probabilities, and a decoding strategy decides which one to use. That strategy is separate from the model itself. The same model can write very differently depending on how you decode it. If scores are new to you, start with what logits are in AI.

The Hugging Face documentation on generation strategies puts greedy search plainly: it selects the next most likely token at each step, and it is the default decoding strategy in the Transformers library. The same page says it works well for tasks with relatively short outputs where creativity is not a priority, and that it breaks down on longer sequences because it begins to repeat itself. Their generation strategies guide also describes the alternatives, sampling and beam search.

What is greedy decoding in AI? A worked example where it loses

The numbers below are invented to make the arithmetic easy. They are not from a real model.

The prompt is "The", and the model must pick two more words.

Greedy decoding looks at step 1 and takes the highest number, "cat" at 0.40. At step 2 it takes "sat" at 0.50. The sentence is "The cat sat", and its overall probability is 0.40 x 0.50 = 0.20.

The better overall path was "The dog barked": 0.35 x 0.90 = 0.315. Greedy never saw it, because "dog" lost at step 1 by five points.

That is the core weakness. Greedy picks the best local move and can walk straight past the best whole sentence. Beam search was designed to fix this by tracking several candidate sequences and keeping the one with the highest overall probability. The same documentation says that if the number of beams is 1, beam search is equivalent to greedy search. Our post on what beam search is in AI walks through that fix.

Why greedy decoding repeats itself

Once a phrase is likely, it tends to stay likely. After the model writes "the results of the results of", the most probable next token is often the same pattern again. With no randomness and no penalty, greedy has nothing to push it out of the loop. That matches the behaviour the Hugging Face docs describe for longer outputs.

You can see the same pressure from the other side. A frequency penalty exists precisely to lower the score of words that have already appeared, which is a fix for exactly this symptom.

Greedy decoding vs sampling

Sampling picks a token at random in proportion to its probability. In the example above, "dog" would win about 35 times in 100 first steps. That is why sampling gives varied and sometimes more natural text, and why it sometimes goes off the rails.

Settings such as temperature and top-p shape how sampling behaves. As temperature approaches zero, sampling behaves more and more like greedy decoding, which is why many people set temperature to 0 when they want predictable answers.

Is a temperature of zero the same as greedy?

Close, not identical. Many APIs treat very low temperature as near-greedy, but you should not assume a hosted model returns byte-identical text on every call. Differences in how requests are batched and computed can still change the result in rare cases. If exact repeatability matters, test it on your own prompts rather than assuming. For why answers still differ run to run, see why AI gives different answers to the same question.

For the bigger picture of how these pieces fit together, read our guide to how AI models work.

When greedy decoding is the right choice

• Short, factual output. Extracting a date or a category, where the top answer is almost always the right one.

• Structured output. When you want the same format every time.

• Testing. Repeatable output makes bugs easier to find.

Avoid it for long-form writing, brainstorming and anything where variety counts.

Frequently asked questions

What is greedy decoding in AI?

It is a decoding strategy where the model picks the single most likely next token at each step. It has no randomness, so the same prompt gives the same output.

Is greedy decoding the same as temperature 0?

They are very close in practice. Very low temperature makes sampling nearly greedy, but hosted systems may still vary slightly, so test your own case.

Why does greedy decoding repeat itself?

Because a likely phrase keeps being likely. Nothing in plain greedy decoding pushes the model away from a pattern it has already started, so long outputs can loop.

What is the difference between greedy search and beam search?

Greedy search keeps one sequence and always takes the best next token. Beam search keeps several sequences and picks the one with the best overall probability. With one beam, beam search is the same as greedy.

When should I use greedy decoding?

Use it for short, factual or structured tasks where you want the same answer each time. Prefer sampling for creative or long-form text.