What Is Beam Search in AI? A Plain Explanation
Beam search tracks several candidate word sequences at once instead of one, often finding a better overall sentence than greedy decoding alone.
What Is Beam Search in AI? A Plain Explanation
Beam search is a decoding method that lets a language model weigh several possible word sequences at once instead of locking in one word at a time. At each step, it keeps a fixed number of the most promising partial sequences, called the beam width, and expands every one of them before narrowing back down to the best few. This differs from greedy decoding, which simply grabs the single highest-probability next word at each step and never looks back. Because beam search compares whole candidate sentences rather than isolated words, it often lands on a more coherent full sequence than greedy decoding does, even starting from the same underlying probabilities.
Why Not Just Pick the Best Word Every Time?
A language model produces a probability distribution over its vocabulary for the next token, then repeats that for every following token. Greedy decoding takes the shortcut of always choosing the single most likely token at each step. It is fast and works fine much of the time, but it is locally optimal, not globally optimal: a word that looks slightly better right now can lead down a path where every later word is a weaker fit, while a word that looked slightly worse a step earlier might have opened the door to a stronger overall sentence. Greedy decoding never notices, because once it picks a token it never reconsiders.
How Beam Search Works
Beam search fixes that blind spot by keeping several candidate sequences alive at once. A beam width of k means the model tracks the k highest-scoring partial sequences at every step:
Score the next-token probabilities for each of the k sequences currently being tracked.
Extend each sequence with plausible next tokens, creating many new candidates.
Rank all candidates by cumulative probability across their full token history so far.
Keep only the top k, drop the rest, and repeat until an end token or max length.
Return whichever surviving sequence has the highest total score.
Set beam width to 1 and beam search behaves exactly like greedy decoding, since only one sequence survives each round. Widen it and the model runs several parallel "what if" continuations before deciding which one paid off.
A Worked Example: Greedy vs. a Beam Width of 3
The numbers below are invented for illustration only, not output from any real model, but they show the mechanic clearly. Say a model has generated the partial sentence "The chef opened the" and scores four candidate next words:
Candidate next word | Probability |
|---|---|
oven | 0.40 |
fridge | 0.35 |
book | 0.15 |
window | 0.10 |
Greedy decoding takes the top score and commits to "oven," never revisiting it. Suppose the next-step probabilities after "oven" favor "and" (0.30), and after "oven and" favor "the" (0.55). That path multiplies to roughly 0.40 x 0.30 x 0.55, about 0.066, and trails off unfinished: "The chef opened the oven and the..."
Beam search with width 3 keeps the top three first words, oven, fridge, and book, instead of discarding fridge and book right away. Down the fridge branch, "door" is the strongest continuation (0.60), and after "fridge door" an end-of-sentence token is strongly favored (0.65). That path multiplies to roughly 0.35 x 0.60 x 0.65, about 0.137, more than double the greedy score, and finishes clean: "The chef opened the fridge door."
Path | Tokens | Cumulative score |
|---|---|---|
Greedy (width 1) | oven -> and -> the | ~0.066 |
Beam search (width 3) | fridge -> door -> [end] | ~0.137 |
Greedy never sees the fridge branch, because "oven" edges it out at the first step and greedy only ever follows one thread. Beam search keeps fridge, book, and oven running in parallel long enough to discover that the second-best opening word actually leads to the best-scoring, most complete sentence. That is the core payoff: trading extra computation for the ability to compare whole candidate sequences rather than betting everything on the first move.
Beam Search vs Greedy Decoding
Quality: beam search generally finds a higher cumulative-probability sequence, since it compares more full candidates before deciding.
Compute cost: beam search does roughly k times the work of greedy decoding at each step.
Guarantees: neither approach guarantees the single best sequence out of every combination that exists; beam search widens the search, it does not make it exhaustive.
Best fit: greedy suits speed-sensitive generation; beam search suits tasks where one well-formed best answer matters more, such as translation or captioning.
Beam Width Explained: What the Number Actually Controls
Beam width is the one knob in beam search, controlling how many competing hypotheses stay alive at once. A width of 1 equals greedy decoding. Widths of roughly 4 to 10 are common in machine translation, usually capturing most of the quality gain over greedy without excessive extra cost. Pushing width much higher runs into diminishing returns: cost per generated token keeps climbing, and very wide beams can nudge output toward safer, more generic phrasing, an effect discussed in NLP decoding literature such as Jurafsky and Martin's chapter on decoding strategies. Wider is not automatically better; it is a trade between search coverage, compute cost, and output character.
Where Beam Search Shows Up in Language Models Today
Beam search has long been a default in neural machine translation, speech-to-text captioning, and summarization, tasks with usually one clearly best rendering of the input. It is one piece of the broader question of how AI models work once training is done and a model has to produce output token by token.
Open-ended conversational models tend to favor sampling-based decoding, such as temperature or nucleus sampling, since always chasing the highest-probability sequence can make replies feel repetitive. Beam search's core idea, tracking multiple candidates instead of one, still shows up conceptually in adjacent techniques; speculative decoding also evaluates multiple token candidates before committing, though to speed up generation rather than improve sequence quality.
A wider beam also means more parallel sequences to score at each step, which ties into why output tokens cost more than input tokens: each added beam multiplies the work done during decode. Teams managing that cost often pair decoding choices with other efficiency techniques, such as quantization to shrink the model, or model distillation to replace it with a smaller one.
Limitations Worth Knowing
Not truly optimal: it only compares the k sequences kept at each step, so a good sequence pruned early for a temporarily low score can be lost for good.
Cost scales with width: each unit of beam width roughly multiplies per-token compute.
Length bias: cumulative probability shrinks as sequences get longer, so plain beam search can favor shorter outputs unless scores are length-normalized.
Can flatten creative output: always picking the highest-scoring path tends toward safer, more predictable phrasing.
Beam search is part of the model-behavior vocabulary worth knowing alongside how a model's training data itself gets built. See our companion explainer on what synthetic data is, the companion "Understanding AI models" glossary entry on data that's generated rather than collected.
Frequently Asked Questions
Is beam search still used in modern language models?
It remains common in tasks with a single clear best output, such as translation, transcription, and summarization. General-purpose chatbots often favor sampling methods instead, since beam search's bias toward the highest-scoring path can make open-ended conversation feel repetitive.
What is a good beam width to use?
There is no universal answer, but widths around 4 to 10 are typical starting points in tasks like translation, capturing most of the quality gain over greedy decoding before diminishing returns set in.
Does a bigger beam width always produce better text?
No. Quality gains taper off quickly, compute cost keeps climbing with width, and very wide beams can push output toward blander, more generic phrasing.
What is the difference between beam search and greedy decoding?
Greedy decoding commits to the single highest-probability token at every step and never reconsiders. Beam search keeps several candidate sequences alive in parallel and picks the best one at the end, letting it recover from a merely-good early choice that leads somewhere better.
Does beam search guarantee the best possible output?
No. It is a heuristic search over a slice of all possible sequences, not an exhaustive one, so it can still miss the true best sequence if a strong candidate is pruned early for scoring lower than the surviving beams at that step.
How did this land?
About the author
Senior Editor, AI & Product
Cecilia leads the Swarmz editorial desk. She has spent a decade turning complex AI and product topics into writing people actually finish, and she owns the blog's quality bar.
