What Is Top-P in AI? Nucleus Sampling Explained

Top-p is the setting that decides how big the model's shortlist is before it picks the next word. Set top-p to 0.9 and the model considers only the smallest group of candidate tokens whose probabilities add up to 90 percent, then samples from that group and throws the rest away. It is also called nucleus sampling, and it is one of two dials most APIs expose for controlling randomness.

That is the whole mechanism. What follows is the arithmetic, why it is not the same as temperature, and the one rule that saves most people a wasted afternoon.

How top-p actually picks tokens

Every time a model produces a word, it first produces a ranked list of candidates with probabilities attached. Suppose the text so far is "The capital of France is" and the model's next-token probabilities come out like this:

Top-p walks down that running total column and stops the moment it reaches the threshold.

• At top-p 0.5, the walk stops at the first row. Paris is the only survivor, because 0.72 already clears 0.5. The model has no choice left to make.

• At top-p 0.9, four tokens survive: Paris, the, a, located. The model samples among them, weighted by their probabilities, so Paris still wins most of the time.

• At top-p 0.95, Lyon enters the pool. It is a fairly unlikely token that produces a wrong answer, and you just gave it a two percent chance of being selected.

The pool size is not fixed. That is the important part, and the part people miss. On a sentence where the model is certain, top-p 0.9 might admit one token. On a sentence where it is genuinely torn between twenty phrasings, the same 0.9 might admit thirty. Top-p adapts to the model's confidence, which is exactly what it was designed to do. The technique was introduced in The Curious Case of Neural Text Degeneration, which showed that always taking the most likely token produces flat, repetitive text, while sampling from the full distribution occasionally produces nonsense from the tail.

Top-p is not temperature

Both dials affect randomness, so they get used interchangeably in blog posts and then people wonder why nothing behaves as expected. They do different things to the same list of numbers.

Temperature flattens or sharpens the whole distribution. Low temperature makes likely tokens even more likely and unlikely ones nearly impossible, but nothing is formally excluded. Top-p leaves the odds alone and draws a hard line, below which tokens simply do not exist for this step.

The rule: change one, not both

Raising temperature flattens the distribution. A flatter distribution means each token carries less probability, which means you need more of them to reach the same cumulative threshold. So raising temperature silently widens the top-p pool even though you never touched top-p.

Run the numbers on the table above at a higher temperature and Paris might drop from 0.72 to 0.55. Now top-p 0.9 admits seven tokens instead of four. You moved one dial and changed two behaviours, and when the output degrades you have no way to attribute it.

Pick the one that matches how you think about the problem. If you want to describe how adventurous the model should be, use temperature and leave top-p at its default of 1.0. If you want to describe how much of the tail you are willing to tolerate, use top-p and leave temperature at 1.0. Most provider documentation says the same thing, and it is the rare piece of API advice that is worth following literally.

When top-p is the right dial

Top-p earns its place when the failure you are trying to prevent is specifically a tail failure: the model was doing fine and then produced one bizarre word that derailed the rest of the response.

• Long-form generation that drifts. In a 900-word draft, a single low-probability token early on can send the whole piece somewhere strange, because everything after it is conditioned on that mistake. Dropping top-p to around 0.85 removes the tail without making the prose robotic.

• Names, identifiers, and enumerated values. If the model must choose from a known set, the tail contains only wrong answers. Tighten hard, to 0.5 or below.

• Creative work where you want range but not chaos. Around 0.95 keeps genuine alternatives available while cutting the genuinely broken options.

Where top-p is the wrong tool: getting reliable structured output. Tightening it helps a little, but the real fix is schema enforcement, which is a different mechanism entirely. We covered that in getting JSON output from AI reliably.

How to tell top-p is your problem

The symptom pattern is specific. Top-p is worth investigating when the output is mostly good and occasionally, unpredictably, contains something that no reasonable model should have said. One wrong proper noun. A sentence that changes topic. A number pulled from nowhere.

If instead the output is consistently bland, consistently the same across runs, or consistently structured wrong, top-p is not your problem. Bland means your threshold is too low or the prompt is underspecified. Identical across runs usually means temperature is at or near zero, which is worth understanding because the same question can otherwise produce different answers. Structured wrong is a prompt and validation issue.

A useful diagnostic: run the same prompt twenty times at your current settings and read only the failures. If the failures are all different from each other and each contains one obviously odd token, tighten top-p. If the failures are all the same failure, the sampling settings are innocent and the prompt is guilty.

Defaults you can assume

Nearly every major API defaults top-p to 1.0, which means no truncation at all: the full distribution is available and temperature is doing all the work. That default is deliberate. Providers would rather you tune the dial you understand than have two interacting parameters producing behaviour nobody can explain.

Reasoning models are a separate case. Several providers now ignore or restrict sampling parameters on their reasoning tiers, because the internal reasoning process is what is generating the variation, not the sampler. Check the current API reference for the specific model rather than assuming your settings are being applied. If you want the wider picture of what is happening underneath these parameters, how AI models work covers the generation loop these dials sit inside.

Frequently asked questions

What is a good top-p value?

Leave it at 1.0 unless you have a tail problem. If you do, 0.9 is a sensible first move and 0.85 is about as low as you can go for prose before it starts sounding stilted. For constrained outputs like categories or identifiers, values below 0.5 are fine.

Does top-p 1.0 mean maximum randomness?

No, it means no truncation. The model still samples according to the probabilities, so a confident model at top-p 1.0 will still say Paris almost every time. Maximum randomness would require raising temperature as well.

Is top-p the same as top-k?

No. Top-k keeps a fixed number of candidates, say the top 40, regardless of how confident the model is. Top-p keeps a variable number based on cumulative probability. Top-p is generally preferred because it adapts: it stays narrow when the model is sure and widens when it genuinely is not.

Should I set top-p to 0 for factual answers?

You cannot usefully set it to 0, and near-zero values just reproduce greedy decoding, which is what temperature 0 already gives you. Worse, it does not make answers factual. Restricting the sampler cannot fix a model that does not know the answer. It only makes the wrong answer arrive more consistently.

Do top-p settings carry over between requests?

No. Sampling parameters apply per request. If you are seeing behaviour change between calls with identical settings, something else is different: the model version, the system prompt, or the amount of context you are sending.