What Is Top-p and Top-k in AI Text Generation
Top-p and top-k both trim which words an AI model can pick next, but one uses a fixed count and the other a moving probability threshold. Here is the practical difference.
What Is Top-p and Top-k in AI Text Generation
Top-p and top-k in AI text generation are two settings that control how a model chooses its next word. Top-k limits the model to a fixed number of the most likely next tokens, then picks among them. Top-p, also called nucleus sampling, instead limits the model to the smallest group of tokens whose combined probability crosses a threshold you set, so the pool of choices shrinks or grows depending on how confident the model is at that moment. Both settings sit next to temperature in most API playgrounds, and mixing up what each one actually does is a common reason generated text comes out either flat and repetitive or unpredictable and rambling.
This guide covers what each setting does on its own, how they interact with temperature, which providers expose which controls, and a practical starting framework for picking values by use case.
What Top-k Sampling Does
Top-k sampling restricts the model to a fixed number of candidate tokens before it samples the next word. If top-k is set to 40, the model looks only at the 40 most probable next tokens for that position, ignores everything else in its vocabulary, and samples from that shortened list. A top-k of 1 is greedy decoding: the model always takes the single most likely token, which makes output deterministic and often repetitive.
Google's Gemini API documentation puts the mechanic plainly: a topK of 1 means the selected token is the most probable among all tokens in the model's vocabulary (also called greedy decoding), while a topK of 3 means the next token is selected from among the 3 most probable candidates. Raise top-k and the model has more room to pick something less obvious, which adds variety but also risk.
What Top-p (Nucleus) Sampling Does
Top-p sampling works on probability mass instead of a fixed count. You set a threshold, like 0.9, the model ranks possible next tokens by probability, adds them up starting from the most likely, and stops as soon as the running total crosses that threshold. Everything outside that shortlist gets zero chance of being picked.
Per the same Gemini documentation, tokens are selected from most to least probable until the sum of their probabilities equals the topP value. The size of that shortlist moves on its own: when the model is confident about what comes next, the list is short and tight; when it is genuinely unsure, more tokens qualify and the output has more room to vary.
Where nucleus sampling comes from
The technique was introduced in 2019 by Ari Holtzman and coauthors in “The Curious Case of Neural Text Degeneration,” as a fix for the repetitive, looping text that older decoding methods like beam search tended to produce. Nucleus sampling caught on because it adapts to context automatically: a sentence with an obvious next word gets a tight, safe shortlist, while an open-ended prompt gets a wider one, without you having to tune anything mid-generation.
Top-p vs Top-k: The Practical Difference
Both settings trim the list of tokens the model is allowed to consider. The difference is whether that list has a fixed size or a size that changes with context.
Setting | What it limits | Shape of the limit | What we know about defaults |
|---|---|---|---|
Top-k | Number of candidate tokens | Fixed count, same regardless of model confidence | Not exposed by every provider (see below) |
Top-p | Cumulative probability of candidate tokens | Dynamic, expands or shrinks with model confidence | Google's Gemini API defaults topP to 0.95 |
Which AI Providers Expose Which Parameter
Support for these two settings is not universal, which matters if you are writing prompts meant to work across more than one model provider. According to Vellum's comparison of provider parameters, OpenAI's Chat Completions API exposes top_p but does not expose a top_k control at all. Anthropic's Messages API supports both, with top_k as an optional parameter. Google's Gemini API also supports both, as topP and topK.
Provider / API | top_p | top_k |
|---|---|---|
OpenAI Chat Completions | Yes | Not exposed |
Anthropic Messages API | Yes | Yes, optional |
Google Gemini API | Yes (topP) | Yes (topK) |
The same source notes that top-p and top-k generally should not be tuned at the same time, since both restrict the same candidate pool from different angles and stacking them makes it harder to know which one caused a change in output.
How Temperature Fits Into This
Temperature reshapes how sharply the model favors high-probability tokens before top-p or top-k trims the eligible list. A low temperature flattens the odds toward the single best guess, a high temperature spreads probability more evenly across plausible options. Top-p and top-k then decide how much of that reshaped distribution is even allowed to be sampled from.
A discussion on the OpenAI developer community points out that OpenAI's own documentation suggests changing temperature or top_p, not both, so that when output changes you can tell which knob caused it. Some experienced users report combining a higher temperature with a moderate top_p for creative work, but that is a deliberate experiment, not the default recommendation.
A Starting Framework by Use Case
These are starting points to test against your own prompts and model, not fixed rules. The pattern that holds across providers: tighten the settings for tasks where correctness matters more than variety, loosen them where variety is the point.
Use case | What you want | A reasonable starting point |
|---|---|---|
Code, SQL, structured or JSON output | Predictable, syntactically valid | Low temperature, tight top-p (roughly 0.1 to 0.3) |
Factual answers, summaries, extraction | Accurate wording, low invention | Low to moderate temperature, moderate top-p (roughly 0.3 to 0.5) |
Marketing copy, brainstorming lists | Some variety without going off the rails | Moderate temperature, top-p near the common default (roughly 0.9 to 0.95) |
Creative writing, fiction, brand voice exploration | Maximum variety, willing to risk an odd line | Higher temperature, top-p near 0.95 to 1, or a generous top-k where available |
Worth noting: for its newest Gemini 3.x models, Google's own documentation says it strongly recommends keeping temperature, topP, and topK at their defaults, warning that changing them, including setting temperature below 1.0, can cause looping or degraded performance. Model-specific guidance like that should override any generic table, including this one.
Matching the Symptom to the Right Knob
When output isn't working, it helps to diagnose which setting is actually responsible before changing anything.
Output repeats phrases or loops: raise top-p or top-k slightly first, and check that temperature isn't set too close to zero.
Output rambles, invents details, or drifts off-topic: tighten top-p before touching temperature, since a wide token pool is often the more direct cause.
Output feels the same every run when you wanted variety: raise top-p a little, or nudge temperature up in small steps.
Output breaks formatting, like malformed JSON or broken code syntax: lower both temperature and top-p together, since either one alone can still let in an unpredictable token at the wrong spot.
An Example Generation Config
Most APIs that expose these controls accept them as part of a generation config alongside the prompt. An illustrative example, using field names modeled on Gemini's GenerationConfig, for a moderate, general-purpose setup:
{
"temperature": 0.7,
"topP": 0.9,
"topK": 40,
"maxOutputTokens": 800
}Treat those numbers as a place to start a test, not a setting to copy blindly. Change one value, generate a handful of samples, and compare before touching a second value.
Where This Fits With Other Prompting Controls
Top-p and top-k are decoding settings, separate from the wording of the prompt itself. If your actual problem is that identical prompts return different answers on different runs, sampling settings are one piece of that puzzle alongside model version and context. For a broader look at getting the same AI output every time, start with prompt structure before reaching for these parameters. And if you haven't tuned prompts systematically before, a general framework for writing prompts that work is a useful place to start, since sampling settings only matter once the prompt itself is solid.
Temperature is the setting most people reach for first. If you want a closer look at how temperature shapes AI output on its own, or a deeper, single-parameter dive into nucleus sampling, both cover territory this piece only summarizes.
Questions
What is a good top-p value?
There is no single correct value. Providers that expose it often default to something in the 0.9 to 0.95 range for general use (Google's Gemini API defaults topP to 0.95), while tightening it toward 0.1 to 0.3 favors safer, more predictable text. Test against your own prompts rather than copying a default from another use case.
Should I use top-p or top-k?
Use top-p if your provider only exposes one control, since it is the more widely supported of the two and adapts to the model's confidence rather than using a fixed count. If your provider exposes both, most guidance points toward adjusting one at a time rather than stacking them.
Does ChatGPT use top-p or top-k?
OpenAI's Chat Completions API exposes a top_p parameter and does not expose a top_k control, per Vellum's provider comparison. What the ChatGPT product itself uses internally isn't published parameter by parameter, but developers building against the API only get a top_p dial to turn.
Can I use temperature and top-p together?
You can, but OpenAI's own documentation, as referenced in its developer community forum, suggests changing temperature or top_p rather than both at once, so it stays clear which setting produced a given change in output.
What happens if top-p is set to 1?
A top_p of 1 means the model considers its full probability distribution for the next token, which effectively turns top-p filtering off and leaves temperature, and top-k if it is set, to do the work of shaping the output.
How did this land?
About the author
Developer Advocate
Steve builds something with Swarmz every week and writes up what worked, what broke, and what he'd do differently. Tutorials and hands-on guides are his lane.
