Why Your Prompt Works in Chat but Not in the API
The chat product is an application built on top of the model. When you call the API you get the model without the application, and everything the application was quietly doing stops.
Your prompt works in the chat window and returns something worse through the API because you are not talking to the same thing. The chat product is an application: it wraps the model in a system prompt you never see, a set of tools, conversation memory, file handling, its own sampling defaults and a formatting layer. Call the API and all of that disappears. The prompt did not get worse. Its environment did.
This matters practically because the fix depends entirely on which piece went missing, and there are only seven candidates.
Why a prompt works in chat and not in the API: seven differences
# | What the chat product adds | What you get in the API |
|---|---|---|
1 | A hidden system prompt with tone, formatting and behaviour rules | Nothing, unless you write one |
2 | Tool access: web search, code execution, file reading | No tools, unless you define and wire them |
3 | Conversation history managed for you | Only the messages you send, every time |
4 | Document parsing: PDFs, spreadsheets, images turned into text | Raw bytes you must handle yourself |
5 | Tuned sampling defaults chosen for chat | Provider defaults, often different |
6 | A specific model version behind a friendly name | Whatever alias or pinned ID you passed |
7 | Retry, truncation and safety post-processing | The raw response, including the ugly cases |
In practice, numbers one, two and three account for most of it. Someone tests a research-flavoured prompt in a chat product where the model quietly searched the web, then calls the API, gets a confident answer with no sources, and concludes the model got dumber. It did not. It lost its search tool.
Isolate the cause in four steps
Do not rewrite the prompt yet. Bisect first, the same way you would with a failing build.
Reproduce the chat behaviour in the chat product using a brand new conversation. If it fails there too, your original success depended on earlier conversation context, which is difference number three.
Send the identical prompt to the API with an empty system prompt and default settings. Compare the outputs side by side and write down what specifically is different: format, length, refusal, accuracy, or made-up facts.
Match on one axis at a time. Add a system prompt. Then set temperature explicitly. Then pin the model version. Then add tools. Change one thing per run and keep the outputs.
Stop at the change that fixes it. That is your answer, and it is usually reached by step two or three.
This takes about fifteen minutes and beats an hour of prompt rewriting, which is the reflex and usually the wrong move. The same reasoning applies when a prompt that used to work stops working, which we covered separately in how to debug a prompt that stopped working.
Difference one: the system prompt you never wrote
Chat products ship with substantial instructions about tone, formatting, hedging, refusals and how to handle ambiguity. Your prompt was written on top of that foundation and inherits it. Through the API the foundation is an empty string.
The practical consequence is that behaviours you assumed were the model's personality are actually instructions. Markdown headings, a friendly opener, asking a clarifying question instead of guessing, refusing to speculate: all of it can be configured, none of it is default. If your API output feels flatter or blunter, write the missing instructions down explicitly:
system:
You are answering questions from a technical audience.
Use short paragraphs and plain prose. No headings unless the answer
has three or more distinct parts.
If the question is ambiguous, ask one clarifying question rather than
guessing. If you do not know, say so instead of estimating.
Never invent citations, URLs, or version numbers.If the split between system and user roles is fuzzy for you, system prompt versus user prompt is the shorter version of that distinction.
Difference two: tools are not implied
A prompt that says check the latest documentation and summarise it will produce a summary either way. In the chat product it fetched the page. Through the API it wrote what a plausible page would say. Nothing in the response announces the difference, which is what makes this failure expensive.
Rule of thumb: if your prompt contains a verb that requires the world, look up, check, browse, run, calculate precisely, read this file, then it needs a tool. Either define one, or rewrite the prompt to work from material you paste in.
Difference five: sampling defaults are not the same
Chat interfaces pick sampling parameters tuned for conversation. API defaults are chosen for general use and often differ. If your API results are more erratic run to run, set the parameters explicitly rather than inheriting them, and read what temperature does before picking a number. Zero is not always right; for anything with a single correct answer it usually is, and for anything generative it usually is not.
Note that even at temperature zero you will not get byte-identical output across runs, for reasons unrelated to your settings. Why AI gives different answers covers the rest of that, and it is worth reading before you spend a day chasing determinism you cannot have.
Difference six: the model name is not a version
A chat product's model picker points at a specific build that changes when the vendor updates it. An API alias behaves the same way. If reproducibility matters, pin the dated model ID, log it with every request, and treat a version change as a deployment. Half the reports of a prompt mysteriously degrading overnight are a silently rotated alias.
A short checklist before you blame the model
Did the chat conversation have earlier turns the API call does not?
Does the prompt implicitly require a tool?
Is there a system prompt on the API call, and does it say the things you assumed?
Are temperature and max tokens set explicitly?
Is the model ID pinned, and logged?
Are you sending file contents as text, or hoping the API parses a PDF you attached?
Are you comparing a single sample against a single sample? Run five of each before concluding anything.
If everything on that list checks out and the output is still worse, then it is a prompt problem, and the general techniques in our prompt engineering guide apply. Most of the time you will not get that far.
Questions
Is the API model weaker than the chat model?
Usually it is the same weights. The difference is the application around it. Where a vendor does ship a chat-specific variant, they document it, and pinning an explicit model ID removes the ambiguity.
Why does the API not format its answers as nicely?
Because formatting instructions live in the chat product's hidden system prompt. Write them yourself and the formatting comes back.
How do I give the API the same conversation memory?
Send the prior turns in the messages array on every request. There is no server-side memory by default, and how much history you resend is a cost and quality decision you now own.
My API responses cut off mid-sentence. Same cause?
No, that is your max output tokens setting. Chat products manage that for you and often continue automatically. Raise the limit or handle the stop reason in your code.
How did this land?
About the author
Developer Advocate
Steve builds something with Swarmz every week and writes up what worked, what broke, and what he'd do differently. Tutorials and hands-on guides are his lane.
