System Prompt vs User Prompt: What Goes Where

The system prompt is where you put everything that stays the same across every request. The user prompt is where you put the one thing that changes. Get that split right and a flaky AI feature usually becomes a reliable one, because you stop re-explaining the job on every call and the model stops getting a slightly different job each time.

If you are building anything on top of a model, this is the highest-leverage structural decision available to you, and it costs nothing.

The two slots

Chat models take messages with roles. Two of them matter for this.

The system message sets standing instructions: who the assistant is, what it does, what it must never do, what format the output takes, what it should do when it lacks information. It is read as the operator speaking, and models weight it accordingly. Anthropic's prompting documentation and OpenAI's prompt engineering guide both treat it as the place for role and standing rules.

The user message carries the request. The customer's question, the document to summarise, the row to classify. It is the variable in the equation.

The distinction is not cosmetic. Instructions in the system slot survive the conversation. Instructions buried in a user message compete for attention with everything else in that message, and are more easily displaced by whatever the user typed after them.

The test

One question resolves almost every case. Would this sentence be identical on the next request?

Identical every time, goes in the system prompt. Changes per request, goes in the user prompt.

The awkward middle case is retrieved context. It changes per request, so mechanically it belongs in the user message, and that is where most implementations put it. What belongs in the system prompt is the instruction about how to treat it: cite it, prefer it over prior knowledge, admit when it does not cover the question.

What a good system prompt contains

Six parts, and you rarely need more.

1. Role and domain. Two sentences. Enough to set vocabulary and assumptions.

2. The task. What this assistant does on every call, stated as a job rather than a personality.

3. Constraints. Length, tone, reading level, what it must never do. Be specific. "Be concise" is not an instruction, "under 120 words" is.

4. Output format. If you parse the output, describe the exact shape and give one example. Ambiguity here costs you more downtime than anything else on the list.

5. The uncertainty rule. What to do when the answer is not available. Without this, the model will invent something, for reasons covered in what an AI hallucination is. This single line prevents a large fraction of production embarrassment.

6. Escalation. When to stop and hand to a human. Any assistant touching money, legal commitments, or customer accounts needs this.

Things that do not belong: the current user's question, today's date if you can inject it dynamically, anything you will want to change per customer, and long preambles about being helpful and harmless that the model already does by default.

A worked before and after

A real pattern from support tooling. First version, everything in one user message:

> Answer this customer question. Be helpful and professional. Do not make up policies. Here are our docs: [8,000 words of documentation]. Customer question: do you ship to Norway?

This works about seventy percent of the time. The failures share a shape: on long documents the instruction at the top has a lot of text between it and the answer, and when the customer question itself contains instruction-like phrasing, the model sometimes follows the customer rather than you.

Second version, split:

System: You are a support assistant for a plumbing supply company. Answer only from the documentation provided in the user message. If the documentation does not cover the question, say so and offer to connect the customer to a human. Never state prices, delivery dates, or return terms that do not appear verbatim in the documentation. Reply in under 120 words, plain prose, no bullet points.

User: Documentation: [retrieved sections only]. Customer question: do you ship to Norway?

Two changes did the work. The rules moved somewhere they are not competing with 8,000 words, and the documentation shrank to the relevant sections, which is a retrieval problem rather than a prompting one. If you are sending a large corpus on every call, the context window is doing work you should be doing with retrieval, and you are paying for it every request.

Common mistakes

Writing a personality instead of a specification. "You are a friendly and enthusiastic assistant who loves helping people" does nothing that "answer warmly and directly" does not, and it costs tokens on every call.

Putting the format example in the user message. Then it changes when the user's input changes, and your parser breaks intermittently, which is the worst kind of broken.

Assuming the system prompt is a security boundary. It is not. It is a strong steer, not enforcement. A determined user can often talk around it. Anything that genuinely must not happen belongs in code that checks the output, not in a sentence asking nicely.

Never revising it. The system prompt is the one artifact that touches every request. When output quality drifts, it is the first place to look. The rewriting method in how to fix a bad AI prompt applies to system prompts too, and pays off more there because the fix applies to every future call. That is also a good reason for keeping track of prompt versions as they change, so you can see exactly what shifted and revert when a change makes things worse.

Cramming tool instructions into prose. If the model is calling tools, describe the tools in the tool definitions where they belong. Modern tooling standards like the Model Context Protocol exist so that tool descriptions live with the tools rather than in a paragraph the model has to parse.

For app builders specifically, the same split shows up under different names, and writing prompts for AI app builders covers how it translates when there is no visible system field.

Frequently asked questions

Does the system prompt count toward my token cost?

Yes, on every single request. A 500-word system prompt sent 10,000 times a day is a real line item. This is a good reason to keep it tight, and a good reason to check whether your provider offers prompt caching for the static portion.

Is the system prompt more powerful than the user prompt?

It carries more weight in how models are trained to prioritise instructions, but it is not absolute. Treat it as a strong default that a sufficiently determined user message can sometimes override, and never as a guarantee.

What if the tool I use has no system prompt field?

Many consumer chat interfaces and app builders hide it. Custom instructions, project instructions, or a persistent first message usually fill the same role. If nothing does, put your standing rules at the very start of the message and repeat the critical one at the end, which is a workaround rather than a fix.

Should the system prompt include examples?

One or two, if the output format is non-obvious. Examples are the most reliable way to communicate format and the most expensive in tokens, so keep them minimal and make them representative rather than exhaustive.

Can I change the system prompt mid-conversation?

Technically yes on most APIs, since you construct the message list each turn. It can confuse the model if the change contradicts what it has already been doing, so prefer starting a new conversation when the job genuinely changes.

Related: debugging a prompt that suddenly stopped working