How to Make AI Say I Don't Know Instead of Guessing
Models guess because guessing is the only move you left them. Four changes that make abstention a valid answer, and the eval that proves it worked.
Adding "if you don't know, say you don't know" to a prompt does almost nothing. Everyone tries it, it works in testing, and then the model invents a policy number in production anyway.
Learning how to make AI say I don't know is less about wording and more about design. A model guesses when guessing is the only available move. Fix that, and abstention starts happening without being begged for.
Why the polite instruction fails
Three reasons, all structural.
The instruction is one line competing with everything else in the prompt, most of which is telling the model to be helpful, thorough, and complete. Under that pressure, a non-answer reads as a failure to comply.
Negative instructions are weak. "Do not make things up" describes a space of things to avoid rather than an action to take, which is a known weakness in how models follow prompts.
Most decisively, there is often nowhere to put the uncertainty. If your output format is a paragraph of prose or a JSON object with a required answer field, the model has to fill it. It is not being dishonest. You built a form with no "unknown" box.
Four changes that work
1. Make abstention a named output
Stop asking for a refusal and start asking for a classification. Every answer picks one of a fixed set of statuses, and "insufficient information" is one of them.
Return one of:
answered - the sources contain a direct answer
partial - the sources cover part of the question
not_in_sources - the sources do not contain thisOnce "not in sources" is a legitimate value rather than a failure, the model selects it. You have moved abstention from a behaviour you hope for to a branch in a schema. This works considerably better as structured output than as free text, because a field with an enum is enforceable and a paragraph is not.
2. Require the evidence before the answer
Make the model quote the source span it is relying on, in a field that comes first.
{
"evidence": "<exact quote from the provided sources, or empty>",
"status": "answered | partial | not_in_sources",
"answer": "<answer, or empty if not_in_sources>"
}Ordering matters. Filling evidence first means the model has already committed to having or not having support before it writes the answer. An empty evidence field with a confident answer becomes visibly inconsistent, both to the model and to you.
This is also a check you can run in code: empty evidence plus a non-empty answer is a rejection, no judgement required.
3. Separate retrieval failure from knowledge failure
These are different events and blending them costs you information.
"The documents you gave me do not cover this" is a retrieval problem, and the fix is better search. "I do not have reliable knowledge about this" is a coverage problem, and the fix is finding a source. Give the model distinct statuses so your logs tell you which one is happening. This distinction is the practical side of grounding.
4. Show it what abstention looks like
Instructions describe, examples demonstrate, and examples win. Include one or two cases in the prompt where the correct answer is a non-answer.
Q: What is the refund window for enterprise plans?
Sources: [pricing page covering monthly and annual plans only]
{"evidence": "", "status": "not_in_sources", "answer": ""}One negative example does more than three sentences of instruction. Keep it short, because examples consume context and two well-chosen ones beat six.
Wording that does help
Prompt phrasing is secondary to structure, but some phrasings still measurably outperform others.
Name the cost of guessing in concrete terms: "A wrong answer here will be sent to a customer as a commitment" beats "be accurate".
Give abstention a positive frame: "Reporting missing information is the correct and expected outcome when sources do not cover the question" rather than "don't guess".
Scope the boundary explicitly: "Answer only from the sources provided. Your own background knowledge is out of scope for this task."
Ask for the gap: "If you cannot answer, state what specific information would be needed." This produces a useful artefact instead of a dead end, and it feeds straight back into fixing retrieval.
That last one is the highest-value line in the list. It turns every abstention into a bug report about your source coverage.
Test it, or you do not know
Abstention is easy to check and almost nobody checks it, which is how the "overreliance" failure in the OWASP Top 10 for LLM applications keeps showing up in shipped products.
Build a small adversarial set, twenty questions is enough:
Ten questions your sources genuinely answer. Correct behaviour is answering.
Five plausible questions your sources do not answer, phrased in the same style as the real ones.
Three questions about entities that do not exist, phrased as if they obviously do. "What is the cancellation policy for the Platinum tier?" when there is no Platinum tier.
Two questions where the sources contradict each other.
Score abstention rate on groups two and three separately from accuracy on group one, because the two move in opposite directions. Push abstention too hard and the model starts refusing questions it can answer, which is its own kind of broken. You are looking for the setting where group one stays high and groups two and three are near total.
Run this every time you change the prompt, the model, or the retrieval. It takes minutes and it is the difference between believing your system abstains and knowing it does. The wider practice is building evals for your own use case rather than trusting benchmarks built for someone else's.
FAQ
Does lowering temperature make the model admit uncertainty?
No. Lower temperature makes it more consistent, which mostly means it produces the same confident wrong answer every time instead of varying wrong answers. Consistency is not calibration.
Should I ask for a confidence score?
Self-reported confidence numbers are poorly calibrated and cluster around whatever value the prompt implies. A small set of discrete statuses backed by an evidence quote is more reliable than a number between 0 and 1 that the model made up.
What if abstentions annoy users?
Design the abstention to be useful. "That is not covered in the product documentation, here is who can answer it" is a good customer experience. A bare "I don't know" is not, and neither is a fabricated answer that gets discovered later.
How is this different from spotting hallucinations after the fact?
This is prevention at generation time. Detection afterwards is a separate and still necessary layer, covered in how to tell if an AI answer is hallucinated. Doing both is the point.
How did this land?
About the author
Developer Advocate
Steve builds something with Swarmz every week and writes up what worked, what broke, and what he'd do differently. Tutorials and hands-on guides are his lane.
