How to Stop AI From Making Up Citations

Ask an AI model for its sources and it will hand you a tidy list: author names, years, sometimes page numbers. Some of those sources are real. Some are invented, and the invented ones read exactly like the real ones, because the model is not retrieving anything. It is predicting what a citation-shaped string of text looks like at that point in the answer.

How to stop AI from making up citations comes down to one structural change: stop asking for an answer with citations baked in, and start asking for an answer and a claimed-source list as two separate outputs, then verify the second list before you trust it. That is the whole method. The rest of this is why it works and exactly how to run it.

Why AI hallucinated citations look so convincing

A language model generates text one token at a time based on probability, not lookup. Ask it for a citation and it is not querying a bibliographic database, it is continuing a pattern it saw thousands of times in training: author surname, year, title, journal. It fills that pattern with whatever surname-year-title combination is statistically plausible given the topic. Plausible is not the same as real, and nothing in the generation process checks the difference unless you build a check yourself.

The same mechanism produces a fabricated software package that was never published in coding tools. The output matches a familiar shape, a package name that sounds right for the task, and the model has no built-in step that checks the shape against a real registry unless something outside the model does that checking for it.

Why retrieval-augmented generation helps, and why it does not fully fix it

Retrieval-augmented generation, RAG, changes the setup by fetching real documents at query time and handing the model actual passages to work from, instead of relying purely on what it memorized during training. That should make fabricated citations rare. It makes them less common, not gone.

A 2024 Stanford study of commercial legal research tools built specifically on RAG, the kind designed to ground every answer in real case law, found they still hallucinated between 17% and 33% of the time (Hallucination-Free? Assessing the Reliability of Leading AI Legal Research Tools). For comparison, general-purpose models answering similar legal questions without retrieval hallucinated 69% to 88% of the time in a related Stanford study (Large Legal Fictions). Retrieval cuts the failure rate by more than half. It does not zero it out.

Two specific failure modes survive retrieval. First, misattribution: the model is handed several retrieved chunks, states a fact correctly, but cites the wrong one of them, or blends two chunks and cites neither accurately. Second, synthesis: when a retrieved chunk is topically close but does not fully answer the question, the model can still generate a citation-shaped completion, an author and year that feels consistent with the surrounding retrieved material but was never actually stated in it.

For contrast, Anthropic's Citations feature only returns exact quoted passages from documents you explicitly hand the model, chunked and matched at the sentence level. That narrower guarantee, quoting only what it was actually given, is closer to a real fix than open-ended citation of sources the model was never handed in the first place.

The verification-first prompt template

Stop asking one prompt to both answer and cite in the same breath. Split the task into two fields and move the verification burden to a second pass instead of the model's word. This is a narrow application of a broader prompt engineering principle: separate what the model claims from what it concluded, so each half can be checked on its own terms.

Answer the question using only verifiable knowledge.

Return two separate sections:

CLAIMED_SOURCES:
List every source you are relying on, one per line, numbered, in the format:
- Author/Organization, Title, Year, and a URL or DOI if you have one.
- Mark each one "high confidence" or "uncertain" based on how sure you are it exists exactly as stated.

ANSWER:
Write the answer. Do not put citations inline. Reference sources only by the number
they have in CLAIMED_SOURCES, like [1].

Now verify CLAIMED_SOURCES before you touch the answer. If you have tool access, fetch each URL or DOI and confirm the title, author, and year actually match what is on the page, not just that the page returns a 200. Doing this in a second, separate pass matters: asking the same turn to both write and check its own citation just repeats the same pattern-completion that produced the fabrication in the first place.

If you are checking by hand, three signals catch most fabrications without any special tooling: search the exact title in quotation marks, if nothing relevant appears on the first page of results, be suspicious. Check whether the DOI resolves at doi.org, a fabricated DOI usually returns an error or points at an unrelated paper. Check that the journal or publisher is a real, indexed outlet rather than a name that merely sounds like one.

Before and after: catching a fabricated-looking citation

A generic prompt, "summarize the research on remote work and productivity, with citations," tends to produce a sentence like this: "Remote workers show a 13% productivity increase according to a 2021 study published in the Journal of Organizational Behavior by Chen and Alvarez." That sentence is well-formed enough that most people would leave it in a draft untouched. It may also be entirely invented, a plausible-sounding author pair, a plausible journal, a plausible number, none of it checked against anything real.

Run the same request through the two-field template and the CLAIMED_SOURCES section surfaces a line reading "Chen and Alvarez, 2021, Journal of Organizational Behavior, uncertain, no URL available" sitting next to the answer instead of buried inside it as a settled fact. That "uncertain" flag is the model's own hedge showing through once it is asked to grade its own confidence separately. A thirty-second search for the exact title and author pair that returns nothing relevant is enough to pull the line before it reaches a reader. The model did not get smarter. The workflow around it now has a place to catch the failure before publishing, which is the actual goal here, to fix AI fake sources at the point where a human can still intervene.

A quick table for spot-checking a claimed source

Two habits worth stacking on top

Pair the verification-first template with asking the model to check its own work in a distinct follow-up turn, not the same reply, specifically flagging any claim in ANSWER that is not backed by a source marked "high confidence." And build in the option to abstain entirely: getting AI to say it does not know instead of guessing removes the pressure that produces a fabricated citation in the first place, a model that is allowed to return "not_in_sources" has less reason to invent one.

Questions

Why does AI make up citations that look so real?

Because it is predicting the shape of a citation, author, year, title, journal, based on patterns from training data, not retrieving a real record. A fabricated citation is grammatically and stylistically identical to a real one because both are generated the same way. Only a check against an outside source can tell them apart.

Does telling AI to only cite real sources actually work?

Not on its own. Adding "only cite real sources" to a prompt is a negative instruction competing with the model's underlying drive to produce a complete, helpful-looking answer. It does not give the model a way to check whether a source is real, so it keeps generating plausible-looking ones. Structure, separating claims from citations and verifying the citations, works where wording alone does not.

Are AI hallucinated citations still a risk with tools that search the web or use RAG?

Yes, at a lower rate. As the Stanford figures above show, retrieval-grounded tools built specifically for accuracy still produced fabricated or misattributed citations in roughly one out of every three to six answers. Treat retrieval as a meaningful reduction in risk, not a guarantee.

How do I check if an AI-generated citation is real without paid tools?

Search the exact title in quotation marks, check whether any DOI resolves at doi.org, and confirm the journal or publisher is an outlet you can independently verify exists. Any one of those failing is a reason to drop the citation rather than fix its formatting.

Does asking the model for a confidence score help catch fake citations?

Not much. Self-reported confidence scores from language models are poorly calibrated and tend to cluster around whatever number the prompt implies is expected. A discrete "high confidence" or "uncertain" label attached to each source in a separate list, checked independently, is more useful than a number the model made up alongside the citation itself.