How to Build a Knowledge Base for AI Customer Support
An AI support bot is only as good as the articles it reads. Here is an article template, a 20-question test set, and a weekly routine that keeps answers correct.
A knowledge base for AI customer support is the set of articles your bot is allowed to answer from, and its quality decides the quality of every answer. Write each article around one customer question, put the answer in the first two sentences, name an owner and a last-verified date, and test the whole set against 20 real questions every week.
Most bad bot answers are not model failures. They are content failures: two articles that disagree, a policy that changed last spring, or a page that answers a question nobody asked. Fix the content and the same model suddenly looks smarter.
Start from real questions, not from your website
Copy 50 recent customer messages from email, chat or DMs. Group them by topic. You will often find that a handful of topics cover most of the volume: delivery, refunds, opening hours, pricing, account access, and a few product questions. Write articles for those first.
An existing website is a poor source because it is written to sell, not to answer. Use it as raw material, then rewrite.
The article template
Retrieval systems find articles by matching the customer's question to the text, so a title that mirrors a real question helps. A consistent shape also makes mistakes easy to spot.
Title: Can I return an item after 30 days?
Answer: No. Returns are accepted within 30 days of delivery.
After that we can offer a repair or store credit at our discretion.
Details:
- Return window starts on the delivery date shown in your tracking email.
- Items must be unused and in original packaging.
- Refunds go to the original payment method within 5 working days of us receiving the item.
Exceptions: Faulty items are covered by a separate process, see
"My item arrived damaged".
Do not say: that we guarantee refunds, or quote a deadline for store credit.
Owner: Priya (operations)
Last verified: 2026-09-14Each part earns its place. The answer comes first so the bot leads with it. Exceptions point to a sibling article instead of repeating it. The do-not-say line stops over-promising. The owner and date let you find stale content later.
Rules that prevent contradictions
One fact, one article. If the return window appears in six places, you will update five of them. Link to the single source.
No marketing words. Replace fast, easy and best with the actual number or step.
Write limits explicitly. Say what you do not do, such as phone support or weekend delivery. Silence invites the bot to improvise.
Keep policy and how-to separate. A policy article states the rule. A how-to article lists the clicks. Mixing them creates long articles that answer neither well.
Delete, do not append. When a rule changes, rewrite the article. Adding an update note at the bottom leaves the old rule readable.
Tell the bot what to do when the answer is missing
The system instruction matters as much as the content. Tell the bot to answer only from the articles, to say it does not know when no article applies, and to pass the conversation to a person. Anthropic's guidance for long-document work also suggests having the model quote the relevant passage first, which makes unsupported answers easier to catch. The handover itself is covered in how to hand off an AI conversation to a human.
The reason this works is called grounding, explained in what is grounding in AI.
The 20-question test set
Write 20 questions with the correct answer for each. Make 12 of them ordinary, 4 phrased badly or with typos, and 4 that your knowledge base should not be able to answer. Those last four test whether the bot admits it does not know.
Question type | Count | A pass looks like |
|---|---|---|
Ordinary, covered | 12 | Correct answer that matches the article |
Messy phrasing or typos | 4 | Same correct answer |
Not covered by any article | 4 | States it does not know and offers a person |
Run all 20 every week and after every edit. Keep a score sheet. The first week will be humbling. The point is the trend, and the failures tell you which article to fix. The practice of keeping a fixed set of checks is covered in what is an AI eval.
A weekly upkeep routine
Read ten real bot conversations, newest first, and note any wrong or awkward answer.
For each, decide: article missing, article wrong, or article fine but the bot ignored it.
Fix content first. Change the system instruction only if several failures share a pattern.
Re-run the 20 questions and record the score.
Check any article whose last-verified date is older than your review window.
Setting up the bot itself is a separate job. How to build a customer support chatbot with AI covers that, and automate customer support with AI shows what is sensible to hand over. For the broader context, see AI for small business.
FAQ
How many articles does an AI support bot need?
Fewer than you might expect. A small set of well-written articles on your most common questions is easier to keep correct than hundreds of rough ones.
Can I use my website as the knowledge base?
As raw material only. Marketing pages rarely state exact policies, limits and exceptions, which are what customers ask about.
How often should I update the knowledge base?
Whenever a policy, price or process changes, and on a fixed weekly check. Stale articles cause confident wrong answers.
What if the bot answers something not in the knowledge base?
Tighten the instruction to answer only from the articles, then add the question to your test set so you notice if it comes back.
How did this land?
About the author
Senior Editor, AI & Product
Cecilia leads the Swarmz editorial desk. She has spent a decade turning complex AI and product topics into writing people actually finish, and she owns the blog's quality bar.
