Prompt AI to Write an FAQ From Support Tickets

To prompt AI to write an FAQ from support tickets, do it in three passes rather than one. First make the model count: group the tickets into question clusters and report how many tickets fall into each. Then throw away anything below a frequency floor you set. Only then ask for the written answers. Hand a model 400 tickets and a single instruction to "write an FAQ" and you get eight plausible questions chosen for how interesting they look, not for how often anyone asked them. The counting pass is what makes the output a real FAQ.

Why the one-shot prompt fails

A model reading 400 tickets is not tallying anything. It is producing text that reads like a reasonable FAQ for a company with those tickets. The failure modes are consistent:

• It favours vivid tickets over frequent ones. One furious message about a billing edge case outranks ninety identical password questions.

• It merges distinct questions that use similar words, so "how do I cancel" and "how do I get a refund after cancelling" collapse into one entry with a vague answer.

• It invents answers. The tickets contain the questions. The correct answers live in your product, and a model with no instruction to abstain will fill the gap.

• It writes in support-agent voice, apologising for problems your FAQ page should simply explain.

Each of these is fixable, but not inside one prompt.

Pass one: cluster and count

Export your tickets to a text file or spreadsheet with one row per ticket: id, subject, first customer message. Strip names, emails and account numbers before anything leaves your systems, which matters more here than in most prompting work because ticket bodies are dense with personal data. How to prompt AI without leaking sensitive data covers the mechanics if you are doing this at volume.

Then ask for structure, not prose:

Below are customer support tickets, one per line, as id | subject | message.

Group them into distinct underlying questions. A group is one question a
customer was actually asking, regardless of wording.

Return a markdown table with these columns:
- question (phrased the way a customer would type it into a search box)
- ticket_count
- ticket_ids (up to 5 examples)
- confidence (high if the grouping is obvious, low if you are guessing)

Rules:
- Do not merge two groups unless a customer asking one would be
  satisfied by the answer to the other.
- Do not invent groups. Every group must trace to specific ticket ids.
- Sort by ticket_count, descending.
- Do not write any answers yet.

The ticket_ids column is the audit trail. Spot-check three rows against your actual tickets before you trust the counts, because the model may have assigned loosely and the table will look identical either way. The confidence column catches the groupings it was unsure about, which is where errors cluster. There is more on getting a model to admit uncertainty in how to prompt AI to flag what it is unsure about.

The frequency floor, and why you set it before you look

Decide the cut-off before you read the table. A reasonable default is any question appearing in at least 1% of tickets, or at least five tickets, whichever is larger. Write the number down.

If you set the floor after reading the list, you will keep the questions you find interesting, which puts you back where the one-shot prompt left you. One-off tickets are not FAQ material even when they are the most memorable thing in the dataset. They belong in a help article or a bug report.

Two exceptions worth keeping below the floor. Anything with a legal or safety dimension, because a rare question about data deletion still needs a published answer. And anything whose absence from the FAQ is costing you sales, which you will know from the tickets that arrive before purchase rather than after.

Pass two: answers from your material, not the model's memory

Now you have a shortlist. The second pass needs your documentation in the prompt, not just the tickets, or the model will write confident answers to questions only your product can answer.

You are writing FAQ answers for [product], a [one line description].

AUTHORITATIVE SOURCES (use only these for facts):
<docs>
[paste help centre articles, pricing page, policy text]
</docs>

QUESTIONS TO ANSWER:
[paste the shortlist from pass one]

For each question:
- Answer in 40 to 80 words. Lead with the answer, then the caveat.
- Use only facts present in <docs>. If the answer is not there,
  write exactly: NEEDS INPUT, followed by what you would need to know.
- Plain second person. No apologies, no "we're sorry to hear".
- No links unless the URL appears in <docs>.

The NEEDS INPUT instruction is the one that earns its place. On a first run through real tickets, expect a third of the questions to come back marked, because customers routinely ask about things no document covers. That list is more valuable than the FAQ: it is the exact set of gaps in your documentation, ranked by how many people hit them.

Wrapping the sources in tags rather than pasting them loose makes the boundary explicit to the model, which is the same reason XML tags are useful for structuring prompts generally, and the reason Anthropic's own guidance recommends using XML tags to separate the parts of a prompt.

Pass three: match your own voice

The answers are now correct and generic. The third pass is short:

Rewrite these FAQ answers to match the voice in the examples below.
Keep every fact identical. Do not add or remove information.
Do not change any answer marked NEEDS INPUT.

<voice_examples>
[three paragraphs of your existing published copy]
</voice_examples>

Keeping facts and voice in separate passes means a voice edit cannot quietly change a refund window. If you do this regularly, store the voice examples once rather than pasting them each time, as described in following a style guide without repasting examples.

A worked example of the counting pass paying for itself

A small invoicing tool ran this on 380 tickets from one quarter. The team's own guess at the top five questions, written down beforehand, was: how do I connect my bank, how do I add VAT, can I send recurring invoices, how do I change my plan, is my data encrypted.

The counted table came back differently:

Three of the team's five guesses were below the frequency floor. The top item, a date format question, was not on their list at all, because support had been answering it individually for months without anyone aggregating it. The email delivery question turned out to be a spam filtering issue, which belonged in a troubleshooting article rather than an FAQ, and the bank connection question was rare because the feature was easy.

That gap is the normal result, not an unusual one. Teams estimate from the tickets they personally remember, which skews towards the difficult and the recent. The counting pass is the correction.

Variations worth knowing

If your tickets have resolution notes. Include them in pass one and ask the model to add a column for how the ticket was resolved. Where the resolutions within a group disagree, you have found either an inconsistent support process or two questions that look alike, both worth knowing before you publish a single answer for them.

If you have search logs instead of tickets. Use the same three passes on your site search queries or help centre searches. Search logs are better than tickets in one respect, because they capture people who never contacted you, and worse in another, because a query is shorter and more ambiguous than a ticket.

If you support multiple products. Run pass one per product rather than across all of them. A combined run will group by surface wording and merge the same question asked about two different products, which produces an answer that is wrong for both.

If the FAQ is for a sales page rather than a help centre. Filter to pre-purchase tickets only, which changes the list substantially. Questions that arrive before payment are objections, and the answers need to resolve a doubt rather than explain a feature.

What to check before publishing

• Every number, date and policy detail traced back to a source document, by a human.

• No answer that contradicts another answer. Models produce these in pairs when two questions touch the same policy.

• Questions phrased as a customer would search, not as your team describes the feature internally.

• The NEEDS INPUT list routed to whoever owns those answers, with the ticket counts attached so they can see the cost of leaving them unwritten.

Run the whole thing again each quarter against the newest tickets. The counts move, and the questions that drop off the list are the ones your product fixed. If your tickets are more useful as a roadmap input than a content input, turning customer reviews into a feature roadmap uses the same clustering pass for a different output, and triaging support tickets handles the routing side.

FAQ

How many tickets do I need before this works?

Around 100 gives usable clusters. Below about 50 you can read them yourself faster than you can set up the prompt, and the frequency counts are too noisy to rank.

Should I use the ticket text verbatim in the FAQ?

No. Use it to find the question and to check your answer addresses what was actually asked. Customer wording is often specific to one account and sometimes includes details you cannot publish.

Can I skip the counting pass if I already know my top questions?

If you are confident in your ranking, yes, go straight to pass two with your own list. The counting pass exists to correct the gap between what a team thinks customers ask and what the tickets say, and that gap is usually larger than expected.

What if the model groups two questions I think are different?

Trust yourself and split them. The model groups on wording similarity, which is a poor proxy for whether one answer satisfies both askers. The instruction in pass one reduces this but does not remove it.