How to Prompt AI to Triage Support Tickets

To prompt AI to triage support tickets, give it a closed list of categories, a required "unsure" option, and a strict output schema, then measure it against fifty tickets you have already triaged by hand. The unsure option is the part people leave out, and it is the difference between a triage system you can trust and one that quietly misroutes the tickets that matter most.

Triage is a good first job to hand over because it is high volume, low creativity, and every decision has a correct answer somebody already knows. It is a bad job to hand over carelessly, because the cost of a mistake is wildly uneven.

Start with the asymmetry

Misrouting a billing question to the product queue costs someone twenty minutes. Misrouting an outage report to the general queue costs you a customer. Any triage prompt that treats all categories as equally important will be optimised for the wrong thing.

So before you write a word of prompt, sort your categories into three tiers:

• Must never be missed. Outages, security reports, data loss, legal threats, anything about a minor. These should be over-detected on purpose. A false positive here is cheap.

• Route accurately. Billing, account access, feature questions. Normal accuracy is fine.

• Safe to get wrong. Feedback, thank-yous, spam. Nobody is harmed by a mistake.

That tiering goes into the prompt as an explicit instruction, not as a hope.

The prompt, piece by piece

Five components. Each one fixes a specific failure you will otherwise hit in week one.

1. A closed category list. Name every category and define it in one line. If you leave the list open, the model invents categories, and you discover this three weeks later when you try to build a dashboard.

2. An explicit unsure option. Give it a category called needs_human and tell it plainly when to use it: when the ticket fits two categories equally, when it contains something not covered by the list, or when the customer's actual problem is not stated. Models will assign a category to anything if you do not give them an exit. Making AI say it does not know is the general version of this technique, and triage is where it pays fastest.

3. A strict output schema. JSON with fixed fields, nothing else. Something like category, urgency, needs_human as a boolean, a one-line reason, and any product area mentioned. The reason field is not decoration: it is what you read when you audit misroutes, and it costs almost nothing. Getting reliable JSON output covers the mechanics of making the shape hold.

4. Field order that matches thinking order. Put evidence before conclusion. Ask for the quoted phrase that decided it, then the category. Reversing this makes the model commit first and rationalise after, and the quality drop is measurable.

5. Three to five examples, chosen badly on purpose. Not the clean ones. Pick the tickets that a new hire got wrong: the billing complaint that is actually a bug report, the feature request phrased as a compliment, the outage report that opens with an apology for bothering you. Easy examples teach nothing.

A worked prompt skeleton

You triage inbound support tickets. Output JSON only.

Categories:
- outage: the service is down or broken for the customer right now
- bug: something behaves incorrectly but the customer can still work
- billing: invoices, payment methods, refunds, plan changes
- account: login, access, permissions, seats
- howto: the product works, the customer does not know how to use it
- feedback: praise, complaint, or a feature request with no blocked task
- needs_human: fits two categories equally, or is not covered above

Priority rule: if there is any chance the ticket is an outage or a
security report, classify it that way even if another category also fits.

Output:
{"evidence": "<quote the phrase that decided this>",
 "category": "<one of the above>",
 "urgency": "high|normal|low",
 "needs_human": true|false,
 "reason": "<one sentence>"}

Ticket:
"""{{ticket_text}}"""

Note the priority rule sits above the output schema, not buried under it. Instructions placed after a long schema get followed less reliably, which is a small thing that matters at volume.

Measuring it, properly

Do this before it touches a live queue. It takes an afternoon.

1. Pull fifty tickets from the last month that a human already triaged. Include at least five from each category and every awkward case you can remember.

2. Run the prompt over all fifty. Do not look at the outputs yet.

3. Build a simple confusion table: what the human said against what the model said.

4. Read only the disagreements.

That last step is where the value is. You are not looking for an accuracy percentage, you are looking for *which pairs* it confuses. Bug against outage is a problem you must fix. Feedback against howto is not. A model at 82% overall with clean tier-one performance is better than one at 91% that occasionally files an outage as feedback.

Two rules that fall out of doing this honestly:

• If needs_human fires on more than about one ticket in five, your categories overlap and the fix is in the definitions, not the prompt.

• If needs_human never fires, the model is guessing and you have not made the exit credible enough. Add an example that uses it.

For repeatability across runs, getting consistent output every time covers temperature and the other settings that stop the same ticket landing in two categories on two days.

Rolling it out without regret

Three stages, roughly a week each.

Shadow mode. The model triages, nobody acts on it. You compare against what your team did. This catches the categories you forgot existed.

Suggest mode. The model's category is pre-filled in your tool and a person confirms it. Most of the time saving arrives here, and the risk is still near zero.

Auto mode, partially. Auto-route only the tiers where a mistake is cheap. Keep tier one in front of a human indefinitely. There is no version of this where an outage report should be routed without anyone reading it.

Most teams stop at suggest mode and are right to. The remaining gain from full automation is small, and the failure mode is large.

Where it goes next

Once triage is stable, the natural next steps are drafting first responses for the highest-volume category and summarising long threads before escalation. Both reuse the same evidence-first structure. Automating customer support with AI covers the wider workflow, and the prompt engineering guide covers the underlying patterns if you want to build the rest yourself.

FAQ

What categories should a support triage prompt use? Yours, taken from how your team already routes tickets, plus an explicit needs_human option. Do not import a generic taxonomy; the categories only work if they map to a queue somebody actually watches.

How accurate does ticket triage need to be? Accurate enough on the categories where a mistake is expensive, which is a different bar from overall accuracy. Measure per category and weight by what a misroute costs.

Should the AI reply to tickets as well as sort them? Sort first, run it for a few weeks, then consider drafting replies for one high-volume category. Combining both on day one makes it impossible to tell which half is failing.

What if a ticket fits two categories? That is what needs_human is for. Tickets that genuinely span two queues are usually the ones where a person adds the most value.

Do I need to fine-tune a model for this? Almost never. A clear category list, an evidence-first schema and five well-chosen examples get you further than fine-tuning on a few hundred tickets, and cost nothing to change when your categories do.