Why Your AI App Costs More Than You Expected

Your AI app costs more than expected almost always for one of four reasons, and none of them is that the model is expensive. The input you resend on every turn, the retries you are not counting, the embeddings you recompute, and the logging you forgot was priced. Work out which one is yours before you start optimising, because three of the four are fixed by configuration rather than by using a cheaper model.

The diagnostic question is simple: is your bill dominated by input tokens or output tokens? That one number points at the cause.

Line item one: you resend the whole conversation every turn

This is the big one, and it catches people who have reasoned correctly about everything else. A chat request is stateless. Turn ten does not send one message, it sends the system prompt plus all nine previous exchanges plus the new message. Input grows with every turn while the user experiences a constant-feeling conversation.

Ten turns into a conversation with a 2,000-token system prompt, you have sent that system prompt ten times. A long document pasted at the start is re-sent on every subsequent turn for the life of the conversation. If your bill is overwhelmingly input tokens, this is your answer, and the arithmetic is the same one described in our explainer on how context length drives cost.

The fix is prompt caching, and it is worth understanding properly because the default surprises people. Anthropic's prompt caching documentation specifies a 5-minute default cache lifetime, refreshed each time the cached content is used, with a 1-hour option available. It also specifies that cache hits require 100 percent identical prompt segments, and that prompts below a minimum cacheable length, between 512 and 4,096 tokens depending on the model, are processed without caching and return no error.

Three practical consequences. A user who steps away for ten minutes comes back to a cold cache and pays full price for the next request. A system prompt that includes a timestamp or a user name near the top will never cache, because the prefix is no longer identical. And a short system prompt may not be caching at all while you assume it is. The usage fields tell you the truth: cache_read_input_tokens versus cache_creation_input_tokens versus input_tokens. If reads are near zero, your caching is not working.

Line item two: retries nobody is counting

Every retry is a billed request. A failed call that returns a malformed response, hits a timeout or trips a validation check still consumed input tokens, and if it produced partial output it consumed those too.

The pattern that quietly doubles a bill is a retry loop around a parse failure. The model returns JSON with a stray code fence, your parser throws, your code retries, and the retry succeeds. Everything works, your users see nothing, and you are paying twice for a meaningful share of requests. Nothing in your error dashboard flags this because the end state is success.

Count retries explicitly. If retries are more than a few percent of requests, fix the cause rather than the cost: constrained output modes are more reliable than format instructions, and a validation failure that recurs is a prompt problem wearing a billing costume.

Line item three: embeddings you recompute

If you built retrieval over a document set, embedding is a one-time cost per chunk that becomes a recurring cost the moment something re-runs it. The usual culprits are a reindex job that rebuilds everything instead of only what changed, a deploy that clears the vector store, and a document pipeline that re-embeds a file whenever any part of it is touched.

Embedding is cheap per call, which is exactly why this hides. A full reindex of 50,000 chunks is individually trivial and collectively a line on your invoice. Check whether your indexing job is incremental. Most first implementations are not.

Line item four: the things you forgot are priced

• Logging full prompts and responses. Not a model cost, a storage and log-ingestion cost, and prompts are large. Teams discover this on the observability bill rather than the AI one.

• Evals running in CI. An eval suite on every pull request is a model call per test case per run. Worth paying for, worth knowing about.

• Streaming abandoned early. A user who closes the tab after two sentences may still be billed for the full generation, depending on your provider and whether you cancel the request.

• Development and staging traffic. Frequently on the same key as production, and genuinely invisible until you split them.

Separate keys for production, staging and development is a ten-minute change that makes every subsequent cost question answerable. Do it before you need it.

Which reason your AI app costs more than expected

1. Pull the input and output token split for a week. Input-dominated points at line items one and three; output-dominated points at verbosity and retries.

2. Check your cache read and write fields. Near-zero reads with a long system prompt means caching is misconfigured, not absent.

3. Instrument retries as a counter, not a log line. You are looking for a percentage.

4. Check whether your reindex job is incremental.

5. Split your API keys by environment so next month's question is easier.

Only after that is it worth reaching for general efficiency work, and the levers there are ordered by payoff in how to reduce AI API costs. Starting with model selection is the common mistake: a cheaper model applied to a resend problem saves a fraction of a cost you could have removed most of.

If a client is paying for this

Variable costs are a commercial problem as much as a technical one. Two things make the conversation easier.

Price with the variability visible rather than buried in a flat fee you will regret. The structures in how to price an AI product with variable costs cover the options, and the mechanics of passing usage through are in how to bill clients for AI API usage.

Then explain the resend behaviour early, before the first invoice rather than after it. A client who understands that a longer conversation costs more than a shorter one reads their bill differently, and our notes on explaining AI costs to a client have language that works for non-technical buyers. The broader question of which revenue models survive variable costs is covered in AI monetization strategies.