How to Estimate Tokens for an AI Task

How to estimate tokens for an AI task comes down to three numbers: how many tokens your input contains, how many tokens the output is likely to need, and how much buffer to add for instructions and formatting overhead. Count the words in your input, divide by roughly 0.75 to convert words to tokens, then multiply by an output multiplier based on the type of task. Summarization runs 10 to 25 percent of input length. Classification returns a fixed handful of tokens no matter the input size. Code generation can come out longer than what went in. Add input and output together, then test the math against a small sample before you commit a full document or batch to the API.

Why bother estimating before you run the task

Every API call has a cost and a context window limit attached to it. If you guess wrong on either, you either pay more than expected or the call fails partway through. Estimating tokens first is the same instinct as checking a gas gauge before a road trip: you don't need to be exact, you need to know if you're going to run short. This matters most for batch jobs, where a small error per item multiplies across hundreds of runs, and for anything with a hard budget, since a task that runs out of credits mid-run loses whatever partial output it produced. Estimating up front is also the first lever most teams pull when they start looking at how to reduce AI API costs, since you can't cut what you haven't measured. Token estimation is a small, mechanical piece of the broader prompt engineering skill set, worth doing before you write the prompt itself.

Step 1: convert your input to tokens

Start with a word count. For English prose, OpenAI's own guidance is that one token is roughly 4 characters, or about 0.75 words, which works out to around 1.3 tokens per word. That means a 1,000-word document lands around 1,333 tokens. Anthropic's Claude models tokenize a little differently but land in a similar range for plain English text, and their models produce a free, exact count through a token counting API if you want a real number instead of an estimate. If you're new to the concept, it helps to first get clear on what a token in AI actually is, since it isn't the same thing as a word or a character.

The formula for this step is simple: tokens ≈ word count ÷ 0.75. Don't forget the instructions you're sending alongside the content. System prompts and formatting instructions add tokens too, and if you're unsure how much to write, how long a prompt should be is worth checking before you assume your instructions are negligible. A short one-line instruction is a rounding error. A five-paragraph system prompt with examples is not.

Step 2: apply an output multiplier by task type

Input tokens are the easy half. Output tokens depend entirely on what you're asking the model to do, and the relationship to input length varies a lot by task. Use these as starting multipliers, then correct them once you have real data from your own runs.

For open-ended chat or drafting tasks where the model could keep going indefinitely, skip the multiplier and set a hard max_tokens cap instead. An estimate only helps when the task has a natural stopping point.

Step 3: test on a small sample before committing

Back-of-envelope math gets you in the right neighborhood, not the exact number. Before you run a full document or a batch of hundreds, run one representative unit through the actual model or a tokenizer and compare the real count to your estimate. If your estimate was off by 10 percent on one page, it'll be off by roughly the same margin on the other nine, so you can correct the whole batch estimate with one test call.

This step matters even more for multi-turn tasks, where each new message resends the accumulated conversation history unless the provider supports prompt caching for the repeated portion. A five-turn conversation isn't five separate small calls, it's five calls of growing size, and estimating only the first turn will badly undercount the total. Testing one full multi-turn exchange, not just one message, catches this before it surprises you on a real bill.

Worked example: how to estimate tokens for a 10-page document

Here's the full method run against a real task: summarizing a 10-page report down to a short brief. Assume roughly 500 words per single-spaced page, a common working estimate for standard business documents.

The first-pass estimate and the tested estimate land within about 2 percent of each other here, close enough for planning purposes. That gap won't always be this small; tables, code snippets, or number-heavy text tokenize denser than plain prose, which is why the test step matters. Scale to a batch of 50 similar documents and the total comes out to roughly 8,020 × 50, or about 401,000 tokens, worth knowing before you queue the job rather than after.

Where these estimates break down

• Dense or technical text. Tables, code, JSON, and text heavy with numbers or symbols produce more tokens per word than plain prose, sometimes by 30 percent or more.

• Non-English text. Languages that don't use spaces between words, or that use non-Latin scripts, often tokenize at a worse ratio than English.

• Tokenizer differences across models. Anthropic's newer tokenizer, used in its current model generation, produces roughly 30 percent more tokens than earlier Claude models for identical text, according to Anthropic's own documentation. A count measured on one model version doesn't transfer cleanly to another.

• Growing multi-turn context. As covered above, conversation history compounds call over call unless it's cached.

Given these gaps, treat any back-of-envelope number as a floor, not a ceiling, and add 10 to 15 percent headroom when the task involves anything other than clean, plain-language prose.

Frequently asked questions

How many tokens is 1,000 words?

Around 1,300 to 1,400 tokens for plain English prose, using the common rule of thumb of about 0.75 words per token. Dense text with a lot of numbers, code, or unusual formatting will push that number higher.

How do I estimate output tokens before running a task?

Apply a multiplier based on the task type rather than treating output as a fixed guess. Summarization typically produces 10 to 25 percent of the input token count, classification produces a small fixed number regardless of input size, and code generation can exceed the input by several times depending on scope.

Is token estimation accurate enough to predict AI API cost?

It's accurate enough for planning and budgeting, usually within 10 to 20 percent on plain text, but not accurate enough to bill against exactly. For a real number, run a small sample through the provider's own tokenizer or count endpoint before committing a large batch, and treat the estimate as a starting point for decisions about model choice and context trimming.

Do token counts differ between AI models?

Yes. Each provider uses its own tokenizer, and even within one provider, newer model versions can use an updated tokenizer that counts the same text differently. Anthropic has noted that its newer tokenizer produces roughly 30 percent more tokens than earlier models for identical input, so counts don't transfer cleanly across model generations.

What happens if I underestimate and run out of budget mid-task?

The call typically stops wherever it is, and depending on the provider and task, you may keep the partial output or lose it entirely. This is exactly the scenario that makes estimating worthwhile in the first place, and it's covered in more depth in what happens when AI credits run out mid-task.