What Is Batch Inference and When Is It Cheaper?

Batch inference is sending a large pile of requests to a model at once and collecting the answers later instead of waiting for each one. You give up immediacy. In return, both major providers charge half price. If your work can wait, that is a 50% discount for a change that takes an afternoon to implement.

The trade is simple enough that the interesting question is not what batch inference is, but which of your workloads qualify, and what happens when a batch does not finish.

The mechanics

A normal API call is synchronous: you send one request, you block, you get one response, typically in seconds. Batch inference inverts that. You upload a file of many requests, the provider queues them, they run whenever there is spare capacity, and you poll for results.

That "whenever there is spare capacity" is what you are actually paying less for. Providers have idle capacity between traffic peaks, and batch work is what fills it. You are buying the off-peak fare.

Three consequences follow, and all three matter operationally:

• You cannot show the result to a waiting user.

• Requests are independent. Nothing in a batch can depend on the answer to something else in the same batch.

• Completion is a window, not a guarantee, which is the part that bites.

What the providers actually commit to

The published numbers, side by side.

Sources: Anthropic's batch processing docs and OpenAI's Batch API guide.

The row worth reading twice is the deadline behaviour. Anthropic states plainly that batches expire if processing does not complete within 24 hours, that you can access results when all messages finish or after 24 hours whichever comes first, and that under heavy demand you may see more requests expiring. You are not billed for expired requests, which is fair, but you do have to handle a partial result set. Code that assumes every request in a submitted batch returns will break the first busy week you hit.

Which workloads qualify

The test is not "is this important". It is "does anyone wait for it".

Good fits:

• Classifying or tagging a backlog. Support tickets, product records, documents, transcripts.

• Generating content for a catalogue. Descriptions, summaries, alt text, translations.

• Running evaluations. A test suite over hundreds of prompts is the ideal batch job, and it is often the biggest single line on a small team's bill.

• Extraction over a document archive.

• Nightly enrichment jobs where the output is read in the morning.

Bad fits, regardless of price:

• Anything in a request-response path with a person at the end.

• Multi-step agent work, where step two depends on step one.

• Interactive tools, obviously.

• Anything with a deadline shorter than 24 hours and no fallback.

That last one deserves care. "Usually under an hour" is a useful expectation and a terrible SLA. If a job absolutely must be done by 9am, either submit it the previous evening with hours of slack, or build a synchronous fallback that reruns whatever expired.

The hybrid that most teams end up with

The pattern that survives contact with production is not batch or synchronous. It is both, split by who is waiting.

Take a document processing product. A user uploads a file and wants a summary now: synchronous, full price, worth it. That same product also re-scores every document in the archive when the prompt changes: batch, half price, nobody waiting. Same model, same prompt, two paths, one line of config deciding which.

Setting that up costs you a queue and a results handler. It is the same infrastructure you need for any background work, so if you already have a job runner, batch inference is a small addition rather than a new system.

How it interacts with caching

One non-obvious interaction, and it costs people money.

Prompt caching and batching both reduce spend, but they work on different timescales. Cache entries are short lived by default, and a batch can sit queued for far longer than a standard cache entry survives. Anthropic's docs note this directly and suggest the longer one-hour cache duration for batches with shared context, so your cached prefix is still there when the requests actually run.

If you are batching thousands of requests that share a long system prompt, that detail is the difference between a large saving and a small one. What prompt caching is covers the mechanism, and the other cost levers put batching in context against the rest.

Worth doing?

Run the arithmetic before the engineering. Take last month's bill, work out what share of it came from work nobody was waiting on, and halve that number. If the answer is fifteen dollars, do something else with your afternoon. If it is several hundred, batching is probably the highest-return change available to you, ahead of switching models and well ahead of prompt golf.

For the surrounding concepts, what inference actually means is the foundation, provider throughput limits explain why batch queues exist at all, and how AI models work covers what is happening inside each request.

FAQ

How much cheaper is batch inference? 50% off standard input and output pricing at both Anthropic and OpenAI, for the same models.

How long does a batch take? Anthropic says most batches complete in under an hour, with a hard 24 hour expiry. OpenAI uses a 24 hour completion window, often finishing sooner. Neither guarantees a time shorter than the window.

What happens if my batch does not finish? With Anthropic, unfinished requests expire at 24 hours and are not billed, so you receive a partial result set and must resubmit the remainder. Handle partial results in code from day one.

Can I use batch inference for a chatbot? No. Anything with a person waiting for the answer needs a synchronous call. Batch is for work that is read later.

Is batch inference the same as batching requests in my own code? No. Sending several requests concurrently to the normal endpoint is just concurrency, at full price. Batch inference is a separate endpoint with its own queue and its own discount.